Find Jobs
Hire Freelancers

Vk Video Scraper using PHP (plus added features)

$30-350 USD

Completed
Posted over 9 years ago

$30-350 USD

Paid on delivery
I am looking for someone that can create a sort of scraper/script that will do the following: Scrape a page/pages of [login to view URL] for videos and return the video urls ([login to view URL]) as well as generate an iframe for each video that has been scraped. As well as these basic functions, which are a must: 1. Be able to enter a search string and return the videos. (Including the ability to be able to search with the safe-search filter OFF) 2. Be able to enter a community-page and return the videos of the page. 3. Be able to enter a profile page and return the videos the user has uploaded. 4. Including the above, be able to scrape multiple pages and not just the first few results. The reason for the huge budget range? If you can do these extras you can request a higher budget: A. Your script adds results (without duplicates) to a database, including meta data such as the video name, length, thumbnail url, uploader, etc. B. Your script can be run manually as well as automatically via cron jobs, detecting any new videos from a page, from a community or from a search term. C. Your script is highly optimized and can scrape thousands of videos a minute. (The code will be ran on a custom server hooked up to a 1GBit line, and the usual 32GB ram and ssd storage) So a clean and fast working script will be much more appreciated than a sluggish one! Some things to note: - The script will need to allow the use of an account, since VK is closed and only registered users can access content. This includes username,password and phone number. - Vk occasionally pops-up a window that will ask for the last four digits of your phone number, which need to be entered in order to continue to the content. Script must be able to recognize and fill in the form. - Vk is a Russian social website, though they have an English page. Some source code may be in Russian which may come to complications during scraping? - Encoding needs to be in UTF8. A great place for you to start would be checking this, old, code: [login to view URL] If you'd like to use a language other than PHP then that is completely acceptable, provided you include excellent documentation of the code. You have up to two months after starting the project to finish it. Which is more than enough time for a side-project. Last but not least, please bid reasonably, if you can accomplish all of the features including all the extra-plus features and you can create good documentation, as well as include some of your own ideas and features, then by all means go for the max bid.
Project ID: 6699616

About the project

3 proposals
Remote project
Active 10 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
I'm looking for my first project here so I'm willing to do quality work for a low price. I can easily do everything you asked, including all the extra features. The database is definitely the best way to organize this data. On the speed part there is something you should be aware of. Most sites have a rate limit, this means that if our scraper does too many requests too fasts it will get temporally banned (by ip address). I have lots of experience in trying to find exactly what the rate limit is so that we can get the most content possible per hour without triggering alarms. That being said, the code will still be very fast, using a very efficient framework that works in parallel. If there turns out to be a forgiving rate limit we could go to the limit of the network connection / computer resources without the code being a bottleneck. I plan on using python with the scrapy framework which is a framework that allows scraping any kind of content easily and efficiently. It is also easy to understand and very customizable. I would document the code extensively, making it easy for someone who never coded in python understand. Possible usage: -vk_scraper --search='cat' -vk_scraper --search='[login to view URL]' --profile_page='[login to view URL]' -> Where on the files are all items to scrape (it would skip items already scraped like you asked) Of course, please let me know of anything you would like done differently. If you choose to go with me you wont be disappointed!
$180 USD in 12 days
0.0 (0 reviews)
0.0
0.0
3 freelancers are bidding on average $243 USD for this job
User Avatar
A proposal has not yet been provided
$200 USD in 3 days
4.9 (90 reviews)
6.7
6.7
User Avatar
Hello! PHP/JS/HTML experts here for your needs. All the features you listed are not a problem (even those on extra list). We are from Ukraine so we are used with VK social web. The only thing I'm worrying about is if VK asks only for phone number (which is not a problem to handle), and doesn't ask captcha (which can make it all much more difficult to built). Also VK has an official API, which can be helpful for some of listed tasks. As for the script productivity, we'll have to test script on your server, to be sure we achieved desired scrape speed. We have an experience in building full sites, modules and plugins for popular CMSs, and standalone PHP scripts. We can start today! Looking forward for your comments! Regards
$350 USD in 30 days
5.0 (20 reviews)
4.9
4.9

About the client

Flag of GERMANY
Breidenbach, Germany
5.0
1
Payment method verified
Member since Nov 7, 2014

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.