Find Jobs
Hire Freelancers

Create a cloud web scrapper for me

$250-750 USD

Closed
Posted almost 6 years ago

$250-750 USD

Paid on delivery
I have a web scrapper software that is written in C#....The software loads an input file with all the URLS to scrape and return all data that i need such as stock status and price etc....Once the software starts, the software goes to each link one by one and return the data and populates it in a grid....i have over 10 thousand URLs to scrape and the software takes a lot of time... i need someone who is experienced to make the same software run on a cloud and to check each link simultaneously so that it wont have to go thru each link after the other...meaning once software starts, it will check all links at once...the website to scrape is highly protected against bots...therefore on my current software, i use proxies....please see in the attachment for a screenshot of my current software scraper
Project ID: 16783652

About the project

29 proposals
Remote project
Active 6 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
29 freelancers are bidding on average $581 USD for this job
User Avatar
Hi,yavaar19! I saw your requirements and I'm ready to recreate the code you have in Python using best practices of multithreading and multiprocessing (so data could be scraped in parallel and not url-by-url) Please contact me to discuss project details, Best regards, Vlad
$888 USD in 10 days
5.0 (74 reviews)
8.1
8.1
User Avatar
Hi there,I am Miljan,Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immediately and finish it within the agreed deadline. Check out my profile, portfolio, and former clients feedback - that'll let you know everything about me. Please feel free to contact me so that we can discuss further details. Thank you for taking the time to read my proposal.I am looking forward to hearing from you. Kind regards
$680 USD in 6 days
5.0 (200 reviews)
8.2
8.2
User Avatar
Hi, I have developed similar scraper/crawler and data/web automation projects. Please let me know if you are interested and I am available to start right away.
$250 USD in 5 days
4.8 (288 reviews)
8.3
8.3
User Avatar
Here is a senior C# scraper, langlang from China. I have read your job description with exciting mood :) In one word, I can say <I can do your project> proudly !!!! I have really GOOD exp in C# Programming, PHP, Python, Software Architecture, Web Scraping, any other many things ... Please check my reviews; www.freelancer.com/u/langlangfan I am interested in your projects and I believe I can deliver rich features of the application to beat your imagination. Let's work together to make it happen. Let me know if you are interested and we can always start with something small.
$1,111 USD in 10 days
4.9 (62 reviews)
7.6
7.6
User Avatar
Hi. I did lots of crawlers in python and I think this is the right path for it. Yes we can add concurrency to the crawler. Do you need as well a web interface? Let me know details and we can discuss. Thx.
$1,111 USD in 10 days
5.0 (147 reviews)
7.8
7.8
User Avatar
Greetings of the day! I am the best fit to your requested requirement. I can help you in Create a cloud web scrapper My Expertise & Experience: I am skilled in C# Programming, PHP, Python, Software Architecture, Web Scraping I would love to work with you on this project. Carrying total experience of 6.5 years. I can assure you, you won't disappointed with class of the work I produce. I can start the work on the go as per your need! Communication: I will be available on the freelancer.com chat 18 hours in the day. (FLEXIBLE WITH TIMINGS IN CASE OF THE NEED) I am just a message away in case of any queries.
$750 USD in 10 days
4.9 (210 reviews)
7.7
7.7
User Avatar
Dear Sir, I'm very much delighted to let you know that i did data scraping with PHP-cURL, PhantomJS, Node.js, Selenium from many sites. I just scraped the data from web site and then wrote the data in mysql database or excel or csv or xml file. I worked on many similar projects, I have big experience in data mining projects. I have written hundreds of web scrapers which scrape millions of pages each day. I'm ready to fulfill your requirement. I can finish this task in short time, with the best quality. I can assure 100% accuracy. Please give me the opportunity to do the work. With Kind Regards, Debdulal Roy Proshanta
$250 USD in 7 days
4.9 (102 reviews)
7.6
7.6
User Avatar
Hello yavaar19, We have gone through the details you have provided for "Create a cloud web scrapper for me" and would be pleased to work on this with you to deliver the results that you have expected Also I do have similar experience doing something similar to yours therefore I am looking forward to discuss and move ahead. Relevant Skills and Experience I have 10+ years of experience in C# Programming, PHP, Python, Software Architecture, Web Scraping I can do this within your budget too $250 - $750 USD Let's have chat so i can explain you in deep. Thank you, Regards, Mmadi, Flowgica Technologies.
$525 USD in 12 days
5.0 (49 reviews)
7.1
7.1
User Avatar
Hi there I can help your URL checks to run in parallel over proxies. For my bid I will create a service that performs the necessary work, which you can deploy in the cloud. I would recommend something like Azure Functions - it is very cheap to run and you only pay per time that you call your function. Then I can update your client app to call those functions in parallel instead of one by one. You will get the same result but at massive scale. You will need to register for an Azure account to run the app, I can help you set that up - it is super easy to do. Hope to hear from you soon! Regards, Joon
$555 USD in 10 days
5.0 (63 reviews)
7.3
7.3
User Avatar
Hi there. I can create you a scraper to do this task. I have created such scrapers before in C# and can create you exactly one you are looking for. Please share the website as i can't see it clearly on the picture you have attached. looking forward to hearing back from you. Have a good day. Thanks
$444 USD in 4 days
4.9 (160 reviews)
7.0
7.0
User Avatar
how are you,sir? I am a professional developer who has rich experience in this field. If you contact me, you and i will all be happy. Thank you for your reply in advance. Create a cloud web scrapper for me C# Programming, PHP, Python, Software Architecture, Web Scraping,
$555 USD in 4 days
4.9 (30 reviews)
6.3
6.3
User Avatar
Hi I have more than 5 years of hand on experience in web development using PHP and MySql I have expertise in following technologies and tools: HTML CSS Bootstrap Anguarjs Jquery PHP My Sql Wordpress codeigniter I have good eye on design and details and can convert a psd to html if required. I have good analytical ability to breakdown the complex problems in to small units. I have ability to write object oriented and maintainable code. I am hard working and honest person and looking for along term relationship. Quality code in timely manner is my highest priority. I can communicate in English and can we available on skype. I am in IST Time zone but ready to work in any time zone. I can give you 40 hrs/week. Looking for a positive response. Thanks
$555 USD in 10 days
5.0 (10 reviews)
5.3
5.3
User Avatar
Ready to start the work to Create a cloud web scrapper, We can discuss more over chat, Thanka Arjun S.
$500 USD in 10 days
4.5 (33 reviews)
6.1
6.1
User Avatar
Your project comes under our area of interest, together we can finish your project with our technical expertise and dedicated team.
$555 USD in 10 days
4.7 (7 reviews)
5.6
5.6
User Avatar
Interesting project you've got. My experience with building web scrapers ranges from static web pages for simple periodical report creation of product inventories and as far as dynamically loaded pages via JavaScript with bot prevention systems provided by Cloudflare and Distil Networks protecting: ticketing solutions websites, stock market, and Forex analysis. Depending on the level of sophistication of the websites you need to scrape we can evaluate the proper amount of complexity in response. We can use tools such as browser kit spoofing, responsible scraping methods, human like actions, and of course the use of rotating proxies; In the scope of reducing the visibility and increasing the longevity of the scraper. For this kind of job, I personally feel more comfortable using a JavaScript framework such as PhantomJS or SlimerJS inside a Node.js application. I'm not sure what production platform on the cloud you have been looking in but Node.js should fit well and we can also look into some alternatives depending on your needs. Please do get in touch with me, I’d be curious to hear more about your project and I’ll be glad to answer any questions you may have or provide advice if you need. All the best, Andy
$900 USD in 21 days
5.0 (6 reviews)
5.0
5.0
User Avatar
Hi, In the end it depends on how well protected the site is. If there are limitations, you can't do too many things... you have to have some delays. But you are correct - running in a cloud environment might help. For example if you run the scraping process on 5 servers that means 2000 records/server. If we calculate an average of 3 seconds/page = 6000 seconds = 100 minutes to scrape all the records. BUT as i have said - that depends on the limitations of the server... maybe you need 5 seconds for a page (including sleep time). The scenario would be: store all liks/urls in either a mysql or redis database (redis has some advantages). Using DigitalOcean as a cloud provider you can create 5 vms that are using the same backend database of urls to run the scraping scripts. If you need more info feel free to contact me via chat.
$777 USD in 10 days
5.0 (30 reviews)
5.0
5.0
User Avatar
Hey There !! We have seen the job post of yours and very interested to start work with you and we completely understand the subject of yours and what is required to tailored out. I am having major 13+ years experience in Asp.net,C# mssql,html,html5, .net, MVC, MVC4,MVC5,php, MySQL,css,css3,laravel, yii, wordpress, codeigniter and also with its related frame work. I have developed various application in ajax,.net-framework,C#,css ,html5, Javascript, jquery, MVC4,MVC5,MsSql using Graphicdesign, php, laravel,yii,MySQL and deployed over the internet,If you want then i will show you the robust applications which i have developed so far. Some of the software projects and websites development that i have undertaken and successfully delivered include:( Customized ERP system for Schools and Universities, Management System for Tour and Travels industry,Real states,ticket booking software,complete ecommerce solution, Various MLM portals,bitcoin webiste Web Application for Media Company,Booking Software,Various types of CRM/ERP). Please feel free to contact us for any further discussion. Hoping for a fruitful business and chance to serve you. Regards Dunitech
$555 USD in 10 days
5.0 (8 reviews)
4.5
4.5
User Avatar
I did scarping with Amazon Seller Center using C#. For this kind of scraping I need to know the site details (from where you want to get data). So let's discuses.
$555 USD in 10 days
5.0 (1 review)
1.0
1.0
User Avatar
i have read your requirements. i have done already this job . you can get better service here. i am expert in web scrapping
$555 USD in 10 days
5.0 (1 review)
1.0
1.0
User Avatar
My specialty is web scraping and multi threaded applications. Testing today on a dual xeon 32gb server, i was able to simultaneously run 400 threads. With that being said, it really depends on your server. I can write a thread engine around your customized needs. My specialty is in the ticket industry, which is web scraping and purchasing of tickets multi-threaded, proxies, etc. Quite complicated. Thanks, Kevin
$605 USD in 2 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of CANADA
Etobicoke, Canada
5.0
114
Payment method verified
Member since Aug 12, 2013

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.