Design of web crawler/web spider

Hi there! I am an expert American programmer specializing in webscraping with experience developing custom applications and collecting data from hundreds of websites for clients here on Freelancer. For this project I would develop an application in VB.NET which runs on any windows PC and goes to each URL in your list, crawling every sub-link it finds, looking for the keyword's you specify. Each time it find's a match it will then output the data you need (100 characters to either side of the keyword "hit"). The app would let you input a simple spreadsheet with a column list of URL's and a column list of keywords which you can change anytime you need. Please send me a message so we can speak further about the project details! Thanks, Mike

Dear Sir, I'm very much delighted to let you know that i did data scraping with PHP-cURL, Node.js, Selenium from many sites. I just scraped the data from web site and then wrote the data in mysql database or excel or csv or xml file. I worked on many similar projects, I have big experience in data mining projects. I have written hundreds of web scrapers which scrape millions of pages each day. I'm ready to fulfill your requirement. I can finish this task in short time, with the best quality. I can assure 100% accuracy. Please give me the opportunity to do the work. With Kind Regards, Debdulal Roy Proshanta

Hi, I have more than 14 years of Web scarping exp and I am expert in this kind of work. I have completed more than 270 projects. Please look at the feedback left by my employers to know more about my work. Waiting for your positive response. Thanks.

Hi. I can start work on your project right now. But I need more details about your requirements. I have experience in scraping different sites from simple to weight rich sites, that uses javascript for content generation. Thanks anyway.

Hello, I can write a php code for you to collect data from your desires website and in your desires format to store into the database. as well as we can set that script to collect data with specific tie intervals. Please let me know the website from where you want to collect data. so that i can give you the time-frame for this project. Have a nice day. Thanks, Muhammad Jawad

Hi there. I've got some questions about your project: 1- Do you want a scraper that navigates a particular website in a predefined way and extracts data from known sections, or do you want a more "generic" scraper that can navigate (almost) any site and tries to extract data from it (these are actually called crawlers)? If what you need is the first option, and you have between 10 and 30 sites in mind, then you will also need between 10 and 30 scrapers. 2- If you are thinking in the second option, how should it behave? Should it follow every link it encounters or just stay in the home page? I've written several scrapers with Scrapy, which is Python framework. Although all of the scrapers I write are of type #1, Scrapy has features to support type #2 scrapers (crawlers) an it would be a fun challenge for me. You can check my reviews as evidence of my work or I can provide some code if you want. Feel free to contact me.

Hi, I can do it very quickly with Scrapy. It's a very fast python crawler. Let me know. 3 days max. probably less. thank you

Hello! I understood the task and I can implement the required functionalities. I have great experience in performing tasks like this and I have positive feedbacks about it in my profile. Before I get paid I will provide a proof in order to guarantee that it works correctly. I can begin to implement the task immediately. Thanks.

Hello, Having experience of crawling quite handful websites in scrapy in Upwork and freelancer, I assure you I can provide the deliverance that you require. I have used scrapy for almost two years now. I am familiar with working aroung IP bans using rotation, using concurrent requests and time request per minute, using selenium to crawl visible only data (with or without PhantomJS as ghostdriver), and avoiding honey pots and tarpits. I work fast and diligently. My work history can affirm that. Apart from scrapy, I am well versed in selenium, requests library and creating socket class level scrapers, if needed, for TCP stream. If the data is present in the site I have been able to deliver them to the client in the format they require. I hope we can work together as I am very much interested to work in this project. Also, I have succesfully setup crawlers in client remote server, setup cron jobs to periodically scrape them. Apart from that I am actively involved in Natural Language Processing, hence any semantically related data crawling using intelligent algorithms too is my forte. PLEASE CLARIFY ME ON THIS SENTENCE: "It will record the details of key word hits, to 100 characters either side of the hit on an excel document." I know that you want to count the search words in those 10-30 websites and save in excel. But, that sentence is quite vague can you explain it to me? Regards Ashmit

Hi, expert programmer and web/data scraper here with over 19 years experience in programming and RDBMS. Please see my reviews. I'm using python for this kind of jobs.

Hi There, I am an English speaking native and I've written many python webscraping scripts for my own projects. I am bidding a lower amount, as I understand that I don't have the Freelancer reputation. If you partner with me, I will: - Work with you to ensure clarity on your requirements - Prompt communication with an English-first speaker - Provide a minimum product to you within 3 days of confirmation - this will give you confidence in my skills. If you're interested in the python packages: requests - for HTTP requests to the websites in question BeautifulSoup - to parse webpages dependant on your needs; this might include crawling through hyperlinks as well xlsx - it appears at though there will be an excel requirement as well. Simply, either I deliver in full to your expectations or it's free. This is a no-risk situation. Please contact me should you have any other questions. Mark