[login to view URL] crawler
$30-250 USD
Paid on delivery
We want a crawler to capture data about jobs and companies advertising those jobs from Monster.com. It should be written in c# and use a SQL Server database.
1. Start crawl at [login to view URL]
2. Browse jobs by Category, looping through each category
3. Jobs are returned by Ajax. The program should retrieve all available jobs, including the ones that can be accessed by repeatedly clicking the "Load more jobs" button in the web user interface.
4. For each jobs, capture/save to database: Job title, company name, When posted, city, state. Also, use a regular expression to extract any email addresses from the job ad text and associate those with the job ad in the database.
5. Program should be able to use Proxies
6. Allow proxy IPs to be configured in a database table which includes proxyip:port, username, password. If username and/or password are not null, they should be used as the proxy server credentials.
7. Program should switch to a different proxy IP (the one used least recently) for each job title it searches
8. activity of the crawler should be logged to log files that include the date in the filename. For example, ljs[date].log.
9. at startup, the application should delete any logs that are more than 4 days old
10. crawler should be able to run unattended by the windows task scheduler
11. when crawling before retrieving a job, check to see if it's already in the database. If it is, skip it
12. crawler should pause a configurable amount of time after each HTTP request (default 500 miliseconds)
Project ID: #18575829
About the project
Awarded to:
Hey I'd love to develop you a Windows Desktop Application to crawl Monster and scrape up all the jobs based on the categories and such. This is an interesting project to me and would love to work on it and provide y More
13 freelancers are bidding on average $428 for this job
Dear Employer, I will do everything that you need but for that I would need atleast 500usd to configure this. Please message me if fine with you. Thanks and Regards, Ash
I would like to inform you also that I am highly skilled with : • 10+ years of professional experience in C#/.NET software development; • 5+ years Experience with ASP.NET MVC , LINQ , EF , SQL; • 10+ years Experie More
Hello, I believe i can develop this software for you, as i've worked alot with HTML elements in C#. I can gather that information and place it inside the SQL server quite easily. I look forward to working with you. B More
Hi, I have now gone through the requirement in detail. I have completed although not same but similar projects. . Please feel free to contact me to discuss in detail.
