Google News crawler

Google News Crawler Need a program to crawl [login to view URL] -- the program should be able to run unattended and log its results to a log file. It should automatically create a new log file every day (even if the program doesn't stop at the end of the day), so that the log may have a filename format of news[date].log. The program should use a SQL Server database (you can create the db structure). It should be written in C# It should crawl each of these top-level categories: Business, Entertainment, Sports, Technology, Science, Health, Society, Politics, World In addition, it should crawl the top 200 metros in North America using the &geo= querystring parameter (we will provide a list of the top 200 metros) The first step the crawler should take when it navigates to a new top-level category is to parse the list of subcategories (which appear in the left-side navigation in the Google News interface) Then it should navigate to each of the subcategories to retrieve the matching articles. If Google News returns new subcategories, those new subcategories should be saved to a DB table and associated with their Master category. The primary objective of the crawler is to obtain contact information for Journalists. So the program should save the following information to the database: 1. Journalist name 2. Journalist’s title 3. Journalist’s beats 4. Journalist’s sub-beats 5. Journalist’s tags 6. Journalist’s location (city, state, country) 7. Journalist’s email 8. Journalist’s social media links (LinkedIn, Tweeter, Facebook pages) 9. Journalist’s website 10. Journalist’s association (news media) 11. Journalist’s About info 12. Dates of publications and Titles The crawler should use Proxies. The configuration of the crawler should specify if it should only focus on news articles added within the past day, or if it should crawl articles more than 1 day old. There should also be an ability to specify which top level beats to crawl. This way, we could potentially have multiple instances of the crawler running simultaneously. We may want to have one instance that crawls just recent articles, and then two or more additional instances that crawl historical articles (which could each be assigned half of the beats). The program should be able to crawl multiple pages of Google News results. Some news sites include journalist contact information in the articles themselves. Some have a link to the journalist's bio & contact details from the article. And these are in different places on different sites. Because different news sites put contact info for journalists in different places, the program should have some flexibility to be able to try a number of the likely places where the contact info may be to try to retrieve this information. However, if the crawler checks more than 3 articles on a given news site and does not find an email address associated with the author of the article any of those times, it should stop trying to search for journalist contact info on that site going forward (in other words, ignore that site when it appears in Google News results).