Crawler to Automatically Collect Data from Websites

I need an experienced developer that can write an application that has crawler type of functionality that will do the following things: 1. Have the crawler go to the specified website every day at the time that I specify – login with the username & password that I will also specify (I should be able to type in this information into the application’s interface). 2. Then the crawler should download the daily five .csv files that contains various user type of information into a folder that the application should automatically create on the local hard drive. 3. Once these files are downloaded then your code should parse these files and extract the user’s first name, last name, and email address. Keep in mind that each one of these .csv files could have its own unique format when it comes to storing this information. 4. And once your application has parsed these files and retrieved the user’s first name, last name and email address then your application should automatically insert this data into a database that I will provide for you. This is the main functionality of this application – but, in addition to all of the functionality that I outlined above, your application should also do the following: 1. On the day of the month that I specify – please have your application login to the same website as specified above and download a monthly .csv file that your code should parse and extract all first name, last name and email address data and automatically insert into the same database & table as above. In other words – this functionality is exactly the same as above, except that it will only happen once every month. 2. Your application should also go to another website and automatically log in with the username and password that I will provide you with. Then your application should download the daily .txt file (only 1 file per day) and extract the first name, last name, and email address data from the .txt file and insert this data into the same database (different table). 3. On the day of the month that I specify – please have your application login to this second website and download a monthly .txt file where your code should parse and extract all first name, last name, and email address and automatically insert into the same database as specified earlier. 4. Then I want this application to automatically launch a webpage and allow the code to run within the background so that I don’t see it on my computer screen. Now, this webpage has already been coded – so you won’t have to do any of that. All your application needs to do is to automatically launch and allow this page to execute until I stop it. Of course the application’s interface should allow me to enter this page’s URL that should be automatically executed in the background. Now, that concludes the functionality description for this application – and now, here are some of the features that I want for this application. 1. I want this application to be written as a client server application that can be minimized to the system tray where it could automatically run in the background without any intervention from me. In other words, I want everything to be fully automated and if there’s any kind of problem then I want to be notified. 2. I want this application to give me the ability to type in all of the settings that I’ve talked above. In other words – I want to be able to specify the hard drive location where all of the files will be saved, the username and password for both of these websites, the day of the month for both monthly downloads, and I want to be able to type in the connection string that this application uses to connect to my database. 3. I want this application to never export the same data files to the database. In other words once a day’s data has been exported to the database then it should keep track so that it doesn’t do it again. In addition, I do want some type of reporting so I can make sure that everything is working properly. 4. I want to have the final word on how this application looks. In other words – in the beginning you can make the application look in anyway that you like, but at some point I will provide you with a design that you should implement for the application. 5. This application must be completed within 3-4 business days – I’m on a very strict and tight schedule and I urgently need this application to be completed as soon as possible. And, once the first four items have been completed then and only then I will fund escrow. In others I want to make sure that you can do this type of work and once I start seeing some results then I’ll gladly fund escrow while you complete this application. But, if you can’t complete the first four items that I’ve listed above then I reserve the right to look and hire another developer that can do the job. 6. In addition, I want a developer that I could come back to and re-hire to add other websites with their corresponding downloads and importing of data as needed. In other words – if you do a good job, you’ll get additional work from me... So, if you’re an experienced developer that can do this application as specified above then please place your bid. In addition, please be sure to tell me what kind of previous experience you have that qualifies you to do this project. Here’s the criteria that I will be judging all bidders on: 1. Your Bid Amount 2. Time of Delivery 3. Experience Thank for taking the time to read this and good luck to you… Gabe U.