Scrapy and Python - Web Scraping

Use Scrapy to go through a list of websites and crawl all pages of the website, then save data for analysis. (A) Save page info in a text file: (1) Creates and overwrites existing folder (a) folder starts with web site name (b) folder ends with date (yyyymmdd) (2) create pipe (|) delimited text file (3) Data headers: - URL of each page - main website address ( same as website in folder name) - page title, if found - page category (later phase, leave blank) - sub page category (later phase, leave blank) (B) Save links for each page - URL of each page link is on - main website address ( same as website in folder name) - link name - link URL (C) populate page category and subpage category fields (1) Uses rules to determine (2) Blank if unknown (3) Must be in a tables, and able to easily add and edit rules, as well as indicate if multiple rules or a single rule is required for a positive result Examples Link text to page = "Find a Doctor" then "provider search" Link text to page = "Find a Provider" then "provider search" Code must be commented. - Scrapy used for web scraping - data will feed into MySQL with pipe delimited text files - rotates through 20 sites at a time, alternating between which site is scraped, so as not to overload or trigger DOS attack.