Build a Python Based Web Scraping Application

Closed Posted Feb 8, 2015 Paid on delivery
Closed Paid on delivery

I'm looking for an expert in Python and Web Scraping to take on a very specific project. The ultimate project will actually NOT be Django, but rather using a microframework called Flask (If you have Django experience, Flask is about a one day learning experience).

For Python, we prefer people with Scrapy experience, and would like to stick with Scrapy and Flask.

Essentially, you'll be building a spider that will hit a specific website, and your job is to crawl approximately 20K pages. The first time, we do a full scrape, if the page exists in the Database then you'll check for changes and updates. Accuracy is 100% vital.

All data should be collected using JSON, the spider will of course run server-side, but there needs to be a corresponding web interface for manually performing CRUD operations. You should also note, that we expect the collected data to be stored in a MySQL database (which should be appropriately indexed and normalized).

Finally, it is an absolute must that you have GIT experience. Not just doing code checkins 1x per day, but routinely doing pushes to our repository. If you don't understand GIT and how team development works with a DVCS, please don't waste the time bidding. Also, if you don't get anything herein, again don't waste our time.

If you send me 20 useless website URLS I'll passover you entirely. I'm a little jaded by people that keep submitting the same "blanket response" to every project. You have enough information to accurately bid within reason, but ask questions if you don't.

Django Python Software Architecture User Interface / IA Web Scraping

Project ID: #7108070

About the project

25 proposals Remote project Active Mar 11, 2015

25 freelancers are bidding on average $1269 for this job

phpXpertbd

Dear Sir, I'm a seasoned web scraper. Have written hundreds of web scrapers which More

$1250 USD in 20 days
(72 Reviews)
7.3
flashsaiful

Hi, I can do this for you. Please send a massage in the PMB for details.......Best Regards flashsaiful

$1250 USD in 20 days
(127 Reviews)
6.6
johnpo

A proposal has not yet been provided

$1250 USD in 20 days
(56 Reviews)
6.3
anuyadav1

hello i have good experience in writing crawlers in python . And also with scrapy and crawlera . i can work with django,flask and git . thanks

$1400 USD in 10 days
(77 Reviews)
6.2
stasshtin

Hello, I would like to help you with this project. I've worked with scrapy before, and 20k items is certainly doable with it. I must say, if I were to choose web framework, it would be django rather than flask, as i More

$1400 USD in 15 days
(23 Reviews)
5.7
Fortut

A proposal has not yet been provided

$750 USD in 20 days
(107 Reviews)
5.9
JoshuaCru

Hi, I have experience working with flask, scrapy and mysql and GIT. I'm also proficient in regular expressions and general web development which come in handy for web scraping. I've done web scraping projects in the More

$1250 USD in 20 days
(13 Reviews)
5.2
oeermanz

Hello, sir. My name is Alex, i'm professional python developer. About my bid first - i'm just making portfolio on freelancer, so my rates are low now, later it will cost twice =) I like your project and i'm good can More

$833 USD in 15 days
(3 Reviews)
4.9
sunsl

Hello, I have experience with Flask/SQLAlchemy, will use Scrapy to do the crawling. My solution will keep the historical downloaded data, and will provide a web interface to view it. Updates will be done using HTTP H More

$1000 USD in 10 days
(1 Review)
2.7
Nicolastrognot

Hello Sir, I can develop your web scraper, I have already developed this type of work using BeautifulSoup. It's nota problem for me to use Scrapy. For example, I get the entire database of citizens in France usin More

$1250 USD in 25 days
(0 Reviews)
0.0
timsjackson

I have substantial experience developing services and applications in Python, Django and Flask which would be ideal for this project. I've around 5 years commercial experience with Python, and over the past year have u More

$1111 USD in 7 days
(0 Reviews)
0.0
BigBigface

This is a expert python developer with experience in Django and Scrapy. Hope to establish cooperation relationship. Regards.

$1250 USD in 15 days
(0 Reviews)
0.0
rrargh

I used Python and Scrapy to extract trading card data from four different HTML-based sites. The data was extracted in JSON format, and the script was manually run from the server side, with no available CRUD web inter More

$1400 USD in 15 days
(0 Reviews)
0.0