Instagram / Twitter / Flickr Python Data Crawler


We need to crawl 10M geotagged data from Flickr / Instagram / Twitter to do a data visualization on the map. To achieve something like

[url removed, login to view]

Freelancer will need to deliver


1. register Flickr / Instagram/ Twitter dev account

2. research their API to write a crawler to grab the data within the geofence bounding box. e.g. San Francisco bounding box: [url removed, login to view], 37.6040, [url removed, login to view], 37.8324.



1. three daemon/service-like python programs to crawl the geotagged data from Instagram / Twitter and Instagram and stores these data into the NoSQL database MongoDB.

2. It should be stable enough to crawl the data 24/7.

3. It should crawl 1 millions geotagged data per week even given the rate limit of the APIs.

4. the programs must have scalibility and multithread ability like queue library e.g. Celery in Python.

GEOTAG is a must! we don't need data with no GPS information.

Qualities needed to be successful

Python Experience to write service / daemon like

MongoDB, Redis, Celery

Twitter / Instagram / Flickr API experience.

Other Skills: Data Science Data scraping MongoDB Python Redis Web Crawler

You will be asked to answer the following questions when submitting a proposal:

(1)Have you written a Python crawler to use Twitter / Instagram / Flickr API before?

(2)Have you used any queue library (e.g. Celery) with multithreaded workers in Python to write daemon/service like program?

(3)Have you used any noSQL database before to store data like mongoDB?

(4) We want to estimate how much time you need to put on this whole project.

(5) And we want to set up with a small interview milestone to test: simply use your API to grab 10+ Instagram, Flickr and Twitter raw json data with GEOTAG (latitude and longitude).

(6) Next question will be how can you deal with rate limitation while crawling data? Multiple IPs / accounts ?

Skills: NoSQL Couch & Mongo, Python, Software Architecture

See more: software read write data smart card, web crawler example python, simple web crawler using python, python twitter crawler, twitter crawler python, web crawler data scrap, web scrape movie data python, web crawler data excel, need web crawler data, flickr crawler python, python script query twitter data, web crawler data mining java, crawler write data, project python twitter data mysql, java web crawler data mining, web crawler data extraction, twitter data crawler, twitter crawler data, data gathering crawler python, software edid data write

About the Employer:
( 0 reviews ) Atherton, United States

Project ID: #11816564

11 freelancers are bidding on average $859 for this job


Greetings!! I am very thankful for this opportunity. It’s really exciting that we have similar kind of expertise and work experience. Scrapping data from domain site: [login to view URL] htt More

$2882 USD in 25 days
(17 Reviews)

Answers 1. Yes and many many more social networks. 2. I have worked with multithreading in python. I use a custom que and concurrency managing 3. Yes i have worked with mongo 4. 7-10days 6. Multiple accounts with More

$500 USD in 14 days
(58 Reviews)

I want to discuss this project with you further, let me know the best suitable time for you to schedule the meeting, Feel free to message me at any time, i used to be online 14 hrs in a day on this website so probably More

$882 USD in 18 days
(7 Reviews)

Hi, Nice project you have there, let us help you with it! Our team is Russian-Canadian. We code in PHP. Check our recent projects here [login to view URL] Here’s a little video about our team too! ht More

$750 USD in 7 days
(9 Reviews)

Can do this in PHP instead of Python. No need for multi-threading. You'll run into rate limiting before tread limitations if coded properly

$600 USD in 10 days
(22 Reviews)

My name is Mike and I’m from UK. I work with individual clients and also provide outsourcing services for a number of UK and USA based agencies. Your project description sounds interesting to me and I do have skills & More

$555 USD in 10 days
(3 Reviews)

Dear Hiring Manager, First of all thanks for creating an opportunity. Hope you are doing well. I will not propose what the maximum Freelancer do. I dont believe in copy paste Cover letter. I read your project details, More

$705 USD in 7 days
(7 Reviews)

I have attached sample video of crawling instagram [login to view URL] Also I can do what you have mentioned in your description. Secondly I am okay if initially you More

$500 USD in 10 days
(1 Review)

Hi,I have gone through your project description. I could be confident if we can proceed towards more discussion. I am an individual developer and you will be working directly with me if we proceed work on this project More

$833 USD in 15 days
(0 Reviews)

Dear Re: Freelance VA position Please accept this as my application for the position of freelance VA with Peace, Love & Tambourines. Here is the crux of the qualifications that I present: • Track record of providing More

$751 USD in 21 days
(0 Reviews)
$493 USD in 5 days
(0 Reviews)