Web-scraping a big dataset

The task is simple. You have to go to this website [1] and download the publicly available data. Manually it would take forever. That’s why you will have to study the JavaScript that creates the requests. You are expected to write a script which makes POST requests and batch downloads the data.


Currently you can get 24H data for all stations. Using a script create requests for all days one by one (without overwhelming the server). Requests are processed and then the resulting data is dumped on an ftp server [2] with the respective request number. 24H data for all stations should be about 1.5 GB of data. Download them and put them on a server where we can access them in bulk.

Download procedure to be automated:

1) Enter an e-mail address

2) Select data format: Mini-Seed

3) Select time interval: Loop over A single day from 1/1/2009 to 16/10/2016

4)Select stations: Select all stations and all channels

5) Submit request

6) Wait and collect the completed request from [2]

[1] [login to view URL]

[2] ftp://[login to view URL]

Skills: AJAX, Big Data Sales, Data Processing, Javascript, Web Scraping

See more: free web phone number scrape, scraping web screens excel, item number inventory access data type, parse web pages number, big woman web, alpha web design number users, adempiere change web port number, website scraping web services, free web sms number number max chacters 160, web page click button access, screen scraping web free java, web menu table based access, scraping web excel, scrolling big image web, web html front end access database

About the Employer:
( 0 reviews ) Zurich, Switzerland

Project ID: #11804239

20 freelancers are bidding on average €170 for this job


Hi, I can download all of those. where you want me to put downloaded data and in which format ?

€199 EUR in 3 days
(221 Reviews)

Hi, my name is Sevinc. I read your "Web-scraping a big dataset" project descriptions carefully before bidding. I checked the urls, and your requirements list as well... I got what you need and ready to go ahead as so More

€250 EUR in 5 days
(267 Reviews)

Dear sir, I am scraping expert, I have did too many scraping projects, please check my reviews then you will know. Can you tell me more details? then I will provide example data/script for you. Thanks, More

€250 EUR in 5 days
(279 Reviews)

Dear Sir/Ma'am, I am a Web research, Data Entry & Webs Scrapping expert. I checked and understood your requirements. I can handle this job very well to your appreciation. I can find and extract the information More

€200 EUR in 5 days
(197 Reviews)

Hi there, I have read the project description.. I can write a web scraping script to get this data from the website. Let me know & we can discuss details.. Thanks..

€100 EUR in 1 day
(122 Reviews)

Hello, I'm professional scraper with 10 years experience in PHP & Ruby. I will be glad to apply my skills :)

€277 EUR in 3 days
(8 Reviews)

Sir i am really interested in your project . sir i am an expert of software and web development. i have already developed many web and windows applications and some are similar to your project . you can check my port More

€184 EUR in 3 days
(7 Reviews)

yes, let's clarify the requirement in details also if you want us to share our skill and previous work let us know.. hope to hear you soon.. Thanks & Regards RIMSHA

€257 EUR in 15 days
(16 Reviews)

Hello sir/madam, good day, i read your job description fully... I have worked on similar projects to what you are looking for, and I am confident that I can exceed your expectations. I can complete your project on time More

€30 EUR in 3 days
(18 Reviews)
€155 EUR in 3 days
(16 Reviews)

Hi...... I am Pritesh and I represent Pthreeinfotech. I can complete your project on time and within your budget . I have worked on similar projects to what you are looking for, and I am confident I can exceed your e More

€100 EUR in 3 days
(6 Reviews)

Hi, I can complete this task by using Selenium Web Automation in C# language. It is not going to take more than 2 days since I have been doing web scrapping in my job on daily basis. Please let me know if you are i More

€73 EUR in 2 days
(3 Reviews)

Hi there! My full-time job is webscraping and I'm available for work. I understand that you want data to be fetched from the websites and dumped in to a server. My question is how do we get a request number? Is More

€166 EUR in 3 days
(1 Review)

I believe/trust that you have come across many others who are equipped for the job in terms of capabilities and abilities; however, going through my past experience and work performance you will find out that I am one More

€155 EUR in 3 days
(0 Reviews)

Price Negotiable! python 3 years+ developer have experience with web scrapping/ web crawling for collecting malware and information and downloading files have experience with web scrapping/ web crawling for co More

€111 EUR in 3 days
(0 Reviews)
€111 EUR in 5 days
(0 Reviews)

I'm web programmer writing java scripts or tools download content from web site is kind of easy to me.

€277 EUR in 5 days
(0 Reviews)
€155 EUR in 3 days
(1 Review)

Hi I am Dinh. You can see my some old auto project in here. [login to view URL] [login to view URL] [login to view URL] Hope to work with you.

€80 EUR in 3 days
(0 Reviews)

Hi, I'm Ken Waribo. Let me introduce me as a hard working and passionate Developer who loves to achieve beautiful looking end product of targets set forth. I read through your project description and can see you need More

€277 EUR in 10 days
(0 Reviews)