Remake a scraper, make a new one and make a working Database

We want to highlight buildings that have apartments through Mapbox. Each of these highlighted buildings should have information about the apartments in it (the one we get from our working scraper which we should remake by adding some new fields). Mapbox will highlight apartments via a certain parameter, in our case this parameter is av = 0 and av = 1 (av - availability), where av = 0 - there are no apartments in the house, it is not highlighted it and av = 1 where there are apartments should be highlighted.

What we need:

A database (we think of SQL, but there may be better options), which will store the following data: coordinates of the house (in which the apartment is), address, links to the apartment on [login to view URL], description of the apartment (need to be added to the scraper), price, area, number of bathrooms and bedrooms, links to images.

We have one big .geojson file with all the buildings in New York City. It serves as the system layer in Mapbox. This file contains parameters such as PID (polygon id), height (for 3d buildings), coordinates of the polygon and the parameter av = 0 (because buildings are not highlighted).

We have a working scraper. It should scrape the necessary information and output it in the format .geojson, with the addition of the parameter av = 1 (because such buildings should be highlighted). The keys in the scraper should be instead of Number of bedrooms key changed to Number_of_bedrooms. When this file is ready, it should be uploaded to the database. In the database it is necessary to synchronize two .geojson (system and scraped one), that is, fill apartments info in the empty rows of the corresponding coordinates, and therefore it is necessary to check whether a certain apartment is not already in this base (the base should be updated every 4 hours, it is necessary to check coordinates and other info whether this apartment is in the DB. As the DB already has a system layer, which contains information about the polygons (ie the buildings themselves) you should not forget that the scraped info doesn't contain polygons info, it carries only the coordinate of the building, so you need to check which of the polygons is the coordinate of the scraped apartment and add the information we need to the database. There are two options: inside() function or others which allow us to check if a certain coordinate is in a certain polygon (in this case it is necessary to check in our system .geojson, there are all polygons) and the second option: our own table "polygon-coordinates": each polygon corresponds to certain [login to view URL] on this table should be faster than programmatically comparing coordinates with all polygons each time (and the .GeoJSON system contains more than 1 million buildings and weighs 400+ mb). We have a csv file that has list of addresses and 700k+ coordinates. So you should make a small scraper with the function inside () or analogue and output it all in one base, in which each coordinate corresponds to a specific polygon. There may be cases when not all the coordinates of all the NYC buildings are in the csv file, which means that when the such data arrives, it must be checked first in the "polygon-coordinates" table and then (if there are no matches), check them separately on the function inside () or analogues. When it's over, the database should be like this: there are many polygons (system layer + those highlighted). The system layer polygons have av=0 parameter, the highlighted ones have av = 1 and information. The database then needs to export it all to a new .geojson file. This file will be updated every 4 hours as scraping will take place every 4 hours. Therefore, it is important to keep in mind that you should constantly check whether a particular apartment is already in the database (starting with the coordinates, description and ending with the price, as there is often a price update or 5 apartments in the same house).

Each polygon corresponds to only one building.

Skills: Web Scraping, Database Development, API

See more: web scraper save mysql database, working database joomla, working database netbeans jsp example reservation, confidentiality agreement someone working database, agriya working database, access working database, working database php images flex, car database make model download, contact database email search scrape doctor chiropractor trainer list, flash working database, jsf framework working database, make script scrape, wpf working database, isa working database, interface matlab working database, working database school, autohotkey script working database, working database wxwidgets, create working database huffman fleet truck maintenance, how to make 5 page website

About the Employer:
( 3 reviews ) Kyiv, Ukraine

Project ID: #20963041

17 freelancers are bidding on average $192 for this job


Hi there,I am Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this project ! I can start immediately and finish it within the More

$450 USD in 7 days
(115 Reviews)

Hi.Dear... I can scrape anything.I know all the angles of website scraping. I have much experience for 10+ years such as avoiding recaptcha, using multi threading, avoiding site security.... I can do your job with craw More

$140 USD in 7 days
(38 Reviews)

Dear Sir, I am interested in your project. I have gone through your requirement. I'm expert in web scrapping and web automation using selenium and jsoup, data management, data mining. I assure, I will do my best to w More

$200 USD in 7 days
(29 Reviews)

I have lots of experience writing web automation scripts. Available to start immediately and finish as soon as possible. Please contact to discuss details if you are interested. Looking forward to work on this project. More

$140 USD in 7 days
(29 Reviews)

Hello, After reading your project details I believe I'm suitable for this project. As I'm expert on it with more than 7 years experience. Please feel free to contact me. I am looking forward to hear from you. More

$70 USD in 4 days
(25 Reviews)

Hello\nI am a python developer.\nI have great experience in web scraping and I am an expert in it.\nI have all necessary skills by which I can scrape any website.\nPlease message me to discuss in detail.

$140 USD in 2 days
(17 Reviews)

Hello, I have Experience with scraping Websites like FB ,IG and Telegram and Experience in Google APIs. I have 5 years of experience with Python. I have worked with many libraries in python for tasks such as Data Analy More

$60 USD in 3 days
(12 Reviews)

Greetings I have 3 years of experience in python, web scraping. I have scraped more than 100 websites. I can help you scrape this website . I have looked at this website. Ping me for further discussion

$140 USD in 7 days
(5 Reviews)

hello sir, i worked on several scrapping projects, i read all the description you write, and i understand what you want, i just need to discuss with you some details. i am available, ready and highly motivated to work More

$140 USD in 4 days
(2 Reviews)

Hello, I am an experienced developer and coder with very good experience in data scraping. I can make the scraper exactly as per your requirements. As the data set contain a large amount of data using SQL type databa More

$300 USD in 20 days
(2 Reviews)

Hi, My portfolio site, [login to view URL] My Github site: [login to view URL] Please do leave a message, would be interested in hearing about the requirement and the objective that More

$140 USD in 7 days
(0 Reviews)

Greetings! My name is Daniel and I am a software developer from Germany, specialized in python applications. I have finished many web scraping projects and after looking at your attached flowchart, I am quite confident More

$250 USD in 7 days
(0 Reviews)

Hi, I already created the scraper which will scrap all these details from trulia site. Kindly message me if you need this. Thanks

$133 USD in 1 day
(0 Reviews)

I have a good scraper already produced for other projects that can easily be customized for this specific use. I also have a business intelligence database system that I developed as an open source project that can be More

$200 USD in 3 days
(0 Reviews)

Good day! This is my woocommerce shoping site. [login to view URL] I have developed it from scratch to the end. and I have many experiences in laravel, wordpress, CI, python as backend. And I had several experiences in angu More

$140 USD in 7 days
(1 Review)

Dear sir. I think you need web scrapping for your data. It's easy. I will suport you by using IronWebScraper-C# framework or NodeJS-Nightmare Library , Dataminer Scraper, and Python in your favourite ways. I have full More

$125 USD in 7 days
(0 Reviews)

Dear Sir or Madam: I am doing crawling, scrapying and ETL for a long time, many proyects.I have also worked with geolocation API of Google. According to your description a succesful system could be achieved within a m More

$500 USD in 30 days
(1 Review)