Find Jobs
Hire Freelancers

Web Scraping Needed - 22 Million Records

$30-250 USD

In Progress
Posted over 4 years ago

$30-250 USD

Paid on delivery
I need a freelancer that can crawl a list of 22,000,000 domains that I will share in four CSV files that you will download from Google Storage. The crawler needs to crawl ONLY THE LANDING PAGE - not the entire site - and capture the following and output a CSV file and stored to Dropbox: 1) Does URL have Google Analytics code - yes or no. Use a search for "Google Analytics" in the source of the page. 2) Is there a link to a privacy policy on the page - yes or no. Use a search for the word "Privacy" in the link text 3) How many unique internal URL links are present on the page. Return href count. 4) Is the URL secure (SSL) - yes or no. 5) Is the URL mobile-friendly - yes or no. Use a search for "meta name="viewport"" in the source of the page. 6) Is the domain parked - yes or no. Search for [login to view URL], [login to view URL], namecheap, uniregistry, adsense anywhere in the source code OR the words "for sale" or "See related links" the meta description. 7) Is a phone number present on the page? yes or no. Capture the phone number. 8) URL being crawled. 9) Does the domain resolve to an actual website (no timeouts, 404s, etc) You MUST have your own server or equipment to do the scraping. Project must be complete in 5 days.
Project ID: 23751802

About the project

11 proposals
Remote project
Active 4 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
Hi, I am a java programmer good at data scrapping. I can download the 22,000,000 landing pages by applying multiple-thread to improve the downloading speed. Once all the pages are downloaded, to generate the result you need is easy task. I am ready to work now . I can finish your work within 5 days.
$160 USD in 5 days
5.0 (30 reviews)
4.8
4.8
11 freelancers are bidding on average $245 USD for this job
User Avatar
Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON (Scrapy, Selenium) based web scraper as well as WINDOWS BASED web scraping software through which I have crawled many sites such as Craigslist, Amazon, Yelp and many others. I have also worked on complex site to bypass CAPTCHA with the use of PROXY IP bouncing techniques.. Let's work together :)
$222 USD in 5 days
4.9 (94 reviews)
5.7
5.7
User Avatar
Hello From your project I understand that you need a developer to web scraping. I want to inform you that we are a team of freelancers who have experience of over 7+ years. And after reading the description I can guarantee you the best outcome of the project if we are working on this together. Please give me a chance to prove my skills and you won’t regret it. Looking for a long term work relation with you. Thanks
$140 USD in 7 days
5.0 (18 reviews)
5.2
5.2
User Avatar
Dear Sir, I am interested in your project. I have gone through your requirement. I have gone through your requirement, we can complete your task within 5 days by sure. please ping me we can start right now. I will provide you a demo within 24 hours. I'm expert in Core Java, J2ee, Javafx, Spring boot, Spring websocket, Spring JPA, Hibernate, JSF, Angular, ReactJs, React native, web scraping, web automation, mysql, mongo, postgreSql. I assure, I will do my best to work with you on your project to present the best possible outcome for you and your customers. I will also do my best to correct any area of work where quality comes into question, I want to have pride in my service to your company and the final product provided. I look forward to working with you. Thanks & Regards
$750 USD in 5 days
4.7 (30 reviews)
5.5
5.5
User Avatar
I have over 7+ years of experience along with in-depth technical knowledge on Python, Flask, Node.js, JavaScript, PostgreSQL Administration, Django, MySQL, MongoDB, XML-RPC services, stores, ERP/CRM portals and implement the fully innovative and creative strategy to deliver with outstanding functionalities and user experiences. I have rich experience in web scrapping and web crawling and recently i have done similar kind of projects. >>I can extract data on any website using python. >> I can do Data RESEARCH from any platform for certain subjects. >>Expertise in python,django,flask. >>I can stretch my time according to your time zone as well. >> I can Find data from INTERNET RESEARCH/ WEB SCRAPING. Please initiate the chat for further discussion. Regards,
$250 USD in 5 days
4.9 (6 reviews)
5.2
5.2
User Avatar
Hi Hiring manager I have read your all given information in description and i am ready to do various type of work for you and available 24*7*365 for you. 1* I can do Data RESEARCH from any platform for certain subjects. 2* I can create the accurate DATABASE (EMAIL/CONTACTS/PERSONAL INFO etc*. 2* I can properly fill a Google Sheet/EXCEL/WORD. Have good working skills of Excel/PDF. 4* I am Expert in DATA ENTRY and do any kind of that work. 5* I can Find data from INTERNET RESEARCH/ WEB SCRAPING. 6* I can Work as a VIRTUAL ASSISTANT for daily any kind of task. 7* I can work as per client TIME ZONE if they want and provideS 15hrs services in a day. 8* I have 3YRS of good experience of as a VIRTUAL ASSISTANT and done various type of task. 9* I have the excellent skill of SEO/DIGITAL MARKETING/SOCIAL MEDIA MARKETING/MANAGEMENT. 10* I can create MEMES/posting,strategy for increasing likes,followers of social media. 11* I have expertise in WORDPRESS sites creating/product Uploading/WOOCOMMERCE. 12* I have Good skills in SHOPIFY, PRESTASHOP, MAGENTO etc for DROPSHIPPING. 13* I Can Write Optimized SEO friendly impressive articles for BLOG and get 1 rank. 14* I am fast Learner. Grab The instructions from client instantly and implement. 15* I can get 100% results and Give Guaranteed work satisfaction from client instructions. for work i am always available and take the challenges and never say no to work always be ready and be professional in work. thanks
$140 USD in 1 day
4.9 (19 reviews)
4.4
4.4
User Avatar
Hello There, Hope you are doing well ! Saw your job posting and had carefully gone through your requirements for website. I hastened to submit my proposal for your review. I have 4 years of experience in the field of design/development, Python, Django, PHP, Angular I promise the highest quality of work that suits your unique styling. Thanks and Regards, Tabish Adnan!!.
$250 USD in 3 days
5.0 (3 reviews)
2.6
2.6
User Avatar
Hi there! I have read your description carefully and I am confident I can full fill your requirement. I provide my skills in hopes of working with you. I am a web scraper and have 7+years of ample experienced in web scraping. I can extract data on any website using python and PHP script(beautiful soup, selenium, scrapy, phantomJS) I can save data scraped into xlsx, CSV, MySQL and etc using CSV module and xlsxwriter Question Would you like to let me know your target website URL? please feel free to contact me if you want. I look forward to working back with you. Best Regards.
$250 USD in 7 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hi, Having read your complete requirements, I would like to assure you that I will produce the most notable results for you, having had a myriad of previous experience with similar projects. I would like to discuss the thorough details with you to fully grasp the requirements. i have my own server - let's talk. fell free to contact me
$140 USD in 7 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hi. nice to meet you. i have experience in web scraping in php and asp.net. Let's discuss more details over chat. Thank you
$250 USD in 7 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of UNITED STATES
Sheb, United States
4.7
20
Payment method verified
Member since Jun 16, 2009

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.