web crawler with javascript support

I need a webcrawler library which is supposed to run in the backend/docker with some JavaScript support. There are many websites which are inserting some JavaScript checks in their page and I want them to pass

What it needs to support:

1. I need a way to send standard HTTP requests with an URL and some parameters (POST/GET/PUT, etc). At best, I would like this call _NOT_ to be asynchronous.

2. A way to store cookies from request to another, like a cookie jar;

3. I need to get the raw body of the request - the way server responded to the client;

4. I need also a formatted HTML (tidy/*) of how the document actually looks like.

I would like the bidders to start their bidding post with "+++". This is how I can tell if you actually read the post. I will simply ignore/remove the others - there are too many to read and many of them are nothing but automated posts/spam.

The language you choose is not important as long as it respects the things above. I would prefer Java or Python, but I can accept others - just let me know about this in your post. Again, this will be running in an linux docker, so be prepared for this.

I am a programmer myself and I know this can be done. I did it in the past, meanwhile technologies changed and it may take a while until I am up to date with everything. This is the reason why I need your help. Also, this means I have a fair idea of the effort needed, so don't go wild with your bid here.


* full source code;

* a small demo in docker on how you will crawl [login to view URL] - 4-5 pages at your choice, no need to interpret the results;

Skills: Java, Javascript, Python, Web Scraping

See more: java web crawler image capture, java spring web crawler, java web crawler username password, java web crawler auto login, web crawler java reviews, web crawler javascript sql dynamic html, dont find java project web crawler free, java multithreaded web crawler, web crawler find persons posts, web crawler supports javascript java, multi threaded web crawler java, multithreaded web crawler using java, java implementierung web crawler, java web scraping javascript, web crawler java, java based applet web phone full support sip protocol java javascript, web crawler javascript

About the Employer:
( 8 reviews ) Sterrebeek, Belgium

Project ID: #17951269

25 freelancers are bidding on average €573 for this job


Hello!! I am Gang Lee and I read your post. But I have something to ask you. Your idea is amazing and it will change the world! I am a magic talented developer in your skill. If you wanna be the success, hire me More

€555 EUR in 10 days
(589 Reviews)

"+++" ... I have designed many webcrawlers. One of them use Web Driver, Selenium, Java, Http POST/GET, login, Cookies (after login), HTML screenshots and few other things. It was pure Java implementation. I already hav More

€900 EUR in 10 days
(122 Reviews)

Hi there,I am Miljan,Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immediately and finish it within More

€555 EUR in 10 days
(84 Reviews)

Hi, I can start work on your project immediately once you confirm me and i am also open for further discussion if you want. I have 8+ years in web design & development with good experience in following skills ... More

€500 EUR in 10 days
(61 Reviews)

This is Vibrant Webtech and I was glad to see that you're looking for help for project web crawler with javascript support. I've delivered more than 400 + projects in the last 5 years and this makes me very confident More

€600 EUR in 6 days
(51 Reviews)
€450 EUR in 12 days
(12 Reviews)

Hi, Dear. Nice to meet you. I've read your post carefully. I'm Python and Web expert. Please discuss more details by chatting. Regards. Gao M.

€555 EUR in 10 days
(41 Reviews)

Hi there, I’d like to be considered for your job position. I’m a Web Developer with a strong background developing with back end & front end. I can turn your requirement in a way that represents your brand and app More

€750 EUR in 10 days
(17 Reviews)

Hi. I saw your description carefully and I think I can help you. I'm Web scraping expert, I have more experience in python web scraping. if you want to know more imformation about me ,please see my profile. Scrapin More

€300 EUR in 10 days
(30 Reviews)

Hello, I read your project description posted. I am a web crawler with javascript expert and have plenty of experience of it. However, I'd like to discuss more detail with you now. I am free to work now. Kindly s More

€555 EUR in 10 days
(16 Reviews)

Hello, how are you? I just read your requirement in details. I am very interested in your job as I am experienced scraping developer. I have some experience in web scraping with python script. So I feel confident t More

€555 EUR in 10 days
(18 Reviews)

Hi there, The requirements are quite clear and straightforward to implement. I am planning to use python along with nodejs. In a common scenario, session and request handling is managed in python and js code is simp More

€550 EUR in 5 days
(18 Reviews)

+++ Hi, I can develop a library for you in PHP which will use a headless browser named PhantomJS behind the scenes to load pages you requested along with all the linked JavaScript and CSS files so you can fetch pages More

€250 EUR in 10 days
(21 Reviews)

+++ Will make docker service along with Scrapy and splash for supporting JavaScript. Can store html and response body. These are my skills related to web scraping and crawling Have done scraping in CasperJS Phantomj More

€666 EUR in 10 days
(30 Reviews)

Hello, I have genuine skills in *Angular, React JS, Node JS. *PHP MVC frameworks such as Wordpress, Laravel, Magento*. *Ruby and Ruby on Rails Don't worry about basic More

€370 EUR in 10 days
(15 Reviews)

Hi,dear. I've just checked all requirements of your project 'web crawler with javascript support'. I'm a senior software developer and I'm able to fulfill your project requirements ,you may be interested in me. Please More

€500 EUR in 3 days
(10 Reviews)


€555 EUR in 3 days
(13 Reviews)

HI, Hope you doing well sir , I go through your message in given detail, I make sure you that i am working as professional web developer . I can work with you on your project as well superior for you, More

€555 EUR in 10 days
(11 Reviews)

Web Developer Hello Greetings! I read your job description is very Deeply . I have developed the same websites just related to your requirements and I have 6 years of experience in Php, Python, Wordpress, Web Scrap More

€555 EUR in 10 days
(8 Reviews)

I am expert in Python and recently implimented google captcha on this [login to view URL] create complex [login to view URL] check previous work [login to view URL] [login to view URL] [login to view URL] http://escap More

€555 EUR in 10 days
(12 Reviews)