Web Crawling Jobs

We've built out a well-tested, docker swarm + typescript + timescale + bull-queue based infrastructure for crawling a variety of food ratings sites. It has full CI/CD to production servers and is ready for crawling the USA. It was fully working a few months ago on runs in SF, so now the job is just fixing up the tests that are breaking in the last couple months, getting proxy rotation working nicely, and scaling it up to run continuous crawling on the US.

Hi, I am looking for web crawler to do medical directory and indexing the documents and URLS.

I need to web crawl e-commerce website to extract the top 100 products in each category and put the results in an excel file each day, then have a report that shows the products that remained in the top 10 (this can be adjusted to be any number of ranks) for the past 30 days (this can be adjusted to any number of ranks). Web Crawling can be done using Python but it need to be executable file.