Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoParsing website with CloudFlare protection
Open
Hello. Is it parsing websites with CloudFlare protection? Could you provide more information about this case?
Hello, and thank you for your interest in Website Content Crawler. This depends on the level of protection the website uses and on the configuration of the crawler. Using a proxy or setting the crawler type to stealthy firefox may help.
sash2s
Should we use a specific proxy of our own or can we use one of the proxies from the apify list? We have problems crawling websites from the arabic ariea.
I'd advise you to experiment with the Apify residential proxy and to test out various country settings, as shown in the attached image. This can be configured in the crawler settings on the input tab.
sash2s
Hi, I'm currently trying to use a proxy from scrapingbee.com, but every request is not processed because there are SSL errors connecting to the proxy (test via "curl -k" works). In the scrapingbee.com manual, in the "Apify Integration" section, they recommend enabling the "Ignore SSL errors" checkbox. But I don't see it in the actor settings.
Apify- 3k monthly users
- 465 stars
- 99.9% runs succeeded
- 3.1 days response time
- Created in Mar 2023
- Modified 10 days ago
