Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoCompany started to use cloud flare and now randomly some pages only return the cookie security prompt. are there settings you can change to make this not happen?
Hello, Cloudflare has strong protection, so we always recommend using proxies with Residential IPs. You can change it on the Proxy configuration.
- 3k monthly users
- 465 stars
- 99.9% runs succeeded
- 3.1 days response time
- Created in Mar 2023
- Modified 10 days ago