Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoWebsite crawler ends successfully but crawls nothing
Open
We have been trying to crawl a public website, the crawler finishes successfully but crawls nothing. From the logs we couldn't figure out anything wrong, so we are contacting support. Thank you in advance for your help.
Hi, thank you for using Apify.
I've checked the run, and there are a couple of warnings about timeouts:
**WARN PlaywrightCrawler:** Reclaiming failed request back to the list or queue. Navigation timed out after 60 seconds.
It seems that the website https://www.belmont.ch/ has been down recently. It might have been unavailable when you attempted to scrape it. Please try again once belmont.ch is accessible.
I'll close the issue for now. Please try recrawling once belmont.ch is available. If the issue persists after the site is back up, feel free to raise the issue again.
innovum_admin
No that's not it, we just rerun it (https://console.apify.com/organization/tR4PJe4yitgRGwTm4/actors/runs/aMVJLIDGlKDxxhUMO#log) and same issue, while the website was available all the time during the crawling.
Could you please check internally if there is anything that breaks the connectivity?
Apify- 3k monthly users
- 465 stars
- 99.9% runs succeeded
- 3.1 days response time
- Created in Mar 2023
- Modified 10 days ago
