Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.0 (41)
Pricing
Pay per usage
1638
Total users
65K
Monthly users
8.5K
Runs succeeded
>99%
Issues response
6.9 days
Last modified
3 days ago
timeout crawler was not working
Closed
2025-07-26T18:53:40.487Z ACTOR: Pulling Docker image of build N5wC2ArUbLgbcEpaX from registry.
2025-07-26T18:53:40.488Z ACTOR: Creating Docker container.
2025-07-26T18:53:40.729Z ACTOR: Starting Docker container.
2025-07-26T18:53:41.035Z Starting X virtual framebuffer using: Xvfb :99 -ac -screen 0 1920x1080x24+32 -nolisten tcp
2025-07-26T18:53:41.037Z Executing main command
2025-07-26T18:53:44.051Z INFO System info {"apifyVersion":"3.4.2","apifyClientVersion":"2.12.4","crawleeVersion":"3.13.8","osType":"Linux","nodeVersion":"v22.9.0"}
2025-07-26T18:53:44.168Z WARN The saveHtml option is deprecated, please use saveHtmlToFile instead.
2025-07-26T18:53:44.700Z INFO Crawling will be started using 1 start URLs and 0 sitemap URLs
2025-07-26T18:53:44.790Z [DEBUG] gradually switching: GotScrapingHttpClient -> ImpitHttpClient (picked: ImpitHttpClient)
2025-07-26T18:53:45.282Z INFO PlaywrightCrawler: Starting the crawler.
2025-07-26T18:54:10.347Z ACTOR: The Actor run has reached the timeout of 30 seconds, aborting it. You can increase the timeout in Input > Run options.
Hello Antonio, thank you for reporting the issue.
Based on the logs and our investigation, the problem is due to the short 30-second timeout combined with the 4GB memory limit, which slows down the internal browser. As a result, the PlaywrightCrawler instance doesn't have enough time to fully load the first page (which includes many images, animations, etc.), and the Actor times out before any results can be returned.
To resolve this, you have a couple of options:
- Increase the timeout. You can do this in the
Input > Run optionssection. Giving the crawler more time will allow the page to load completely before the Actor is terminated. - Use
CheerioCrawlerinstead: If you don’t require full browser rendering (note: Cheerio doesn't support JavaScript execution), this can be a much more efficient and faster option. You can change the crawler mode to Cheerio by changing theCrawler typeinput option toRaw HTTP Client (Cheerio). Here's an example run usingCheerioCrawlermode that worked successfully: CheerioCrawler example run.
I'll close this issue now, but feel free to ask additional questions if you have any.
Cheers!
