Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1.2k
Monthly users
6.3k
Runs succeeded
>99%
Response time
3.5 days
Last modified
a day ago
How to only have the home page or about us page?
Open
I'm looking to scrape 10k domains to just get their descriptions and don't want to crawl through all the pages but just the home or about us page, how would I do so? Thank you so much!
Hi,
Thank you for trying Website Content Crawler.
You can control the crawling depth using the maxCrawlDepth setting.
When you set maxCrawlDepth = 0, the crawler will only scrape the startUrls without crawling deeper.
You also need to set "useSitemaps": false, which you’ve already done.
I’m not sure what you mean by “about us” page.
If you set the startUrls to https://apify.com/about, it will be scraped.
However, if you use https://apify.com as the startUrls, the about page won’t be scraped unless you allow crawl depth > 0.
I hope this helps. I’ll go ahead and close the issue for now, but feel free to ask any further questions or reopen it if needed.
Jiri
xemivo2655
Thank you Jiri!
If I wanted to reduce the cost because I am crawling 10k domains, what would be the best settings?
Additionally, I just want the description (which should explain what the company is about), which I think the text column has all that I need, any other setting or recommendations?
Hi,
There’s always a tradeoff when it comes to scraping. The crawler uses default settings that work well across a wide variety of sites.
If you're looking to optimize for cost and can tolerate occasional blocks or missing content, there are a couple of key settings to adjust:
- Proxy settings: Use datacenter proxies — they are cheaper but may lead to more frequent blocks.
- Crawler type: Choose Cheerio — it’s lightweight and cost-effective but may not work well on JavaScript-heavy sites.
I hope this helps!
Best, Jiri
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
