Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoDownload RSS Feeds
Closed
Is it possible to download RSS feeds, which are in XML, with this actor?
I've tried and the run succeeded, but there was no file downloaded.
The settings I've used are: "startUrls": [ { "url": "https://example.com/rss-feed" } ], "maxCrawlDepth": 0, "maxCrawlPages": 1, "maxRequestRetries": 3, "crawlerType": "cheerio", "saveHtmlAsFile": true, "saveHtml": false, "saveFiles": false, "saveMarkdown": false, "saveScreenshots": false, "useSitemaps": false, "removeCookieWarnings": true
In the Key-Value Store, there are only 4 keys: CRAWLEE_STATE, INPUT, SDK_CRAWLER_STATISTICS_0, and SDK_SESSION_POOL_STATE.
Thanks.
Hello, and thank you for your interest in this Actor!
Website Content Crawler was made to "crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines" (see the Actor description in the README).
If all you need is to download (and parse) the RSS feed, then you can use one of the existing Actors on Apify Store (search for RSS).
If those do not fulfill your needs, you can also use Cheerio Scraper for parsing the RSS feed manually - see my example run and feel free to reuse the input if you like.
Both of these solutions will also be much faster than anything you could achieve with the Website Content Crawler.
I'll close this issue now, but feel free to ask additional questions if you have any. Cheers!
Apify- 3.8k monthly users
- 616 stars
- 99.9% runs succeeded
- 3.4 days response time
- Created in Mar 2023
- Modified 3 days ago
