No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.

Do you want to learn more about this Actor?

207

174

Website crawler ends successfully but crawls nothing

Opened 11 hours ago by innovum_admin, last comment 9 hours ago by innovum_admin

Use of sitemap makes crawler fail

Opened a day ago by fsmedat, last comment a day ago by fsmedat

scraper started failing

Opened a day ago by fsmedat, last comment a day ago by fsmedat

Failed crawling with 403 code

Opened 4 days ago by ballerine, last comment 4 days ago by ballerine

Can't crawl while logged in

Opened 5 days ago by rust_chimta, last comment 5 days ago by rust_chimta

Fails to scrape useful data from AWS documentation page

Opened 6 days ago by methodical, last comment 5 days ago by methodical

Scrolling functionality breaking our scrape job

Opened 8 days ago by developprotege, last comment 4 days ago by Jan Čurn (jancurn)

Malformed sitemap content and 429s

Opened 8 days ago by MavenAGI, last comment 8 days ago by MavenAGI

My run doesn't work. I have 0 results

Opened 18 days ago by contact_plune, last comment 18 days ago by Jan Buchar (janbuchar)

Poor CPU utilization due to low usage limit

Opened 21 days ago by write2souvik, last comment 10 days ago by write2souvik

Crawling takes longer when calling API vs on site

Opened 24 days ago by adi-kamaraj, last comment 24 days ago by Jan Buchar (janbuchar)

How to ignore broken SSL when using PROXY

Opened a month ago by sash2s, last comment 24 days ago by Jan Buchar (janbuchar)

Unable to crawl https://openai.com/index/extracting-concepts-from-gpt-4/

Opened a month ago by imda_peckyoke, last comment a month ago by Jindřich Bär (jindrich.bar)

Crawler does not identify relative links

Opened a month ago by MavenAGI, last comment a month ago by Jindřich Bär (jindrich.bar)

My Runs do not end

Opened a month ago by matthias.amberg, last comment a month ago by matthias.amberg

Parsing website with CloudFlare protection

Opened a month ago by sash2s, last comment a month ago by sash2s

Unable to crawl the whole website

Opened a month ago by simpleworks, last comment a month ago by Jan Buchar (janbuchar)

Automating Web Content Crawling for Real-Time Updates

Opened 2 months ago by glovebubble, last comment a month ago by Jan Buchar (janbuchar)

Getting duplicate URLs in web crawling

Opened 2 months ago by simpleworks, last comment 24 days ago by Jan Buchar (janbuchar)

Memory limit control

Opened 2 months ago by vitthalrao.lavate, last comment 2 months ago by intriguing_game

Developer

Apify

Actor metrics

3k monthly users
465 stars
99.9% runs succeeded
3.1 days response time
Created in Mar 2023
Modified 10 days ago

Categories

Developer tools

Business

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

Puppeteer Scraper

apify/puppeteer-scraper

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.

Apify

4.2k

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

74.7k

409

Smart Article Extractor

lukaskrivka/article-extractor-smart

📰 Smart Article Extractor extracts articles from any scientific, academic, or news website with just one click. The extractor crawls the whole website and automatically distinguishes articles from other web pages. Download your data as HTML table, JSON, Excel, RSS feed, and more.

Lukáš Křivka

3.8k

Facebook Ads Scraper

apify/facebook-ads-scraper

Extract advertising data from one or multiple Facebook Pages. Get page details, reach estimates, publisher platforms, report count, number of impressions, ad IDs, timestamps, and more. Download Facebook ads data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

4.9k

Tiktok Hashtag Scraper

clockworks/tiktok-hashtag-scraper

Scrape TikTok hashtag data. Just add one or more hashtags and extract TikTok videos with that hashtag: URLs, likes, country of creation, video and music metadata, TikTok creator data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

1.9k

Tripadvisor Scraper

maxcopell/tripadvisor

This unofficial Tripadvisor API is a data extraction tool able to get data on hotels, restaurants, things to do, vacation rentals, attractions, tours, and public trips. Get pricing, contact details, amenities, awards, ratings, and more. Download your data in Excel, JSON, CSV, and other formats.

Maximillian Copelli

5.4k

TikTok Sound Scraper

clockworks/tiktok-sound-scraper

Scrape TikTok videos with a chosen sound. Just add one or more sound URLs and extract tiktoks that have it: URLs, likes, country of creation, video and music metadata, creator data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

167

Tiktok User Search Scraper

clockworks/tiktok-user-search-scraper

Clockworks

What is retrieval-augmented generation (and why use it for LLMs)?

LlamaIndex vs. LangChain

6 open-source Pinecone alternatives for LLMs

Build new tools

Are you a developer? Build your own Actors and run them on Apify.

Learn more

Get a custom solution

Get a custom web scraping or RPA solution.

Book a demo

Website Content Crawler