Web scraping code templates

🐝 BeeAI agent

Example of how to use Bee Agent Framework with Apify Actors to create a social media analysis agent.

Mastra agent

Example of how to use Mastra with Apify Actors to create a social media analysis agent.

Start with TypeScript

Scrape single page with provided URL with Axios and extract data from page's HTML with Cheerio.

Crawlee + Cheerio

A scraper example that uses Cheerio to parse HTML. It's fast, but it can't run the website's JavaScript or pass JS anti-scraping challenges.

Crawlee + Puppeteer + Chrome

Example of a Puppeteer and headless Chrome web scraper. Headless browsers render JavaScript and are harder to block, but they're slower than plain HTTP.

Crawlee + Playwright + Chrome

Web scraper example with Crawlee, Playwright and headless Chrome. Playwright is more modern, user-friendly and harder to block than Puppeteer.

Crawlee + Playwright + Camoufox

Web scraper example with Crawlee, Playwright and headless Camoufox. Camoufox is a custom stealthy fork of Firefox. Try this template if you're facing anti-scraping challenges.

Playwright + Chrome Test Runner

Example of using the Playwright Test project to run automated website tests in the cloud and display their results. Usable as an API.

Empty TypeScript project

Empty template with basic structure for the Actor with Apify SDK that allows you to easily add your own functionality.

Standby TypeScript project

Template with basic structure for an Actor using Standby mode that allows you to easily add your own functionality.

Start with TypeScript on Bun

Scrape single page with provided URL with Axios and extract data from page's HTML with Cheerio.

🦜️🔗 LangChain

Example of how to use LangChain.js with Apify to crawl the web data, vectorize them, and prompt the OpenAI model.

LangGraph agent

LangGraph agent in JavaScript for answering questions via web search.

Start with JavaScript

Scrape single page with provided URL with Axios and extract data from page's HTML with Cheerio.

Crawlee + Cheerio

A scraper example that uses Cheerio to parse HTML. It's fast, but it can't run the website's JavaScript or pass JS anti-scraping challenges.

Crawlee + Puppeteer + Chrome

Example of a Puppeteer and headless Chrome web scraper. Headless browsers render JavaScript and are harder to block, but they're slower than plain HTTP.

Crawlee + Playwright + Chrome

Web scraper example with Crawlee, Playwright and headless Chrome. Playwright is more modern, user-friendly and harder to block than Puppeteer.

Crawlee + Playwright + Camoufox

Web scraper example with Crawlee, Playwright and Camoufox. Camoufox is a custom stealthy fork of Firefox. Try this template if you're facing anti-scraping challenges.

Bootstrap CheerioCrawler

Skeleton project that helps you quickly bootstrap `CheerioCrawler` in JavaScript. It's best for developers who already know Apify SDK and Crawlee.

Cypress

Example of running Cypress tests and saving their results on the Apify platform. JSON results are saved to Dataset, videos to Key-value store.

Empty JavaScript Project

Empty template with basic structure for the Actor with Apify SDK that allows you to easily add your own functionality.

Standby JavaScript Project

Template with basic structure for an Actor using Standby mode that allows you to easily add your own functionality.

CLI-based Actor starter

Actorize a CLI utility with the Apify Actor CLI

🦜️🔗 LangGraph AI agent

Example of how to use LangGraph with Apify Actors to create a social media analysis tool-calling agent.

CrewAI agent

Example of how to use CrewAI with Apify Actors to create a social media analysis tool-calling agent.

LlamaIndex agent

LlamaIndex agent to scrape, deduplicate and summarize contact details from a website

PydanticAI agent

A PydanticAI agent designed to generate jokes, showcasing a minimal example using the Pydantic AI framework.

Smolagents agent

An AI news aggregator that fetches and summarizes the latest news based on user-defined interests using DuckDuckGo search and OpenAI models written in Python Smolagents.

Start with Python

Scrape single page with provided URL with HTTPX and extract data from page's HTML with Beautiful Soup.

BeautifulSoup

Example of a web scraper that uses Python HTTPX to scrape HTML from URLs provided on input, parses it using BeautifulSoup and saves results to storage.

Playwright + Chrome

Crawler example that uses headless Chrome driven by Playwright to scrape a website. Headless browsers render JavaScript and can help when getting blocked.

Selenium + Chrome

Scraper example built with Selenium and headless Chrome browser to scrape a website and save the results to storage. A popular alternative to Playwright.

Empty Python project

Empty template with basic structure for the Actor with Apify SDK that allows you to easily add your own functionality.

Standby Python project

Template with basic structure for an Actor using Standby mode that allows you to easily add your own functionality.

Scrapy

This example Scrapy spider scrapes page titles from URLs defined in input parameter. It shows how to use Apify SDK for Python and Scrapy pipelines to save results.

Crawlee + BeautifulSoup

Crawl and scrape websites using Crawlee and BeautifulSoup. Start from a given start URLs, and store results to Apify dataset.

Crawlee + Playwright + Chrome

Crawl and scrape websites using Crawlee and Playwright. Start from a given start URLs, and store results to Apify dataset.