No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.

Do you want to learn more about this Actor?

Get a demo

Back to issues Create new issue

Crawler fails crawling nike.com

Closed

ballerine opened this issue

I get the following error while crawling https://nike.com -

12024-08-14T12:05:17.702Z ERROR PlaywrightCrawler: Request failed and reached maximum retries. frame.content: Permission denied to access property "doctype"
22024-08-14T12:05:17.703Z @debugger eval code line 225 > eval:3:13
32024-08-14T12:05:17.704Z evaluate@debugger eval code:227:17
42024-08-14T12:05:17.705Z @debugger eval code:1:44
52024-08-14T12:05:17.705Z
62024-08-14T12:05:17.706Z     at /home/myuser/node_modules/@crawlee/playwright/internals/utils/playwright-utils.js:417:47
72024-08-14T12:05:17.707Z     at async parseWithCheerio (/home/myuser/node_modules/@crawlee/playwright/internals/utils/playwright-utils.js:414:9)
82024-08-14T12:05:17.707Z     at async genericHandler (/home/myuser/dist/request-handler.js:68:9)
92024-08-14T12:05:17.708Z     at async wrap (/home/myuser/node_modules/@apify/timeout/cjs/index.cjs:54:21)

Jindřich Bär (jindrich.bar)

Hello, and thank you for your interest in this Actor!

Websites like nike.com are notoriously hard to scrape. Companies like Nike implement anti-scraping protection on their website to protect themselves against scalpers. This is likely why you see the error message when scraping this page with this Actor. We built Website Content Crawler to be the solution for the top 90% of all pages. Unfortunately, we cannot handle all of the use cases - simply because of the sheer diversity of web content and varying technologies used.

If you want a more customizable solution, check out our Web Scraper. You can partially program this actor and execute your custom logic directly on the page. Alternatively, a quick search in the Apify Store shows some community-made actors for scraping the nike.com website. Note that these are community-made and not affiliated with Apify in any way.

I'll close this issue now, but feel free to ask additional questions if you have any. Cheers!

Add comment

Developer

Apify

Actor metrics

3k monthly users
465 stars
99.9% runs succeeded
3.1 days response time
Created in Mar 2023
Modified 10 days ago

Categories

Developer tools

Business

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

Puppeteer Scraper

apify/puppeteer-scraper

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.

Apify

4.2k

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

74.7k

409

Smart Article Extractor

lukaskrivka/article-extractor-smart

📰 Smart Article Extractor extracts articles from any scientific, academic, or news website with just one click. The extractor crawls the whole website and automatically distinguishes articles from other web pages. Download your data as HTML table, JSON, Excel, RSS feed, and more.

Lukáš Křivka

3.8k

Facebook Ads Scraper

apify/facebook-ads-scraper

Extract advertising data from one or multiple Facebook Pages. Get page details, reach estimates, publisher platforms, report count, number of impressions, ad IDs, timestamps, and more. Download Facebook ads data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

4.9k

Tiktok Hashtag Scraper

clockworks/tiktok-hashtag-scraper

Scrape TikTok hashtag data. Just add one or more hashtags and extract TikTok videos with that hashtag: URLs, likes, country of creation, video and music metadata, TikTok creator data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

1.9k

Tripadvisor Scraper

maxcopell/tripadvisor

This unofficial Tripadvisor API is a data extraction tool able to get data on hotels, restaurants, things to do, vacation rentals, attractions, tours, and public trips. Get pricing, contact details, amenities, awards, ratings, and more. Download your data in Excel, JSON, CSV, and other formats.

Maximillian Copelli

5.4k

TikTok Sound Scraper

clockworks/tiktok-sound-scraper

Scrape TikTok videos with a chosen sound. Just add one or more sound URLs and extract tiktoks that have it: URLs, likes, country of creation, video and music metadata, creator data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

167