Web Scraper avatar
Web Scraper

Pricing

Pay per usage

Go to Store
Web Scraper

Web Scraper

apify/web-scraper

Developed by

Apify

Maintained by Apify

Crawls arbitrary websites using the Chrome browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

4.5 (22)

Pricing

Pay per usage

563

Monthly users

3.5k

Runs succeeded

>99%

Response time

10 days

Last modified

2 months ago

TO

Scrape ignores all my startUrls except the first one

Closed
toddles opened this issue
9 months ago

I'm passing a list of urls in "startUrls" but the scraper just stops after hitting the first one. Any idea why?

jindrich.bar avatar

Hello, and thanks for reporting this!

The scraper is likely deduplicating the URLs since they only differ in the #fragment part.

To fix this, enable the "URL #fragments identify unique pages" option (or set keepUrlFragments to true if you run the Actor via the API). This should resolve the issue.

I'll close this issue now, but feel free to reach out if you need further help. Cheers!

Pricing

Pricing model

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.