Web Scraper avatar
Web Scraper

Pricing

Pay per usage

Go to Store
Web Scraper

Web Scraper

Developed by

Apify

Apify

Maintained by Apify

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

4.5 (23)

Pricing

Pay per usage

935

Total users

91K

Monthly users

5.1K

Runs succeeded

>99%

Issues response

5 days

Last modified

2 months ago

CS

Maximum startUrls?

Closed

congruent_stylophone opened this issue
2 days ago

I plan to crawl around 1.000 URLs (just a single GET request per URL).

What is the limit of the "startUrls" parameter? The actor runs with 16 GB.

Thanks!

nick_slam avatar

startUrls can handle more than 100k URLs, so crawling 1000 would be totally fine.

I'm going to close this issue for now, but if you have further questions about this setting, feel free to reopen it later.