Web Scraper avatar
Web Scraper

Pricing

Pay per usage

Go to Store
Web Scraper

Web Scraper

apify/web-scraper

Developed by

Apify

Maintained by Apify

Crawls arbitrary websites using the Chrome browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

4.5 (22)

Pricing

Pay per usage

563

Monthly users

3.5k

Runs succeeded

>99%

Response time

10 days

Last modified

2 months ago

PF

performace issue

Closed
potent_fox opened this issue
9 months ago

I only crawl a page for about 30 seconds to 1 minute, is there any way to improve performance.

PF

potent_fox

8 months ago

i'm waiting....

PF

potent_fox

8 months ago

some execute results

jindrich.bar avatar

Hi potent_fox,

Thank you for your patience. To improve your scraping performance, here are a few suggestions:

  1. Scrape Multiple URLs Together: Scraping multiple URLs in a single crawl session can give you a better time per URL. Currently, your Actor spends a lot of time in the setup phase as you run each URL in a separate run.
  2. Increase Resources: Adding more memory or CPU can help speed up the crawling process. Note that this should be more noticeable once you put more URLs together in one run.
  3. Use Cheerio Crawler: If possible, scrape the page without a browser using tools like Cheerio Crawler. This is up to 20x more efficient than the Web Scraper.
  4. Explore Original APIs: For optimal performance, consider scraping directly from the original API. More details can be found here.

Feel free to try these suggestions, and let us know if you encounter any further issues.

We'll be closing this issue now. Cheers!

Pricing

Pricing model

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.