Web Scraper
Pricing
Pay per usage
Web Scraper
Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.
4.5 (22)
Pricing
Pay per usage
659
Monthly users
3.8k
Runs succeeded
>99%
Response time
21 days
Last modified
3 days ago
performace issue
Closed
I only crawl a page for about 30 seconds to 1 minute, is there any way to improve performance.
potent_fox
i'm waiting....
potent_fox
some execute results
Hi potent_fox,
Thank you for your patience. To improve your scraping performance, here are a few suggestions:
- Scrape Multiple URLs Together: Scraping multiple URLs in a single crawl session can give you a better time per URL. Currently, your Actor spends a lot of time in the setup phase as you run each URL in a separate run.
- Increase Resources: Adding more memory or CPU can help speed up the crawling process. Note that this should be more noticeable once you put more URLs together in one run.
- Use Cheerio Crawler: If possible, scrape the page without a browser using tools like Cheerio Crawler. This is up to 20x more efficient than the Web Scraper.
- Explore Original APIs: For optimal performance, consider scraping directly from the original API. More details can be found here.
Feel free to try these suggestions, and let us know if you encounter any further issues.
We'll be closing this issue now. Cheers!
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
