Fiverr Scraper

2024-01-30T10:15:39.142Z INFO CheerioCrawler: current url https://pro.fiverr.com/categories/programming-tech?source=category_tree&page=33 2024-01-30T10:15:39.143Z INFO CheerioCrawler: next url https://pro.fiverr.com/categories/programming-tech?source=category_tree&page=34 2024-01-30T10:15:39.190Z ERROR CheerioCrawler: Request failed and reached maximum retries. TypeError: Cannot read properties of undefined (reading '0') 2024-01-30T10:15:39.191Z at file:///usr/src/app/src/routes.js:48:35 2024-01-30T10:15:39.192Z at runMicrotasks () 2024-01-30T10:15:39.192Z at processTicksAndRejections (node:internal/process/task_queues:96:5) 2024-01-30T10:15:39.193Z at async wrap (/usr/src/app/node_modules/@apify/timeout/index.js:52:21) {"id":"VfE6T0vgJCEv9eg","url":"https://pro.fiverr.com/categories/programming-tech?source=category_tree&page=33","method":"GET","uniqueKey":"https://pro.fiverr.com/categories/programming-tech?page=33&source=category_tree"} 2024-01-30T10:15:40.049Z WARN CheerioCrawler: Reclaiming failed request back to the list or queue. Request blocked - received 403 status code. 2024-01-30T10:15:40.051Z {"id":"jDMCMNPQwiJ4ccp","url":"https://pro.fiverr.com/categories/programming-tech?source=category_tree&page=34","retryCount":1} 2024-01-30T10:15:46.600Z WARN CheerioCrawler: Reclaiming failed request back to the list or queue. Request blocked - received 403 status code. 2024-01-30T10:15:46.601Z {"id":"jDMCMNPQwiJ4ccp","url":"https://pro.fiverr.com/categories/programming-tech?source=category_tree&page=34","retryCount":2} 2024-01-30T10:15:53.484Z WARN CheerioCrawler: Reclaiming failed request back to the list or queue. Request blocked - received 403 status code. 2024-01-30T10:15:53.485Z {"id":"jDMCMNPQwiJ4ccp","url":"https://pro.fiverr.com/categories/programming-tech?source=category_tree&page=34","retryCount":3} 2024-01-30T10:16:03.505Z ERROR CheerioCrawler: Request failed and reached maximum retries. Error: Request blocked - received 403 status code. 2024-01-30T10:16:03.507Z at CheerioCrawler._throwOnBlockedRequest (/usr/src/app/node_modules/@crawlee/basic/internals/basic-crawler.js:688:19) 2024-01-30T10:16:03.507Z at CheerioCrawler._runRequestHandler (/usr/src/app/node_modules/@crawlee/http/internals/http-crawler.js:318:22) 2024-01-30T10:16:03.508Z at runMicrotasks () 2024-01-30T10:16:03.509Z at processTicksAndRejections (node:internal/process/task_queues:96:5) 2024-01-30T10:16:03.509Z at async CheerioCrawler._runRequestHandler (/usr/src/app/node_modules/@crawlee/cheerio/internals/cheerio-crawler.js:137:9) 2024-01-30T10:16:03.511Z at async wrap (/usr/src/app/node_modules/@apify/timeout/index.js:52:21) {"id":"jDMCMNPQwiJ4ccp","url":"https://pro.fiverr.com/categories/programming-tech?source=category_tree&page=34","method":"GET","uniqueKey":"https://pro.fiverr.com/categories/programming-tech?page=34&source=category_tree"} 2024-01-30T10:16:03.576Z INFO CheerioCrawler: All requests from the queue have been processed, the crawler will shut down. 2024-01-30T10:16:04.144Z INFO CheerioCrawler: Final request statistics: {"requestsFinished":0,"requestsFailed":34,"retryHistogram":[null,null,null,34],"requestAvgFailedDurationMillis":3908,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":7,"requestTotalDurationMillis":132877,"requestsTotal":34,"crawlerRuntimeMillis":273484} 2024-01-30T10:16:04.145Z INFO CheerioCrawler: Error analysis: {"totalErrors":34,"uniqueErrors":2,"mostCommonErrors":["28x: Cannot read properties of undefined (reading '0') (at file:///usr/src/app/src/routes.js:48:35)","6x: Request blocked - received 403 status code. ()"]} 2024-01-30T10:16:04.146Z INFO CheerioCrawler: Finished! Total 34 requests: 0 succeeded, 34 failed. {"terminal":true}