Github Search Scraper avatar

Github Search Scraper

Try for free

1 day trial then $25.00/month - No credit card required now

View all Actors
Github Search Scraper

Github Search Scraper

saswave/github-search-scraper
Try for free

1 day trial then $25.00/month - No credit card required now

Github search scraper. Get all data from search results list

LE

Fails on page 10 every time

Closed

aleksandrmoshkov opened this issue
3 days ago

Please see the actor run. I made a resurrect attempt, but it fails at the same page.

saswave avatar

We are looking into it, probably anti bot protections

LE

aleksandrmoshkov

3 days ago

I successfully added my cookies, which improved the process, but there were still some errors. In the end, I managed to run 100 pages. I think it would be helpful to include a line in the README stating that the antibot sometimes fails and that it's better to add a cookie.

saswave avatar

We found the issue, nothing related to anti bot protections:

When not providing cookies, github sometimes returns 429 responses to slow the requests made to the server

Initially, we ask cookies only if you need to search a url with type=code (mandatory for the extraction to work)

Now you should be able to handle the 100 pages without cookies

Developer
Maintained by Community
Actor metrics
  • 3 monthly users
  • 3 stars
  • 94.2% runs succeeded
  • 0.76 hours response time
  • Created in Mar 2024
  • Modified 3 days ago