1 day trial then $20.00/month - No credit card required now

Substack Scraper

qpayre/substack-scraper

1 day trial then $20.00/month - No credit card required now

The Substack Author Scraper is a powerful Apify actor that makes it easy for content creators to scrape and retrieve all posts from their favorite Substack authors. With structured data presented in a user-friendly format, analyzing and processing valuable information has never been easier.

Back to issues Create new issue

Main function - threw an exception

Closed

lusavkaedu opened this issue

The main function of the Actor threw an exception. this is not a subscription absed substack account.

2024-04-03T09:15:39.906Z --- Logging error --- 2024-04-03T09:15:39.907Z Traceback (most recent call last): 2024-04-03T09:15:39.909Z File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 264, in connect 2024-04-03T09:15:39.911Z sock = self.retry.call_with_retry( 2024-04-03T09:15:39.914Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:15:39.920Z File "/usr/local/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry 2024-04-03T09:15:39.922Z return do() 2024-04-03T09:15:39.923Z ^^^^ 2024-04-03T09:15:39.925Z File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 265, in 2024-04-03T09:15:39.927Z lambda: self._connect(), lambda error: self.disconnect(error) 2024-04-03T09:15:39.929Z ^^^^^^^^^^^^^^^ 2024-04-03T09:15:39.930Z File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 627, in _connect 2024-04-03T09:15:39.932Z raise err 2024-04-03T09:15:39.934Z File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 615, in _connect 2024-04-03T09:15:39.936Z sock.connect(socket_address) 2024-04-03T09:15:39.937Z OSError: [Errno 99] Cannot assign requested address 2024-04-03T09:15:39.939Z 2024-04-03T09:15:39.941Z During handling of the above exception, another exception occurred: 2024-04-03T09:15:39.942Z 2024-04-03T09:15:39.944Z Traceback (most recent call last): 2024-04-03T09:15:39.946Z File "/usr/local/lib/python3.11/site-packages/arb_logger/logger.py", line 40, in emit 2024-04-03T09:15:39.948Z self.redis_client.xadd(self.redis_key.split(':')[0], record_dict) 2024-04-03T09:15:39.949Z File "/usr/local/lib/python3.11/site-packages/redis/commands/core.py", line 3551, in xadd 2024-04-03T09:15:39.951Z return self.execute_command("XADD", name, *pieces) 2024-04-03T09:15:39.953Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:15:39.955Z File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 533, in execute_command 2024-04-03T09:15:39.956Z conn = self.connection or pool.get_connection(command_name, **options) 2024-04-03T09:15:39.958Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:15:39.960Z File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 1086, in get_connection 2024-04-03T09:15:39.962Z connection.connect() 2024-04-03T09:15:39.963Z File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 270, in connect 2024-04-03T09:15:39.965Z raise ConnectionError(self._error_message(e)) 2024-04-03T09:15:39.966Z redis.exceptions.ConnectionError: Error 99 connecting to localhost:6379. Cannot assign requested address. 2024-04-03T09:15:39.968Z Call stack: 2024-04-03T09:15:39.969Z File "", line 198, in _run_module_as_main 2024-04-03T09:15:39.971Z File "", line 88, in _run_code 2024-04-03T09:15:39.973Z File "/usr/src/app/src/main.py", line 19, in 2024-04-03T09:15:39.974Z asyncio.run(main()) 2024-04-03T09:15:39.976Z File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run 2024-04-03T09:15:39.978Z return runner.run(main) 2024-04-03T09:15:39.980Z File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run 2024-04-03T09:15:39.982Z return self._loop.run_until_complete(task) 2024-04-03T09:15:39.983Z File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete 2024-04-03T09:15:39.985Z self.run_forever() 2024-04-03T09:15:39.987Z File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever 2024-04-03T09:15:39.989Z self._run_once() 2024-04-03T09:15:39.995Z File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once 2024-04-03T09:15:39.997Z handle._run() 2024-04-03T09:15:39.999Z File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run 2024-04-03T09:15:40.000Z self._context.run(self._callback, *self._args) 2024-04-03T09:15:40.002Z File "/usr/src/app/src/main.py", line 43, in main 2024-04-03T09:15:40.004Z data = await scrap(actor_input) 2024-04-03T09:15:40.006Z File "/usr/src/app/src/main.py", line 31, in scrap 2024-04-03T09:15:40.008Z articles = [ArticleScraper.scrap(post) for post in posts] 2024-04-03T09:15:40.010Z File "/usr/src/app/src/main.py", line 31, in 2024-04-03T09:15:40.012Z articles = [ArticleScraper.scrap(post) for post in posts] 2024-04-03T09:15:40.013Z File "/usr/src/app/src/substack_scraper/article_scraper.py", line 30, in scrap 2024-04-03T09:15:40.015Z LOGGER.error('rate limited, please calm down') 2024-04-03T09:15:40.017Z File "/usr/local/lib/python3.11/logging/init.py", line 1518, in error 2024-04-03T09:15:40.018Z self._log(ERROR, msg, args, **kwargs) 2024-04-03T09:15:40.020Z File "/usr/local/lib/python3.11/logging/init.py", line 1634, in _log 2024-04-03T09:15:40.022Z self.handle(record) 2024-04-03T09:15:40.024Z File "/usr/local/lib/python3.11/logging/init.py", line 1644, in handle 2024-04-03T09:15:40.025Z self.callHandlers(record) 2024-04-03T09:15:40.027Z File "/usr/local/lib/python3.11/logging/init.py", line 1706, in callHandlers 2024-04-03T09:15:40.029Z hdlr.handle(record) 2024-04-03T09:15:40.031Z File "/usr/local/lib/python3.11/logging/init.py", line 978, in handle 2024-04-03T09:15:40.032Z self.emit(record) 2024-04-03T09:15:40.034Z File "/usr/local/lib/python3.11/site-packages/arb_logger/logger.py", line 46, in emit 2024-04-03T09:15:40.036Z self.handleError(record) 2024-04-03T09:15:40.038Z Message: 'rate limited, please calm down' 2024-04-03T09:15:40.039Z Arguments: () 2024-04-03T09:15:40.041Z ERROR Actor failed with an exception 2024-04-03T09:15:40.043Z Traceback (most recent call last): 2024-04-03T09:15:40.044Z File "/usr/src/app/src/main.py", line 43, in main 2024-04-03T09:15:40.046Z data = await scrap(actor_input) 2024-04-03T09:15:40.048Z ^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:15:40.050Z File "/usr/src/app/src/main.py", line 31, in scrap 2024-04-03T09:15:40.051Z articles = [ArticleScraper.scrap(post) for post in posts] 2024-04-03T09:15:40.053Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:15:40.055Z File "/usr/src/app/src/main.py", line 31, in 2024-04-03T09:15:40.057Z articles = [ArticleScraper.scrap(post) for post in posts] 2024-04-03T09:15:40.059Z ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:15:40.060Z File "/usr/src/app/src/substack_scraper/article_scraper.py", line 31, in scrap 2024-04-03T09:15:40.062Z raise ValueError('rate limited, please calm down') 2024-04-03T09:15:40.064Z ValueError: rate limited, please calm down 2024-04-03T09:15:40.065Z INFO Exiting actor ({"exit_code": 91})

QPS (qpayre)

Hello we are looking to fix it. Thanks for your message

QPS (qpayre)

Hello, the issue has been fixed

Add comment

Developer

QPS

Actor metrics

12 monthly users
4 stars
74.1% runs succeeded
7.5 days response time
Created in Apr 2023
Modified 6 months ago

Categories

Automation

Business

Social media

Substack Publication Scraper

easyapi/substack-publication-scraper

Easily extract a wide range of data from Substack publications, including author IDs, copyrights, creation dates, subscriber counts, and more. Whether you're conducting market research, analyzing industry trends, or building a Substack database, this tool has you covered.

EasyApi

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

21.2k

472

Google Maps Extractor

compass/google-maps-extractor

Extract data from hundreds of places fast. Scrape Google Maps by keyword, category, location, URLs & other filters. Get addresses, contact info, opening hours, popular times, prices, menus & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

9.9k

162

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract data from hundreds of Facebook posts from one or multiple Facebook pages and profiles. Get post URL, post text, page or profile URL, timestamp, number of likes, shares, comments, and more. Download the data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

12.4k

129

Instagram Scraper

apify/instagram-scraper

Scrape and download Instagram posts, profiles, places, hashtags, photos, and comments. Get data from Instagram using one or more Instagram URLs or search queries. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

51.3k

311

TikTok Data Extractor

clockworks/free-tiktok-scraper

Extract data about videos, users, and channels based on hashtags or scrape full user profiles including posts, total likes, name, nickname, numbers of comments, shares, followers, following, and more.

Clockworks

14.6k

122

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

74.7k

410

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add one or more Instagram usernames and extract number of followers&follows, URLs, bio, posts, likes, counts, related profiles, captions, highlight reels. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

34.6k

149

Facebook Groups Scraper

apify/facebook-groups-scraper

Extract data from one or multiple public Facebook groups. Get group and post URLs, post text, comments, timestamp, likes and comments count, and basic commentator info. Download the data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

6.1k

Instagram Post Scraper

apify/instagram-post-scraper

Scrape Instagram posts. Just add one or more Instagram usernames and get your data in seconds including text, hashtags, mentions, comments, images, URLs, likes, locations, and metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

18.1k