I'm trying to scrape data from 300 profile URLs. If I combine all of them into a single API call, it takes forever to respond. On the other hand, sending one API call per URL quickly consumes all 32 GB of my RAM.

What’s the most efficient way to process 30 URLs at a time and receive the responses within a reasonable amount of time?

Dev Fusion (dev_fusion)

We're currently optimizing the scraper to improve processing speed, and the updates will be live very soon. In the meantime, you can submit up to 10 URLs per run for faster results.

ewasserman

Thanks! To help me better build our automation, could you clarify whether it's recommended to pass all 300 URLs in a single API request, or if it's better to throttle them—say, in batches of 100?

ewasserman

I am asking for the end state that you're releasing soon?

Dev Fusion (dev_fusion)

batches of 100, will be better.

Add comment

LinkedIn Profile Search Scraper No Cookies ✅ Find all people 📧

harvestapi/linkedin-profile-search

Search for LinkedIn profiles with filters and extract detailed profile information, including work experience, education history, location and more. No cookies or account required.

HarvestAPI

856

5.0

Linkedin Profile Details Scraper + EMAIL (No Cookies Required)

apimaestro/linkedin-profile-detail

Scrape comprehensive LinkedIn profile data including work experience, education history, certifications, and location details. Get structured information from any public LinkedIn profile using their username.

API Maestro

2.1K

4.4

LinkedIn Profile Bulk Scraper +Email ✅ No Cookies $4/$10 per 1k

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles, including complete work experience, education history, skills and more. No cookies or account required. Concurrency + fast response times make mass scraping fast ⚡

HarvestAPI

849

4.6

Linkedin Profile Scraper No Cookies

logical_scrapers/linkedin-profile-scraper-no-cookies

LinkedIn Bulk Profile Scraper with No Cookies Required. scrapers all publicly available data from a given LinkedIn profile URL.

Goldmine

174

1.0

Mass Linkedin Profile Scrapper (no cookies) + ✉️ + ☎️

anchor/linkedin-profile-enrichment

Import your file and see contacts info being enriched : LinkedIn profiles, emails, phones... from your list of LinkedIn URL of people or company, get a fully functional list of leads, that you can qualify simply. Enrich your file with valuable data, including emails and phone numbers. No cookies.

Anchor

2.5K

1.3

Linkedin Profile Details Batch Scraper (No Cookies)

apimaestro/linkedin-profile-batch-scraper-no-cookies-required

(no cookies need) Extract In Bulk comprehensive LinkedIn profile data including work experience, education history, certifications, and location details. Get structured information from any public LinkedIn profile using their username.

API Maestro

767

5.0

Linkedin Profile Full Sections Scraper + EMAIL | No Cookies

apimaestro/linkedin-profile-full-sections-scraper

Scrape complete LinkedIn profile data including work experience, education history, certifications, and location details. Get structured information from any public LinkedIn profile using their username.

API Maestro

241

5.0

Linkedin Profile Scraper

logical_scrapers/linkedin-profile-scraper

🚀 Fastest linkedin profile scraper. Easily extract comprehensive LinkedIn profile data, including name, headline, industry, location, experience, education, skills, certifications, and more. Automates LinkedIn data collection for lead generation, recruiting, research, and competitive analysis.

Goldmine

276

4.9

LinkedIn Profile Scraper

icypeas_official/linkedin-profile-scraper

LinkedIn Profile Scraper powered by Icypeas. Upload a list of LinkedIn profile URLs to find their information. Supports CSV, TSV, and semicolon-separated formats. Returns all the LinkedIn information about profiles.