3 days trial then $15.00/month - No credit card required now

Web Crawler

rigelbytes/webcrawler

3 days trial then $15.00/month - No credit card required now

This web crawler is designed to provide users with complete flexibility by allowing them to use their **own proxies**. The scraper collects all pages from the website and returns extracts the **MetaData**, **Title**, and **Content** of the page in MarkDown.

URL

urlstringRequired

URL of the web page to scrape

Proxy URL

proxystringOptional

Rotating Proxy URL

Developer

Rigel Bytes

Actor Metrics

7 monthly users
0 No bookmarks yet
>99% runs succeeded
Created in Feb 2025
Modified 19 days ago

Categories

Incremental Web Crawler

flamboyant_leaf/IncrementalCrawler-v2

The Incremental Crawler efficiently fetches URLs of recently added or updated web pages on a target site, optimizing resources by focusing only on new content. Ideal for keeping up with the latest updates, it integrates seamlessly into workflows for content monitoring and analysis.

AIRabbit

Algolia Webcrawler

jancurn/algolia-webcrawler

Crawls a website using one or more sitemaps and imports the data to Algolia search index. The text content is identified using simple CSS selectors.

Jan Čurn

Similarweb scraper

saswave/similarweb-scraper

Extract similarweb informations from multiple input format. Web crawler for similarweb.com website. Get data like total visits, bounce rate, traffic sources, competitors, top countries, company info .. Do market research, track potential competitors, or for other research and analytics on companies.

SASWAVE

Dealroom Scraper

saswave/dealroom-scraper

Dealroom company scraper. Collect data from companies using their url / domain name. Dealroom web crawler extract data about social networks, similarweb website traffic, funding rounds, investments, team members, KPI revenues and more ..

SASWAVE

Rent Scraper

getdataforme/rent-scraper

The Rent Scraper is a web crawler that extracts rental listings from Rent.com. It collects key details such as rent price, location, size, number of bedrooms, bathrooms, and listing status. The scraper efficiently handles dynamic content, providing accurate, up-to-date rental insights.

GetDataForMe

Compass Scraper

getdataforme/compass-scraper

The Compass Scraper is a web crawler that extracts real estate listings from Compass. It collects details like price, location, size, bedrooms, bathrooms, and listing status. The scraper efficiently handles pagination and dynamic content, providing up-to-date, accurate insights.

GetDataForMe

Trulia Scraper

getdataforme/trulia-scraper

The Trulia Scraper is a web crawler that extracts real estate listings from Trulia. It collects key details such as price, location, size, bedrooms, bathrooms, and listing status. The scraper efficiently handles pagination and dynamic content, providing accurate and up-to-date insights.

GetDataForMe

Opendoor Scraper

getdataforme/opendoor-scraper

The Opendoor Scraper is a web crawler that extracts real estate listings from Opendoor. It collects key details such as price, location, size, bedrooms, bathrooms, and listing status. The scraper handles pagination and dynamic content, providing up-to-date, accurate property insights.

GetDataForMe

Forrent Scraper

getdataforme/forrent-scraper

The ForRent Scraper is a web crawler that extracts rental listings from ForRent. It gathers key details such as rent price, location, size, number of bedrooms, bathrooms, and listing status. The scraper handles dynamic content and pagination, providing accurate, up-to-date rental insights.

GetDataForMe

1stdibs scraper

saswave/1stdibs-scraper

1stdibs web crawler. With the 1stdibs.com scraper, extract art product details from a search url. Collect object informations and store contact infos. phone, address and more

SASWAVE