No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.

Do you want to learn more about this Actor?

Get a demo

Back to issues Create new issue

Data not being pushed to Pinecone from WCC

Closed

Custombizio opened this issue

After the crawl is successful. The WCC data is not getting pushed to Pinecode. I setup the WCC and Pinecode integration and selected to auto push to Pinecode. Getting erros and No data in Pinecode.

Jan Buchar (janbuchar)

Hello, we investigated this issue and it seems like it's an Apify console issue and we're working on fixing it. Thank you for reporting it!

Oscar Rodriguez (Oscardz)

The fix for this issue was deployed today. Let us know if you have any other problems.

Custombizio

This is what I see under run. But when I check Pinecone it shows zero records.

Custombizio

Here is the link. https://console.apify.com/actors/aYG0l9s7dbB7j3gbS/runs/esrGogmsatyBuaPwr#output

Custombizio

This is what pinecone shows.

Custombizio

Hello, Still on data in pinecone. See my comments I added. Looks like its not triggering.

Custombizio

No Data.

Oscar Rodriguez (Oscardz)

Hello, the problem is that Pinecone saves all the text in the Metadata, so this can happen when the content is big. It's recommended to chunk the data. You should change 'performChunking' to true in the Pinecone integration, which will fix the issue.

Add comment

Developer

Apify

Actor metrics

3k monthly users
465 stars
99.9% runs succeeded
3.1 days response time
Created in Mar 2023
Modified 10 days ago

Categories

Developer tools

Business

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

Puppeteer Scraper

apify/puppeteer-scraper

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.

Apify

4.2k

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

74.7k

409

Smart Article Extractor

lukaskrivka/article-extractor-smart

📰 Smart Article Extractor extracts articles from any scientific, academic, or news website with just one click. The extractor crawls the whole website and automatically distinguishes articles from other web pages. Download your data as HTML table, JSON, Excel, RSS feed, and more.

Lukáš Křivka

3.8k

Facebook Ads Scraper

apify/facebook-ads-scraper

Extract advertising data from one or multiple Facebook Pages. Get page details, reach estimates, publisher platforms, report count, number of impressions, ad IDs, timestamps, and more. Download Facebook ads data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

4.9k

Tiktok Hashtag Scraper

clockworks/tiktok-hashtag-scraper

Scrape TikTok hashtag data. Just add one or more hashtags and extract TikTok videos with that hashtag: URLs, likes, country of creation, video and music metadata, TikTok creator data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

1.9k

Tripadvisor Scraper

maxcopell/tripadvisor

This unofficial Tripadvisor API is a data extraction tool able to get data on hotels, restaurants, things to do, vacation rentals, attractions, tours, and public trips. Get pricing, contact details, amenities, awards, ratings, and more. Download your data in Excel, JSON, CSV, and other formats.

Maximillian Copelli

5.4k

TikTok Sound Scraper

clockworks/tiktok-sound-scraper

Scrape TikTok videos with a chosen sound. Just add one or more sound URLs and extract tiktoks that have it: URLs, likes, country of creation, video and music metadata, creator data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

167