Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demo.json file issues via API
Closed
Is it possible to obtain a URL to a .json file from a dataset via an API, rather than receiving JSON objects directly?
Hello, and thank you for your interest in this Actor!
I think I'm missing something. Can you please clarify your question? What are you trying to achieve?
You likely already know that the Dataset items can be accessed via API (on the https://api.apify.com/v2/datasets/{DATASET_ID}/items endpoint). This returns the JSON array with the dataset items inside.
If you want to find, e.g., all the dataset IDs for your WCC runs, you can first query the https://api.apify.com/v2/acts/apify~website-content-crawler/runs?token={YOUR_APIFY_TOKEN} API endpoint.
This lists all the recent Website Content Crawler runs under your account, along with their durations, statuses, and default dataset IDs.
Does this answer your question? If not, please feel free to clarify your use case. In the meantime, you can also check out our API documentation, where you can find all the API endpoints along with their descriptions and the parameters they accept.
Thank you!
glovebubble
Thank you for your response.
To clarify, I am looking to obtain the actual .json file or a URL to the .json file, rather than receiving a JSON array or objects. Could you please guide me on how to achieve this?
If you want the server to return the dataset with the Content-Disposition: attachment HTTP response header, which forces your web browser to download the data as a .json file (as opposed to showing you the contents of the file), you can add the attachment=true query parameter to your dataset URL.
Your dataset URL then will look something like this: https://api.apify.com/v2/datasets/{DATASET_ID}/items?attachment=true&format=json.
You can explore other dataset export options at the Storage tab of your run - see the attached screenshot. Clicking the blue "Download" button will generate the URL with the attachment=true parameter, which forces your browser to download the file.
Does this solve your problem? Feel free to close this issue if it does - or ask additional questions if it doesn't. Cheers!
Apify- 3k monthly users
- 465 stars
- 99.9% runs succeeded
- 3.1 days response time
- Created in Mar 2023
- Modified 10 days ago
