Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1199
Monthly users
6.3k
Runs succeeded
>99%
Response time
3.5 days
Last modified
2 days ago
how to extract "date" meta data?
Closed
I need to extract the "date" meta data from the html content ? How can I do that? In the JSON fields I only have "authors" and few more fields but "date" is missing and is "crucial" to time sensitive content.
Example:
<meta name='date' content='2025-04-01'>
Hi,
Thank you for using Website Content Crawler.
I believe this should be possible to add—though we'll simply extract string without any other parsing.
The result would look like this: "metadata": "2025-04-01", would that work for you?
Jiri
avkarma
That would be great! Of course I can process the date/time layout later.
Can you provide me an example of a site with meta date? (for testing), I'm not able to quickly find one.
avkarma
Hi,
Thanks again for your suggestion and for providing the example link — and sorry for the late response.
We looked into it in more detail. The date field you're referring to is usually present on pages that use the NewsArticle schema (like blog posts).
In your case, the blog articles on trinnov.com seem to contain both the NewsArticle and Corporation JSON-LD records, so the date can indeed be extracted from those.
Afaik, the industry-standard way of defining a machine-readable publish date for online articles is through structured data in the JSON-LD element — specifically the datePublished or dateModified fields.
Check out our run crawling a few blog articles — all of them seem to contain datePublished information.
As for extracting the raw <meta name="date"> tag — if you have a strong use case that applies more broadly across other sites, feel free to share more details. So far, we haven’t seen this as a standard practice. In most cases, it would return null, making it less useful for most users. Still, we’re happy to evaluate this further if needed.
Let me know if you need help extracting the field from the blog article pages or have more examples to look into.
Best, Jiri
avkarma
Perhaps there should be a way to configure the crawler to extract additional meta data such meta[name="date"]. Anyway I think I will just parse the raw html myself and extract the data we need.
That would definitely do the trick—having this configurable would help.
In that case, you might want to check out our Web Scraper, which allows you to extract anything you need.
| Anyway I think I will just parse the raw html myself and extract the data we need.
Alright, I’ll go ahead and close this issue for now. Jiri
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
