Website Content Crawler

Hello and thank you for your interest in the Actor! Website content crawler configuration is pretty complex and it can cater to many different usecases. Without knowing yours, I cannot just give you a "correct JSON body payload". Could you show us what you have and describe the behavior that you'd like to change?

I appreciate your guidance. To clarify, I'm looking to run an actor via API to crawl websites with the following capabilities:

Ability to adjust memory allocation Control over crawl depth, specifically limiting it to the first page only Option to save images Option to save the HTML file Flexibility to modify other essential parameters as needed For the output, I'm aiming to retrieve:

Images URLs Text content Description HTML file

Could you provide a sample JSON body payload that includes these parameters and output requirements? I understand that website content crawling can be complex, but I'm looking for a starting point that I can then customize based on my specific use case. Any additional guidance on how to structure the payload or which parameters are most crucial would be greatly appreciated.

This is the default input configuration:

1{
2    "aggressivePrune": false,
3    "clickElementsCssSelector": "[aria-expanded=\"false\"]",
4    "clientSideMinChangePercentage": 15,
5    "crawlerType": "cheerio",
6    "debugLog": false,
7    "debugMode": false,
8    "dynamicContentWaitSecs": 10,
9    "expandIframes": true,
10    "htmlTransformer": "readableText",
11    "ignoreCanonicalUrl": false,
12    "initialConcurrency": 0,
13    "keepUrlFragments": false,
14    "maxConcurrency": 200,
15    "maxCrawlDepth": 1,
16    "maxCrawlPages": 9999999,
17    "maxRequestRetries": 5,
18    "maxResults": 9999999,
19    "maxScrollHeightPixels": 5000,
20    "maxSessionRotations": 10,
21    "minFileDownloadSpeedKBps": 128,
22    "proxyConfiguration": {
23        "useApifyProxy": true,
24        "apifyProxyGroups": []
25    },
26    "readableTextCharThreshold": 100,
27    "removeCookieWarnings": true,
28    "removeElementsCssSelector": "nav, footer, script, style, noscript, svg,\n[role=\"alert\"],\n[role=\"banner\"],\n[role=\"dialog\"],\n[role=\"alertdialog\"],\n[role=\"region\"][aria-label*=\"skip\" i],\n[aria-modal=\"true\"]",
29    "renderingTypeDetectionPercentage": 10,
30    "requestTimeoutSecs": 60,
31    "saveFiles": true,
32    "saveHtml": true,
33    "saveHtmlAsFile": true,
34    "saveMarkdown": true,
35    "saveScreenshots": false,
36    "startUrls": [
37        {
38            "url": "http://example.org/%20%de%20"
39        }
40    ],
41    "useSitemaps": false
42}

I adjusted maxCrawlDepth, saveFiles, saveHtml and saveHtmlAsFile for you. For the rest, you'll need to experiment to find out what works best for your use case. Regarding the maximum memory, that is not configured via input, but you can adjust it with a query parameter that you add to the URL of your API call - for example ?memory=4096 to set the memory to 4GB.