Website Content Crawler

Maybe I'm doing something wrong?

This is returned:

[ { "url": "https://aws.amazon.com/sagemaker/studio/features/", "crawl": { "loadedUrl": "https://aws.amazon.com/sagemaker/studio/features/", "loadedTime": "2024-09-14T04:36:01.172Z", "referrerUrl": "https://aws.amazon.com/sagemaker/studio/features/", "depth": 0, "httpStatusCode": 200 }, "metadata": { "canonicalUrl": "https://aws.amazon.com/sagemaker/studio/features/", "title": "Machine Learning Service - Amazon SageMaker Studio Features - AWS", "description": null, "author": null, "keywords": null, "languageCode": "en-US", "openGraph": [ { "property": "twitter:title", "content": "Machine Learning Service - Amazon SageMaker Studio Features - AWS" }, { "property": "og:title", "content": "Machine Learning Service - Amazon SageMaker Studio Features - AWS" }, { "property": "twitter:card", "content": "summary" }, { "property": "twitter:image", "content": "https://a0.awsstatic.com/libra-css/images/logos/aws_logo_smile_179x109.png" }, { "property": "twitter:site", "content": "@awscloud" }, { "property": "og:type", "content": "company" }, { "property": "og:url", "content": "https://aws.amazon.com/sagemaker/studio/features/" }, { "property": "og:image", "content": "https://a0.awsstatic.com/libra-css/images/logos/aws_logo_smile_1200x630.png" }, { "property": "og:site_name", "content": "Amazon Web Services, Inc." } ], "jsonLd": null, "headers": { "content-type": "text/html;charset=UTF-8", "date": "Sat, 14 Sep 2024 04:35:50 GMT", "server": "Server", "x-frame-options": "SAMEORIGIN", "x-xss-protection": "1; mode=block", "strict-transport-security": "max-age=63072000", "x-amz-id-1": "E744CC50C6D54E448144", "last-modified": "Sat, 14 Sep 2024 01:30:40 GMT", "vary": "accept-encoding", "content-encoding": "gzip", "set-cookie": "aws-priv=eyJ2IjoxLCJldSI6MCwic3QiOjB9; Version=1; Comment="Anonymous cookie for privacy regulations"; Domain=.aws.amazon.com; Max-Age=31536000; Expires=Sun, 14 Sep 2025 04:35:50 GMT; Path=/; Secure\naws_lang=en; Domain=.amazon.com; Path=/", "x-content-type-options": "nosniff", "x-cache": "Miss from cloudfront", "via": "1.1 a5ec1cc448d0ca618712f253b7a7adba.cloudfront.net (CloudFront)", "x-amz-cf-pop": "JFK52-P4", "x-amz-cf-id": "YqwyLwlIkpzRmyc1SDQwkIAsWDKBG0tbLcPWd5vFWP2w_c6lEcG67g==", "x-firefox-spdy": "h2" } }, "screenshotUrl": null, "text": "Machine Learning Service - Amazon SageMaker Studio Features\nAWS support for Internet Explorer ends on 07/31/2022. Supported browsers are Chrome, Firefox, Edge, and Safari. Learn more »", "markdown": "# Machine Learning Service - Amazon SageMaker Studio Features\n\nAWS support for Internet Explorer ends on 07/31/2022. Supported browsers are Chrome, Firefox, Edge, and Safari. Learn more »", "debug": { "requestHandlerMode": "browser" } } ]

When crawling this URL:

https://aws.amazon.com/sagemaker/studio/features/?nc=sn&loc=2

Here is what my request looks like:

"request": { "url": "https://api.apify.com/v2/acts/apify~website-content-crawler/run-sync-get-dataset-items", "method": "POST", "body": "{"startUrls":[{"url":"https://aws.amazon.com/sagemaker/studio/features/"}],"useSitemaps":true,"crawlerType":"playwright:adaptive","includeUrlGlobs":[],"excludeUrlGlobs":[],"keepUrlFragments":false,"ignoreCanonicalUrl":false,"maxCrawlDepth":1,"maxCrawlPages":2,"initialConcurrency":0,"maxConcurrency":1,"initialCookies":[],"proxyConfiguration":{"useApifyProxy":true},"maxSessionRotations":10,"maxRequestRetries":5,"requestTimeoutSecs":60,"minFileDownloadSpeedKBps":128,"dynamicContentWaitSecs":10,"waitForSelector":"","maxScrollHeightPixels":5000,"removeElementsCssSelector":"nav, footer, script, style, noscript, svg,\n[role=\"alert\"],\n[role=\"banner\"],\n[role=\"dialog\"],\n[role=\"alertdialog\"],\n[role=\"region\"][aria-label*=\"skip\" i],\n[aria-modal=\"true\"]","removeCookieWarnings":true,"expandIframes":true,"clickElementsCssSelector":"[aria-expanded=\"false\"]","htmlTransformer":"readableText","readableTextCharThreshold":100,"aggressivePrune":false,"debugMode":false,"debugLog":false,"saveHtml":false,"saveHtmlAsFile":false,"saveMarkdown":true,"saveFiles":false,"saveScreenshots":false,"maxResults":5,"clientSideMinChangePercentage":15,"renderingTypeDetectionPercentage":10}", "headers": { "User-Agent": "Retool/2.0 (+https://docs.tryretool.com/docs/apis)", "Content-Type": "application/json", "Authorization": "---sanitized---", "ot-baggage-requestId": "undefined", "x-datadog-trace-id": "4921186735661459620", "x-datadog-parent-id": "5216962788210542958", "x-datadog-sampling-priority": "0", "x-datadog-tags": "_dd.p.dm=-1", "traceparent": "00-0000000000000000444b9143347728a4-48666003e5b8896e-00", "tracestate": "dd=t.dm:-1;s:0", "X-Retool-Forwarded-For": "xx.xx.xx.xx" } }

Here is the same thing, slightly more readable:

{ "startUrls": [ { "url": {{inputURL.value}} } ], "useSitemaps": true, "crawlerType": "playwright:adaptive", "includeUrlGlobs": [], "excludeUrlGlobs": [], "keepUrlFragments": false, "ignoreCanonicalUrl": false, "maxCrawlDepth": 1, "maxCrawlPages": 2, "initialConcurrency": 0, "maxConcurrency": 1, "initialCookies": [], "proxyConfiguration": { "useApifyProxy": true }, "maxSessionRotations": 10, "maxRequestRetries": 5, "requestTimeoutSecs": 60, "minFileDownloadSpeedKBps": 128, "dynamicContentWaitSecs": 10, "waitForSelector": "", "maxScrollHeightPixels": 5000, "removeElementsCssSelector": "nav, footer, script, style, noscript, svg,\n[role="alert"],\n[role="banner"],\n[role="dialog"],\n[role="alertdialog"],\n[role="region"][aria-label*="skip" i],\n[aria-modal="true"]", "removeCookieWarnings": true, "expandIframes": true, "clickElementsCssSelector": "[aria-expanded="false"]", "htmlTransformer": "readableText", "readableTextCharThreshold": 100, "aggressivePrune": false, "debugMode": false, "debugLog": false, "saveHtml": false, "saveHtmlAsFile": false, "saveMarkdown": true, "saveFiles": false, "saveScreenshots": false, "maxResults": 5 , "clientSideMinChangePercentage": 15, "renderingTypeDetectionPercentage": 10 }