After

Before

Hello, thank you for reporting this bug and I'm sorry for a late reply. I'm currently reworking the Actor to address this issue as well as other bugs. I'll keep you updated about the progress here.

Hi again, there has been a recent change of Google News API that caused this bug. The Actor has been fixed today, see the example run: https://console.apify.com/view/runs/oqDhZjFezcTVZHdAi

The Actor first extracts the links in the https://news.google.com/ format from the API, then it decodes them to the actual target links and opens the target pages to extract preview images. It needs to be done this way because the Actor doesn't use a web browser that could resolve the redirects automatically. It uses HTTP requests only to keep the expenses low. Google News has recently made the encoding of target URLs more difficult to deal with, so the Actor now uses rather a hacky way to decode the links. Hopefully the Google News API will be stable now and won't break the Actor's flow again.

If you encounter any other issues, please report them using new issue threads 🙏

Hi, we seems to be getting a lot of links like that causing it to show a good amount of failures on every run.

Any update on this as we are seeing quite a few links come up as https://news.google.com/_/DotsSplashUi/data/batchexecute?rpcids=Fbv4je

I am also having trouble with article urls being "https://news.google.com/rss/articles/..." rather than the actual aricle url. Could this be fixed ?

Hi, could you please share example runs where this issue occurred?

I did a test run with 1800 results and didn't find any https://news.google.com links stored in the link output field (see screenshot). Knowing which query triggers the issue would help me a lot debugging this.

It's also possible it was fixed on Google News' side in the meantime but I need to redo the problematic runs to validate it.

Regarding the URL https://news.google.com/_/DotsSplashUi/data/batchexecute?rpcids=Fbv4je, the Actor uses it to decode links from the https://news.google.com/ format.

I have another idea what could go wrong - did you set the input filed extractImages to true or false? Originally, non-RSS article links were served by the RSS API, so the Actor was able to scrape them without opening article pages. Crawling articles pages could be switched on by extractImages: true, because images were extracted from these article pages (they still are). After the latest change, non-RSS links are no longer available in the RSS API. It became necessary to crawl article pages to get decoded article links. Therefor, you need to use extractImages: true to get non-RSS article links.

I'm sorry for this confusion, I didn't realize that the function of extractImages changed into a switch between (not) getting decoded article links. I'll add a new field fetchArticleDetails instead and possibly deprecate extractImages

So, fetchArticleDetails input field was added and extractImages deprecated. extractImages can still be used in the JSON input editor or when calling the Actor via API but it is no longer displayed in the Manual input editor.

• fetchArticleDetails: true activates additional requests to article pages to decode the links from RSS format and also extract images from the page's metadata.
  • test run: https://console.apify.com/view/runs/huZYRddUxVUsrPxEq
• fetchArticleDetails: false disables requests to article pages so the links are stored in the RSS format https://news.google.com/rss/articles/ and no images are scraped. The runs with this option disabled are much faster and cheaper though, thanks to saving a lot of extra requests.
  • test run: https://console.apify.com/view/runs/oaf7KSCUBS519ARj5

Please let me know if you still struggle to scrape decoded article URLs with fetchArticleDetails option enabled. Thank you!

Hi,

With the option "fetchArticleDetails: true", "link" is pointing to the direct article and not the rss url.

Thank you for the fix.