I’m having an issue with the Scrape URL block — it’s started returning a lot of inconsistent errors. The same URL sometimes fails with an error (as shown in the attached screenshot), but when I retry the exact same request, it often works on the second or third attempt.
Here’s the URL I’m using (for reference, only as an example): Link
This behavior is happening frequently and unpredictably, which makes the workflow unreliable.
Could you please check if there’s something causing this instability or suggest a way to improve reliability when using the Scrape block?
You can not use the scrape URL block to scrape Google, as Google has sophisticated anti-scrape technology in place. You must use the “Search Google” block.
First of all, you can see in the screenshot that I can, and it works well, but sometimes it returns an error, and I want to understand why it happened.
I use scrape instead of a simple google search because I scrape the page where I use advanced google search
What Sean (CTO) is trying to say is that Scrape URL is not the proper way to search Google. The Scrape URL block will not always return correct results because of Google’s anti-scrape tech. Please use the Google Search block for a consistent experience.
