Hi.
I have been trying to play around with Mindstudio and getting to know how it works.
I created a workflow where the AI worker scrapes the openAI news page everyday at 8 AM (using Firecrawl). Then I used a generate text block to get all the news-related urls from the scraped html data.
The problem is that it was not giving the same answer every time!
Then I tried a new project where I scraped the same webpage and defined a global variable to track any changes of articles from the webpage.
This way, I had 2 json objects. One is a global variable containing all articles from the previous scrape of the news page and the other one is the new scraped json data (both with the same format).
I removed 2 articles from the global variable on purpose and used a text generation block and asked it to give me the new articles by comparing these 2 variables but, again, it is unpredictable and every time, it shows a different result.
I did this comparison on the openAI website and it can easily compares these 2 json objects and give back the results every time.
Does anyone know what the issue is?
Thanks
Thanks for sharing those. Please take a look at Tips for Working with Large Variables in the documentation. I suspect that substituting the variables directly in the instruction sentence is the cause of the issue.
If you take a look at the debugger, you will be able to see the raw text that is sent to the model. In your first block, it will look something like:
Extract all URLs from the provided HTML content <html><head><title>Open AI... [1000 more words of random HTML]</html> and output them in a JSON format
While AI models are great at processing text, this can make it very confusing for the model to understand which bits of the prompt are instructions and which bits are context. Your prompt would be improved by changing it to something like:
<website_html>
{{OpenAI_Content}}
</website_html>
Using the provided HTML, extract all URLs and output them in a JSON format. The extraction should include...
