I have a workflow that fetches information from a data source, analyzes it, and gives me the results based on this data source with just a 400-page PDF. It has given me good results, especially because I followed your suggestion and now it divides the question into three and then queries the data source three times in parallel to try to give a more accurate answer, it is working and giving very good answers.
However, it is not consistent. Sometimes the answers are very good, but there are times when it responds that it has no knowledge of this information and returns inconsistent (i.e., empty) results.
This is the workflow:
Could this have to do with some update that needs to be made at the data source level, or is there a way to ensure consistency without having to worry about whether or not it retrieved the respective chunks?
Could you please share the Debugger logs for the run that returned no chunks? That will let me take a closer look to understand what might have caused it.
To do that, open the Debugger, select the run, click Share, and reply with the URL:
Could you let me know how often you see runs where no chunks are returned? Could you also share a Loom showing the steps you’re taking when the Query Data Source blocks don’t return any chunks?
Thanks for following up. Unfortunately, it’s not feasible to constantly monitor the agent. However, whenever I notice that there are no responses, I check it, and it’s always due to the data source not returning results. This can happen multiple times a day, as it has already occurred several times this week.
Could you clarify what you mean by recording a video in LOOM, doing what?
I’ve been trying to reproduce this on our end, which is why I asked for a Loom. I’d like to see the exact steps you’re taking in case I’m testing it a different way.
How often do you see the Query Data Source block return no results? I can see there was a temporary disruption on January 12 to 13, but I’m trying to understand whether this could be a more consistent issue.
If you’re seeing this regularly, could you share the debugger log links for those runs? That should help us narrow down what’s causing it.
I’m trying to figure out whether this was a one-off glitch or something that’s happening more regularly.
Do you have runs where the Query Data Source blocks don’t return any results on a recurring basis? If so, could you share the debugger logs for any runs that happened after January 13? That would really help us pinpoint the cause of this behavior.
In fact, after January 13, every logs returned results, but I still have inconsistencies when I ask the same question and get different results. Could it be that it blocks the data source, or do I need to go there and “force” the data source every day?
Today, the same question was asked twice again. The first time, there was no correct response (empty), and the second time, the response was correct, asked by the same user, a few minutes later:
Glad to hear Query Data Source is pulling chunks consistently again. If you see any “empty” outputs from the Query Data Source blocks again, please let us know so we can take a closer look.
Now, onto your question:
Your Agent is set up like this:
Generate Text: creates 3 queries from the user’s question
3 Query Data Source blocks: retrieve the closest chunks
Generate Text: creates the final answer from those chunks
Let’s compare both runs:
Steps:
“empty” result run
correct response run
Comment
Refine Query
0: “replace a ned” 1: “what does replace a ned mean” 2: “ned replacement process”
0: “who can replace a need” 1: “how to replace a need” 2: “need replacement process”
I don’t have specific information on that topic. For assistance, please email our support team with your detailed question. They’ll provide you with the information you need.
Any user can replace a training need that was contributed by them until the need is approved or denied by an approver. Permission depends on your approval workflow: 1. When Workflow Hierarchical Approval is enabled – Users from higher levels can always replace contributions from lower levels. 2. When Workflow Hierarchical Approval is disabled – You can only replace a need if you were the contributor or if you did an override of needs from contributors at levels lower than your own, during their contribution period. Important restriction: If the configuration “Managers prevented to Replace or Delete end-user requested needs” is active in Task Plans Global Configurations, the replace button will not be available to managerial contributors (all levels) for needs assigned by end-users. When replacing a need, you must select a course using the Search Course Functionality and press the Replace Assignment button to complete the action. In task plans with budget configuration active, the budget is also checked when replacing a need.
Claude 4.5 Haiku generated different outputs from similar inputs
Based on this, it looks like the inconsistency is coming from Claude 4.5 Haiku in the final Generate Text step.
I can’t say exactly why the model responded differently, but in practice, this can happen when you’re sending a lot of text into a model designed for speed/cost. It can sometimes miss what it needs and fall back to a generic answer, even when the relevant chunks are present. While the model itself is pretty good, it doesn’t have reasoning capabilities, which seem to be required for your use case.
Here’s what I’d recommend:
Test models with reasoning capabilities for the final Generate Text block. The Profiler feature should help with the side-by-side comparison
Lower the temperature for more consistent responses
Reinforce and/or shorten the prompt to minimize any ambiguity
Any clue on how can we improve the response performance of the BOT, which currently has an average response latency of 20 seconds? Do you think we can somehow reduce this to 5-6 seconds?
Reasoning models do tend to take longer to respond. I’d recommend testing a few different models to find the best balance between speed and output quality for your use case.
I ask this because every time i test it, it doesn’t give me an answer connected to the agent context, it looses the agent context.
What is the best way to find the best balance between speed and output quality inside mindstudio, but with the right context, to know if the answers are accurate and in real time?
You can paste the exact prompt that was sent to the Generate Text block when it returned an incomplete response.
Profiler doesn’t have access to Data Sources on its own, just like Generate Text blocks don’t until you add Query Data Source blocks and reference those variables in the prompt.
When a prompt is sent to the model, all variables are resolved and the full text is passed to the selected LLM. You can replicate that in Profiler by copying the prompt with resolved variables from the Debugger and pasting it there:
