Using/architecting data sources

Community Discussion
1

I’m building a chat bot to assist users with navigating a large database of primarily product documentation. The underlying data itself is a mix of PDFs and webpages.

I’ve been able to set the PDFs as data sources easily enough and I’ve got a scrape URL block, but I’m not sure the best way to scale this up especially to a lot of URLs. If it’s helpful most documents are very topic based (Any query with ATA100 very clearly goes to one set of documents, whereas ATA200 definitely goes somewhere else).

Thank you!

2

Hi @rdrake,

Here’s a setup that works well for topic-based documentation:

  1. User enters their query
  2. Router block matches it to the closest Data Source for that topic
  3. Generate Text block analyzes the query and produces three similar variations to pull in more related content
  4. Those queries get passed to the Query Data Source blocks

We also have a template for this setup:
https://app.mindstudio.ai/agents/sample-data-sources-agent-38f31065/remix

Let me know what you think!

3

Hi Alex, sorry to bump an old thread. I’m working on a different (but similar) project that has about 200 URLs as source content, but not as many nice distinctions to route on. Is there a way (within mindstudio) to pre-load these URLs as a data source? Thank you!

4

Hi @rdrake,

Yes, you can set this up with two workflows in your agent:

  1. First workflow: Display Content block to hold your list of URLs as JSON, then pass them into a Run Workflow block set to sequential processing
  2. Second workflow: Scrape URL, Extract Text from File, or Analyze Image block (depending on the file format) followed by an Upload Data Source Document block to add it to a Data Source

Let me know if you have any questions!

5

I’ve got that first block seeming to work, but I think I’m running into issues on the second. I’ve that second block receiving the iterative items from the JSON, and then going to scrape URL. What block lets me take the scrape URL and make it into a PDF for upload data source? Thank you again!

6

Hi @rdrake,

You don’t need to convert it to PDF, Data Sources support a number of file types. The Upload Data Source Document block can take the raw text straight from the Scrape URL block and upload it as a txt file.

If you’re still running into any issues, I can take a closer look at your agent. Could you share a remix link?