Odd error on call to Python custom function

MindStudio IDE Bug Reports
1

(Can’t attach evidence here; it contains a proprietary API key. Please contact me via email.)

I’m building an agent for a client, and the following error occurs randomly on every run:

I created a simple Python script to count the characters from the text string returned by Scrape URL. Works about 98% of the time, but the other 2% throws this error:

**7 Executing "ScrapeSize"**
**8 Error: SyntaxError: Unexpected token '<', "<html>**
**<h"... is not valid JSON

**
The function does nothing but count characters and return a numeric value. So I don’t understand where that’s coming from. The output from Scrape URL and input to the function is just plain text, which can be seen in the debugger log.

I can do an imprecise work-around with a Generate Text block, but it would be better (and cheaper) if the Python function was 100% reliable.

Any guidance you can offer would be gratefully appreciated!

Thank you!

2

Hi Ken,

Could you share the Custom Function code and Configurations?

3

Code:

def scrape_size():

\# Get source variables data from config

text = ai\['vars'\]\['scrapedText'\]

max_size = ai\['vars'\]\['perItemTokenBudget'\]

print(f"Input Text Value: {text}")

print(f"Input Max Size: {max_size}")



chars = len(text)

words = len(text.split())

   

\# crude but solid approximations

tokens_est = max(chars // 4, int(words \* 1.3))



dest_var = ai\['config'\]\['tokens'\]

ai\['vars'\]\[dest_var\] = tokens_est



dest_var = ai\['config'\]\['tooLong'\]

ai\['vars'\]\[dest_var\] = (tokens_est > int(max_size or 3000))

# Execute the function

scrape_size()

Configuration:

config = {

transitionType: ‘controlled’,

metadata: {

name: 'Scrape Size',

description: 'Determines whether a block of text is too long',

},

blockStyle: {

label: 'Scrape Size',

},

configurationSections: [

{

  title: 'Input Settings',

  items: \[

    {

      label: 'Scraped Text',

      type: 'inputVariable',

      variable: 'scrapedText',

    },

    {

      label: 'Max Size',

      type: 'inputVariable',

      variable: 'perItemTokenBudget',

    },

  \],

},

{

  title: 'Output Settings',

  items: \[

    {

      label: 'Token Count Estimate',

      type: 'outputVariableName',

      variable: 'tokens',

    },

    {

      label: 'Too Many Tokens?',

      type: 'outputVariableName',

      variable: 'tooLong',

    },

  \],

},

],

}

4

Hi @Ken,

It seems this could be happening when a Scrape URL block saves an HTML error or redirect page to a variable, so the function receives HTML instead of JSON.

Could you send me a direct message with the Debugger logs for that run? To do that, open the Debugger, select the run, click Share, and reply with the URL:

image
image777×616 78.9 KB