I’m working on a new project at Posit that involves a lot of building on the internals of LLM agents. In this project I recently added a new parameter to a tool that had existed for a while without problems. I titled the parameter description and it was used to provide plain-language description of what the code the agent wrote for the tool call was doing. Like how you’d describe a simple script’s contents to a collegue. E.g. “Read in the csv, filter out NAs, print the first 10 rows.”.
This worked great and the new schema got merged but I started noticing that - more often than I would expect - the models that I like to do interactive data science work with (fast and cheap ones that let me explore in flow) were messing up and repeatedly forgetting to provide another required parameter in the tool: “intent”. “Intent” is a short description that is used to label the toolcall for easy browsing. E.g. “load and filter data”. 1
At first my thought was “well duh, I added another parameter that sounds very similar to the first and the model is getting confused.” But then I looked into it further with some evals. It turns out that the model was always omitting the ‘intent’ field and never the ‘description’ one. I would have thought that this split would be about even.
This made me suspicious so I asked a couple agents to describe “standard tool calling schemas for code execution” and they all had ‘description’ as a parameter, but in the role of our ‘intent’. So it seems like the agent was getting confused with two similarly named parameters and then just defaulting to the one that was probably in harnesses that it was reinforcement learned in.
| Variant | Missed intent |
|---|---|
| current schema | 58% |
current, intent moved first in schema |
56% |
current, sharper intent text only |
31% |
description renamed codeWalkthrough |
34% |
renamed procedure |
18% |
renamed pseudocode |
0% |
renamed walkthrough |
0% |
renamed operations |
0% |
renamed stepNarrative |
0% |
The quick fix was to rename the description to something distinct. I tested a few options with evals and found ‘psuedocode’ brought the tool call error rate down to zero and made sense to me.
Takeaways
When coming up with parameter names for agent tools you not only need to use common sense (something I didnt do initially) but you should also see if there is prior art in other harnesses that could be ‘baked in’ to the models training. Also use simple evals to make sure you’re not missing something.
Footnotes
-
You might say, “what about the parameter description?” These made the difference very clear. The model just seems to not care about them at all. ↩