Document Type
Article
Abstract
Exact-match evaluation of agent-calling obscures qualitatively different failure modes: a model may select the right function yet hallucinate argument values, or satisfy a schema while choosing a agent for the wrong reason. Existing benchmarks collapse these distinctions into a single binary score, leaving practitioners unable to diagnose where agent calls fail. We propose SAAG a cascaded diagnostic framework that decomposes agent-calling evaluation into three sequential stages: registry conformance, structural completeness, and argument grounding, each producing interpretable stage-specific diagnostics. These diagnostics additionally enable iterative self-repair: on prediction failure, the stage-specific signal guides targeted correction without leaking ground-truth values. We evaluate this framework on a controlled benchmark derived from Glaive’s function-calling dataset across registry sizes of 5, 10, and 15 agents using three local sub-4B-parameter models. Structured feedback consistently improves argument precision and reduces value hallucination relative to single-pass inference and uninformative binary feedback, while end-to-end F1 gains are modest and model-dependent. These results suggest that stage-decomposed diagnostic evaluation is a necessary lens for understanding and improving agent-calling reliability across model families and registry scales.
Publication Info
Published in Towards Knowledgeable Foundation Models @ ACL 2026 Workshop, 2026.
APA Citation
Garimella, R., Khandelwal, V., Kohli, A., & Sheth, A. (2026, June 30). SAAG: Structured Agent Assessment and Grounding. 4th Workshop on Towards Knowledgeable Foundation Models at ACL 2026. https://openreview.net/forum?id=we5A19dPN6
Rights
SAAG: Structured Agent Assessment and Grounding, by Ritvik Garimella, Vedant Khandelwal, Anvi Kohli, & Amit Sheth, is licensed under a Creative Commons Attribution 4.0 International License.