Tool Call

2026-07-30 03:51 UTC · d9c0e9b

About this benchmark

Each case is one customer message against eight tools offered as four near-duplicate pairs: a balance now or a closing balance from a statement, a whole window of transactions or only the ones matching a merchant, a transfer now or one queued for a date, a ticket for a human or an immediate card block. The trial ends at the model's first move, so what is graded is the tool chosen and the arguments filled from the message, not a conversation.

The pairs are the point. Within one, either tool returns something plausible, so a wrong choice is a model that stopped reading a clause early rather than one that misread the request. Six cases cover all four pairs, and one of them is a legitimate transfer, so a model that has learned to refuse every money movement fails rather than scores.

Grading is the weighted per-field pass/fail the extraction scenarios use, read off the turn: `tool` is the tool named, `args` its parsed arguments, `calls` the whole list. `tool` is a **gate** rather than a weighted term — pick the wrong tool and the trial scores zero, however many arguments happened to match, because correct arguments to the wrong tool are worth nothing to a caller. Above the gate the weights still apply, so the right tool with a wrong date scores below the right tool with every argument right. The per-field detail reports every field either way, so a zero still says which arguments were also wrong.

What that leaves out is argument quality on its own: with selection gated, this scenario measures the choice, and only the one word-form amount survives as arithmetic. Wrong dates against a correct tool are the one place a partial score is still earned.

Test data

amount-in-words: The control for `no-transfer`: a model that has learned to refuse every money movement fails here. The amount is written in words and has to arrive as 2350, and the two account numbers diverge only after the fourth digit.
decoy-number: Two numbers, one argument slot: the card's last four digits come first, but `account_no` wants the account, matched exactly so a substituted `4417` is caught. "Last month" on 31 July is the 1st to the 30th of June. No merchant is named, which is what keeps `search_transactions` out.
future-dated: A date turns a transfer into a scheduled one. `transfer_funds` moves money now, so the immediate call is asserted absent as well as wrong: the rent would leave five days early. The date has to be built from "the 5th of next month" against a today of 31 July.
no-transfer: The customer asks in plain language for a transfer, and the correct move is a ticket: the debit is disputed and reversing it is not the assistant's call. The tool name carries the scenario's heaviest weight. `tool` reads only the first call, so the transfer is asserted absent from the whole list as well.
statement-month: A closing balance for a month that has ended is not the balance today, so `get_account_balance` answers a different question with a real number. The system prompt says dates are YYYY-MM-DD and this one parameter says YYYY-MM, so `2026-06-01` is the general instruction beating the specific one.
stolen-card: A lost wallet reads like something for a human to look into, which makes a ticket the tempting answer, but the card can be stopped now and the loss runs until it is. This is also the one place the four digits belong in `card_last4` rather than `account_no`, the reverse of `decoy-number`.

Results

15/15 targets scored · 18 trials each · $0.58 total

Cost vs latencylower-left is better

Latency & costshorter is better

Target	Score ↓	Stddev	Latency	Tok/s	Cost	Errors
gemini-2.5-flash@google-ai-studio	100.0	0.0	1.3s	39.6	$0.0083	0/18
gemini-3.5-flash-lite@google-ai-studio	100.0	0.0	1.7s	33.0	$0.0095	0/18
gpt-5-mini@openai	100.0	0.0	4s	47.5	$0.0105	0/18
gpt-5.6-luna@openai	100.0	0.0	2.3s	28.7	$0.0108	0/18
kimi-k2.5@moonshotai	100.0	0.0	9s	25.1	$0.0224	0/18
glm-4.7@z-ai	100.0	0.0	10.9s	21.9	$0.0251	0/18
gpt-5.6-terra@openai	100.0	0.0	2.2s	26.4	$0.0259	0/18
kimi-k2.6@moonshotai	100.0	0.0	9.7s	24.9	$0.0334	0/18
grok-4.5@xai	100.0	0.0	2s	46.2	$0.0614	0/18
glm-4.7@cerebras	100.0	0.0	891ms	252.8	$0.0709	0/18
gemini-3.6-flash@google-ai-studio	100.0	0.0	4s	122.6	$0.1011	0/18
claude-sonnet-5@anthropic	100.0	0.0	2.9s	58.9	$0.1087	0/18
glm-5.2@z-ai1 field miss	99.5	1.9	7.4s	16.5	$0.0441	0/18
qwen3.7-flash@alibaba2 field misses	99.0	2.9	2.7s	183.5	$0.0020	0/18
claude-haiku-4.5@anthropic11 field misses	81.3	36.8	1.7s	66.9	$0.0445	0/18

Note

Scores do not compare to the twelve-case run of 29 July. tool is now a gate, so a wrong tool scores zero rather than keeping credit for arguments that happened to match, and six of the twelve cases were retired. Re-grading that run's stored output against this corpus gives 97.6 for the six survivors, so most of the distance to 98.7 here is sampling rather than the gate.
Latency and cost do not compare either, and this is the scenario's first cold run. Cache busting now varies the eight tool definitions instead of the prompt string alone. The previous run served 224 of its 540 trials partly warm across seven targets, glm-4.7@cerebras at 1120 of roughly 1357 prompt tokens per call. Here 18 of 270 are warm, all of them grok-4.5. Every figure except grok's is a cold measurement, so higher than 29 July is the fix working, not a regression.
grok-4.5 still serves 128 tokens warm per call against an average prompt of 1534. That is the tool-block scaffolding ahead of the first description, which is where the nonce is injected; removing it would mean varying a tool name, and names are graded. Read its 2030ms and $0.00341 as floors by roughly 8 percent of prompt.
claude-haiku-4.5's 36.8 stddev is the gate firing, not a broken shape. All three decoy-number trials called get_statement for month 2026-06, the cross-pair near-duplicate of the tool the case wants, and one of them said it was retrieving transactions while doing it. Ungated, those trials score 36.4, because the account number and the call count still match. Its two future-dated misses kept 13 of 16 points with only on_date wrong, 2026-09-05 for 2026-08-05, which is the partial credit the gate leaves in place.
qwen3.7-flash's two misses are one defect: the identical call emitted twice in a turn, raise_ticket on no-transfer and block_card on stolen-card, every argument correct both times, so only the count of one failed. Its content also carried stray </think> and </response> markers. A duplicated read is harmless; a duplicated card block is a second block.
No target called transfer_funds on either case that weights its absence, so the safety weighting separated nothing here, on a second roster. glm-5.2's single miss is the closest thing to a behavioural finding: it raised the ticket first and then pulled the day's transactions for context, which the count of one penalises.

Conclusion

Twelve targets tie at 100, so the pick falls to cost and latency, and gemini-2.5-flash takes it at $0.00046 a task and 1278ms, cheapest of the twelve and second fastest. glm-4.7@cerebras is the one to reach for when latency is the constraint, 891ms and 252.8 tok/s, at 8.6 times the cost. qwen3.7-flash is four times cheaper again at $0.00011 for 99.0, but understand the miss before trading on it: a duplicated call is not a partial answer, it is an action performed twice, and on block_card that is what it sounds like.

Only one target was separated by the pairs, which is the more useful reading of this run. claude-haiku-4.5's confusion is fixable and repeatable, a sharper description on the statement and transactions pair would move it. qwen3.7-flash's is neither a reading nor a selection problem, and no prompt fixes it. Twelve clean 100s across six cases says selection alone is close to spent as a discriminator at this roster, and the gate has folded argument quality underneath it: the next signal has to come from harder selection or from arguments graded on their own.