Tool Haystack
About this benchmark
Forty-four banking tools, one customer message, six cases. An earlier version offered thirty tools drawn from different corners of the bank and every target scored 100: when the answer is about a statement and the distractors are branch lookups, topic settles it without a description being read.
So the list is built in families. Eight tools take a card and its last four digits, six report transactions, six speak about statements, and the clause that separates them sits at the end of the description: a block is permanent, a freeze is reversible, a loss report changes nothing. Family members are scattered rather than grouped, and two cases are built so the tool whose name best fits the message is the wrong one.
`tool` is a gate, so a wrong pick scores zero however many arguments matched, and the arguments stay at transcription throughout. Three messages come from `tool-call` word for word, so the same request can be read at eight tools and at forty-four.
Cases
hold-not-block: `stolen-card` read the other way, from the same eight card tools: the customer says not to cancel, so the answer is the hold they can lift.merchant-search: The customer names the merchant, which rules out listing the window. `list_recurring_charges` answers "a double bill" rather than the request.pending-not-posted: The merchant is named, so `search_transactions` looks right and is the one tool that cannot answer: it returns settled charges, and this one has not settled.standing-not-scheduled: `schedule_transfer` runs once on a named day. The money moves every month.statement-month: Six tools in the list speak about statements and only one returns what the month closed at.stolen-card: `report_card_lost` records the loss but leaves the card chargeable, while a temporary freeze is for a card expected back. This one will not be recovered.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| qwen3.7-flash@alibaba | 100.0 | 0.0 | 3.5s | 115.3 | $0.0015 | 0/6 |
| gpt-5-mini@openai | 100.0 | 0.0 | 6.5s | 41.1 | $0.0082 | 0/6 |
| gemini-2.5-flash@google-ai-studio | 100.0 | 0.0 | 1.5s | 33.6 | $0.0093 | 0/6 |
| gemini-3.5-flash-lite@google-ai-studio | 100.0 | 0.0 | 1.2s | 42.6 | $0.0109 | 0/6 |
| gpt-5.6-luna@openai | 100.0 | 0.0 | 4s | 16.2 | $0.0136 | 0/6 |
| kimi-k2.5@moonshotai | 100.0 | 0.0 | 12s | 27.3 | $0.0204 | 0/6 |
| glm-4.7@z-ai | 100.0 | 0.0 | 38.5s | 8.3 | $0.0261 | 0/6 |
| gpt-5.6-terra@openaierrorno completion choices returned from OpenAI API | 100.0 | 0.0 | 2.5s | 22.8 | $0.0285 | 1/6 |
| kimi-k2.6@moonshotai | 100.0 | 0.0 | 20.4s | 15.0 | $0.0308 | 0/6 |
| claude-haiku-4.5@anthropic | 100.0 | 0.0 | 2.2s | 47.4 | $0.0430 | 0/6 |
| glm-5.2@z-ai | 100.0 | 0.0 | 5.7s | 19.1 | $0.0502 | 0/6 |
| grok-4.5@xai | 100.0 | 0.0 | 2.5s | 45.2 | $0.0739 | 0/6 |
| gemini-3.6-flash@google-ai-studio | 100.0 | 0.0 | 3.7s | 153.9 | $0.0763 | 0/6 |
| glm-4.7@cerebras | 100.0 | 0.0 | 1.2s | 203.6 | $0.0887 | 0/6 |
| claude-sonnet-5@anthropic | 100.0 | 0.0 | 3.9s | 37.9 | $0.1090 | 0/6 |
Note
- gpt-5.6-terra returned no completion choices on
hold-not-block, its third such drop in three runs. Its score covers five cases.
Conclusion
Fourteen targets tie at 100, so cost decides: qwen3.7-flash at $0.00024 a task, or gemini-3.5-flash-lite at 1189ms where latency matters more.
Second corpus, second flat column. This one made the best name-match deliberately wrong and gave two cases opposite answers from the same eight card tools, and no target missed either. Single-turn selection does not separate these models, so the next signal has to come from argument conformance or from turns where a wrong call compounds.