Tool Loop
About this benchmark
Twenty-nine tools, five requests, and no request any one tool can answer. The calls are answered from a lookup table beside each case, so the loop runs for real: the model calls, reads a result, and calls again with what the result told it.
Every chain is locked by an identifier the model cannot know in advance. Places are found by area, so coordinates have to be resolved first; a session is ranked by exchange code and priced by ticker; a fare needs two station ids. A call whose identifier the world does not recognise is refused rather than answered loosely, so guessing costs a turn and shows up in `rejected`.
The answer is a gate, so a right chain ending in a wrong number scores zero while the per-field detail still says which hop broke. Three cases leave the arithmetic to the model. The distractors are close rather than absent: council parking beside a venue's own car park, an e-hailing estimate beside a rail fare, and all four ranking measures populated, so the wrong one returns a plausible company rather than an error.
Cases
dividend-yield: No tool returns a yield: it is 0.63 declared against a last price of 9.84, divided by the model. `get_company_profile` answers for the same ticker with everything except that.index-close: Two calls are enough. Everything else the world answers here is real and adjacent, so a model that keeps going lands on constituents and a market status.mall-parking: `find_places` takes an area and not coordinates, so the geocode is unavoidable. The nearest mall is 0.4 km out and the famous one 2.9 km, and free bays are a subtraction the model does itself.most-traded-stock: Shares changing hands is `volume`, and all four measures are populated, so the wrong one returns a different company at a plausible price rather than an error. The volume list is unsorted and the leader sits third.station-fare: The two middle hops are independent, so grading matches on tool name rather than position. The nearest station is not the one sharing the area's name, and both routes to KL Sentral are priced.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| gpt-5.6-luna@openai | 100.0 | 0.0 | 9.5s | 19.3 | $0.0024 | 0/5 |
| qwen3.7-flash@alibaba | 100.0 | 0.0 | 6.8s | 120.9 | $0.0024 | 0/5 |
| gpt-5-mini@openai | 100.0 | 0.0 | 22.7s | 46.2 | $0.0146 | 0/5 |
| kimi-k2.5@moonshotai | 100.0 | 0.0 | 17.3s | 20.2 | $0.0185 | 0/5 |
| gpt-5.6-terra@openai | 100.0 | 0.0 | 7s | 22.1 | $0.0229 | 0/5 |
| glm-5.2@z-ai | 100.0 | 0.0 | 17.6s | 14.5 | $0.0466 | 0/5 |
| deepseek-v4-pro@together | 100.0 | 0.0 | 19.6s | 20.4 | $0.0513 | 0/5 |
| grok-4.5@xai | 100.0 | 0.0 | 7.1s | 33.6 | $0.0614 | 0/5 |
| claude-haiku-4.5@anthropic | 100.0 | 0.0 | 6.7s | 48.4 | $0.0889 | 0/5 |
| gemini-3.6-flash@google-ai-studio | 100.0 | 0.0 | 7.6s | 109.3 | $0.1304 | 0/5 |
| glm-4.7@cerebras | 100.0 | 0.0 | 3.1s | 134.6 | $0.1744 | 0/5 |
| claude-sonnet-5@anthropic | 100.0 | 0.0 | 9.1s | 33.4 | $0.1997 | 0/5 |
| gemini-2.5-flash@google-ai-studio4 field missesstation-fare: calls (1/1)station-fare: calls.#(name=="get_transit_fare").args.from_station_id (1/1)station-fare: content (1/1)station-fare: rejected (1/1) | 80.0 | 40.0 | 4.3s | 25.0 | $0.0146 | 0/5 |
| gemini-3.5-flash-lite@google-ai-studio2 field missesstation-fare: calls.#(name=="get_transit_fare").args.from_station_id (1/1)station-fare: content (1/1) | 80.0 | 40.0 | 3.4s | 32.5 | $0.0211 | 0/5 |
Note
- First run on this branch's fixture identifiers and match-query assertions, so the scores are not directly comparable with the run of 31 July at 04:01 even though the roster and the trial count are the same.
- No target skipped a graded call. The three that supplied an identifier out of their own knowledge on 31 July, Sonnet 5 and Haiku 4.5 among them, called the whole chain this time.
gemini-2.5-flashmisspelt the same argument key on two successive calls, was refused both times, and then reported a fare that no tool in this run returned. Its first attempt carried the right station id, so the refusal cost it the answer rather than the lookup.gemini-3.5-flash-litemade four well-formed calls with nothing rejected and answered for a station it was not asked about. Its miss is a wrong answer, not a broken shape, and the two 80s on the page do not mean the same thing.deepseek-v4-promade exactly the calls each case expects with none refused, after the previous run left it unscored on one case and spending a whole step budget on another. Nothing in this run exceeded five calls or two refusals.- One trial per case, so the 40.0 in the stddev column is spread across the five cases and not flakiness across repeats. Nothing here separates a target that cannot do the work from one that did not do it once.
- 345,370 of 904,375 prompt tokens were served from cache, and unevenly:
gpt-5.6-lunaread 78% of its station-fare prompt from cache againstqwen3.7-flash's 36%. The cost and latency columns flatter the warm targets.
Conclusion
Twelve targets tie at 100, so cost and latency decide, and they point at
qwen3.7-flash at $0.000486 a task. gpt-5.6-luna is $0.000007 a task cheaper
and gives back 2730ms for it, at 19.3 tok/s against 120.9, and its edge is the
softer of the two numbers because more of its prompt came from cache. Where
latency is the whole requirement, glm-4.7@cerebras answers in 3089ms for
seventy-two times the money.
Both targets below 100 lost the same case, for reasons that do not deserve the
same reading. gemini-3.5-flash-lite returned a clean, well-formed loop and
picked the wrong subject, which is a single visible field and the kind of miss a
sharper prompt can move. gemini-2.5-flash malformed an argument and then
stated a number it had never been given, which is the failure that matters here:
the answer gate caught it this once, and at one trial per case there is no way to
tell whether that is habitual. Run trials against that target before trusting
its 80, and treat the fabricated figure, not the score, as the finding.