Ticket Extract
About this benchmark
Turns a customer support email into a structured ticket: ticket id, customer name and email, product, severity in the customer’s own words, the date the problem started, order number, requested action, and one entry per affected device with serial and model.
The input is plain text pasted into the prompt, so nothing here depends on PDF handling; it isolates comprehension and instruction-following. Each message carries a deliberate trap: numbers that look like the one being asked for, a date the customer corrects mid-thread, and a device list whose length is stated in prose rather than tabulated.
Test data
ticket-a: Two Meridian G4 gateways, urgent, RMA requested. The sender opens by correcting the start date (the fault began 2 June, not the 6th) and an invoice number sits right beside the order number.ticket-b: One Beacon M2 sensor, low priority, replacement requested. The unit was a free sample, so there is no order number and the field must stay empty rather than borrow the ticket reference.ticket-c: Five Halyard R1 controllers bricked by a firmware rollout, high severity, rollback requested. Serials are split across two sentences and a sixth serial is named only to be excluded, so the expected device count is five.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| gemini-3.5-flash-lite@google-ai-studio | 100.0 | 0.0 | 1.2s | 164.1 | $0.0084 | 0/15 |
| gpt-5.6-luna@openai | 100.0 | 0.0 | 3.7s | 47.8 | $0.0229 | 0/15 |
| claude-haiku-4.5@anthropic | 100.0 | 0.0 | 2.7s | 54.5 | $0.0242 | 0/15 |
| grok-4.5@xai | 100.0 | 0.0 | 4.7s | 79.6 | $0.0422 | 0/15 |
| glm-4.7 | 100.0 | 0.0 | 6.1s | 220.4 | $0.0777 | 0/15 |
| gemini-3.6-flash@google-ai-studio | 100.0 | 0.0 | 6.5s | 206.1 | $0.1568 | 0/15 |
| gemini-3.5-flash@google-ai-studio | 100.0 | 0.0 | 5.1s | 240.0 | $0.1714 | 0/15 |
| gpt-5.6-terra@openai1 field missticket-c: severity (1/5) | 99.4 | 2.1 | 1.9s | 66.4 | $0.0470 | 0/15 |
| qwen3.7-plus@alibaba26 field missesticket-a: affected_devices (1/5)ticket-a: affected_devices.0.serial (1/5)ticket-a: affected_devices.1.serial (1/5)ticket-a: customer_email (1/5)ticket-a: reported_at (5/5)ticket-b: affected_devices (2/5)ticket-b: affected_devices.0.serial (2/5)ticket-b: product (1/5)ticket-b: reported_at (5/5)ticket-b: requested_action (2/5)ticket-c: reported_at (5/5) | 83.8 | 11.2 | 32.8s | 51.6 | $0.0339 | 0/15 |
| kimi-k2.5@moonshotai72 field missesticket-a: affected_devices (1/5)ticket-a: affected_devices.0.serial (1/5)ticket-a: affected_devices.1.serial (1/5)ticket-a: customer_email (1/5)ticket-a: customer_name (1/5)ticket-a: order_number (1/5)ticket-a: product (1/5)ticket-a: reported_at (5/5)ticket-a: requested_action (1/5)ticket-a: severity (1/5)ticket-a: ticket_id (1/5)ticket-b: affected_devices (4/5)ticket-b: affected_devices.0.serial (4/5)ticket-b: customer_email (4/5)ticket-b: customer_name (4/5)ticket-b: order_number (4/5)ticket-b: product (4/5)ticket-b: reported_at (5/5)ticket-b: requested_action (4/5)ticket-b: severity (4/5)ticket-b: ticket_id (4/5)ticket-c: affected_devices (1/5)ticket-c: affected_devices.0.model (1/5)ticket-c: affected_devices.0.serial (1/5)ticket-c: affected_devices.4.serial (1/5)ticket-c: customer_email (1/5)ticket-c: customer_name (1/5)ticket-c: order_number (1/5)ticket-c: product (1/5)ticket-c: reported_at (5/5)ticket-c: requested_action (1/5)ticket-c: severity (1/5)ticket-c: ticket_id (1/5) | 54.7 | 44.7 | 58.1s | 31.4 | $0.0825 | 0/15 |
Note
- Clean
ticket-extractre-run at 5 trials over 3 cases: 10 targets, 150 results, $0.67, no errors. Supersedes the numbers from2026-07-22-4e1d7ed, a warm-cache run since removed. - Roster change against that run: both pinned
z-aivision targets dropped andz-ai/glm-4.7added unpinned, served by Cerebras on all 15 calls at a clean 100.0. Nothing else moved. - Seven of ten targets are perfect. gpt-5.6-terra is the near-miss at 99.4
(±2.1), from one trial that missed
severityon ticket-c. - qwen3.7-plus 83.8 (±11.2):
reported_atfails on all 15 trials, plus intermittentaffected_devicesserial drops on ticket-a and ticket-b. - kimi-k2.5 54.7 (±44.7):
reported_atfails on every gradable trial too, and 6 of 15 trials returned an unusable shape scoring 0 — 4 of 5 on ticket-b alone. That is what the 44.7 is made of. - The laggards are also the slowest, at 32.8s and 58.1s per call against 1.2–6.5s elsewhere. gemini-3.5-flash-lite is the cheapest and fastest of the perfect group at $0.0084 and 1.2s.
Conclusion
At 5 trials the scenario separates cleanly, and with seven targets tied at 100.0 the choice falls to cost and latency. gemini-3.5-flash-lite wins both: $0.0084 and 1.2s per call, against $0.078 for glm-4.7 and $0.171 for gemini-3.5-flash in the same tied group.
The two laggards fail differently, and that decides what to do about them. qwen3.7-plus is consistently wrong on a single date field, 15 trials out of 15 — the signature of a prompt or fixture problem, worth one attempt at a fix. kimi-k2.5 is unreliable rather than weak: gradable trials score 90–92 and the rest score 0 on shape, so 54.7 describes the mix and not the model. A target that returns an unusable shape on 40% of calls is not fixed by prompt work.