Plate Recognition
About this benchmark
Reads the plate off the vehicle in a 1920x1015 barrier camera still and returns it as one string. One field, so every trial scores 0 or 1 and the mean across four cases and five trials is a plain success rate.
The frames are real captures from a Malaysian multi-storey carpark: sun glare on a bonnet, a plate recessed behind a grille with dealer text under it, a two-line plate on a dirty bumper, and a small plate on an unlit level. Malaysian plates carry no state colour coding or hyphens, so there is no format prior to lean on when the pixels are ambiguous.
Each fixture is cropped along the bottom to drop a burnt-in camera overlay that printed the plate in clean text, so the plate can only be read from the plate. Faces and a passing motorcyclist are pixelated in the bytes the models are sent.
Test data
bmw-grille: BMW iX at a barrier, VJP 9981, with a dealer strip under the digits inside the plate frame.honda-glare: Honda City on a sunlit rooftop level, VJF 5591, the plate shadowed under a hard windscreen flare.kancil-dim: Perodua Kancil on an unlit level, WNH 8994, on a slanted decorative plate face that turns the leading W into an M.proton-twoline: Proton Saga at an entry lane, BLM 8686, on a two-line plate with letters above digits.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| qwen3.7-flash@alibaba | 100.0 | 0.0 | 4.4s | 10.4 | $0.0014 | 0/20 |
| gemini-3.5-flash-lite@google-ai-studio | 100.0 | 0.0 | 1.5s | 11.6 | $0.0080 | 0/20 |
| gpt-5-mini@openai | 100.0 | 0.0 | 3.8s | 32.6 | $0.0172 | 0/20 |
| gemini-3.5-flash@google-ai-studio | 100.0 | 0.0 | 2.5s | 53.8 | $0.0598 | 0/20 |
| gemini-3.6-flash@google-ai-studio | 100.0 | 0.0 | 2.8s | 61.5 | $0.0610 | 0/20 |
| grok-4.5@xai | 100.0 | 0.0 | 2.2s | 7.9 | $0.0854 | 0/20 |
| kimi-k2.6@moonshotai | 100.0 | 0.0 | 14.7s | 31.3 | $0.0870 | 0/20 |
| claude-sonnet-5@anthropic | 100.0 | 0.0 | 3.3s | 5.1 | $0.1201 | 0/20 |
| gpt-5.6-sol@openai | 100.0 | 0.0 | 5.6s | 3.0 | $0.3157 | 0/20 |
| qwen3.7-plus@alibaba1 field missproton-twoline: plate_number (1/5) | 95.0 | 21.8 | 6.1s | 26.8 | $0.0177 | 0/20 |
| gpt-5.6-luna@openai5 field misseskancil-dim: plate_number (5/5) | 75.0 | 43.3 | 2.5s | 14.8 | $0.0328 | 0/20 |
| claude-haiku-4.5@anthropic5 field missesproton-twoline: plate_number (5/5) | 75.0 | 43.3 | 3.1s | 4.5 | $0.0382 | 0/20 |
| gpt-5.6-terra@openai5 field missesproton-twoline: plate_number (5/5) | 75.0 | 43.3 | 2.1s | 9.9 | $0.0795 | 0/20 |
| kimi-k2.5@moonshotai17 field missesbmw-grille: plate_number (4/5)honda-glare: plate_number (5/5)kancil-dim: plate_number (4/5)proton-twoline: plate_number (4/5) | 15.0 | 35.7 | 10.1s | 24.7 | $0.0467 | 0/20 |
Note
- kimi-k2.5 scores 15.0 while reading every plate correctly on all 20 trials.
It ignores
response_formatand answers with the bare plate,VJF5591rather than{"plate_number":"VJF5591"}, so the grader rejects 17 trials outright. Of the three that pass, two are the right JSON buried in several hundred lines of whitespace and one is clean. Read its score as unusable output, not as blindness; the same pinning on kimi-k2.6 honours the schema every time. - The three 75.0 targets miss the same count but not the same way, and the
shared stddev hides it. gpt-5.6-luna returns
MNH8994for kancil-dim on all five trials and gpt-5.6-terra returnsDLM8686for proton-twoline on four of five, so each is wrong at one fixed string and retrying buys nothing. claude-haiku-4.5 fails the same case with four different answers across five trials,PLM85B5,PLM9596,PLM9595,PLM8555andPLM9595against a trueBLM8686. Its leading B reads as P every time while the digits scatter, which is an unstable read rather than a fixed one. - grok-4.5 served cached prompt tokens despite
cache_bust: true, the same behaviour it showed on2026-07-23-2664bd8. No other target reported any, so its latency and cost are the only two in the table that are flattered.
Conclusion
Nine targets tie at 100.0, so cost and latency decide, and they no longer point the same way. qwen3.7-flash is the cheapest correct reader by a wide margin at $0.0014 across its 20 calls, roughly a sixth of gemini-3.5-flash-lite at $0.0080, but it takes 4394ms against flash-lite's 1533ms. At a barrier the car is waiting, so flash-lite still takes it on the three seconds; for batch work off a recorded feed, where nothing waits, qwen3.7-flash is the better buy. Nothing else in the tie is competitive on price: claude-sonnet-5 is correct and mid-pack on latency at 3337ms but costs $0.1201, fifteen times flash-lite, kimi-k2.6 is slowest in the run at 14700ms, and gpt-5.6-sol costs $0.3157. Read grok-4.5's $0.0854 and 2204ms as a floor rather than a measurement.
The failures split three ways, and only one of them is about seeing. luna and
terra each return one identical wrong plate on every trial of the single case
they cannot read, so their 75.0 describes coverage and not reliability: three
captures they always get, one they never do. haiku-4.5 reaches the same 75.0
from a worse place, because its wrong answers move between trials, which means
a majority vote over repeated calls would not converge on the right plate
either. qwen3.7-plus is the mildest version of the same family, one trial
reading BLM8B86 where the other four are right, a genuine flake rather than a
blind spot. kimi-k2.5 is the different case: its problem is the envelope, not
the plate, and it is the only target here that a schema-repair step would
rescue.