Receipt Parse

2026-07-23 04:27 UTC · 6bc8930

About this benchmark

Reads a short, clean PDF receipt and fills a fixed JSON schema: invoice and receipt numbers, date paid, vendor, bill-to name and email, currency, subtotal, discount, total, amount paid, payment method and the line-item table. The model is told to use only values printed on the page.

This is the floor of the document suite: it asks nothing of long-context reading, only whether a model can hold a whole schema at once without dropping or inventing a field. Grading is per field: exact match on identifiers and dates, case-insensitive containment on names, a numeric tolerance on money, and absolute value on the discount so a sign convention does not decide the score.

Test data

receipt-a: One synthetic single-page receipt from Northwind Labs billed to Jane Doe, in USD. A 50% discount line takes a $22.00 subtotal down to an $11.00 total, so a model that reads the discount as positive, or reports the subtotal as the total, is caught.

Results

10/10 targets scored · 3 trials each · $0.15 total

Cost vs latencylower-left is better

Latency & costshorter is better

Target	Score ↓	Stddev	Latency	Tok/s	Cost	Errors
gemini-3.5-flash-lite@google-ai-studio	100.0	0.0	2.5s	94.3	$0.0023	0/3
gpt-5.6-luna@openai	100.0	0.0	4.1s	48.2	$0.0050	0/3
gpt-5.6-terra@openai	100.0	0.0	3s	46.7	$0.0099	0/3
claude-haiku-4.5@anthropic	100.0	0.0	3.8s	52.1	$0.0105	0/3
qwen3.7-plus@alibaba	100.0	0.0	35.8s	51.8	$0.0135	0/3
grok-4.5@xai	100.0	0.0	7.4s	100.7	$0.0256	0/3
gemini-3.6-flash@google-ai-studio	100.0	0.0	6.5s	183.7	$0.0293	0/3
gemini-3.5-flash@google-ai-studio	100.0	0.0	6.1s	182.8	$0.0324	0/3
glm-4.6v@z-ai40 field misses	16.7	11.8	15.6s	58.8	$0.0088	0/3
glm-5v-turbo@z-ai48 field misses	0.0	0.0	8.5s	76.9	$0.0151	0/3

Note

Running the @z-ai targets at all required dropping OpenRouter's require_parameters filter (commit 6bc8930); it keys on structured_outputs, which no GLM endpoint advertises, so both previously hard-failed before the call.
The two GLM scores in this file are post-processed, unlike every other run here: their stored raw_output was re-graded after stripping the ``` fence that all 6 GLM trials wrap the JSON in, and the result written back. The shipped grader does no stripping, so a re-run at this commit reproduces 0.0 for both.
That post-processing recovers almost nothing: glm-4.6v reaches 16.7 (±11.8), which is 25.0, 25.0 and 0.0 across its trials, and glm-5v-turbo stays 0.0 because its output is not valid JSON even unfenced, carrying a literal newline inside the vendor address. The fence was the trigger, not the cause.

Conclusion

Eight targets tie on score, so cost and latency decide, and gemini-3.5-flash-lite wins both at $0.0023 and 2.5s, a fourteenth of gemini-3.5-flash's $0.0324 for the same score. One case is not enough to separate the eight; treat the tie as "all clear this bar" rather than a ranking.

The GLM results are unsafe rather than fixable. glm-4.6v extracts the right values and puts them in a shape nobody asked for: money as "$11.00", dates as "June 3, 2026", bill_to.name instead of bill_to_name, items instead of a line_items array. Only the 4 plain-string fields ever match, and trial 3 nests everything under receipt and scores nothing. The pinned Z.AI endpoint serves no response_format, so nothing constrains the output. Repinning glm-4.6v to @novita, the only endpoint of either model advertising it, is what a real GLM number here would need.