Report Extract

2026-07-23 03:19 UTC · 2664bd8

About this benchmark

Extracts the header facts of a real equity research report (publisher, date, report type, subject, ticker, recommendation, currency, share and target price, previous target, shares issued, market cap), plus three lists: the credited analysts, the major shareholders, and any peer or sector comparison table.

These are dense multi-column PDFs with tables, disclaimers and footers, and the facts are scattered across pages rather than sitting on page one. The hard part is not reading but discipline: not attributing a peer company’s rating or target price to the subject, crediting only the report’s own authors, and leaving fields null when the document simply does not state them. Fields are weighted, with target price counting heaviest.

Test data

dialog-uobkh: UOB Kay Hian company report on Dialog (DLG MK), 4 pages, BUY with a MYR 2.70 target against a MYR 2.29 share price. No previous target stated.source document withheld
greatec-maybank: Maybank company report on Greatech, 10 pages, the longest document in the suite. BUY, with the target raised from MYR 2.40 to MYR 3.66, so both the new and the superseded target must be picked up.source document withheld
plantation-mbsb: MBSB sector report on Plantation, 5 pages, POSITIVE. The null-discipline case: a sector report has no ticker, no share price and no target price, but its comparison table is full of per-company ratings and targets waiting to be mistaken for the subject’s.source document withheld

Results

8/10 targets scored · 9 trials each · $1.87 total

Cost vs latencylower-left is better

Latency & costshorter is better

Target	Score ↓	Stddev	Latency	Tok/s	Cost	Errors
gemini-3.6-flash@google-ai-studio	100.0	0.0	7s	177.4	$0.1300	0/9
gemini-3.5-flash@google-ai-studio	100.0	0.0	11.5s	126.9	$0.1651	0/9
gpt-5.6-terra@openai	100.0	0.0	4.5s	50.6	$0.6196	0/9
grok-4.5@xai3 field misses	99.0	1.5	9.6s	108.2	$0.3392	0/9
gemini-3.5-flash-lite@google-ai-studio5 field misses	98.3	1.5	2.7s	105.3	$0.0157	0/9
qwen3.7-plus@alibaba6 field misses	96.9	3.6	39.6s	44.8	$0.1619	0/9
gpt-5.6-luna@openai9 field misses	96.4	10.2	5.8s	56.7	$0.2532	0/9
claude-haiku-4.5@anthropic41 field misses	80.4	28.2	5.4s	42.3	$0.1808	0/9
glm-4.6v@z-aierror	n/a	n/a	n/a	n/a	$0.0000	9/9
glm-5v-turbo@z-aierror	n/a	n/a	n/a	n/a	$0.0000	9/9

Note

report-extract re-run at 3 trials over 3 cases: 10 targets, 90 results, $1.87. First run with cache busting on (cache_bust: true in the config), so every trial pays cold-cache prices. Replaces the report-extract numbers from 2026-07-22-4e1d7ed, a warm-cache run since removed.
Roster change against that run: kimi-k2.6 dropped. Nothing else moved.
The busting shows in the money, not the scores: every target reports 0 cached prompt tokens except grok-4.5 (55,040, of which 54,272 land on a single plantation-mbsb trial; xAI's cache is evidently not prefix-keyed). gpt-5.6-terra's nine perfect calls now cost $0.620 against $0.289 in the warm run; the old figure was cache-flattered.
Both pinned @z-ai targets still fail all 9 attempts with 404 No endpoints found, unchanged since the last run. Dead weight until dropped or routed unpinned.
Three targets are perfect: gemini-3.5-flash, gemini-3.6-flash and gpt-5.6-terra at 100.0. grok-4.5 99.0 (±1.5) misses major_shareholders.0.name on two dialog-uobkh trials plus analysts once; gemini-3.5-flash-lite 98.3 (±1.6) misses only currency, 5 of 9 trials.
qwen3.7-plus 96.9 (±3.6), up from 86.3: no block collapse this run, just currency on all three dialog-uobkh trials and two scattered singles, at 39.6s per call against 2.7s to 11.5s for everything above it. gpt-5.6-luna 96.4 (±10.2) is perfect except one plantation-mbsb trial at 68 that dropped the peer_ratings block.
claude-haiku-4.5 80.4 (±28.2) is the laggard: 62/32/32 on plantation-mbsb, losing the whole peer_ratings block on all three trials plus recommendation and target_price on two.

Conclusion

The ranking survives cold caches intact, which settles that the earlier numbers were honest on accuracy and flattered on cost. gemini-3.5-flash-lite stays the pick at 98.3 for $0.0157 and 2.7s per call, an eighth of the $0.130 that gemini-3.6-flash, the cheapest perfect target, now costs. Its one defect is still currency, though at 5 of 9 trials it has gone intermittent, which weakens the fixture-problem reading; terra's true price is $0.620, not the $0.289 the warm run suggested.

The failures left in the table split two ways. haiku's 80.4 averages clean cases against a plantation-mbsb where the peer_ratings block vanishes every trial. The mean hides a case it cannot do, so it is unsafe for these documents rather than merely behind. qwen's jump to 96.9 with no block collapse suggests its old 86.3 mixed in the same instability; one more run would say which number is the model. The pinned @z-ai targets remain unable to serve the scenario at all.