The Demo Works. Production Is the Benchmark.

We tested frontier models and document-AI engines on 1,000+ real documents, measuring the capabilities required for straight-through processing. anyformat tops all four studies, and is the only system that pairs frontier-level accuracy with visual citations and calibrated confidence.

What's your accuracy rate?

It's the first question every document-AI buyer asks. But as I argued in Beyond Accuracy, it's rarely the one that decides whether a system works in production.

A clean demo document is not the hard part. Production is where documents become longer, layouts change, tables span dozens of pages, and every failed extraction creates another manual review.

The real test is straight-through processing: whether a document can go from input to validated output without someone fixing it in the middle.

So we ran the benchmark everyone wanted to see: frontier models from several providers and the main dedicated OCR and document-AI engines, tested on 1,000+ real documents.

Not clean examples. Not single-page PDFs. Real documents with the failure modes that break automation.

Four studies: parsing quality, long document extraction, complex layouts, and confidence calibration.

Here's what we found.

What we tested

Four studies, one principle: real documents, production scoring.

Parsing. PDF-to-Markdown quality on 1,000+ real documents across 30+ document types, evaluated on content extraction, tables and document structure.

Long documents. Real documents of up to 50 pages (around 2,400 rows), measuring how many rows were correctly recovered.

Complex layouts. Nearly 100 real, mixed-format invoices, scored by full document completeness: every field and every line item captured.

Confidence. Whether the confidence attached to each extracted value is calibrated. When we say 90% confident, are we right about 90% of the time?

Parsing: frontier-level accuracy, with production infrastructure

Measuring document parsing quality is complex. A good parser needs to preserve content, reconstruct tables, and maintain the structure that gives the document meaning. Each dimension requires a different way of measuring quality, which is why academic efforts like OmniDocBench evaluate parsing as an end-to-end pipeline rather than a single metric.

Our Parse Score follows the same principle. It combines these dimensions into a single benchmark by averaging specialized evaluation metrics for text, tables, and document structure. We tested it across 1,000+ real-world documents against a carefully curated ground truth.

On this combined score, anyformat reaches state-of-the-art accuracy at a fraction of the cost of frontier models, and stays ahead of every dedicated OCR engine we tested. Here is the full leaderboard:

Accuracy is only half the picture. The other half is what that accuracy costs to run:

Up and to the left is where you want to be: frontier-level accuracy at $25 per 1,000 pages, against $102 for GPT-5.5. Self-hosted engines (Docling, PaddleOCR) excluded

Built on open data, given back to the community

Before the full methodology write-up, a note on where the numbers come from. The parsing benchmark builds on an open-source, MIT-licensed sample of real PDFs. The original dataset shipped its ground truth only as JSON output, which works for extraction but can't judge parsing: it doesn't tell you whether a table survived, whether the reading order held, or whether the document's structure was preserved.

So we rebuilt it by hand. For every document, we manually created the ground truth in Markdown with inline HTML, the format that has become the industry standard for evaluating document parsing. We'll publish the improved dataset and the full scoring methodology in the coming weeks, returning the benchmark to the community it came from, so every number in this post can be checked and reproduced.

The limitation of vanilla frontier models is not just the quality of the output, though. They return a prediction, but they don't provide the infrastructure needed to operate document workflows in production.

That is what anyformat adds on top of the model: visual citations that link every extracted value back to its exact location in the document, calibrated confidence signals that tell you what can be automated, validation, and the workflow layer required to process documents reliably at scale. The harness is the product.

Turning a model output into a system you can trust and operate.

Extraction: where documents actually break

Extraction is where production exposes the edge cases. The challenge is not extracting a clean, short document once. It is maintaining quality when documents become longer, more complex, and less predictable.

In production, parsing rarely stands alone. Parse → extract is the most common workflow our customers run: parsing turns the document into faithful Markdown, and extraction turns that Markdown into typed, JSON-schema-validated fields. Normalizing data into an ERP, a TMS or a database requires typed outputs; no downstream system ingests prose.

The parse → extract pipeline in anyformat Studio. Parsing produces faithful Markdown; extraction turns it into schema-typed fields

Two failure modes appear consistently in production: documents that grow in length, and documents with complex structures. We measured both.

Long documents: when outputs start to decay

The document keeps going. Most extraction systems don't.

A short table is easy. The challenge starts when that table grows to dozens of pages and thousands of rows. Rows disappear, headers split, and the output can look correct until someone realizes information is missing. This degradation is well documented in the research literature: language models lose track of information in long contexts, even when the input is perfectly readable.

Our row-scaling benchmark tested exactly that: a single line-item table growing from 1 to 50 pages, reaching around 2,400 rows.

Frontier models degraded as documents grew, eventually recovering almost none of the rows. anyformat stayed at ~99% row recovery, keeping the output complete even as the document scaled.

GPT-5.5 and GPT-4.1 collapse first, Gemini 3.5 Flash by page 20, Claude Opus 4.8 by page 50. The blue line is the one your operations depend on

We've written before about why long documents break extraction systems and the parse-extract architecture behind this result.

Complex layouts: complete documents, not partial results

The second failure mode is completeness.

Real invoices, delivery notes and statements contain repeating entities, nested structures, changing layouts and tables that vary from supplier to supplier. Getting most of the document right is not enough for straight-through processing. The extraction has to be complete.

So we measured the strict version: a document only passed if every field and every line item was captured.

Across our complex-layout benchmark, anyformat extracted 94% of documents perfectly.

On the hardest documents, anyformat reached 83% perfect extractions, almost 2× the next best system, which stopped at 43%.

Percent of documents extracted perfectly, by difficulty tier:

Notice the pattern: everyone is near-perfect on the easy tier. The demo works for everybody. It's the hard tier, the invoices with nested line items and layouts that change from supplier to supplier, where the field drops off a cliff.

These are the cases that decide whether automation runs end to end, or whether someone is still reviewing every document manually.

Confidence: a number you can act on

Every value anyformat returns carries a calibrated confidence score. Calibrated is the important part: a confidence score is only useful if it reflects reality.

When we say we are 90% confident, the extraction should be correct around 90% of the time. Across the benchmark, anyformat reached 99.1% calibration accuracy (Adaptive ECE 0.009).

That means teams can set thresholds, automatically approve confident extractions, and send only uncertain values to review, with visual citations attached. We've documented how we score and calibrate confidence in structured data in detail, building on the research around calibration for language models.

What's next

The benchmark results are now live on our homepage.

Explore the four studies, compare the models, and see where document AI still breaks.

Over the next few weeks we'll publish deeper dives into each benchmark: the datasets, scoring methodology and failure modes behind the results.

If you want to see where your own documents land, start with the hardest ones you have.

Plug-and-play document infrastructure you can trust.

Frequently asked questions

Related reading: Beyond Accuracy: The Document AI Metrics That Actually Predict Production Success · Long Documents Are the Production Case · Making AI Data Extractions Trustworthy

See how anyformat compares: vs ChatGPT, Claude & Gemini · vs Azure Document Intelligence · vs AWS Textract · All comparisons →