The reference gap: visual grounding in a production document AI pipeline

Why this study exists

Most writing about document AI argues over recognition accuracy — can the model read the characters. That fight is largely won: modern vision models transcribe even hard scripts well. The problem that actually stalls a production pipeline is quieter and rarely measured. A model can read a page correctly and still be unable to point at what it read — to bind a specific claim back to the exact place on the page it came from. DeepSeek named this the reference gap in its Thinking with Visual Primitives work (a paper it briefly published and then pulled): a model can see a detail perfectly yet cannot reliably refer back to it when it reasons in words.

We hit that gap in production, in its purest form, building a pipeline that turns a scanned book into a faithful translated document. This paper reports what the gap cost us on a real ~700-page text, how we engineered around it, and what happened when we tested DeepSeek's proposed fix — emitting coordinates — with a cheap, off-the-shelf model instead of a trained one. It is a single operator reporting its own system, so read the limitations before you generalize. The numbers are specific, measured, and, we think, useful to anyone buying or building document AI.

Background: where this sits

Four lines of work bracket this problem, and none of them solve the specific case here — cheap, off-the-shelf, in-line marker-to-exact-line anchoring on right-to-left, diacritized text.

• Grounding as model output. DeepSeek's visual primitives and Allen AI's Molmo and PixMo both teach a model to emit points or boxes as grounded output. Their route is training — a custom encoder, a pointing dataset. Our question is the opposite: can a model that was not trained to point still do it well enough to be useful?
• Document OCR to structure. DeepSeek-OCR, Meta's Nougat, and Allen AI's olmOCR convert a page into structured text, including footnotes. They produce clean text; they do not solve robust in-line marker-to-line anchoring for RTL diacritized script.
• Reference and footnote extraction. GROBID extracts reference and footnote callouts from scholarly PDFs with trained sequence models (~.87–.90 F1 on references). That is the established approach for Latin scholarly documents; ours is the Arabic, cheap-model-in-the-loop analog. The geometry substrate underneath both is the hOCR line-box format.
• Arabic and RTL difficulty. The recent KITAB-Bench benchmark shows widely used engines (Surya, Tesseract, PaddleOCR) lack Arabic-specific diacritic and segmentation handling. Right-to-left flow with left-to-right numerals, optional diacritics, four positional letter forms, and mandatory ligatures make the script unusually hard — which sharpens the reference gap rather than causing it.

The pipeline and the gap

The system converts a scanned PDF of a book into a page-faithful translated document. It renders each page to an image, runs several OCR engines and reconciles them into page-aware layout — headers, body, and footnotes kept as separate blocks — protects quoted passages, translates the body while leaving the protected spans intact, and renders a document with real footnotes. The book here is a ~700-page diacritized classical Arabic religious text with heavily quoted passages; we keep the specific work anonymized and report only aggregates.

The footnote subsystem is where the reference gap lives. The OCR reliably reads the footnote text at the bottom of the page. What it routinely drops is the tiny superscript marker — a parenthesized digit like (3) — sitting inside the running text. So the note exists, but the spot in the body it attaches to (its anchor) is missing. And by deliberate design, the vision model is not allowed to answer with a position: it must read the text back and name the exact Arabic word next to each marker, never a coordinate. That rule earns its keep — feeding the model our earlier guesses only made it confidently agree with wrong ones — so the prompt asks, in effect, "read this back and list the markers you can see." The pipeline then has to find that word again by fuzzy text-matching against the lines it read.

That round-trip is the reference gap made concrete: the model reports what (an Arabic word) but never where (a spot on the page), and the search that has to re-find the spot fails often. Of 4,667 footnotes across the 646 pages that carry them, the first automated pass placed only 2,339 with confidence; 1,657 needed review and 682 could not be placed at all. Put plainly, roughly half the footnotes had no reliable spot to attach to after that pass. Everything downstream exists to close that half without disturbing the quoted passages.

The grounding tax: a 13-module cascade

Closing the gap took a cascade of thirteen modules and about 7,300 lines of code. Each stage is cheap on its own; together they are the price of a model that reports what but not where. This is the grounding tax. The main stages, and how far each moved the number:

Stage	What it does	Real numbers
AI review	Text-context pass that auto-clears the easy anchors before spending vision calls	2,339 rows reviewed, $2.44 (gpt-5.4-mini)
Visual review	Vision pass over page crops for anchors text review could not settle	288 pages / 545 rows (~$1.18)
Snap-repair	Local, no API call: reuses an earlier visual decision, snapping a near-miss to one safe line	369 rows, 43 accepted / 326 held for review (~88%)
Marker inventory	Reframed prompt: crop a narrow strip of the page, read it back, list every marker seen	3,630 page-strip crops across 198 pages
Line-index repair	Fixes anchors pointing at the right text but the wrong line	4,339 anchors scanned, 227 repaired
Translated-anchor projection	Projects each source anchor onto an exact English substring so the renderer can place it	drove document warnings 725 → 289 → 70 → 0

The single most useful move in the cascade was not a bigger model; it was a reframed prompt. Switching the vision task from "find marker X" — which invited the model to confidently hallucinate a marker that was not there — to "transcribe this band and report every marker you can see, with the Arabic word beside it" made the model far more honest, because it removed the leading guess. That change, plus the strict no-coordinates contract, is the load-bearing methods insight here.

It worked. The English-side placement funnel closed all the way: 725 unplaceable footnotes fell to 289, then 70, then 0, and the finished document carried about 4,332 correctly placed footnotes with none blocked. But look at what that cost: thirteen modules, ~7,300 lines, and a human confirming residual cases by hand. Almost none of that code would exist if the model could have handed back a coordinate in the first place.

The grounding experiment: does a cheap point close the gap?

DeepSeek's fix for the reference gap is deceptively simple. Instead of letting the model describe a location in words — "the marker is near the top, on the right" — it teaches the model to answer with an actual point: a pair of coordinates on a fixed grid laid over the image, numbered 0 to 999 across and down. The model literally points at what it means, and holds onto that point as it reasons. A word can be vague; a point cannot.

Their version of it takes a specially trained 284-billion-parameter model, which is out of reach for a small pipeline. So we tested the cheap version. We added one line to the existing prompt: alongside the Arabic word, also return a point for the marker on that 0-to-999 grid. Because the grid is relative to the image, the point still works after the crop is resized. Then plain geometry does the rest — take the point, and pick the line of text it lands on, using the line boxes the OCR had already produced. One model call now yields both a word and a point, so we could compare word-matching against point-geometry on the exact same output, on markers whose correct answer we already knew. The model is the same cheap one already in production, gpt-5.4-mini. The whole experiment cost about $0.30.

The result is honest rather than triumphant, which is why we trust it.

• The mechanism works. The model did not just stamp the same spot every time, and when it gave a usable point, that point landed on the correct line about 87.5% of the time — the same rate on two separate samples.
• It is safe once you guard the edges. The one dangerous failure — a point sitting on the boundary between two lines, quietly assigned to the wrong one — is fixed by a rule that flags the case for review instead of guessing. After that guard there were zero wrong-line errors. A footnote sent for review is a nuisance; a footnote silently attached to the wrong line is a defect, and the design refuses the second.
• It beats word-matching, cleanly. On a full same-sample comparison of 24 labeled markers, the point path scored 8 correct, 0 wrong, 16 flagged against word-matching's 7 correct, 0 wrong, 17 flagged — one more placed, one fewer sent to a human, and not a single regression: on no marker was the point path worse. Every win was the same kind of case: the coordinate held up even when the model's written-out word failed to match.

One honest correction, because it is the discipline that makes the rest trustworthy: an earlier, looser run looked like +2. When we insisted that every accepted footnote also survive the later document-building steps, one of those two wins dropped out — it would not have held up downstream. The real, safe margin is +1. And it does not grow with scale: measured on 7, then 13, then 24 markers, the relative gain shrank from +14% to +8% to +4%. The point path rescues a small, roughly fixed set of hard cases; it does not multiply.

The experiment surfaced a way to wire this in, which we call point-first validation. Today the pipeline decides a footnote's home from whether the model's written-out word matches. Point-first reverses the order: trust the point first — accept the marker when it has a good point, place it on the line the point lands on, flag the case for review when the point sits on a boundary, and keep the word only as a backup check. Across every marker we tested it never did worse than word-matching and sometimes did better, which is what makes it worth adopting.

The ceiling is detection, not localization

Here is the finding that matters most, and the one most likely to be misread as a disappointment. Coordinate grounding does not solve the pipeline's hardest failures, because those failures are not about localization at all. About 40% of the hard markers emitted no usable point — not a wrong point, no point — because the model never detected the marker on the crop in the first place. Geometry can only compete with text once there is something to map; when the model does not see the marker, there is nothing to place, and the whole coordinate idea has no purchase.

Asking for a point did nudge detection up on one sample (the model reported markers it had otherwise skipped), but it also lowered the reliability of the word the model wrote out — a real interaction, not a free lunch. The clean read is a split: localization is bridgeable for cents with an off-the-shelf model; detection is not. That is precisely the boundary DeepSeek's own thesis draws. Closing the reference gap where the model already sees the entity is cheap; teaching it to see the entity reliably is what needs a trained model. Our result is the empirical version of their argument, measured on a real, full-length document rather than a benchmark.

What this means if you are buying or building document AI

Three things follow directly from the numbers, and all of them are about the system around the model rather than the model itself.

• Recognition accuracy is the wrong question. A vendor demo that reads your document flawlessly has answered "can it see the words," which is the easy part. The load-bearing question is whether it can bind what it read back to the right place — anchor a footnote, tie a total to its line item, attach a clause to its section. Ask to see the reference gap handled, not the transcription.
• Budget for the grounding tax. On real documents the placement and reconciliation code dwarfs the model call. If a plan prices only the OCR and not the cascade that makes its output trustworthy, it is pricing the easy 20%.
• Cheap grounding is worth trying before training. Adding a point to the prompt is essentially a one-line change and about +12% cost, and on our data it recovered placements that word-matching missed, with no new errors. Reach for a trained or larger model only after you have hit the detection ceiling that cheap grounding cannot cross.

Across everything we tested, the point path never did worse than word-matching, never once placed a footnote on the wrong line, and cost almost nothing extra. On that evidence, turning it on by default is defensible — after a single dry run over a whole book to catch any last edge cases, because each time we widened the test it found one. The size of the win is not the point. The point is what a disciplined, cheap experiment told us about where the effort should go: not into a cleverer prompt, and not yet into a bigger model, but into detection — the one place with real headroom left. That is the kind of answer a benchmark score or a vendor demo will never hand you, and it is why we ran the pipeline in the first place and measured what it actually did.

Limitations

This is a single operator studying its own system, and it should be read that way. The pipeline is ideius's own R&D, run on one book — a single text, a single domain — not a controlled benchmark; the source work is anonymized and only aggregate counts and reason-codes are reported, never the text itself. The grounding experiment is small: two dozen labeled markers across six pages, on the cheap model we already run. The win is a single marker, which is within noise at this size, and every time we widened the test it exposed a new edge case — so a full-book dry run, not this sample, is the real test before turning it on by default. We recorded token and dollar cost but not wall-clock speed. The ~50% figure and the funnel numbers come from one production run; treat any single figure here as directional until other operators publish their own. And detection — the dominant ceiling — is an upstream recognition problem this study measures but does not solve.

How to cite this

Ahmed, F. (2026). The reference gap: visual grounding in a production document AI pipeline. ideius. https://www.ideius.com/papers/document-ai-reference-gap/

The figures are derived from aggregate telemetry; no source text is published. For the accessible version of this argument, read the companion article on what document AI still gets wrong. If you are evaluating or building document AI and want the reference gap pressure-tested before you commit, that is where our AI technical due diligence and LLM/RAG evaluation work starts.