AI Document Extraction Accuracy: Citation Checks

AI document extraction accuracy improves when the model must prove each value. Require a verbatim quote and a page number with every field, check in code that the quote really appears on that page, and check the value against its quote. Drop whatever fails, re-check the risky fields, and send disagreements to a person instead of guessing.

TL;DR: Change the output contract to value, quote, and page. Normalise the page text and the quote the same way, then search for one inside the other. Parse the value back out of the quote. Allow fuzzy matching only on OCR pages. Anything that fails is dropped and shown to a reviewer with the reason, so a guessed number never reaches a spreadsheet.

This post covers the check as a general method you can add to any extraction job. SigmaCPM, our construction platform in early access, uses it on tender PDFs; the full tender pipeline, from OCR to the review queue, is in reading tenders with AI.

Why can't you trust plain extraction?

Because a wrong value looks exactly like a right one. Ask a model to "extract the bond amount and completion date" and it returns two plausible values. Sometimes they're right. When they're wrong, nothing flags it: the model doesn't mark its own invented number, and a confident mistake in a bid or an invoice costs real money.

Telling the model to "only use the document" helps a little. It doesn't give you proof. Proof comes from making the model say where each value came from, and then checking that claim with code that can't hallucinate. SigmaCPM's rule is short: every extracted value must come with a quote, the quote must be found on the cited page, and if it isn't there the value is dropped.

What should the model return for each field?

Every field carries its own evidence:

Field	Meaning
value	The extracted value, normalised (number, date, name)
quote	The exact text in the document that states it
page	The PDF page index the quote is on
confidence	Optional; useful for sorting review, never for trust

Ask for structured output so you can parse responses reliably. OpenAI's Structured Outputs, for example, "ensures the model will always generate responses that adhere to your supplied JSON Schema" (OpenAI docs). A schema guarantees shape, not truth, though. A perfectly valid JSON object can still hold an invented quote, which is why the next step exists.

How does the quote check work in code?

It's a string search on text you already have. You extracted each page's text earlier, from the PDF's text layer or from OCR (Apache PDFBox, which SigmaCPM uses for PDF work, lists Unicode text extraction among its features: PDFBox). Normalise both sides the same way, then look for the quote inside the page:

import re
import unicodedata

QUOTES = {"\u2018": "'", "\u2019": "'", "\u201c": '"', "\u201d": '"'}
DASHES = {"\u2013": "-", "\u2014": "-", "\u2212": "-"}

def normalise(text: str) -> str:
    text = unicodedata.normalize("NFKC", text)        # ligatures, full-width forms
    for src, dst in {**QUOTES, **DASHES}.items():
        text = text.replace(src, dst)
    text = re.sub(r"-\s*\n\s*", "", text)              # hyphenation at line breaks
    text = re.sub(r"\s+", " ", text)                    # collapse all whitespace
    return text.strip().lower()

def quote_on_page(quote: str, pages: list[str], page: int) -> bool:
    if not 0 <= page < len(pages):
        return False
    return normalise(quote) in normalise(pages[page])

Why NFKC? Python's unicodedata docs explain that the compatibility forms "replace all compatibility characters with their equivalents" (Python docs), and the Unicode standard's own example is that the "fi" ligature becomes "f" and "i" (Unicode UAX #15). PDFs are full of ligatures, so without this step a correct quote of "certificate" can fail to match.

Does the value actually match its quote?

A model can quote the right sentence and still return the wrong number from it. So parse the value back out of the quote. If the quote says "Rs. 12,50,000" and the value is 1250000, a small parser for Indian digit grouping confirms it. If the value is a date, parse the date in the quote and compare. A mismatch is a failure, even when the quote itself checks out.

Where does exact matching break on real documents?

Real documents fight back. Plan for these cases before the first run:

1. OCR noise. Scanned pages confuse O and 0, l and 1. Allow a small, character-aware fuzzy match on OCR pages only, and mark those values for closer review. Never loosen matching on pages with a real text layer.
2. Tables. A value in a table has no sentence around it. Ask for the row label and column header as the quote, and check that both appear on the page.
3. Text across a page break. A clause can start on one page and end on the next. Let a quote span two consecutive pages, and record both.
4. Printed page numbers. The number printed on a page is often not the PDF's page index. Store the index internally and show both to reviewers.
5. Repeated boilerplate. The same sentence can sit on many pages. Verification still passes, but show the reviewer the exact page cited.

Is fuzzy matching a back door for invented quotes? It can be, which is why it stays narrow: limited to OCR pages, limited to look-alike characters, and visible in the review screen.

SigmaCPM pulls line items and bond terms out of tender PDFs. Line items in a tender are usually laid out as a schedule, so for that kind of field the table rule matters more than any other: a sentence-style quote simply doesn't exist there to be found.

When is a second pass worth the cost?

Only on fields where a mistake is expensive. A second pass doubles model time for those fields, so spending it on low-risk fields buys little. In SigmaCPM, bond terms are re-checked by a second, adversarial model run, and when the two runs disagree the field lands in a disputed state for a person to resolve.

A second pass works best when it's set up to disagree:

1. Give it the first value and quote and ask it to find evidence that the value is wrong or incomplete.
2. Or run an independent extraction with a different prompt or model, and compare.
3. Treat agreement as "ready for review", never as "certainly true".

What happens to values that fail the check?

They leave the automatic output and go to a person, with the reason attached. Every field ends in one of three states:

State	Meaning	Who acts
Verified	Quote found on the cited page, value matches, no disagreement	Reviewer spot-checks
Dropped	No verifiable quote, or value doesn't match its quote	Reviewer fills it in
Disputed	Two passes disagree	Reviewer picks a reading

The reviewer's screen should put each value next to a preview of the cited page with the quote highlighted, list dropped values with their reason, and float disputed values to the top. Corrections are worth saving: each one is a ready-made test case for a RAG evaluation set or an extraction regression suite.

How do you measure accuracy once checks are in place?

Count states, not vibes. Per document type and per field, track how many values verify, how many drop, how many end disputed, and how many verified values a reviewer later corrected. That last number is the honest one: it tells you how often the check passed something wrong.

Watch for changes, not absolute levels. A jump in dropped values after a new batch usually means a new layout or an OCR problem, not a worse model. A rise in corrected-but-verified values points at the value parser or a fuzzy rule that's too loose.

Where else does this method fit?

Anywhere a wrong value is expensive and the source is a document someone can open. Invoices and purchase orders feeding accounting. Contracts, for notice periods, renewal dates, and liability caps. Insurance and compliance forms. Lab reports and certificates. The contract stays the same: value, quote, page, then a check in code.

Would training a model on your documents remove the need for checks? No. A fine-tuned model can still produce a confident wrong number, and it still can't show you where the number came from; our RAG vs fine-tuning page explains why grounding beats training for document facts. The general version of this setup is on our OCR data extraction page.

FAQ

Does citation checking work with any model provider?

Yes. The check runs in your own code after the model responds, so it doesn't depend on the provider. What you need from the model is structured output with a quote and a page per field. Switching models later changes how often values verify, which your counts will show immediately.

What if a document has no text layer at all?

Run OCR first and store text per page, keeping the page index. Then the same check works, with fuzzy matching allowed only for look-alike characters on those OCR pages. Expect more dropped values on poor scans; that's the check doing its job, not failing.

Can the model cite several pages for one value?

Allow it only where a clause genuinely runs across a page break, and only for consecutive pages. For anything else, ask for one page per quote. Wider citations make verification weaker, because a short quote is more likely to appear by coincidence somewhere in a long span.

Does verification slow extraction down?

Barely. Normalising text and searching for a substring takes far less time than the model call that produced the quote. The cost that matters is the optional second pass, which is why it should be reserved for the few fields where a wrong value is costly.

Extracting values from PDFs, scans, or tenders and need to trust them? See our AI document processing work, and test the approach on a sample of your own documents first.