AI Tender Document Extraction: How the Pipeline Works

AI tender document extraction that estimators can trust runs in six stages: take in the files, get text from every page (OCR for scans, with retries), pick the pages likely to hold each field, extract values that quote their cited page, re-check the risky fields, and send anything uncited or disputed to a person. Each stage is a queued job.

TL;DR: Don't send a whole tender package to a model and hope. Split it into pages, OCR the scans, send only candidate pages per field, demand a quote and page number for every value, drop what can't be verified, and give an estimator the leftovers. That's how SigmaCPM reads tenders, and it's the shape we'd build again.

This is the pipeline behind tender intake in SigmaCPM, our construction project management platform for civil contractors, which is in early access. This post owns the end-to-end flow. The quote check itself is covered in depth in citation checking for AI extraction, and the job queue in Postgres as a job queue.

Why is AI tender document extraction harder than it looks?

Tenders are long, inconsistent, and full of numbers that go straight into a bid. One package can mix typed pages, scanned annexures, line-item tables, and boilerplate repeated across sections. The values an estimator needs (quantities, rates, bond amounts, terms) are scattered, and a single wrong one costs real money.

That last part shapes everything. A misread quantity or an invented bond amount doesn't stay in a test report; it ends up in a submitted bid. So SigmaCPM is built on one rule: an AI that guesses a number in a bid is worse than no AI at all. If a value can't prove where it came from, it doesn't get through, and code, not the model, has the last word before a person does.

What does the pipeline look like end to end?

Six stages, each with one job and one main way to fail. The failure column tells you what to log.

Stage	Input	Output	Fails when
1. Intake	Uploaded tender files	Stored files, page count, jobs created	A file is huge or corrupt
2. Text and OCR	Each page	Page text, from the text layer or OCR	A scan is faint, skewed, or the OCR call errors
3. Page targeting	All page texts	Candidate pages per field	The page that matters isn't sent
4. Extraction	Candidate pages	Value, quote, and page per field	The model misreads or invents
5. Verification	Extracted values	Verified, dropped, or disputed	A plausible wrong value slips past
6. Review	Dropped and disputed values	Accepted or corrected values	Reviewers get tired and click through

Everything after intake runs as background work. SigmaCPM uses a Postgres-backed job queue with leases, orphan reclaim, and skip-locked selection, so there's no separate broker to run. Postgres documents SKIP LOCKED as a way "to avoid lock contention with multiple consumers accessing a queue-like table" (PostgreSQL docs). If a worker dies halfway through a big package, its lease runs out and another worker picks the job up. We cover the database side on our PostgreSQL page.

A walk-through: one bond clause from scan to review

Here's a single field followed through. The tender and its numbers below are illustrative, not taken from a SigmaCPM customer. Suppose an estimator uploads a package whose performance security clause sits on a scanned annexure near the back.

Stage 1 and 2: the page has no text

Intake stores the files, counts pages, and queues a text job per page. Most pages have a text layer and come back directly. The annexure page comes back nearly empty, which is the usual sign of a scan, so it's queued for OCR.

The first OCR attempt times out. The job records the failure against that page and retries, and the second attempt returns text. Downstream stages only see a page whose text arrived a little late.

Stage 3: the clause is among a handful of pages sent

For the bond fields, page targeting searches the page texts for the language bond clauses tend to use and picks a short list of candidate pages. Because the annexure now has OCR text, it's on that list. Without stage 2 doing its job, it wouldn't be, and the field would simply be "not found".

Stage 4 and 5: a value, a quote, and a disagreement

The model returns the bond amount, a verbatim quote, and the page number. Code then checks that the quote really appears on that page. It does, so the value survives the first check. Bond terms then get a second, adversarial model run. Say the second run reads the validity period differently. The field lands as disputed.

Stage 6: the estimator decides

The estimator opens the review screen, sees the scanned page next to both readings with the quote highlighted, picks the right one, and moves on. The correction is saved. No guessed number went quietly into the bid.

When we lay the flow out like this for contractors, the question we hear is rarely about the model. It's "what happens to the pages it couldn't read?" Having a clear answer to that, a person sees them with the source alongside, is most of what earns trust.

How should scanned tender pages be handled before OCR?

Treat text extraction and OCR as their own stage with its own retries, and keep text per page. SigmaCPM handles PDFs with Java PDFBox, an Apache toolkit whose feature list includes "Extract Unicode text from PDF files" (Apache PDFBox). Pages without usable text go through OCR with retry handling before extraction, so old scanned packages still work.

What to plan for at this stage:

1. Detect which pages need OCR. A near-empty text layer usually means a scan.
2. Retry failures per page, and count the retries, so one bad page doesn't fail a whole package.
3. Keep the text page by page, with the page index. Every later stage cites pages, so never merge the document into one blob.
4. Record OCR quality if your engine reports it, so weak pages can be flagged for review.
5. Clean the image first when scans are poor.

Image quality matters more than people expect. Tesseract's documentation, for example, says it "works best on images which have a DPI of at least 300 dpi" and recommends rotating skewed pages so text lines are horizontal (Tesseract docs).

Page targeting: send the model only the right pages

Because most pages don't contain the fields you want, and sending all of them is slow, expensive, and makes it more likely the model cites the wrong place. A package can run to hundreds of pages. SigmaCPM retrieves candidate pages per field before extraction, so the model reads the right pages instead of the whole package.

Two habits help. Keep a few extra candidate pages rather than too few, since a missed page is a missed value. And log which pages were sent per field: when a field comes back empty, check first whether its page was ever sent.

Extraction that cites its page

Three things per field: the value, a verbatim quote, and the page the quote is on. SigmaCPM extracts line items and bond terms this way. Anything that can't come with a quote isn't accepted as a value. The citation checking post covers the contract and the matching rules in detail, including where exact matching gets awkward.

SigmaCPM calls models through OpenRouter, so the model can change without rewriting the pipeline, and it records tokens and cost on every LLM call. OpenRouter returns a usage object with prompt and completion token counts and the cost of each response (OpenRouter docs). Store it against the tender and the stage, and you know what one tender costs to process before anyone asks.

The second pass is where judgement comes in. In SigmaCPM it's reserved for bond terms, where a mistake is costly. Running every field twice adds cost without much benefit on low-risk fields, so pick the few that would hurt most.

What does the estimator actually review?

Not "the answer". The pipeline hands back values in three groups, and only two need a person:

1. Verified: the quote was found on the cited page and nothing disagreed.
2. Dropped: no verifiable quote, so the estimator fills it in.
3. Disputed: two passes disagreed, so the estimator picks or corrects.

A good review screen puts the source page next to each value, highlights the quote, and makes accepting or correcting one click. Watch for fatigue: if reviewers see dozens of trivial items per tender, they start approving without reading. Good targeting upstream keeps the queue short.

Corrections are the best test data you'll ever get. Each one is a real page, a real field, and the right answer, chosen by someone who knows tenders. Save them with the page and the model's original reading, and you have an evaluation set built from your own documents for free.

Once accepted, line items become the start of the project record. In SigmaCPM they feed vendor quote requests and later progress billing, so nobody re-keys them; our construction software page covers the rest of that record.

How do you monitor AI tender document extraction?

Measure per tender and per stage, not only the final output. A six-stage pipeline has six places to stall, and a single "accuracy" figure won't say which one did. The useful set is small: time in each stage, OCR pages and retries, fields extracted, dropped, and disputed, and cost.

A sudden rise in dropped values usually means a new document layout or an OCR problem, not a model problem. Is the jump on scanned pages only? Look at stage 2. Fields empty with no quote? Check which pages stage 3 sent.

If we were advising a contractor building this themselves, we'd say: don't accept anyone's accuracy figure until it's measured on your own tenders; insist every value shows its page; start with the fields your estimators re-key most; keep people on disputed and uncited values permanently; and track cost per tender from week one.

FAQ

What share of tender values should we expect the pipeline to verify?

There's no honest single figure, because it depends on your documents: how many pages are scanned, how clean they are, and how consistent the layouts are. Measure it on a sample of your own past tenders, counting verified, dropped, and disputed values per field, before trusting anyone's number, ours included.

Can it pull tenders directly from government e-procurement portals?

Uploading documents downloaded from a portal always works. India's Central Public Procurement Portal lets bidders download tender schedules free of cost (eProcurement System, Government of India). Fetching them automatically is a separate question that depends on each portal's terms and login rules, so we treat it as its own piece of work.

Why use page-level text instead of one long document?

Every value has to cite a page, and the check confirms the quote on that exact page. If the text is merged into one blob, the page is lost and the check can't run. Page-level text also lets you retry OCR for one bad page instead of reprocessing the whole package.

Does the second model pass double the cost of each tender?

No, as long as you limit it. SigmaCPM runs the second pass on bond terms only, not on every line item. Because tokens and cost are recorded on every call, you can see exactly what the extra pass adds per tender and decide which other fields, if any, are worth it.

Still re-keying tenders by hand? Our tender management software page shows how this pipeline fits a contractor's workflow, and our AI document processing service builds the same kind of checked extraction for other document types. Ask to see SigmaCPM in a demo.