Multi-Agent Invoice Pipeline

Why it’s built this way

The LLM reads.
Code decides.

Invoices are the kind of input LLMs handle well: an email that says “Amt”, a PDF that writes 2026 as “2O26”, a CSV with no currency. But an invoice is also a request for money, and some of them push. One sample says “URGENT - Pay immediately to avoid penalties!!!” I didn’t want a model that can be talked into something, or that rounds a total, anywhere near the decision to pay. So I split the work by what a mistake costs. The agents read, propose, and explain. Code computes, checks, decides, and pays, and when the two disagree, code wins.

1. Invoices arrive as text, PDF, JSON, CSV, and XML, with typos, OCR errors, and missing fields.
2. Four agents do the reading. Intake extracts the fields, Validation proposes catalog matches, and the VP and critic write and audit the decision.
3. Code does everything that touches money: arithmetic in integer cents, stock, duplicates, the policy gate, and payment.
4. Code treats what the agents say as proposals. When it disagrees, it overrides them and logs the override as an event.

Orchestration · Pydantic AI

A transition table,
not a graph framework.

Grok was a given, so the open question was what should drive the agents. LangGraph was the obvious candidate, but nothing here needed it. No agent picks its own next step, every path is known in advance, and the one loop, between the VP and the critic, has a fixed cap. So I used Pydantic AI for what the agents do need, typed outputs, tools, and output validators, and wrote the orchestration as a small state machine: a Stage enum, a table of transitions keyed by stage and condition, and one handler per stage. Each handler returns a condition and the table says where it leads, so a path missing from the table fails the run instead of wandering somewhere new. Any exception ends at failed.

Extraction · Intake agent · output validator

Every value has to be
on the page.

The Intake agent turns any format into one typed Invoice schema. The schema only holds what the document states, like stated_total and stated_tax_rate, never a computed value, so an invented number has nowhere to hide. Pydantic enforces the shape, and an output validator checks the content: every number, name, and date must appear in the source text, after normalizing case, punctuation, and OCR zeros. A value that doesn’t raises ModelRetry with a sentence the model can act on, up to two retries. The animation replays the test that covers this, on invoice 1010.

1. Grok returns an Invoice: every field copied as written, and null wherever the document says nothing.
2. Before the output is accepted, code looks for each value in the document’s own text.
3. The first attempt states a 5% tax rate. Invoice 1010 lists $335.00 of sales tax but never writes a rate.
4. ModelRetry sends the problem back. The second attempt returns null, and the check passes.
5. A wrong total that is printed on the page still passes here. That is the vendor’s mistake, and Validation flags it.

Item matching · Validation agent · reconcile()

A similar name is
not the same item.

Invoices name items loosely: “Widget A”, “WIDGET A”, “WidgetA (rush order)”. The Validation agent handles that with three read-only tools over the inventory table, lookup_item, search_inventory, and check_stock, and proposes a catalog item for each line. But a model that decides WidgetC is probably WidgetA would pay for a part that doesn’t exist. So code computes its own match, exact apart from spaces, letter case, and a trailing note in parentheses, keeps it, and records every disagreement as a match_override event. In the recorded run Grok got WidgetC right. The scene replays the test that makes sure it wouldn’t matter if it hadn’t.

1. Invoice 1016 bills for WidgetA, WidgetB, and WidgetC. The catalog has no WidgetC.
2. The Validation agent looks each line up with its tools and checks stock for what it finds.
3. Suppose it maps WidgetC to WidgetA, the closest name. The test scripts exactly that.
4. Code matches names exactly, keeps its own answer for the line, and logs a match_override.
5. The rules then run on code’s matches. WidgetC is an unknown item, an error, so the policy gate will reject the invoice.

Policy gate · approval/policy.py

Most outcomes are settled
before an agent drafts one.

Validation turns everything code found into typed findings, each with a code like out_of_stock or price_above_catalog and a severity. The policy gate reads only those findings and picks one of four modes, which sets how much the approval agents may choose. A reject or a human review is a fixed outcome; only strict and standard leave the decision open. Of the 17 sample invoices that weren’t duplicates, the gate fixed 11. The animation runs the whole batch in order.

1. Any error-level finding, such as a missing vendor, an unknown item, short stock, or wrong arithmetic: reject. Eight invoices.
2. A non-USD currency, a price above catalog, a revision, or an invoice number another vendor already used: a person decides. Three invoices.
3. A total over $10,000, or a risk signal the Intake agent quoted: strict review, with up to three VP and critic rounds.
4. Everything else is standard. The VP approves unless it finds a concrete problem, and the critic checks it. Six paid.

Approval · VP and critic agents

The agents can escalate.
They can’t loosen.

The VP agent never sees the raw document. It gets a case file that code wrote: each line with its catalog match, stated against computed amounts, every finding, and the policy mode with the outcomes it allows. It drafts a decision with a rationale of at most 120 words, and a critic agent audits the draft against the same case file, checking every claim and amount. Under strict review they get three rounds to agree, otherwise one. After every draft, one function of code has the last word.

1. Code writes the case file, and the VP drafts a decision and a rationale from it.
2. The critic audits the draft. An objection goes back as concrete issues, and the VP revises until the critic agrees.
3. If they still disagree after three rounds, the invoice goes to a person.
4. A fixed outcome stays fixed. On 1002.txt, a test scripts a VP that approves 20 GadgetX with only 5 in stock, and a policy override sets it back to reject.
5. In standard mode a VP reject becomes needs_review, so no agent can reject a clean invoice on its own.

Duplicates and payment · duplicates.py · payment.py

Pay each invoice once,
whatever file it comes in.

A filename says nothing about whether an invoice was paid, so identity comes from content. The key is the normalized number plus the vendor, and the fingerprint is a SHA-256 of the fields every format carries: number, vendor, date, the lines as a multiset, total, and currency. The same key and fingerprint is a duplicate. The same key with a new fingerprint is a revision, which a person must approve. Payment then sends only what is still owed, under an idempotency key, inside one BEGIN IMMEDIATE transaction, so two approvals at the same moment can’t both read “nothing paid yet”. Money is integer cents throughout.

1. INV-1012 arrives twice: as a PDF, then as a text file with a different layout.
2. Both reduce to the same core fields, whatever the layout.
3. Their fingerprints match, 6d751f33753066a3, so the text file ends as a duplicate and pays nothing.
4. INV-1004 comes back revised: same key, new fingerprint, and 5 more GadgetX. A person has to approve it.
5. Approved, it pays only the difference, $4,050, under its own key. Approving either version again pays nothing.

Persistence · SQLite · WAL

A paused run
is just a row.

The CLI and the dashboard are separate processes over one SQLite file in WAL mode, and both call the same two entry points, run_pipeline() and resolve_review(). Because the run’s state is saved before every stage, a run that stops for review holds nothing in memory, and either process can load it and carry on. The dashboard’s FastAPI server runs invoices on one worker thread in the order they were submitted, so every duplicate check sees the invoices that came before it.

1. Each stage starts by writing the run’s full state to its row.
2. A run waiting at needs_review needs no process. The CLI can exit, and the dashboard picks the run up with Approve and Reject.
3. Two people approve at once. An UPDATE that only matches a run still at needs_review claims it for one of them; the other is told it’s no longer waiting.
4. A process killed mid-run leaves a row with no new events. After 15 quiet minutes, the next run fails it, or puts a claimed review back in the queue.

Testing · FunctionModel · ANSWER_KEY

Record Grok once,
replay it in every test.

A pipeline of LLM calls is hard to test when every run costs money and answers a little differently. So the offline suite runs with model requests switched off. A script saves each agent’s outputs per invoice from a live run, and in tests each of the four agents is replaced by a FunctionModel that replays its queue and fails the test if anything is left unused. The golden test runs all 20 samples end to end and checks each against an answer key: status, amount paid, and finding codes.

1. A live run records each agent’s outputs, per invoice, as a JSON fixture.
2. In tests, all four agents become FunctionModels that replay their queues, with no API key and no network.
3. The whole batch replays end to end against ANSWER_KEY, one test in a suite of 126 that runs in 4.6 s.
4. pytest -m live checks the same answer key against real Grok: 20 of 20.