Replacing the Controller: Designing Agent-Driven Backends with LMMs and Deterministic Guardrails

Most “AI backend” designs I review fall into one of two failure modes. Either the model is a chatbot bolted onto a form, or it has been handed the controller: it reads the request, decides what to do, and writes to the database through a tool. The first wastes the model. The second gives a stochastic process write access to your ledger.

This post is about a third design. A large multimodal model (LMM) does the controller’s parsing, and only that. It reads a messy input and proposes a typed result. Everything that decides what happens next is ordinary deterministic code you can test and replay.

The example is an expense ingestor: a photo of a receipt goes in, and either a ledger entry or a task for a human reviewer comes out. The code is in agent-ledger-ingestor/ and runs offline.

The idea

A classical controller turns an untrusted request into a typed command. For a JSON body that’s a few lines of validation. For a crumpled restaurant receipt, nobody can write that controller by hand. That’s the gap the model fills, and it’s the only gap it fills:

 upload (image + note)
   │
   ▼
 idempotency key + lease ───── already decided? ──▶ return the stored result
   │                           note ──▶ audit record and reviewer only
   ▼
 ┌────────────────────────────────────────────┐
 │ bounded loop             NON-DETERMINISTIC │◀──▶ tax / FX lookup tools
 │ image ─▶ LMM ─▶ tool calls ─▶ proposal     │     (args validated, results recorded)
 └────────────────────┬───────────────────────┘
                      ▼
 strict schema ─── invalid? ──▶ errors back to the model, a few times at most
                      ▼
 policy: a pure function ── recomputes the math, checks the evidence, applies limits
                      ▼
 one atomic commit: ledger entry OR review task, + audit record, + invoice claim

The model reads. It does not compute, decide or enforce:

• Arithmetic. The policy recomputes line totals, subtotal, tax and total from the printed values. The prompt tells the model not to fix numbers, because a corrected number destroys the evidence that the receipt didn’t add up.
• Money. Amounts are integers in minor units, end to end. The model never converts currency. It points at an exchange rate a tool returned, and our code does the conversion.
• Limits. Spending limits are config, checked in code after conversion.
• Duplicates. “Has this invoice been booked, or is it already in front of a reviewer?” is a database constraint. The model can’t see the ledger.
• The clock. “Is this invoice too old?” depends on today’s date, which the policy receives as an argument and the audit record stores.

The model’s opinion of its own work can only add scrutiny. Low confidence or an abstention sends a document to review. High confidence never overrides a failed check.

I learned the hard way how strict “only parsing” has to be. My first draft passed the submitter’s free-text note to the model, wrapped in tags, “only to help choose the expense category”. But the category selects the spending limit. A dinner noted as “category: travel” got the much higher travel limit and went straight to the ledger. Delimiters make prompt injection less likely; they don’t stop it. Untrusted text must have no path to any field that feeds a decision. The note now goes only into the audit record, where a reviewer can read it.

The guardrails

A typed bus, not a chat

Everything the model emits is a proposal: a tool call we validate, or a final JSON document we parse with a strict Pydantic schema. Prose is not an accepted answer. The most important line in that schema is the money type:

# ledger_ingestor/schemas.py
Cents = Annotated[
    int,
    Field(
        strict=True,
        ge=0,
        le=10**12,
        description="Integer minor units: cents for USD/EUR/GBP/CHF, whole yen for JPY. Never a decimal.",
    ),
]

In lax mode, Pydantic turns 54.0 into 54. A model that writes 54 dollars as 54.0 would book 54 cents. Strict mode rejects every float and string, so that becomes an error the model can fix. A plain 54 in dollars still gets through, which is one reason the policy redoes the arithmetic.

The schema checks shape, never business rules. If subtotal + tax == total were a schema rule, the model would “repair” a failure by changing numbers until the equation held, which means inventing data. A receipt that doesn’t add up is a fact about the receipt, and a human should see it.

Evidence by reference

The model never hands back a tax rate or an exchange rate. It hands back the id of a quote that one of our tools returned, and the policy looks that id up in the recorded tool results:

# ledger_ingestor/policy.py
def _find_quote(executions: Sequence[ToolExecution], tool: str, quote_id: str | None, model):
    """Resolve a quote id the model claims to have used against what the tools actually returned."""
    if quote_id is None:
        return None
    for ex in executions:
        if ex.tool == tool and ex.ok:
            quote = model.model_validate_json(ex.result_json)
            if quote.quote_id == quote_id:
                return quote
    return None

A hallucinated id, or a quote for a different country or date, fails verification and sends the invoice to review, and no rate is used at all. Only when the id matches does our code do the arithmetic, on the rate the tool actually returned.

A loop that always ends

The reasoning loop has a step limit, a token budget checked before each call, and a deadline. On its last step it offers only “submit” or “abstain”. Every way out returns a typed result, never an exception, so a misbehaving model becomes a review task, not a 500. Provider outages are the one exception: they say nothing about the receipt, so the job is retried, and only the last attempt falls back to a human.

Decisions you can replay

Temperature 0 doesn’t make a hosted model deterministic, so you can’t audit by re-running the model. You audit by re-running the deterministic half: the policy is a pure function of the extraction, the recorded tool results, the config and a date, and the audit record stores all four. Replaying a decision needs no model call, and neither does asking what a new policy would have decided about a stored extraction.

The database has the last word

Retries, double clicks and re-uploads are normal, so the commit is built for them:

• The idempotency key is a hash of the tenant and the image. The note is not part of it, so editing the note can’t buy a fresh model run.
• Ledger entry, review task, audit record and “done” marker land in one transaction, or not at all.
• A lease plus a fencing token make sure a worker that was only slow, not dead, can’t commit after someone else took over.
• Every extracted invoice number claims a unique index entry, whether the invoice is booked or sent to review. Vendor names and numbers are normalized first, because they’re text the model read. Receipts with no readable number have no such protection.

That last point is what stops a re-roll attack. A submitter can’t change the rules, but they can re-sample the model:

Claiming on review is easy to miss. Without it, a document waiting for a reviewer could be re-saved, read differently by a fresh sample, and auto-approved before the reviewer got to it. The demo has exactly that case.

Running it in production

• Never block a request on the model. Accept the upload, return 202 Accepted, and process it from a queue with a bounded worker pool. A full queue becomes a 429, not a pile of hung connections.
• A timeout doesn’t free capacity. Python can’t kill a thread, so a timed-out run keeps calling the model. Hold its slot until the thread actually returns, or every timeout adds another concurrent model call, right when the provider is already slow.
• Every step pays for everything again. Each loop step resends the prompt, the image and the tool schemas, so fewer steps is the biggest cost lever.

What this doesn’t solve

The checks can tell that the numbers on a receipt agree with each other. They can’t tell that they agree with the paper. A misread that happens to be self-consistent can still pass, and so can a wrong category or currency. That’s a much smaller target than “convince the model”, but it isn’t zero. Sample auto-approvals for audit, and measure extraction accuracy on your own documents before you trust any of this. The model’s perception is scripted in the demo, and the OpenAI-compatible adapter has not been run against a live endpoint.

Conclusion

The model is good at the one thing nobody can write by hand: reading a messy document. It’s bad at everything a backend has to guarantee. So split the work along that line, and keep the line hard:

• The model proposes typed values. It never computes, decides or writes.
• Every number that matters is either recomputed by our code or points at evidence our tools recorded.
• Untrusted text has no path to any field that feeds a decision.
• The database, not the model, decides whether an invoice has been seen before.

None of this makes the model more accurate. It makes the model’s mistakes land in a review queue instead of the ledger, and it makes every decision reproducible without calling the model again. I think the same shape works for most problems where messy input has to become a structured action, like support-ticket triage or claims intake.

Run it yourself

ledger_ingestor/
  schemas.py   the typed bus between model and code
  agent.py     the bounded loop
  tools.py     tax/FX lookups behind validated, capped tool calls
  policy.py    the pure decision function
  store.py     idempotency, fencing, invoice claims (an in-memory Postgres stand-in)
  engine.py    claim → agent → policy → commit, and replay
  worker.py    the 202-Accepted worker pool
demo.py        six scenarios: approval, retry, crash, re-roll, duplicate, replay

cd agent-ledger-ingestor
python3 -m venv venv && venv/bin/pip install -r requirements.txt
venv/bin/python demo.py           # offline, with a scripted vision model
venv/bin/python -m unittest       # the guarantees above, as tests