How I Built an AI Agent That Automated 70% of Our Manual Data Processing

Every morning someone opened a shared inbox holding fifty to two hundred messages: supplier invoices, delivery confirmations, customer change requests, and a long tail of things that did not fit a category. They read each one, pulled out the fields that mattered, decided which system it belonged in, and typed it there. Three hours, every day, done by people who were expensive and bored.

Before writing any code I sat with them for two days and wrote down what they actually did. That produced the most valuable artefact of the project: a list of eleven document types, the fields each one needed, and the destination system for each. Automation projects fail when they start from the tool. This one started from a taxonomy, and the taxonomy is what made the rest tractable.

The instinct is to write one enormous prompt that reads a document and returns everything. That performs badly, because the model is simultaneously deciding what kind of thing it is looking at and what to pull out of it. Splitting the problem in two improved accuracy immediately: a cheap, fast classification call returns one of eleven labels plus a confidence, and only then does a second call run the extraction schema for that specific type.

The split also controls cost. Classification runs on the cheapest model with a tiny output budget, and it is right the overwhelming majority of the time. Extraction, which needs more care, runs on a stronger model but against a narrow, type-specific schema, so its prompts are short and its outputs are constrained. Two calls ended up cheaper and more accurate than one, which is not intuitive until you look at the token maths.

Free-text responses are unusable in a pipeline. I define a JSON schema per document type and require the model to emit exactly that shape, then validate the result before it goes anywhere. If validation fails, the item is retried once with the validation error appended to the prompt, and if it fails again it goes to the human queue. That retry-with-the-error trick fixes a surprising share of failures, because the model usually just missed a required field.

Types matter more than they look. An invoice total parsed as a string is a bug waiting to happen three systems downstream. Dates, currencies, and identifiers get explicit formats in the schema and are re-validated after parsing, so anything malformed is caught at the boundary rather than discovered in a reconciliation report at month end.

The system does not try to be right every time. It tries to be right when it is confident and to abstain otherwise. Every extraction carries a confidence signal, and anything below the threshold routes to a review queue where a person confirms or corrects it in seconds rather than doing the whole job manually. The threshold started conservative and came down as the accuracy data accumulated.

That review queue turned out to be the most important part of the design, and not because of the corrections. It is the feedback loop. Every correction is logged with the original document, which gave me a growing set of exactly the cases the system gets wrong — the ideal material for improving prompts and schemas, and the evaluation set I now regression-test against.

Documents arrive in bursts, and downstream APIs rate-limit. A queue between ingestion and processing absorbs the burst and lets me control concurrency, which also keeps the model spend predictable rather than spiky. Each item carries a stable ID derived from the source message, and every write downstream uses that ID as an idempotency key, so reprocessing the same message never creates a duplicate record.

Failures are categorised rather than lumped together. A transient API error retries with backoff. A validation failure retries once, then escalates. A classification the model is unsure about skips straight to a human. Treating those three as one generic error was the bug I shipped first, and it produced a queue full of items retrying forever against a schema they were never going to satisfy.

The honest result is that roughly 70 percent of volume now flows through untouched, and the remaining 30 percent reaches a human already classified and pre-filled. The three-hour block became a review pass of about twenty minutes. I deliberately do not quote a 100 percent automation figure, because the long tail of genuinely ambiguous documents is real and pretending otherwise is how these projects lose credibility.

The second-order effects were larger than the time saving. Data entered systems within minutes instead of a day, so downstream reporting stopped lagging. And because every decision is logged, the team could finally answer questions like which supplier sends the most malformed invoices — a question nobody could have answered when the process lived in someone's head.