When an AI Agent's Explanation Looks Right and Isn't: A Reconciliation Case
The risk in AI-assisted accounting work is rarely an obvious error. It is a plausible explanation that is wrong in a way no one catches, because the explanation fit the pattern.
This is a composite case, built from a pattern common in reconciliation work. It is not a description of a specific utility, and the figures are illustrative.
The Task
A finance team asked an agent to reconcile the accounts payable subledger to the general ledger at month-end. The variance was $48,215 across 90 line items. The agent was asked to identify the items driving the variance and explain each.
It returned six items needing attention, each with an explanation. Five were straightforward. The sixth was a $31,400 invoice dated September 29 and posted October 3.
The Explanation
The agent classified the item as a cut-off timing difference: invoice dated in September, posted in October, expected to clear in the next period. That explanation fit the pattern of the other items and the team's posting calendar, as far as the agent knew it.
The explanation was plausible. It was also incorrect.
What Actually Happened
The invoice was a duplicate of one already posted on September 30. The supplier existed in the vendor master under two records, and the two invoices had been entered against different ones.
The agent had searched for a matching invoice by vendor ID and amount. With two vendor IDs, no match existed to find. Finding no match, it concluded the invoice was a timing item rather than a possible duplicate.
Left alone, the October posting would have been paid. An AP clerk caught it because she recognized the supplier and amount from the prior week's payment run.
Why It Was Easy to Miss
| Factor | Effect |
|---|---|
| The explanation fit the pattern | Five similar explanations made the sixth look routine |
| It was stated without qualification | Nothing signaled that the match had been on limited evidence |
| The review tied totals | A total can agree while an individual explanation is wrong |
| The root cause was in the vendor master | The agent had no reason to look there |
The third factor is the controllable one. A reviewer who ties the agent's totals to the ledger confirms the arithmetic. That does not test whether any individual explanation is true.
What the Team Changed
1. Require evidence with each explanation. The prompt now tells the agent to state what supports each explanation, including which fields it matched on, and to label an item "unconfirmed" when the support is a single field or an absence of a match.
An absence of a match is weak support for a timing conclusion. The prompt now says so.
2. Trace the largest explained item to source. Each month the reviewer takes the largest item the agent explained and traces it to the source invoice and the posting record, not to another system report. The review note records that this was done.
3. Add a duplicate vendor check to the month-end procedure. The root cause was two vendor records for one supplier. That is a control issue independent of AI, and it would have produced a duplicate payment in a manual process too.
4. Update the known-answer test. The team added a prior-period reconciliation containing a known duplicate to the prompt's test set. If a future model or prompt change causes the agent to miss it, the test shows it before current-period work is affected. See Prompt Rot for how the known-answer test works.
What This Does and Doesn't Show
It doesn't show that the agent was poorly suited to the work. It found the variance and reduced a ninety-line review to six items. For a month-end process, that is real time saved, and the other five explanations were correct.
It shows where the agent's work stops. The agent can identify which lines need attention. Deciding which explanation is true depends on information it doesn't have: your vendor master history, your posting calendar, which suppliers have been entered twice before. It will not ask for that information unless you tell it to flag what it cannot confirm.
The division of work is the same one described in What Utility Accountants Still Need to Know When AI Does the Close. You hand over pulling, comparing, flagging, and drafting. You keep the judgment on which explanation is true and the sign-off.
Checklist for Your Own Reconciliation Prompts
- Does the prompt require the agent to state its evidence for each explanation?
- Does it tell the agent to label items it could not confirm, rather than choosing the most likely category?
- Does the reviewer trace at least the largest explained item to a source document?
- Does the review note say what was traced and what was changed?
- Does your known-answer test include a case where the plausible explanation is wrong?
If the answer to any of these is no, the review step is relying on the explanations being right. In most months they will be. The month they aren't is the one that matters.
Bottom Line
An agent's explanation is a draft of a judgment. It is often correct and consistently well written, which is why the review has to test it against source documents and not against its own consistency. Build that into the prompt and the review procedure, and the agent's speed is worth having.
Related Articles
