When an AI agent pays the wrong party, or the wrong amount.

An agent decides who to pay from what is in front of it: the task, the conversation so far, the results of the tools it called, and whatever the seller's own page says. Every one of those can carry a payee that is not the right one. A conversation that mentioned two vendors earlier can leave the agent paying the first when the task named the second. A search result can surface a seller with a name close to the one the agent was told to use. A tool response can contain a payment address that was correct last month.

The agent is not being careless in the way a person would be. It is doing what language models do, which is to continue the most plausible pattern from its context. When the context contains a plausible wrong answer, the agent can choose it with complete confidence and describe the choice in a sentence that reads perfectly well.

Payment details are especially exposed to this because they look alike. Two addresses, two account references or two invoice numbers that differ in a few characters are hard for a person to tell apart at a glance, and an agent that is matching patterns rather than checking identifiers can treat them as the same.

The amount goes wrong in a few recurring ways. The agent pays a list price when the task specified a negotiated one. It pays the total for several items when it was asked for one. It confuses units, paying per thousand when the price was per item. Or it pays a figure that appeared earlier in the conversation for a different purchase.

Duplicate payments deserve their own mention because they are the failure that looks most like correct behaviour. A payment request times out, or returns an error after the money has already moved. The agent, reasonably, tries again. Now it has paid twice. Nothing in the transcript looks wrong: the agent hit an error and retried, which is what any careful developer would want. Only the ledger shows two payments where there should be one.

If you grade an agent by reading what it wrote, a wrong-payee failure is nearly invisible. The agent will say it paid the vendor, name the vendor correctly, and quote the amount it intended. The mismatch is between the sentence and the payment, and the only place the payment is recorded accurately is the ledger.

The same holds for a judge model reading the transcript. It sees a confident, well-formed account of a correct payment, and scores it as correct. That is why our payee and amount checks are not judgements at all: the grader compares the recipient and the amount on the ledger with the ones the task specified, and counts the payments per item.

A good payee test puts the right answer and several plausible wrong ones within the agent's reach: a similarly named seller, an outdated payment detail in the conversation history, a second vendor mentioned in passing. Then it checks which one actually received the money.

An amount test varies the shape of the price: per unit and per batch, list and agreed, with figures for unrelated purchases nearby. A duplicate-payment test makes the payment request fail in the ways real requests fail, including after the money has moved, and counts what the agent does next.

Each failure in the report comes with the ledger record that shows it: the payee that was paid, the one that should have been, the amount sent and the amount owed. A finance or risk team can check each line itself.

A failure rate on these tests is rarely fixed by a better prompt alone. The usual fixes are structural: give the agent one authoritative source for payee details instead of letting it read them from the conversation, make payment requests safe to retry so a second attempt cannot pay twice, and require the agent to confirm the payee identifier rather than the payee name. Each fix is then retested, because a change that removes one failure can introduce another.