Who pays when the agent is wrong?

Building agents for regulated industries, where "oops" has a paper trail

Say you run claims at an insurance company and you've deployed an agent that settles the routine ones. It works well for eight months. Then one week it quietly denies forty claims it should have paid, a customer complains to the regulator, and someone from legal is standing at your desk with a question that sounds simple: whose fault is this?

There are four names on the list. The company that deployed the agent. The developer who built it. The QA team that signed off. The model provider whose model did the actual reasoning. I think the answer is less mysterious than the current hand-wringing suggests, though parts of it are genuinely unsettled. One disclaimer before we start: I'm an engineer, not a lawyer. Read this as field notes from someone building these systems, not as legal advice.

Regulators do not read git blame

Start with the one case everybody can name. In 2024, Air Canada's website chatbot offered a bogus discount, a passenger booked a flight relying on it, and the airline refused to honor the fare. In front of a British Columbia tribunal, Air Canada argued, with a straight face, that the chatbot was "a separate legal entity that is responsible for its own actions". The tribunal member called the argument remarkable, which in a legal ruling is not a compliment, and the airline paid. The amount was small, around eight hundred Canadian dollars. The principle wasn't: if it speaks with your logo on it, it speaks for you.

Regulated industries settled this even earlier, they just didn't call it AI. The US banking guidance on model risk, SR 11-7, has said since 2011 that a bank owns the risk of every model it uses, including the ones it bought from vendors. The EU AI Act points the same direction: using someone else's model doesn't move your duties onto them. Regulators fine license holders. They do not read git blame.

So the first question has a boring answer. Facing customers and regulators, it's the company, essentially always. The interesting questions are all internal.

The developer and the QA team work for you

Inside the company, the instinct after an incident is to find the person. The developer wrote the prompt and picked the model. QA blessed it. Surely one of them owns this?

Legally, mostly no. Employees doing their jobs are covered by the company's liability. That's centuries-old employment law doing exactly what it was built for. Personal liability needs something close to recklessness: skipping a check a regulation explicitly required, faking a test report, shipping over a known failure.

With agents specifically, individual blame gets even less coherent, because nobody wrote the failing line. The developer wrote instructions. The model turned them into behavior. QA tested that behavior on the cases they thought of. The failure emerged between the three, on a case nobody thought of. So "which employee is at fault" is the wrong question. A better one is "was the process defensible?", and that's the one regulators actually care about. Were there written evals, and were they wired into the deploy pipeline rather than run by hand when someone remembered? (An eval you run manually is just better vibes.) Did someone senior sign off on how much autonomy the agent got? If yes, you had an incident. If no, you had negligence, and it's institutional, not personal.

The model provider wrote the contract first

The party whose model actually made the bad call is the party you're least likely to collect from, because their lawyers got there years before you did. Read any provider's terms of service: liability is capped, typically at whatever you paid them in the last twelve months, and consequential damages are excluded. The indemnities providers do offer are about copyright. OpenAI and Microsoft will help defend you if the model's output plagiarizes someone. Nobody indemnifies you for the model being wrong. Being wrong sometimes is the documented, known behavior of the product, and catching it is your job.

Large enterprises negotiate custom contracts and get somewhat further than the standard terms. But I've seen no version of this where a model provider underwrites your loan decisions or your claim denials, and I wouldn't hold my breath. From the provider's side it would mean insuring outcomes in a business they can't see, on prompts they didn't write.

So, insurance?

This market now exists, which surprised me the first time I looked. Lloyd's-backed policies covering AI underperformance started appearing, and there's a small crop of startups underwriting agent deployments specifically. So the question isn't hypothetical anymore. Should you buy it? For a regulated deployment, probably yes, but not mainly for the payout.

The most useful thing about insurance is the underwriting. To price your policy, the insurer has to audit you: where are your evals, where are your logs, what actions can the agent take without a human, what's your kill switch. Insurers have played this role before. Boiler insurers in the 1800s effectively invented the industrial safety inspection, and fire insurers did more for building codes than most legislatures. If underwriters converge on "no policy without enforced evals and an audit trail", that will discipline this industry faster than any regulation. And if an insurer looks at your agent and declines to quote, that's information you should probably act on.

The caveat: insurance covers money, and money is the recoverable kind of loss. A suspended license, a consent order, customers who leave, those aren't claimable in any way that makes you whole. So insurance is the floor under the risk that remains after the engineering. It never substitutes for the engineering.

"But it passed QA"

Now the scenario that should keep you up at night. Testing was thorough. QA did great work. The mistake never appears in the test environment, and in production it happened once, on a Tuesday, and you can't reproduce it. Whose fault is that?

I'd say engineering first: for agents, this is the default shape of failure, and it's structural. Real inputs drift away from whatever your test set froze. Sessions run longer than tests do, and context piles up in ways fixtures never capture. Tools return live data instead of mocks. And the model itself can change underneath you, same endpoint, same version string in your config, different behavior after a provider update.

The legal reading is less forgiving than you'd hope. "We tested thoroughly" is good evidence you weren't negligent up to the day you launched. It says nothing about the days after. In regulated settings the duty keeps running in production: the EU AI Act expects high-risk systems to be monitored after deployment, and bank model-risk guidance has demanded ongoing monitoring for years. If the agent drifted in production and your first signal was a customer complaint, the unreproducible bug is not your problem. The missing monitoring is.

This is where traces earn their keep. I've written about what an agent trace has to capture, and the liability version of that argument is blunt: a production mistake you can't reproduce but can replay is an incident with a fix attached. A mistake you can neither reproduce nor replay is an unexplained event in a regulated system, and "unexplained" is the word regulators punish hardest. In some of these industries, failing to explain a decision is a violation all by itself. US credit law requires telling a declined applicant the specific reasons, and the regulator has said plainly that "the model did it" is not a reason.

Autonomy moves the fault around

I believe fault isn't fixed, and that's why the four-suspects question is misleading. Where fault lands depends on how you build the system, and mostly on one thing: how much the agent is allowed to do on its own. I've argued that the most important design decision is what your agent can't do, and it turns out to be the most important liability decision too. Every irreversible action you put behind a human gate converts "the agent did something wrong" into "a qualified person made a call with bad advice on their screen". The second category is legally old and boring. Professionals have been acting on flawed advice forever, and every regulated industry already has rules for it. An autonomous wrong action is the new, scary category. A reviewed wrong recommendation is Tuesday.

Sometimes the gate isn't even optional. EU data protection law gives people the right not to be subject to purely automated decisions that carry legal effects, which for the consequential calls means a human belongs in the loop regardless of how good your agent gets.

So for high-stakes work my honest version of the fault list looks like this: the agent proposes, a human disposes, the company owns whatever happens, and insurance sits underneath as a floor. Boring on purpose.

An old machine for absorbing mistakes

Here's the thought I keep coming back to: we already employ millions of fallible decision-makers. We call them employees. Institutions spent a very long time learning to absorb their mistakes, through training, supervision scaled to seniority, sign-offs on the dangerous stuff, insurance, incident reports, and a legal structure where the institution answers for the person. None of that machinery ever assumed the decision-maker was infallible. A corporation is, among other things, a very old technology for turning individual mistakes into survivable institutional risk.

Agents fit into that machine more neatly than the panic suggests. Supervise them like juniors: narrow authority at first, widened with track record, everything logged, the irreversible stuff gated. The real danger isn't deploying an agent that will eventually make a mistake. Every person you hire will too. The danger is handing an agent, on day one, the kind of unsupervised authority you would never give a human in their first week, and then having no answer ready when legal shows up at your desk.