Evaluating an underwriting agent before it touches a real file

How one insurer tested an agent on 600 historical files before letting it near a live one, what the senior underwriters found, and the one category the agent missed every time.

AO

Written by Dr. Amara Okafor · Head of Clinical Research

  • Updated Nov 13, 2026
  • 3 min read
Cover art: layered violet strata on a pale violet ground, crossed by a thin violet line

An underwriting team at a mid-sized commercial insurer had a working prototype of an agent that could read a submission, pull the relevant data and recommend a decision. It looked impressive in demos. Before letting it near a live file, they asked a simple question: would it make the same calls as our best underwriters?

This is how we helped them answer it.

Start with the files you already decided

The fastest way to evaluate an agent on real work is to give it work that has already been done well. The team pulled 600 closed files from the past two years, chosen to cover their full book: small and large accounts, renewals and new business, accepted and declined.

  • Every file had a final decision made by a senior underwriter.
  • Every file had the full submission, loss history and correspondence.
  • None of the files had been used to build or tune the agent.

The underwriters wrote the rubric

We didn't bring our own definition of a good decision. The insurer's three most senior underwriters wrote it, over two working sessions, with one of our evaluation leads facilitating. The rubric covered the decision itself and the reasoning a regulator or reinsurer would expect to see.

  1. Is the accept or decline decision the same as the senior underwriter's?
  2. Is the premium within the band the team would have quoted?
  3. Are the conditions and exclusions appropriate to the risk?
  4. Would the written rationale satisfy an audit?

Two graders on every file

We ran the agent on all 600 files. Then two underwriters from the insurer's team, working independently, graded every recommendation against the rubric. Where they disagreed, the senior underwriter who made the original decision broke the tie.

What they found

The agent agreed with the senior underwriters' decisions on 78% of files. On straightforward renewals with clean loss histories it was close to perfect. On complex new business it was weaker, but not alarmingly so.

That single category explained a large share of the disagreements, and it is exactly the kind of failure that a demo never shows. Nothing in the agent's output looked wrong. It was confident, well written and internally consistent.

If we'd gone live, it would have quietly underpriced our riskiest renewals for a year before anyone noticed.

What happened next

The team didn't scrap the agent. They did three things instead:

  • Routed every file with a disputed claim to a human, permanently.
  • Commissioned a small set of expert-written examples of disputed-claim files to retrain the agent.
  • Set up a monthly re-run of the same 600-file evaluation, so any regression shows up before it reaches a customer.

Six weeks later, the agent's agreement on the disputed-claim slice had moved from zero to 71%, and the team started a supervised pilot on live renewals.

If you're deploying an agent into regulated work and want to know how it performs on your own files, book a demo. We'll show you how the evaluation is set up, using your own underwriters as the standard.

Start with one workflow. Prove it, then add the next.

Tell us the system, the data and the outcome you need. An engineer from SixthEra replies within one working day.

Contact us