Measuring agent work honestly

“Time saved” is the easiest number to claim and one of the hardest to defend. We have reported it, we have been asked to prove it, and we have quietly stopped reporting some of the ways we used to measure it. Here is what we trust now and what we let go.

Honest measurement is less flattering and more useful.

The numbers that flatter

Raw automation rate flatters. So does average handling time in isolation. An agent can post a beautiful automation rate by handling the easy things and escalating everything hard, and a beautiful handling time by giving fast wrong answers that generate a second contact.

These numbers go up and to the right and tell you almost nothing about whether the work got better. We stopped leading with them because they reward the wrong behaviour.

The numbers we trust

We trust resolution that holds: threads closed that do not reopen within a sensible window. We trust the escalation rate read alongside what happens after the escalation. We trust the count of interactions where a person had to redo the agent's work, which should fall over time.

These are harder to game because they measure outcomes, not activity. An agent cannot fake a thread staying closed, and it cannot hide reopened work in a good average.

Time saved, done carefully

Time saved is legitimate if you measure it against what a person would actually have done, not against a fantasy of infinite manual effort. The honest version is: here is the work the agent handled, here is the realistic time that work took a person, here is what a person's time is worth. No inflation.

We report it that way and it is a smaller, defensible number. The inflated version wins slides and loses trust the first time someone checks it.

Measure the log, not the promise

Every one of these numbers comes out of the same audit trail that makes the system safe. Because every action is recorded, the measurement is not a separate reporting layer that can drift from reality. It is a query over what actually happened.

That is the point of measuring from the log. The metric and the truth are the same object, so the temptation to quietly pick the flattering framing has nowhere to hide.

Where this tends to go wrong

The failure mode is almost never the agent inventing something wild. It is the small, plausible miss: a reply that is correct in general but wrong for this one account. That is exactly what the approval step and the log are for, and it is why we tell teams to read the log before they widen scope.