How to measure whether automation paid off

Why “it feels faster” isn't enough

People are poor judges of their own speed. In a 2025 study by METR, 16 experienced software developers worked on 246 real tasks, some with AI tools allowed and some without. Before starting, they expected the tools to make them 24% faster. Afterward, they believed they had been 20% faster. Measured, they took 19% longer.

METR has since said those results are out of date, since the tools have improved. The lesson that lasts is the gap between how work feels and what the clock says. Your team will feel faster too. You need numbers to know.

Gains are also rarely spread evenly. A study of 5,179 customer support agents by Brynjolfsson, Li and Raymond found that an AI assistant raised issues resolved per hour by 14% on average. For newer and less skilled agents the gain was 34%. For the most experienced, it was minimal. If you only look at the team average, you'll miss where the value is.

Capture a baseline before you build

Once the new system goes live, the old way of working disappears. Any number you didn't write down is lost, and the debate about whether it worked turns into one person's memory against another's.

For the process you plan to change, a handful of measures is enough. Three that almost always matter:

• Hands-on time per item. The minutes a person spends on one order, claim or invoice.
• Wait time. Calendar time from arrival to done, including every hour it sits in a queue.
• Backlog. How many items are still waiting on Friday afternoon.

Hands-on time and wait time are different numbers. An invoice might take six minutes of work and sit for four days. Automation often helps the second number far more than the first.

Volume and rework usually earn a line too. Which measures fit your process, and where the honest numbers already live in your systems, is something a good builder works out with you before anything gets built.

Write down the context as well: who was on the team, whether it was a busy season, and what else changed at the same time. Launch in your quiet month and the after numbers will flatter you.

Measure without a big study

You don't need a consultant with a stopwatch. You need enough data to be honest.

Sample instead of counting everything. A few dozen real items across two normal weeks, including some awkward ones, tells you most of what you need. Use dates your systems already record, such as when an email arrived or a ticket closed. Wait time usually falls out of those for free.

Then define each measure in one sentence and keep it fixed. If “done” means shipped before and approved after, the comparison is broken. The same definitions and the same method apply after launch, once the system has settled.

Count the full cost

Most return estimates count the build and stop there. The real cost has more lines. Three that get missed most:

• Your own team's time: workshops, testing and data cleanup all land on your people.
• Running costs: hosting, subscriptions and usage charges. If the system uses AI, those services usually bill per use, so cost rises with volume. Check the real monthly bill, not the estimate.
• Review time: if people check what the system produces, that time is a cost. In the first months it is often the biggest one.

Upkeep and training belong on the list too. Which lines matter most depends on the system, and a builder who has run one before should be able to tell you where yours will land.

When you turn hours into money, use the loaded cost of an hour, not just the wage. In US private industry, benefits made up 30% of employer compensation costs in June 2026, according to the Bureau of Labor Statistics. Your finance team will have its own figure. Use theirs, so nobody can argue with the math.

Hours saved versus hours redeployed

This is where most automation reports quietly overstate. “We saved 120 hours a month” sounds like 120 hours of value. It only is if those hours went somewhere useful.

Saved time ends up in one of three places.

• Redeployed: the team took on work that was waiting, such as follow-ups, a backlog or a new account. This is real value.
• Avoided: volume grew and you didn't need the extra hire you had planned. Also real, and often the easiest to prove.
• Absorbed: the time spread out into slightly easier days. That has some value, but keep it out of the business case.

Ask the team lead one question each month: what did your team do with the time? Write the answer down. A short list of concrete things beats a big number nobody can trace.

Benefits that aren't hours

Some of the best results never show up as time saved. A quote that goes out in an hour instead of two days can win more work. Each error you no longer make had a cost further down the line, such as a credit note or a second shipment. And managers can see the queue without asking around, because reports that used to be built by hand are simply there.

Put a number on these where you honestly can. Where you can't, list them plainly and leave them out of the money math.

A worked example (made up)

A made-up distribution company processes supplier invoices by hand. Three people match each invoice against the purchase order and the delivery record, then key it in. The company builds a system that reads each invoice, matches it and sends only the mismatches to a person.

The gross saving is about 49 hours a week. Review time is already inside the after numbers. Running costs and upkeep are real monthly costs, so they come off the top.

Then the honest part. The team lead reports that about 30 of those hours now go to supplier follow-ups and to paying early enough to take discounts that used to be missed. The other 19 or so are spread across the week, so the report doesn't count them as savings.

That is the story to tell: fewer errors, approval in a day and a half instead of six, higher volume with the same team and discounts now captured. All of it rests on numbers written down before the build.

Reporting to leadership

Leaders want a short answer: did it work, what did it cost and what happens next. One page is enough. Put the before and after numbers side by side, with the same definitions. Show the full cost and what the team did with the time, in plain words. End with a recommendation: keep going, expand, change course or stop.

Report on a fixed rhythm, such as 30, 90 and 180 days after launch. Set expectations early. The first month often looks worse, because people are learning and checking everything twice.

When to stop or change course

Agree on stop rules before launch, while nobody is attached to the project. For example: review time isn't falling after eight weeks, which means people may not trust what the system produces. Or running costs are growing faster than volume. Or people are keeping a side spreadsheet “just in case.”

Stopping doesn't always mean switching off. Often it means narrowing the system to the cases that work well. The worst outcome is a system nobody trusts that stays on because nobody wants to say so.

What most guides miss

Measuring changes the work. Once people know a process is being timed, they often tidy it up. Some of your gain comes from that alone. Give the team the credit, not the software.

Averages hide the story. Look at the spread as well. If most items got faster but a few now take much longer because they fall outside the rules, the slow ones are what customers notice.

Review time should fall. Early on, people check everything. As trust builds, they move to spot checks. A falling line is one of the best signs the system works. A flat one is a warning.

Watch the people who handle exceptions. Automation often moves the hard cases to a few senior staff. If their load goes up, part of the saving elsewhere is borrowed from them.

How we do it

We agree on the baseline measures with your team before we build anything, and take the first measurements together during our process audit. After launch, we report against the same numbers for the systems we build. And we stay on after launch, because the measurements keep pointing at the next thing worth fixing. Tell us about the process and we'll help you set the baseline.