The easy trap: "it saved time"
You built the automation. It runs. Nobody complains. Someone estimates "this probably saves a few hours a week," writes it down as a win, and moves on.
That's not an audit. That's a guess wearing a lab coat.
Time saved is the metric everyone reaches for first, because it's easy to imagine and hard to disprove without real data. But time saved on its own can hide two failure modes: the automation quietly creates more errors than it removes, or nobody actually uses it and the "savings" only exist on paper.
A real Check-stage audit needs three numbers, not one: time, error rate, and adoption. Track all three and you'll know whether you built something good or just something that runs.
The three numbers that matter
Each number answers a different question. Time tells you if it's faster. Error rate tells you if it's better. Adoption tells you if it's actually being used instead of quietly worked around.
| Metric | Question it answers | How to instrument it | What "good" looks like |
|---|---|---|---|
| Time | Did this free up real hours? | Timestamp logs, before/after time studies, or a simple weekly self-report for 2 weeks pre- and post-launch | Actual savings land within ~20% of your estimate; if it's wildly off, your baseline or your automation is broken |
| Error rate | Did quality improve, stay flat, or get worse? | Count defects, rework tickets, or manual corrections per 100 transactions, before and after | Errors drop — or at minimum don't shift downstream into a queue nobody's watching |
| Adoption | Is it actually being used? | % of eligible cases that flow through the automation vs. handled manually or worked around | 80%+ adoption within 4–6 weeks; lower than that means friction or trust problems, not a tech problem |
Miss any one of these and you can end up celebrating a "win" that's really a wash — faster, but sloppier, or fast for the tool and unused by the team.
Instrument before you flip the switch
You can't measure a "before" you never recorded. If you're about to automate something, capture the baseline first: how long the task takes today, how often it goes wrong, and how it's currently handled. We cover exactly how to do this in Baseline or Bust — it's worth doing even if it feels like overhead, because it's the only thing that makes your Check-stage numbers meaningful later.
If you haven't automated anything yet and aren't sure where to start, Automate One Task This Week walks through picking a single task and setting it up cleanly — which also makes it much easier to instrument, since you're not untangling a dozen moving parts at once.
Instrumentation doesn't need to be fancy:
- A spreadsheet with three columns — date, minutes spent, errors caught — kept for two weeks before launch and two weeks after is often enough to see a real signal.
- For adoption, tag or filter your system logs: how many invoices, tickets, or leads actually went through the new path versus the old one.
Reading the results: what good actually looks like
Numbers without context don't tell you anything. Here's how to read each one honestly.
Time. If you projected saving 5 hours a week and you're seeing 4.5–6, that's a solid result — normal variance. If you're seeing 1 hour, either the automation is only handling part of the task, or the "before" number was inflated. If you're seeing 8+ hours saved, double-check you're not missing new manual work the automation created elsewhere (someone now has to review its output, for instance).
Error rate. A drop is the goal, but watch for errors that didn't disappear — they moved. A classic pattern: manual data entry errors go down, but a new "exceptions" queue quietly fills up with edge cases the automation can't handle, and nobody's assigned to clear it. That's not a fixed error rate. That's a hidden one.
Adoption. This is the number people skip, and it's the one that catches the most fake wins. An automation with great time and error numbers but 40% adoption isn't delivering those numbers at business scale — it's delivering them for the minority of cases that go through it, while the rest still happen the old way. Low adoption usually means one of three things: the tool is harder to use than the old process, people don't trust it yet, or it doesn't cover the real-world variations of the task.
A before/after example
A property management office automated its rent reminder process. Here's what the audit looked like.
Baseline (4 weeks, manual):
- Time: 6.5 hours/week across two staff, sending reminders and updating a spreadsheet
- Errors: 4% of reminders had wrong amounts or wrong tenant names, caught mostly by tenant complaints
- Adoption: N/A (fully manual)
After (6 weeks post-launch):
- Time: 50 minutes/week, mostly spent reviewing exceptions
- Errors: 1.2%, almost all from a handful of tenants with non-standard lease terms the system didn't handle
- Adoption: 94% of reminders sent automatically; the remaining 6% were flagged and handled manually by design
That's a genuine win on all three fronts — and the honest audit also surfaced a fix: build a small rule for non-standard leases, since that's where the remaining errors cluster. Without measuring error rate specifically, that gap would've stayed invisible under a "we saved 5+ hours a week" headline.
Red flags — and a checklist to catch them
Watch for these patterns when reviewing your own automation, or anyone else's report of one:
- No baseline exists. Time saved is just being estimated after the fact, which almost always rounds up.
- Errors moved, not gone. A new manual review step or exceptions queue is absorbing the errors the automation was supposed to eliminate.
- Adoption is assumed, not measured. "People are using it" isn't a number. Pull the actual usage split.
- Cherry-picked window. Results measured during a quiet week, holiday period, or right after a training session tend to look better than steady-state performance.
- Vanity metrics substitute for outcomes. "We built 4 automations this quarter" says nothing about whether any of them worked.
Run through this before calling anything done:
- Baseline recorded before launch: time, error rate, current handling method
- Time measured post-launch over at least 2–4 weeks, not a single good day
- Error rate tracked at the same granularity as before (per 100 transactions, not "it feels better")
- Adoption rate calculated as a real percentage of eligible volume
- Exceptions queue, if one exists, has an owner and a review cadence
- Results compared against your original estimate, with variance explained
If you're not sure which of these three numbers matters most for a given automation, it usually depends on which type of leverage you were going for in the first place — Match the Metric to the Leverage breaks down how to pick the right measure for efficiency, scale, insight, or risk-reduction projects specifically.
An automation that saves time, holds or improves quality, and gets used by nearly everyone it's meant for is a real win. Anything less than all three isn't finished — it's still in progress, whether or not it's live.