How to tell if your automation is actually working

We've watched a lot of business owners trust a dashboard they've never actually tested. It's not their fault. The dashboard looks finished, the numbers are moving, and asking "but is this actually working" feels like paranoia when everything appears fine.

That question is the one most automations never get asked.

What's the fail-on-purpose test, and why does it matter?

Here's a scenario we walk people through when we're teaching someone the basics of building automations. Say you've just wired up a system: it pulls in an order, processes it, logs the result. You test it with a normal order. It works. Green light. Done, right?

Not quite. The part of any build most likely to be wired backwards is the error handling. A job that completes cleanly can end up logging as a failure, and a job that actually failed can end up logging as a success. Nobody notices, because from the outside, both look identical. The work seems to get done either way, until the day it doesn't and nobody finds out for weeks.

The fix is simple and almost nobody does it. Once a build is finished, make it fail on purpose. Feed it a broken input, cut its connection to a tool mid-run, give it something it can't process. Then go open the log and check that it actually says what happened.

If a deliberate failure logs as a success, you've found the exact same bug that would otherwise sit there hidden until it costs someone a missed invoice or a customer nobody followed up with. If it logs correctly, now you actually know the log can be trusted, instead of assuming it can.

This works because the run history is the only place a miswired system would ever show itself. And in a real business, almost nobody opens it. The log sits there quietly, either telling the truth or not, and the only way to know which is to force it to lie to you on purpose, once, before you trust it.

Why do adoption numbers lie about whether the work got done?

AI and automation tools get measured by adoption. Seats filled, logins, prompts sent, "active users this week." None of that tells you whether the actual work happened.

A login is just a login. It tells you someone opened the thing. It says nothing about what the thing did once they were in there, whether the invoice still got typed twice, or whether the report that used to take someone two hours on the last day of the month still takes someone two hours on the last day of the month.

The number that actually matters is different: what share of the manual work stopped being manual. Did the invoices stop getting retyped from the supplier PDF into the accounting system by hand. Did the monthly report stop being rebuilt from scratch. Did someone stop copying the same customer details into three different tabs every morning.

We wrote about how that copying adds up in the AED 10,000 question, which walks through what one hour of manual re-typing a day is actually worth over a year. It's the same blind spot. A login is easy to count. Work that quietly disappeared takes someone actually going and looking for it, comparing what used to happen against what happens now. The easy number wins by default, not because it's the right one.

We saw the difference clearly on a bike rental check-in process we rebuilt. The number that mattered wasn't how many staff had opened the new system that week. It was that registering a customer went from just over 7 minutes to about 70 seconds, an 84% drop, because the manual re-entry step was actually gone. That's a work number. "Everyone's logged in" would have told us nothing.

If you want the fuller picture of why systems drift away from the real workflow in the first place, we go into it in why AI implementations fail.

What should you actually check every month?

You don't need a technical background to run this. A ten-minute monthly check catches most of what quietly breaks:

• Force one failure. Feed the system a bad input and confirm the log says it failed, not succeeded.
• Pick one task it was supposed to eliminate. Watch someone do that part of their job for five minutes. Is the manual step actually gone, or did it just move somewhere less visible?
• Open the run history, not the summary dashboard. Summaries smooth things over. The raw log doesn't.
• Ask the person using it daily whether they've built a workaround. A quiet workaround is usually a sign the system doesn't handle something it was supposed to.
• Check the last "success" against reality. Pick one recent record the system marked as done and confirm the actual output exists, in the actual place it's supposed to be.

None of this requires touching code. It requires deciding to look, which is the part almost everyone skips because the dashboard already told them it was fine.

If a spreadsheet quietly turned into your real operating system somewhere along the way, that's a related pattern worth checking too. We cover the signs in when a spreadsheet becomes a system.