It's easy to check what an AI assistant tells you.  It's hard to check what it doesn't.  Most teams only measure the first one, and then call the assistant accurate.

The trouble is the silence.  Silence can trick a monitor, because a quiet system and a healthy one look exactly the same, especially when the monitor only checks its own list.  Then it goes on to kill the productivity the whole setup was built to buy: you trust the all-clear, stop looking, and find out later what it never told you.

Two weeks ago it was a monitor that stayed quiet whether it was working or broken.  Last week it was errors that landed in an inbox and waited for someone to notice.  This week is the same question, one step harder: how do you measure a miss when a miss, by definition, leaves nothing behind?

I tried three times.  This is the honest account of how that's going.

Some context for anyone new here.  I run an AI assistant on my own work.  It reads my email, my calendar, and my queue of customer support cases.  Every morning it sends me a briefing: what's on the calendar, which cases are open, which orders need attention, how many emails are unread.  Every evening it sends a wrap: what got done.

The question isn't whether those briefings are right.  It's what they leave out.  If a message lands in my inbox and the briefing skips it, I won't know the briefing skipped it.  So since July I've been trying to measure its misses.  Three attempts: ask myself every Friday, compare its output against the other ways news reached me, and run an automated check every week.  All three fell short, and the third one failed without a sound.

When someone says their assistant is accurate, they usually mean: of the things it said, most were right.

That's worth knowing.  It's also the easy half.  You can check it by reading the output.  Every answer is sitting right there, so you can grade it: right, wrong, close enough.  (For the architects: that's precision.)

The other half is: of the things it should have told you, how many did it?  (That's recall.)  An assistant can be right every single time it speaks and still stay quiet about the one thing that mattered this week.  Nothing in the output points at what's missing from it.

To count what an assistant missed, you need a separate list of everything that was there to catch.  An answer key.

In a classroom, somebody writes the answer key ahead of time.  In your inbox, your calendar, your ticket queue, nobody does.  The only complete record of "what mattered this week" is the person who lived it.

So the one instrument most teams have for measuring misses is a human paying close attention to everything, then comparing notes with the assistant.  That's the exact work the assistant was bought to take off their plate. 

I run an assistant for my own work.  It sends me a briefing every morning to prepare for the day and a wrap every evening to look back at how productive I was.  A before and after for the day.

The briefs are only as good as what they've been told to watch.  If one misses something I decide mattered, that should raise an alert, so I can set a new threshold or point it at new data.  That's how the list grows.  So I wanted a way to find the holes, and I built the obvious instrument: once a week, it asked me.

Every week from early July to mid-August, that row on the scorecard read the same thing: no data.  Not zero.  No data.  The reasons rotated. The cause never did.  I was traveling, the wrap had fired late, I'd been heads-down. 

The only instrument for measuring what it missed cost me exactly the thing it exists to save: my attention, on a Friday afternoon.  I checked whether it had been a fair test.  Everything was running, I was at a keyboard, and I still didn't answer.  So it wasn't a bad week.  The design was the fault.

If you've ever sent a survey to your own team and wondered why it came back empty, it’s the same disease.

You don't need to record everything.  You just need two timestamps.

Log what the assistant tells you, and when.  Then, separately, note when the same fact reached you by some other route: a meeting, an email from a person, a call.  Subtract.  Some items the assistant gets to first.  Some it gets to late.  Some, you hear and it never says a word.

It costs me nothing per item, because it reads the logs and my own inbox and calendar instead of asking me anything.  The answer key becomes "What reached you another way?".  That's not everything that mattered.  It can't see a hallway conversation.  But it's a sample drawn from exactly the things that turned out to matter, and it needs no extra effort from me.

I backfilled eight days of my own records to see whether it would find anything.

So it declined.  It would not give me a verdict.

I'll take that.  An instrument that refuses to grade itself on thin data is the one I'd trust to grade later on fat data.

But it did turn up one thing, and it wasn't a statistic.  It was a name.  That week, roughly forty-seven error emails from an integration had landed in my inbox in one burst.  My assistant had said nothing about them.  Nothing in its briefings or its wraps.  Those were the same errors from last week's issue: the ERP-to-CRM sync that failed while every dashboard stayed green.

It watched support cases, open orders, and a count of unread email.  An integration error is none of those.  It shows up as email from a system, in a mailbox where the only thing counted is how many are unread.  The assistant wasn't wrong about anything it said.  It just had never looked at that channel at all.

That's the part that's easy to forget.  A monitor, an alert, an AI assistant: none of them has a sense of what you care about.  They have a list.  Whatever's on the list gets watched.  Whatever isn't, doesn't exist to them.

You know the list in your head is longer than the list you gave them.  "Obviously it would tell me if the integration broke."  Obviously nothing.  Nobody said so.  The assistant had no rule for that channel, so it stayed silent about it, perfectly, forever, and every report it sent was still accurate.

The things you think are being watched, but never told anything to watch, are the exact things that fail without a sound.  The guardrails have to be written down, or they don't exist.  Ask it a different way: if you wrote out everything you assume your tools are watching, and then wrote out everything they've actually been told to watch, how long is the gap?

That gap is where last week's outage happened: nobody had told anything to watch those error emails.  It's where my weekly check died, too.  Nobody had told it to watch itself.

The weekly check was my third attempt.  Once a week it was supposed to compare what the assistant told me against what reached me some other way, and write down the gaps.  I set it up, it ran for a while, and I stopped thinking about it.  That's what automation is for.

Then it quietly stopped.  No error.  No alert.  No “the report didn't run.”  Nothing.  The last entry in its log is from late August.

I only found out when I went to pull the numbers for this issue, and there weren't any to pull.  Weeks of nothing, from the thing I built to catch what goes missing.

Why did it stop?  I don't know.  There's been no root cause analysis yet, and I'm not going to guess in print.

Why didn't I restart it sooner?  The dullest possible reason: nothing told me it had stopped.  An alert was supposed to say so, but I never saw it, so as far as I knew the check was running.

Read that again, because it's the whole month in one paragraph.  The check I built to catch what my assistant misses missed its own failure.  Same shape as the smoke detector with no battery (#19).  Same shape as the integration that failed while every dashboard stayed green (#20).  A quiet instrument and a dead one look identical.  I knew that.  I'd written it down.  I still built a check that couldn't tell me it had died.

Here's what I've taken from it, and it applies well beyond AI:

Skip any one of the three and the thing you're counting on to tell you the truth will fail quietly, and you'll find out the way I did: by going to use it.

A monitor only watches what you told it to watch.  The same is true of every assistant.  And that's also where an assistant can help you the most, if you give it something to work with.

Here's the shift.  Don't start with "what should I have it watch?"  Start with the trip.

Where does it end?  Here is what "done" looks like.  Describe the finished thing: the report, the cleared queue, the decision made.  Be specific enough that a stranger could tell whether you got there.

Where does it start, and who is it for?  This data, in this condition, going to this audience.  The journey I want to complete starts here, ends there, and looks like this.

Which path?  There are always many ways to get from here to there.  Some are fast and risky.  Some are slow and safe.  Some you'd enjoy walking, and some you'd dread.  The assistant can travel any of them with you.  It can't tell you which one is safest, which one carries the least risk, or which one you'll want to be on at 6:30 on a Tuesday morning.  That one's yours to choose.

What do you check along the way, and who sees it when it fails?  This is the monitor, the test, and the visible alert, written into the plan before the trip, not bolted on after the breakdown.

So here's the question I'd put to anyone about to hand work to an AI: can you define the journey?  If you can, the assistant has something to measure itself against, and so do you.  If you can't, it will choose a path for you, and you'll find out which one on the day it breaks.

I don't have a miss rate.  The first instrument went unanswered.  The second refused to grade eight days of data.  The third, the one meant to collect the real thing, died in August and told no one.  If I printed a number today, it would be one the tools never gave me.

If a vendor has quoted you one, ask what it was measured against, and who was watching the measurement.

If someone tells you their assistant is accurate, ask five questions:

Next week closes the month.  I think the reason assistants miss is simpler than it sounds: they're built to understand your data, and they're built to understand themselves.  Almost none are built to understand you. 

There's a 1994 idea from education research that shows what that would take.

How do you know when your assistant missed something?  Reply, I read every one.

Go hug your ERP.  Then check its pulse.

Plant Floor to Cloud: The Newsletter goes out every Tuesday.  The podcast runs monthly.  Same charter, more room to think out loud.  Watch or listen on the website, or wherever you get podcasts: Apple Podcasts, Spotify, YouTube, iHeart, and more.