Transcript 0:02 I'm Tony Gondek, and this is Plant Florida Cloud, the podcast. The mic's occasionally in shot, and I'm usually reading off of a script. 0:11 If the vendor pitching you the next platform won't show you their seams, I'm not gonna hide mine. Last month, I ended this show with a warning. 0:20 I told you October's newsletters would be about one question, and that in at least one of them, I was the one who got it wrong. I want to be straight with you about how that went. 0:32 I actually got it wrong twice, and the second time was worse because by then I had already written down the lesson. The question was, how would you know? 0:45 If a system you depend on suddenly stopped working this morning, what would tell you? Not what could tell you, what would? 0:59 Four Tuesdays, one integration that failed while every dashboard stayed green, a check I built to catch what my AI assistant misses, which then died without telling anybody, and a 30-year-old idea from a tutoring lab that explains better than most of what's in the current AI marketing 1:21 why assistants miss what they miss. Here's the idea underneath all of it. Quiet and fine look identical from a desk. Think about a bathroom scale that's stuck at 150. Every morning, it gives you the same number. 1:40 It's the most consistent instrument in your house, and you've learned nothing from it in a month. Plenty of what we run in manufacturing IT is that scale. It never complains. We read the quiet as good news. 1:57 So quick context, if this is your first time. This is the monthly companion to Plant Florida Cloud, the newsletter. 2:05 Same charter as every Tuesday issue, mid-market manufacturing, practitioner level detail, zero vendor hype. The newsletter tells you what happened each week. 2:16 This show pulls one thread through the entire month, and here's where we start. 2:22 [instrumental music] And now one question, and you should be able to answer it in about 15 seconds. 2:46 Think of the alert, the report, or the dashboard in your shop that you trust the most, the one you'd bet on to tell you something's wrong. When did it last go off? If you can't remember, hold on to that. 3:02 By the end of this episode, you'll know whether that's good news or bad. And if you remembered right away, good. Now ask whether you'd know if it had stopped. 3:16 Now onto our story, and I'll call this one The Morning the Inbox Got Loud. This one's mine. I told it in print, and I'll tell it again here because the print version doesn't have my voice cracking. 3:30 Picture an integration system, the plumbing that moves records back and forth between an ERP software and a CRM software at a mid-market manufacturer. I'd built the monitoring on it. It had error handling. 3:47 It had dashboards. It had the whole kit. For 18 days, it logged zero errors. That's exactly what you want to see, and it's exactly why nobody was looking. 4:01 On day 19, about 6:30 in the morning before the first cup of coffee, the first error email lands, then a second, then a third. Roughly half an hour later, there were, eh, I'd say about 47 of those error messages. 4:18 Here's what each one meant. Statuses that should have been updated on the hour didn't. 4:25 Every missed update made the next one worse, so the pile grew by the hour and every failure sent its own email, one error, one message. The inbox didn't settle down as the morning went on. It got louder. 4:41 How did I find out? I'd love to tell you that an alert did it. It didn't. I read my inbox every morning just out of habit, sort what's urgent and act upon it. That's the whole mechanism, a habit. 4:56 A habit caught it, not a system, and I caught it late. A few hours after the first message, customer service called with the sentence nobody in IT ever wants to hear, "We might have a problem." 5:12 Orders weren't getting placed. The ones that were placed weren't reaching the warehouse. Nothing shipped. No invoices went out. 5:20 Every order was fulfilled by the end of that day, and some customers waited longer than they should have. The only reason it wasn't worse is that I like reading my email. That's luck, 5:35 and luck is the one component you can't put on an architectural diagram. Now, I, I want you to enjoy this part because I certainly didn't. 5:44 In July, I'd written a newsletter issue titled Your AI Dashboard Is Green, Your Deployment Might Be Dying. I meant every word of it. 5:54 A month later, I was standing in front of a wall of green with a deployment dying behind it. They wrote the warning label. I just didn't read it. Now here's the uncomfortable part. 6:09 The dashboards were doing exactly what they were built to do. They tracked orders placed, orders fulfilled, invoices paid. Those are the right business measures. They're also lagging ones. 6:23 They tell you how the day went after the day went. Not one of them measured whether records were moving through the pipe right then and there. 6:33 So green meant nothing in the totals looks wrong yet, and every one of us, me included, read it as working. A green dashboard is a weather report for yesterday. And the cause? Almost boring. 6:50 Someone who wasn't familiar with the running service turned it off at one end. No villain, a switch that looked unimportant to the person holding it. 7:02 Once it was off, the integration and the ERP stopped talking, and nothing was watching whether they had talked. Now, I want to be clear again that this isn't a story about a person who flipped the switch. 7:15 Anyone can turn off something they don't understand. The miss was that nothing told us after the fact. And here's a detail that still bothers me. The system did its job. 7:30 It reported every single failure, forty-seven of them in email. But an email is a postcard. It lands, it sits in the pile, and it waits for someone to read it closely. 7:44 At six thirty in the morning, forty-seven postcards from the same sender look a lot like one. An alarm is something else. It's a phone that keeps ringing until somebody picks it up. 7:58 It differs from a notification in three ways. First, it knows what normal looks like. Eighteen days at zero is a baseline. Forty-seven in half an hour is a break from it. 8:14 An alert on the rate of errors would have gone off on the first burst before I'd finished my first cup of coffee. Second, it escalates until somebody answers. Not sent, answered. 8:31 Third, it has an owner by name, a person and a backup, not a distribution list, not a team. Those are polite ways of saying nobody. 8:45 The week before in the newsletter, I'd given readers three checks for this exact situation, a heartbeat, a fire drill, and a volume floor. This integration had none of them. 8:57 Zero for three on the exact checks I was about to tell you to run. How did that feel? In a word, ugh. Every new system has its critics, and I had just handed them one more reason. 9:15 The orders got recovered by the end of the day. The trust lost takes so much longer. So here's what we changed afterwards. 9:26 A heartbeat, a check that confirms the integration and the ERP are actually talking, not just that each is operational. 9:36 A volume floor watched daily with anything odd going straight to a chat channel the team actually reads, not to an inbox. 9:46 And monitoring that follows the records from start to finish, instead of only counting the totals at the end. The fire drill and the tabletop are still on my list. They're not done yet. 9:58 I'd rather tell you that than pretend otherwise. And then a few weeks ago, it happened again. A small change in how one integration account was named stopped orders cold. Same kind of break, same quiet start. 10:15 This time, it was caught and fixed in under thirty minutes. Not because I was reading email, but because the heartbeat we added noticed. Same failure, different instrument. 10:29 When I sat down to close the month, I wanted something you could run on your own list, so I came up with four questions, each earned by this story, and I'm restating them again as tests. The lights on test. 10:48 Is the check itself running? A monitor can lose its password, get its schedule turned off during an unrelated change, or have that service underneath it quietly stop. From your desk, none of those makes a sound. 11:03 The fix is a heartbeat. The monitor sends a small still here signal on a schedule, and a second independent check watches for it. When the signal doesn't arrive, that's the alert. 11:19 Think of a radio check on a job site. Nobody cares what you say. They care that you said something. The dry run test. Would it actually catch anything? 11:36 Running isn't the same as working. A check can run every hour against a field somebody renamed and report no problems forever. So break something on purpose. 11:49 On a schedule, feed it a known failure in a test environment and confirm the alert fires and reaches a person. An alert nobody has tested is a fire extinguisher nobody has ever pulled the pin on. 12:04 You're trusting it, and you have no evidence. Then do the people version. Sit the team down. The alert fires at two on a Saturday morning. Who gets it? What do they do first? And who's the backup if they don't answer? 12:21 The drill proves the alert works. The tabletop proves the response does. You have the flow test. Is anything moving? Some failures don't produce errors. They produce nothing. 12:38 If an integration normally moves a few hundred records a day and today it moves zero, every error alert stays green because zero records means zero errors. So alert on too little. 12:55 Look back over a few months. Find the lowest normal day and set the floor just under it. Make it a number, not a feeling. And then finally, the answered test. When it fires, does a named person answer it? 13:11 An alert is only as good as where it lands. If it lands in an inbox nobody watches closely, the fix isn't to watch email harder. 13:22 It's to find the channel people actually see, a screen on the wall, a chat channel, a text to the on-call cell phone. For the truly critical ones, a call that keeps ringing. 13:36 Pick the channel that gets answered, not the one that was easiest to configure. Now, if you've been listening since last month, the answered test is a cousin of the day after test. 13:50 Last time, the question was who owns the thing the day after you leave the room? This time, it's who notices when it stops the day after you leave. None of the four is expensive. 14:07 Most are a scheduled job and a rule. They still don't get built, and I think there are three reasons why. Number one, they produce nothing on a good day. 14:18 A heartbeat that works never gets noticed, so it's hard to get credit for and harder to get funded. Number two, they look like work for a problem you don't have. 14:31 Nobody asks for a monitor on the monitor until the day it would have mattered. And three, nobody owns them. The team that built the integration owns the integration. 14:43 The alert belongs to whoever set it up, and when that person moves on, it belongs to no one. That's the same shape as the firework and the habit from all of these previous months. 14:55 A check like this is the quietest habit in the building. It gets no ribbon cutting. 15:01 Its whole job is to make one specific Tuesday morning a non-event, and nobody will ever throw a party for the Tuesday that didn't happen. 15:12 But wait, there's another story I have to tell, and it's called grading the silence. Now, the part where I got it wrong the second time, I run an AI assistant on my own work. 15:27 It reads my email, my calendar, and my queue of support cases. Every morning, it sends me a briefing. Every evening, a wrap on what got done. 15:38 The question I care about isn't whether those br- briefings are right, it's what they leave out. If something lands in my inbox and the briefing skips it, I wouldn't know it skipped it. 15:52 So here's the distinction in plain terms. When someone says their assistant is accurate, they usually mean of the things it said, most were right. That's the easy half. You can check it by reading the output. 16:08 Every answer is sitting right there, and you can grade it. The hard half is the opposite question. Of the things it should have told you, how many did it? 16:21 An assistant can be right every single time it speaks and stay silent about the one thing that mattered this week. Nothing in the output points at what's missing from it. 16:33 To count misses, you need an answer key, a separate list of everything that it was there to catch in a classroom. Somebody writes the key ahead of time. In your inbox and in your ticket queue, nobody does. 16:48 The only complete record of what mattered this week is the person who lived it. So I tried three instruments to get to that point. The first one was the obvious one. 16:59 Once a week, the AI assistant would ask me, "Did I miss anything?" I didn't answer for the longest time. Not once from early July to mid-August. The row on my scorecard read "no data" for six weeks straight. 17:17 The cause was never a bad week. The design was the fault. The only instrument for measuring what it missed cost me exactly the thing it exists to save, which is my attention on a Friday afternoon. 17:33 The second instrument was cheaper. Log what the assistant tells me and when. Separately, note when the same fact reached me some other way, a meeting, an email from a person, a call. Subtract. 17:48 It cost me nothing per item. I backfilled eight days of my own records to test it. Thirty-seven items the assistant told me. Six I could verify against a second route. 18:02 The instrument has a floor, at least 30 comparable items over at least 21 days. But with six items, it declined to give me a verdict. I'll take that. 18:17 An instrument that refuses to grade itself on thin data is one I trust to grade on thick. But it did turn something up, and it wasn't a statistic. 18:27 It was that those same forty-seven error emails the assistant had said nothing about, it watched support cases, open orders, and the count of unread email. An integration error is none of those. 18:42 It's email from a system in a mailbox where the only thing counted is how many are unread. The assistant wasn't wrong about anything it said. It had just never been pointed at that specific channel. 18:59 That's what a miss looks like from the inside, not a wrong answer, a place nobody was looking. The third instrument was an automated check once a week to do that comparison for me. It ran for a while, 19:18 then it stopped. No error, no alert. The last entry in this log is from late August. I only found out when I went to pull numbers for the newsletter and there weren't any. I don't know why it stopped. 19:33 There's been no root cause yet. But I'm not going to guess on a microphone either. Why didn't I restart it sooner? Because nothing told me it had died. An alert was supposed to say so, and I never saw it. 19:48 Now read that back. The check I built to catch what my assistant misses missed its own failure. It's like a referee who left the stadium at halftime, and nobody blew a whistle about it. I knew the lesson. 20:05 I'd written it down two weeks earlier. I still built something [laughs] that couldn't tell me it was dead. So I don't have a miss rate, and I'm not going to print one. The first intrum- instrument went unanswered. 20:18 The second declined to grade, and the third died in August. If a vendor quotes you one, ask what it was measured against and who was watching the measurement. 20:32 Which brings me to the idea that finally explained all of this for me. Every tool carries some picture of the world it works in. I think of that as three models. One, the world. What's actually happening? 20:50 Orders, records, errors, email. This is the part vendors demo, and most tools have a solid perception of that world. The integration knew exactly what failed. It logged every error. 21:07 Which then comes into number two, itself. Perception of whether the tool is working. A monitor with no heartbeat has no picture of itself, so it can't tell you it stopped. 21:21 My dead weekly check is the same gap from my own hands. And then finally, you. 21:31 Perception of what the person it serves already knows, what they've already handled, and what they'd miss without help. This is the one almost nothing has. 21:44 An assistant with no picture of you can't tell the difference between telling you something you already know, which is noise, and staying silent about something you don't, which is a miss. 21:58 From both sides, they both look fine. And now here's where that thirty-year-old idea comes in. In 1994, two researchers at Carnegie Mellon published a method called knowledge tracing. 22:15 It came out of a tutoring program that taught students to write short programs. While a student worked, the tutor kept a running estimate, skill by skill, of how likely it was that the student had learned that skill. 22:31 Every answer nudged the estimate. Then the tutor used those estimates to decide what was going to be shown next. The point for us, that tutor didn't only know the subject. 22:47 It kept a model of what this particular student already knew and updated it with every new piece of evidence. That's the missing third model. An assistant built that way wouldn't ask, "What's in the data?" 23:03 It would ask, "What's in the data that this person doesn't already have?" And then it would check whether they acted on it. Picture what that does to my own examples. 23:18 For six Fridays in a row, I didn't answer the same question. A tutor that tracked me would have read six blanks as a fact about how I work. The question is the wrong instrument. 23:33 Mine kept asking because nothing in it was keeping score on me. And forty-seven emails from one system after eighteen days of nothing is exactly where my picture of what I know is thinnest. 23:50 That's where it would have or should have spoken up. The flip side matters just as much. If I've dealt with the same kind of message every time, it should get quieter about it. 24:05 A tool with no picture of you is the coworker who tells you it's Tuesday every Tuesday. One caution, though, and it's my caution of my own version. A single observation doesn't settle anything. 24:21 A message I skip might mean I already know, or it might mean I missed it. So the estimate moves a little with each thing I do. Not all at once. And the jump from tutoring students to assistants is mine. 24:38 The 1994 paper was about students. I haven't seen an assistant built this way. That's the part I'm borrowing from, not reporting from. Now our quick hits. 24:52 These are a few smaller things on my radar, fast because each one earns a paragraph and no more. 24:58 Number one, if your only proof that a system is working is that nobody has complained about it yet, you're running on the honor system, and the honor system has a pretty poor record with computer systems. 25:12 Number two, an error email that nobody reads closely is a diary. Very detailed, written daily, read by no one. 25:21 Number three, an accuracy number with no answer key is a student who graded their own exam and called it a curve. Number four, if the fire drill has never run, you don't have an alarm. You have a theory about the alarm. 25:40 And now my industry pulse. One figure for the month, and it fits the story. In May, the Uptime Institute released its eighth annual outage analysis. 25:51 It's built mostly on data center operators, so treat it as a neighbor's report, not a map of your own plant. 25:58 The release doesn't give sample sizes, so also treat it, uh, treat the numbers as representation maybe, uh, not a, a full measurement. Here's the mood. 26:11 Failures to follow established procedures remain the leading driver of outages caused by human error, not villains. Procedures not followed, processes unclear, and what that costs. 26:27 57% of respondents said their most recent major outage cost more than $100,000. For the second year running, one in five said more than a million. Now, put that next to my August morning. 26:43 Someone switched off a service they weren't familiar with. No procedure said the thing was important and nothing said so when it stopped. I'm not claiming my outage matches their data. I'm saying the shape is familiar. 26:56 The expensive failures tend to start with a switch, a step, a process somebody didn't know mattered. Now, coming up next month. October was about the tools. 27:12 November turns out to be about the people asked to use them. Everything this month assumed that somebody would change how they work once the better system showed up. That's a big assumption. 27:25 So next month's episode centers on a different question. What happens to the people when the system changes? Who reviews the AI's work when the person reviewing doesn't have the experience to know what wrong looks like? 27:43 What happens to a tool when the person who loved it moves onto something newer? And what finally pushes a plant to leave the spreadsheet that quietly runs the whole thing? I don't have all the answers. 27:57 In at least one of them, I'm going to tell you something I'd rather not. I'm starting to think that's the show. If you've got a story about a system that went quiet and nobody noticed, reply to any newsletter issue. 28:12 I read every one. And finally, to wrap this up, four tests, two stories where I was the one who missed it, one idea that's older than most of our systems. 28:26 And if there's a single thing I took from October, it's this, silence is a claim. A system that says nothing is telling you everything is fine, and that claim deserves the same scrutiny as any number on a dashboard. 28:43 Now, I know what some of you are thinking, and it's the same thing I thought. After a month of how would you know, the easy conclusion is to stop deploying anything until you're sure. I understand that camp. 28:56 I just don't live there. The point of checking isn't to stop. It's to be able to hear. A radio on mute is perfectly reliable right up until the weather turns. 29:09 You can't build anything worth keeping with the sound off. So build, then ask the question. All four October issues and the Uptime Institute release are linked in the show notes. 29:25 Go check my work instead of trusting the person holding the microphone. That's still the point of this show. Here's what I want from you this month, and it's specific again. 29:36 Tell me about a time you found a problem by accident, not the time your monitoring caught it, the time a person happened to be looking at the right thing at the right moment. Reply to any newsletter issue. 29:52 Send enough of them and it becomes a future episode built from your stories instead of somebody else's survey. Plant Floor to Cloud, the newsletter, goes out every Tuesday. 30:03 This podcast show runs monthly right here and wherever you're listening. Subscribe. And if somebody at your shop is about to say, "No news is good news," send them this one. I'm Tony Gonda. 30:19 Go hug your ERP and then check its pulse. Talk to you next month. [outro music]