Killing an alert flood: from 50 iMessages an hour to signal only
Five repeat loops, one fix, and a lesson every ops engineer learns the hard way.▌
## THE NEWS THAT HIT CLOSE TO HOME
Yesterday I read about an AI agent that burned through its operator’s budget scanning a network it wasn’t supposed to — runaway loop, no circuit breaker, no dedup. I laughed for about three seconds before I remembered I had exactly that problem on my own phone. Not money, but attention. My alert system was sending 50 iMessages an hour.
## WHAT BROKE
The automation system I run on Art3ry OS watches a lot of moving parts — scheduling, follow-ups, disk health, invoice cadence, the whole back office. Each of those subsystems had its own alert path. And somewhere along the way, five of them developed repeat loops. Same event, same text, firing every few minutes. Not malicious. Just no one checking whether the message had already gone out.
At 50 texts an hour, you stop reading them. That’s the real failure mode. A noisy alert system is the same as no alert system. You start ignoring the phone and then the one real problem disappears into the feed.
## THE FIX
Went through each of the five sources and added deduplication at the send layer — track a fingerprint of the last message per channel, suppress repeats within a window. Also wired the new 36-hour ops-watch tick to fire only on anomalies. No anomaly, no text. That’s the rule it should have had from day one.
An alert that
- Comment
- Reblog
-
Subscribe
Subscribed
Already have a WordPress.com account? Log in now.
Leave a comment