/ CONTEXT
One person, one pipeline, no ops team
I run a small digital-products business where an agent pipeline does most of the mechanical work: research, drafting, illustration, PDF rendering, publishing to a storefront, then posting to five social platforms. It runs unattended most days. That is the whole point.
Unattended is also how you find out what your monitoring actually measures. Over two weeks I hit three outages that had nothing in common technically, and everything in common structurally. Here they are, in the order I found them, with the same detail I would want if you were telling me.
/ FAILURE 01
The file every tool said was fine
One product would not publish to Instagram. Every other product went through. That asymmetry immediately rules out credentials and permissions, which kill everything or nothing, so the problem had to be the payload.
The video looked perfect. The metadata tool reported the right codec, duration, resolution and bitrate. The file downloaded over HTTPS with a 200 and the correct content type. Meta accepted the upload, chewed on it for seventy seconds, and returned a bare ERROR with no reason.
The catch is that metadata tools read headers. Headers were intact; the actual frame data was garbage, most likely from two render passes racing on the same output path. Reading the container told me nothing. Decoding every frame told me everything:
ffmpeg -v error -i output.mp4 -f null -Clean file prints nothing. The broken one printed a wall of invalid NAL units and missing pictures. Worth knowing: YouTube had accepted the same file without complaint, because it re-encodes aggressively and papers over damaged input. One platform being strict is not the platform being difficult. It is the only one telling you the truth.
The fix was not just re-rendering. Renderers now write to a temporary file, decode it fully to prove it is valid, and only then atomically move it into place. A corrupt render can no longer replace a good file.
/ FAILURE 02
The backup that quietly stopped
My offsite backup runs nightly on a systemd timer. Checking it showed a timer marked active and enabled, and a service that had exited successfully. Both green. The last snapshot in the remote repository was three days old.
What happened: one night the prune step hung on a dead connection to the storage backend. The unit had no start timeout, so systemd waited. Forever. A timer will not start a service that is already starting, so every subsequent night was skipped silently. The unit state I was reading, activating (start), is not a failure state, so nothing anywhere considered this a problem.
Two changes. A start timeout, so a stuck run is killed instead of blocking every future one. And a check that reads the age of the newest snapshot rather than the state of the unit, because the unit answers "did I run" while the business question is "do I have a recent backup". Those are different questions and only one of them matters at restore time.
/ FAILURE 03
The account that hit zero
This one is embarrassing because it is the most ordinary. The account funding every model call ran out of credit. Every LLM-dependent job began failing instantly: research, drafting, quality critique, translation. The pipeline was effectively dead for two days.
What I saw was one notification per day saying a scheduled discovery job had failed. That reads exactly like a transient upstream hiccup, which is a thing that genuinely happens and which I had trained myself to ignore. The actual message was buried in a stack trace nobody surfaces: your credit balance is too low.
The cost was not the credits. It was two days of a production system doing nothing while its dashboard stayed green, because a billing failure and a flaky API look identical from the outside.
The fix is a special case that pays for itself: scan failed-job errors for billing markers and raise a distinct, loud alert that names the provider and links its billing page. Once per provider per day, so a scheduled job cannot spam it. Billing failures are not transient and must never be shaped like transient errors.
/ THE PATTERN
Every check asked the wrong question
Three unrelated systems, one shape. In each case something reported success while producing nothing usable:
- The encoder exited zero and wrote a file that could not be decoded.
- The timer was active and had produced no snapshot in three days.
- The API returned a valid HTTP response that happened to mean the account was empty.
Every check I had was measuring the process. None was measuring the artifact. Exit codes, unit states and response codes all describe machinery, and machinery can run beautifully while producing nothing you can use.
So the rule I now apply everywhere: assert on the output, not on the run. Not "did the render finish" but "does the file decode". Not "is the timer enabled" but "how old is the newest snapshot". Not "did the call return" but "did it return content". Each of these costs a few lines and catches a class of failure that dashboards are structurally blind to.
/ IF YOU RUN AGENTS
Autonomy raises the cost of a quiet failure
This matters more with agents than with ordinary services, for one reason: nobody is watching. A web app that breaks gets reported by users within minutes. An unattended pipeline can produce nothing for days and the only witness is a log nobody reads. Autonomy converts every silent failure into a long one.
If you take one thing from this: go look at your own health checks and ask, for each, whether it could pass while the thing it monitors is completely broken. In my case the answer was yes three times, and I only found out because a customer-facing platform happened to be stricter than my own tooling.
