The quiet cost of operational drift
Almost every incident was drift first: a stale doc, a skipped step, a number that quietly stopped matching. Here is how small gaps compound into failure, and what it takes to catch them while they are still cheap.
Outages rarely begin as outages. They begin as a doc nobody updated, a step quietly skipped under pressure, a threshold that drifted a few points past where it should have stayed. By the time anyone files an incident, the gap has been open for days.
The shape of drift
Drift is the distance between how your operation is supposed to run and how it actually runs on a given Tuesday. It is not a bug and it is not an outage. It is the slow, ordinary divergence that precedes both. A runbook that no longer matches the service. An access grant that outlived the project. A reconciliation that used to balance and now rounds away a few pounds a week.
None of these are alarming in isolation. That is exactly the problem. Each one sits below the threshold that would trigger a page, so nothing fires, and the gap simply waits.
Almost every incident was drift first. The failure is just the moment the gap finally got expensive enough to notice.
Why it stays invisible
Most monitoring is built to catch events, not divergence. A dashboard will happily tell you a number moved; it will not tell you the number was supposed to hold, or that the move means a process has quietly stopped working. Observability shows you state. It does not encode intent.
So the work of noticing drift falls to people, and people are busy, rotate off, and reasonably assume that if something were wrong, an alert would have said so. The result is a blind spot that widens precisely where things are calm.
A useful test: for any signal you collect, can you say what it should be right now, not just what it is? If not, you can observe that thing but you cannot detect drift in it.
The cost curve
Drift compounds, and so does the cost of fixing it. Caught the day it appears, a stale permission is a one-line correction. Caught a quarter later, after three systems have come to depend on the wrong state, it is a migration, a review, and an awkward conversation.
The curve is steep because each unaddressed gap becomes load-bearing. Other work quietly builds on top of the wrong assumption, and unwinding it later means unwinding everything that trusted it. Early correction is cheap not because the fix is smaller, but because nothing has grown on top of it yet.
Catching it while it is cheap
The leverage is entirely in the timing. To catch drift while it is still cheap, you need two things most operations lack: a model of how the operation is meant to run, and a continuous comparison of live signals against that model.
That is the whole job. Encode intent once (the tools, the workflows, the thresholds, the relationships), then score every incoming signal against it. When a signal disagrees with the model, you have found drift at the moment it opened, not the quarter it failed.
Detection is not the same as action. Surfacing a gap early is valuable on its own; deciding what to do about it (and how much an agent may do without you) is a separate, deliberate choice.
Closing the loop
Detection only pays off if it feeds back in. Every correction you approve can become a playbook: the next time the same kind of gap appears, the response is already known and, within the bounds you set, already handled. The operation gets a little more self-correcting each week.
That is the loop we keep coming back to: observe real behaviour, detect drift, correct continuously. The quiet cost of operational drift is simply what it costs to run without it.