Login
Why observability doesn’t explain what happened
I’ve had a version of the same conversation probably 30 times in the past year. A VP of engineering or a senior SRE describes their observability stack — the dashboards they’ve built, the alerting they’ve tuned, the Datadog or Grafana setup they’ve spent months getting right. And then they describe the last major incident. How the alert fired in seconds. How the dashboard showed exactly where the degradation was. And how their team still spent two to three hours figuring out what caused it before they could fix anything.
The framing I hear most often goes something like this: “We can see everything. We just can’t explain it.”
That gap between seeing and explaining is worth digging into, because it’s quietly become one of the more expensive problems in operations.
What observability was built to answer
Observability tools are excellent at what they were designed for. They surface what is happening inside systems — latency spikes, error rates, resource exhaustion, dependency failures — and they do so at scale, in real time. That capability is foundational. Without it, you wouldn’t know something broke until a customer told you.
But observability was built around a specific question, what is happening inside the system right now? It answers that question well. A different question, why is this happening and what triggered it, sits mostly outside its reach.
Those sound similar. They are not the same question.
“What is happening” lives in the infrastructure layer: metrics, traces, logs. The observability stack captures it. “Why is it happening” often lives outside the infrastructure entirely, in a deployment that went out two days ago, in a support queue that started spiking before the alert fired, in a change record that engineering approved last sprint. That context isn’t in the telemetry because it was never meant to be.
The investigation that precedes the fix
When a customer-facing incident occurs, the information that would explain it is rarely concentrated in one place.
Engineering is looking at the infrastructure signal. They know what the system is doing, but not which customers are affected or what those customers reported before the monitoring alert fired.
Support has the cases, the complaints written in plain language by people describing symptoms they can’t name technically. Support often knows something is wrong before engineering does, but the deployment history isn’t a system they can search.
Somewhere in Jira or a CI/CD pipeline is a record of what changed and when, with a timestamp, an author, and a scope. That record doesn’t know about the support cases or the anomaly.
So somebody has to connect all of it by hand. That means reading the customer complaints, cross-referencing the deployment history, holding multiple systems in their head simultaneously until the pattern surfaces. That is what the investigation actually is; not problem-solving, but reconstruction.
It takes hours. And it runs entirely before the resolution process can begin. Mean time to resolution (MTTR), the metric most teams are tracking and trying to improve, measures what happens after the investigation ends. The investigation itself is outside the frame.
Who actually does this work
This is the part I think gets systematically under-reported.
The manual correlation work does not fall to anyone with spare capacity. It routes to whoever knows the system well enough to read across these different views simultaneously. The engineer who remembers what shipped last week and why it was unusual. The support lead who can translate a customer’s description of a problem into a technical hypothesis. The person who has been around long enough to recognize the pattern.
That person gets pulled from whatever they were doing. Not because they are available, but because the investigation requires them specifically.
In my experience this is not an edge case. It is a structural feature of how incident response works when the relevant signals are distributed across systems that do not talk to each other. The investigation always finds the person with the most context. And that person is never the most available; they are the most consequential.
The hours they spend reconstructing a timeline are hours they are not doing the other things only they can do. That cost is real and it compounds. It shows up in velocity. In recurrence. In the quiet attrition of senior people who have been in the war room one too many times. It does not show up in MTTR or any incident metric most organizations track.
Two kinds of work
There is a distinction that rarely gets made in incident response conversations: the difference between investigation work that requires human judgment and investigation work that is, at its core, data assembly.
Deciding whether to patch or roll back requires judgment. So does knowing which customers are actually affected and which alerts are noise. These are high-context calls that require judgment that takes years to develop.
Cross-referencing timestamps across Jira, Salesforce, and a deployment dashboard to determine which change shipped before which customer complaint started; that is data assembly. It is tedious and time-consuming. It is what consumes the first two to three hours of almost every serious incident.
The problem is these two types of work have always arrived bundled together. The person who can make the judgment call also has to do the assembly, because no system was built to do the assembly for them. So your most experienced engineer — the one whose judgment you actually need — spends the majority of the investigation doing work that is far below their capability. Work that is numbing, not demanding. Work that clears their day and produces, at the end of it, a starting point for the actual decision.
That’s the waste that rarely gets named. Not the hours themselves, but the quality of what those hours contain.
What changes when they’re separated
The signals that would explain why an incident happened are already sitting in your systems. The customer cases, the deployment records, the infrastructure signals. They were there before the alert fired.
The investigation cost is a correlation problem, not a data problem. The data exists. It has just never been assembled automatically.
This is where a newer category of tooling is starting to take shape, sometimes described as operational intelligence: software that sits across observability, ticketing, and deployment systems and does the cross-referencing automatically, before a human ever opens a second tab.
When that assembly work is separated from the judgment work, the picture changes. The engineer in the war room arrives with a structured view of what happened, the deployment, the cases, the timeline. Their job becomes confirming it, refining it, and deciding what to do next. That’s the work they’re actually good at, and the reason they became engineers in the first place.
The organizations that get ahead of this won’t necessarily have better engineers or better observability stacks. They’ll have recognized that the investigation itself is a workflow, not an inevitability, and that data assembly and human judgment are separable. As AI-driven correlation tools mature, that separation is where the real leverage in incident response is likely to come from.
The observability stack tells you an incident is happening. What teams need next is the context that explains why. One of those capabilities has matured significantly over the past decade. The other is where the work is now.
—
New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.
