For years, the cloud market has made a simple promise: Move workloads to large-scale platforms, gain better resilience, and worry less about downtime. That promise was never entirely wrong, but it is becoming less complete. The latest findings from Uptime Institute’s seventh Annual Outage Analysis suggest that the outage landscape is changing in ways that should concern both cloud providers and cloud customers. The biggest risks are no longer limited to broken physical infrastructure. They are increasingly tied to the complexity of the systems used to run, coordinate, update, and recover that infrastructure.

The most alarming number in the report is that IT and networking issues accounted for 23% of impactful outages in 2024. Uptime Institute links these increases to growing IT and network complexity; the long-term shift toward colocation, cloud, and third-party digital services; and the resulting increase in change-management failures and misconfigurations. That number is more than a statistical footnote. It points to a structural change in how outages happen and why cloud outages are becoming such a stubborn problem.

Hardware redundancy can protect against component failures, but it doesn’t help much when the outage stems from a bad configuration, an automation error, a faulty network change, or an underappreciated control-plane dependency. In those cases, the infrastructure itself may remain intact while the system that governs it breaks down. The industry is learning that resiliency is less about duplicating equipment and more about managing complexity. Today’s increasingly distributed and software-defined environments cannot operate safely at scale.

Failures at the operational level

Uptime’s findings show that power remains the leading cause of major outages, underscoring that traditional infrastructure engineering still matters a great deal. But even as providers continue to improve physical resilience, outages can still arise from the digital and procedural layers above it. Cloud platforms are now dense stacks of services, APIs, orchestration systems, software-defined networks, identity controls, failover logic, and third-party dependencies. That complexity creates more possible points of interaction and more opportunities for an error in one layer to cascade into several others.

This helps explain why outages can feel more surprising today than they did a decade ago. In older data center models, an outage often had a more apparent root cause, such as a power event, a cooling failure, or a hardware fault. In cloud environments, the trigger may be a small configuration change that propagates across regions, a policy update that unintentionally blocks service communication, or a network control failure that affects seemingly unrelated services. These are not failures of raw infrastructure capacity. They are failures of complexity management.

The report’s language around change management and misconfiguration is especially important because it challenges one of the most common assumptions in the cloud market: that scale automatically produces better operational outcomes. The reality? Scale can magnify both strengths and weaknesses. Large cloud providers have more engineering talent, more sophisticated tools, and more redundancy than almost any enterprise customer. But they also run far more interconnected systems at far greater speeds with far more automation. A single process failure can have a wider blast radius.

Another important lesson from the Uptime analysis is that automation has not removed the human factor. If anything, it has changed its form. Even in highly automated environments, human error remains central to the problem. The report notes that in 2025, the share of outages caused by human failure to follow procedures rose by 10 percentage points compared with 2024. A related industry summary of the report notes that 58% of human error-related outages were caused by staff failing to follow established procedures.

That matters because cloud providers often position automation as the answer to reliability. Automation is essential, but it only works as well as the operational model that surrounds it. If teams deploy changes too quickly, rollback paths are weak, approval chains are bypassed, or procedures are incomplete, automation can accelerate failure rather than prevent it. In a modern cloud environment, a human mistake is rarely just a single keystroke. It is more often a design weakness in process, governance, testing, or accountability.

This is also why customers should resist the comforting notion that outages are somebody else’s problem once workloads move to the cloud. Provider-side mistakes remain real, but customer architectures are increasingly entangled with provider networking, identity, observability, and platform services. When an outage occurs, the customer may not have caused it, but they still bear the business impact. The shared responsibility model does not end with security. It extends to resilience planning as well.

Better change management

The Uptime data points to a clear conclusion: Cloud providers need to treat operational discipline as a first-class design requirement. That starts with better change management. High-risk changes should be tested more aggressively, staged more gradually, and accompanied by stronger rollback mechanisms. Providers also need better dependency mapping to understand how a change in one control layer can affect services far beyond its immediate scope. If the system is too complex to clearly explain, it is too complex to operate.

Providers also need to improve procedural quality. The rise in outages caused by failing to follow procedures suggests that procedures are being ignored under operational pressure or that they are too cumbersome, outdated, or unclear for real production conditions. Neither explanation is comforting. Stronger runbooks, better training, more realistic failure drills, and tighter operational guardrails are not glamorous investments; they are increasingly central to resilience.

Another pressure point is visibility. Uptime notes that software-based and distributed resiliency tools can improve availability, but they also introduce new risks and complicate root-cause analysis. Cloud providers need more transparent and faster incident diagnosis, not just more layers of abstraction. Customers cannot build trust in resilience if every major incident becomes a long exercise in reconstructing opaque service dependencies after the fact.

Design with outages in mind

What’s the financial impact of more frequent problems? Uptime’s 2024 analysis found that 54% of respondents reported that their most recent significant outage cost more than $100,000, and 20% said it cost more than $1 million. These are not edge-case losses. They show that outages remain costly even if they are less frequent than in earlier years.

Customers need to stop evaluating cloud resilience through uptime promises and start evaluating it through failure behavior. How does a provider isolate faults? How transparent is incident communication? How portable are workloads if a major service degrades? How dependent is the architecture on a single region, network path, identity service, or control plane? These are not just technical questions; they are now critical business questions.

The core lesson from Uptime’s data is simple. Outages are becoming a bigger problem for cloud providers and customers because the cloud’s biggest vulnerabilities are increasingly tied to complexity, process failures, and control-plane mistakes, not just broken infrastructure. In addition to adding redundancy, the next phase of cloud improvement will focus on building systems that are easier to understand, safer to change, and more disciplined to operate.

We’ve all heard the mantra from the quants in the business community: you can’t manage what you can’t measure. And if that’s true for human intelligence, it should be true for the artificial kind too.

How do we measure agents and large language models (LLMs)? We’re just beginning to come up with statistical metrics. Here are several of the most common metrics that designers and users toss about when they’re evaluating a model.

[ See also: 27 questions to ask before choosing an LLM ]

Time to first token

How long does it take to generate the first token? For real-time applications with time constraints, faster responses can be essential. It’s well-known that people hate waiting even a few milliseconds. The teams that develop user interfaces learned decades ago that it’s important for the software to respond quickly when a human is waiting for an answer. Even a few seconds of delay mean that the human will wander off to another window to check some email or place some bet on a prediction market. Time to first token is a good measure for models that will be working directly with the fickle human intelligences and their latent attention deficit disorder.

Time per output token

Take the total time it takes to respond and divide by the total number of tokens. The time to first token measures how long it takes to start a response and this measures the average speed as the model through all of the tokens. In basic LLMs, this value is generally fairly constant. Once the prefill is done and the LLM enters the decode phase, the output tokens usually appear at a constant stream. When the output is long enough, the startup time to first token is amortized away. In some of the more complicated architectures with loops for planning or gathering data from various tools, the average speed can vary as the model shifts in and out of making agentic decisions.

Tokens per second

This is just the reciprocal of the average time per token. Sometimes it is reported separately for different stages in the pipeline.

Throughput (requests per minute)

If a system supports more than a single user, tracking the number of different requests makes sense. These throughput numbers can be quite useful for measuring the power of some of the newer pipelines that are more efficient when they’re answering multiple prompts at the same time.

Error rate

Not every request gets an answer. The error rate tracks how often rate limits, timeouts, or model “refusals” get in the way. Better accounting tracks each independently because the number of failures in each category can be very different.

Token efficiency

Not all work tokens are visible and not all tokens are part of the final outcome. This measures how much work is done to produce the final result. As models become more complex or agentic and the pipelines become more sophisticated, the efficiency tends to drop. Agentic reasoning and strategic planning typically require more tokens that don’t appear in the final answer. This is generally a measure of how expensive a model might be to run.

Tail latency

It’s all well and good to measure the average time to answer, but in some cases a few very slow responses can really color people’s judgement. Some applications require good performance all of the time. Would you want to ride in an autonomous car that gets steering instructions very quickly “on average” instead of always? What if that’s only 99% of the time? Tail latency uses a mixture of queuing theory and detailed measurements to track the worst moments in the long tail of the latency graph. It’s useful when even occasional delays are problematic. 

Total cost of ownership

Projects that use an API or buy output from providers just look at the cost per 1M tokens. They’re effectively renters. The groups that are buying GPUs and paying for electricity, though, will add up these costs and other indirect costs like depreciation and maintenance to come up with a number that estimates how much the tokens really cost to produce. This value will depend upon demand and utilization rates—that is, on how many users are sending in prompts and how efficiently the model fits in a particular GPU and its RAM.

Parameters

Many models have numbers in their name followed by a B. This is meant to roughly capture the number of parameters, or the number of variables the model uses to generate outputs from inputs. The number “70B” means that there are about 70 billion parameters in the model. This is a good estimate for the complexity of the model and the size of the training set that has been stuffed into it. Generally bigger numbers mean a larger amount of information is hiding inside the model. It often means that it will take a bigger GPU with more RAM to generate an answer with it. It’s not a very precise number, though, because there are many other areas of the architecture that can influence whether the model can generate the answer you want inside your budget. There continue to be advances and it’s not uncommon for someone to claim that a new model with X parameters is better than an old model with 2X or 3X parameters.

Hallucination rate

While everyone wants LLMs to generate accurate output, measuring it can be difficult because deciding what’s accurate is sometimes complicated. One approach is to ask the LLM to summarize a document. Then another model evaluates how well the summary matches the original. While this may not catch all subtle slips, it will capture enough of the worst departures from reality. Some researchers have built complex test sets with curated answers. The LLMs that deliver the expected answers get the highest scores. Some common benchmarks are TruthfulQA, HaluEval, QAFactEval, and Vectara’s Hallucination Evaluation Model (HHEM).

Toxicity and bias scores

If measuring accuracy is difficult, building a metric to detect toxic or biased output is even more challenging because the definitions can be so protean. Still, some teams have built LLMs that key on particular concepts or word choices. They can detect some of the most obvious red flags that could generate political trouble. Some well-known versions include Granica Screen and Perspective API.

PII leakage

One of the biggest fears is that LLMs will somehow absorb information that may be considered personal and private. Some of the simplest measures can be as simple as regular expressions that look for the sixteen digit numbers used for credit card transactions. Many of the model builders work on eliminating personally identifiable information (PII) from the training set before beginning.

Tool-calling accuracy

As models grow more complex and agentic, they often gain access to various tools or Model Context Protocol (MCP) gateways that can help them find the best answers. Not all models take advantage of this help. The tool-calling accuracy scores count how often the models choose the best tool for the job. One particular example of this measurement is BFCL (Berkeley Function Calling Leaderboard).

Prompt sensitivity

The value captures how small changes in the language of the prompt induces the model to produce different results. It’s like a derivative from calculus class, although it’s generally computed experimentally using some collection of test prompts. There are a number of different approaches that depend upon different types of changes. Some test sets are built with small rephrasing of the request that are semantically the same. Others mix together different ways of specifying the problem, some with examples, say, and some without. Some specific examples include PromptSE and ProSA.

Semantic similarity and conciseness

Some metrics evaluate the answers by comparing them to a set of gold standard answers. This often involves feeding them to a vector embedding model and searching a retrieval-augmented generation (RAG) database for similar answers. This can track how concise or fluffy the answers might be as well as looking for how much variability might be introduced through changing parameters like the temperature. One common example is the BERTScore.

Grounding score

Many systems that combine an LLM with a vector search tool for RAG measure the effectiveness of the combination with a benchmark like the grounding score. The LLM is presented with extra data from the vector search and the benchmark measures how closely it follows this extra information. That is, how much of the answer comes from the provided source documents and how much is synthesized using the data in its training set. Some examples include RAGAS, TruLens, ARES (Automated RAG Evaluation System), RGB (Retrieval-Augmented Generation Benchmark), HaluEval, and HalluHard. A similar concept is called “context adherence,” “context precision,” “context recall,” or “faithfulness.”

Model variability

Most LLMs fold in a certain amount of random entropy, and this amount is often controlled by a parameter called the “temperature.” The model variability is a measure of how much the answers will change between runs. Some applications like chatbots require a certain amount of variability because the randomness adds a bit of “life” to the answers. Other applications like those in mission-critical areas like law or medicine will undermine confidence if the answers vary.

Format compliance rate

In some roles, LLMs are asked to produce data in strict formats like JSON or CSV. This is often important if the data will be fed into some pipeline for further processing or storage. The format compliance rate tests a number of common formats and measures how often the LLM returns semantically correct data. Agentic systems that glue together multiple LLMs and other tools rely heavily on LLMs with good scores on this benchmark.

Instruction following

Some prompts include very specific instructions and the adherence can be measured empirically. For example, some prompts will ask the LLM to produce exactly 300 words or a poem in rhyming couplets. These tests use a collection of sample prompts that ask for answers that can be easily measured. Some specific examples include IFEval, FollowBench, and the BFCL (Berkeley Function Calling Leaderboard), a value that is mentioned above in the section on tool usage.   

Subgoal success rate

As agentic models become more common, it’s helpful to track how well the model performs on each of the various parts of the agent’s strategic plan. All of the metrics here can be broken down and tracked for each of the subgoals.

Plan stability

Agentic models start with a plan. Some of them are smart enough to abandon the plan or at least adjust it as the work evolves. Plan stability measures how often the plans are adjusted. A high rate of adjustment could mean that the agent is a bad planner or just flexible or maybe both.

Self-correction score

Some agents are able to dive deeper and recognize their mistakes. The self-correction score measures how often the model will make a mistake and then recognize it, either on its own or after being prompted with the question, “Are you really sure?”

Jailbreak resistance

Some users try to find clever ways to lure the LLM into tossing aside any restrictions on topics or answers. In the past, some LLMs could be fooled by being told the answer was part of a play or a work of fiction. So discussing forbidden subjects wasn’t a problem because it was all pretend. Newer models have more elaborate defenses. Measures of the ability to resist deception include JailbreakBench, AgentHarm, and Tele-AI-Safety. 

Prompt injection vulnerability

Sometimes untrusted data from extra sources or skills may include malicious instructions that can exploit the LLM. Benchmarks such as Skill-Inject and SPIKEE (Simple Prompt Injection Kit for Evaluation and Exploitation) work with known attack vectors and measure how susceptible a model is to targeted prompt injection attacks.

Copyright infringement score

Some LLMs can regurgitate the data in their training corpus in a way that seems like plagiarism or copyright infringement. This can be an issue when the training material wasn’t carefully licensed. The copyright infringement score measures how often the LLM may parrot the training material a bit too closely. Tools for defending against this include CopyrightCatcher and DE-COP.

RULER

How well can a model extract information from the entire context? NIAH (needle-in-a haystack) benchmarks measure how well a model can retrieve small, crucial bits of information from long contexts. RULER takes NIAH tests further with the ability to vary the types and quantities of needles, the size of the haystack, and the complexity of the task.

GSM8K 

The developers of GSM8K (Grade School Math 8K) set out to benchmark an LLM’s ability to tackle multistep mathematical problems, so they gathered 8,500 problems that are common in grade school math classes. While the focus is explicitly on solving math homework problems, the benchmark also measures the ability to construct reasoning chains.

GPQA

The Graduate-Level Google-Proof Q&A is composed of hundreds of hard questions that might normally be answered by humans in graduate school, generally in science. To make the benchmark harder, the researchers focused on questions that non-experts often get wrong. The term “Google-proof” means that the benchmark includes questions that can’t be easily answered by asking a search engine.

MMLU-Pro

The MMLU-Pro benchmark builds on the Massive Multitask Language Understanding dataset to test a model’s understanding of a broad set of scientific knowledge. It includes more than 12,000 questions about general scientific fields like biology, chemistry, economics, and law. 

MBPP

Google created MBPP (Mostly Basic Python Problems) to evaluate how well a model was solving coding questions. Each problem comes with a statement, a gold standard solution, and several similar test cases. The number of accurate answers to these questions is a good measure of how well the model will solve many of the simpler Python coding problems presented by users.

SWE-bench

This collection of several thousand software engineering challenges evaluates how well a model solves programming problems. The developers created it by selecting a number of issues and corresponding pull-requests from a dozen or so Python projects. After some limitations appeared, the creators expanded the set by creating SWE-Bench+, SWE Bench Verified, and SWE-Bench Pro.

LMSYS Chatbot Arena

Instead of creating a fixed set of test prompts, the Large Model Systems Organization’s Chatbot Arena is a dynamic system that feeds the same prompt to different models and then asks humans to pick the best results. These head-to-head contests produce an Elo-like rating that is similar to the one used to score chess players.

Price

The rest of these metrics are useful, but as the real estate agents say, the three most important numbers on a property listing are price, price, and price. The cost is a bit less important for measuring AIs, but only a bit. Price can make a huge difference between a project being profitable and a moneysink. When the cost for each inference is a tad too high, it’s impossible to make it up with volume.

The key caveat is that a cheaper model isn’t a good idea if it generates answers that are filled with hallucinations or worse. The quality of the answers can differ greatly, and saving a few pennies can be a mistake. To make matters more complicated, there’s an explosion in different styles and approaches. Sometimes it makes sense to pay a bit more for a model that delivers answers with the right vibe.

Enterprises are increasingly investing copious amounts of cash in AI without a lot to show for it. This could be, in part, because the wrong people are leading the change.

As I’ve argued before⁠, AI isn’t likely to eliminate developers so much as change what we need from them. For example, we keep asking whether junior developers are needed in a world where large language models can write code faster and cheaper. What this overlooks is the reality that these younger developers and their relative inexperience may be exactly what we need to rewrite the rules of software development.

This thought hit me while reading James Governor’s riff on something Ben Griffiths⁠ wrote about our industry’s habit of confusing age with authority. Griffiths remembered sitting through a conference talk in which a speaker tried to shame a young audience for not recognizing some of the older men who had shaped computing. The irony, Ben noted, was that many of those “old men” had done their world-changing work when they were younger than the people being lectured. Bill Joy wrote vi when he was 22, John Carmack created Doom at 23, Linus Torvalds launched Linux at 22, etc. Many of our industry’s titans made their biggest contributions before they had decades of experience.

The point isn’t that young people are smarter. They’re not. The point isn’t that the key to AI success is to ignore more experienced developers. That’s dumb. Rather, it’s a suggestion that Griffiths’ larger point is right: At the beginning of big shifts, experience can be a mixed blessing. It can help you see risk, but it can also make you overconfident in old ways. The most successful enterprises will find ways to balance youthful innovation with experienced guardrails.

The factory doesn’t redesign itself

Zara Zhang recently pointed to Paul David’s classic 1990 paper, “The Dynamo and the Computer,”⁠ as a way to understand why so many companies have “adopted” AI without much to show for it. David’s argument, simplified, is that electricity didn’t immediately transform factories. For a long time, factories simply swapped out the central steam engine for an electric motor while keeping the same layout, the same workflows, and the same assumptions.

Electricity was new, but we largely stifled its potential by force-fitting it into old factory systems.

The big productivity gains came later, when factories stopped treating electricity as a cleaner steam engine and started redesigning work around smaller motors distributed throughout the factory. Once each machine could have its own motor, the factory no longer had to organize itself around a single driveshaft. Work could instead be reorganized around the flow of production.

That’s a decent description of where many enterprises are with AI. Enterprises today are buying copilot licenses by the thousands, wiring agents into existing applications, etc., and then wondering why the results are so uneven, as I’ve written⁠. This is the equivalent of swapping the steam engine for an electric one and declaring that the AI modernization work is done. It’s not. Not even close.

The real payoff won’t come from asking AI to write the same tickets a bit faster. It will instead come from changing how teams define work and how (and what) developers build. The “factory” has to change.

So here’s the uncomfortable question: Who is most likely to build the new factory?

Experience cuts both ways

There’s an obvious danger in romanticizing youth. Plenty of bad software has been written by people with unlimited confidence and limited context. Enterprises need software that works, yes, but “works” also means it complies, scales, respects security boundaries, and more.

This is where experienced developers matter. A lot.

As I pointed out recently, the agent era makes engineering judgment more important than ever. After all, AI makes it easier to generate code, but easier code generation can become easier technical debt generation. Hence, the limiting factor becomes less of “Can we create something?” and more of “Can we create the right thing, in the right place, with the right constraints?” Taste is required, in other words.

Senior engineers are often better at seeing those constraints because their experience gives them “taste.” They know why the weird validation rule exists, and they remember the customer who depended on the undocumented behavior. They understand why a simple schema change can turn into a multi-week migration.

But experience also has a shadow side, because it can make the current process feel inevitable. A senior engineer may see an AI assistant as a faster autocomplete because that’s the easiest way to fit AI into their existing mental model. A junior developer, less invested in the old workflow, may ask the more interesting questions: Why are we doing this ticket at all? Why isn’t the spec executable? Why can’t the agent generate the test harness first?

It’s not that the more experienced developers don’t know these questions. Rather, they may simply not have the energy to rage against the machine, as it were.

The value of inexperience

The worst way to use junior developers in the AI era is to treat them as cheaper versions of senior developers. That was always a bad idea, but AI makes it worse. If the job is “take this ticket, generate some code, and send it to a senior person for review,” the junior developer becomes a human wrapper around a coding assistant. That helps no one. The junior doesn’t learn much, the senior gets buried in review, and the enterprise ends up with more code, which, as I’ve said, is hardly a good thing.

Instead, junior developers should be given room to explore new workflows, with just enough oversight from experienced colleagues. That might mean giving these newer developers interesting questions to answer, such as:

• How would we redesign onboarding if every internal API had an AI-readable contract and examples that actually worked?
• How would we change code review if the agent produced a change summary, test evidence, dependency risk, and rollback plan with every pull request?
• How would we build features if product requirements were written as executable acceptance tests rather than vague prose?
• How would we reduce toil if agents could safely perform routine migrations, dependency updates, or incident triage within clearly defined boundaries?

These are not toy problems. They’re not “junior work.” They’re exactly the sort of process redesign that enterprises need but generally avoid because everyone is too busy running on the existing hamster wheel.

Finding the balance

So what should engineering leaders do? First, stop treating AI adoption as an individual productivity contest. We seem to be moving quickly away from the idea that “lots of tokens” equals “great engineer,” but the fact that we even flirted with it is damning. I love how Santiago Valdarrama eviscerates this vanity metric: “Measuring AI productivity in number of lines written is a stupid mistake. One day, everyone will have always been against this.” Instead we should be asking questions like, “What part of our software delivery process no longer makes sense?” AI’s biggest gains will come when we change how we specify, test, review, and ship software.

Second, mix up your AI workflow teams. No, not committees or PowerPoint-producing centers of excellence. I’m talking about combining two or three newer developers who are already fluent in AI-native tools with two or three senior engineers who understand production, security, architecture, and organizational constraints. Then give them a real workflow to redesign, such as dependency upgrades or test creation.

Third, make the senior engineer’s job less about saying no and more about defining the guardrails within which others can say yes. I’ve argued that golden paths are key to using AI effectively. Good senior engineers should define the paved roads: approved patterns, test requirements, observability standards, etc. Then let junior developers and agents move quickly inside those boundaries.

Fourth, reward deletion. This may be the most important point. Going back to the factory electricity metaphor, we’ll fail with AI modernization if we simply add AI without removing outdated processes.

Bring everyone to the table

The future of software development won’t belong to the young. It won’t belong to the old, either. It will belong to teams that combine the talents of both.

Newer developers often bring impatience. They’re less likely to accept the existing workflow as sacred. They’re more likely to try weird tools, compose them in unexpected ways, and wonder why enterprise software development feels like a ritualized exercise in waiting for permission.

Experienced developers bring judgment. They know that software has users, auditors, attackers, budgets, latency, history, and consequences. They know that the right answer is often boring, and boring is good.

Enterprises need both. They need the developer who asks why the factory is still organized around the old drive shaft, and they need the developer who knows which machines will kill someone if moved casually. In sum, every development team needs people who know why the old system exists… as well as those who don’t.