❌

Normal view

GitHub Outage

28 August 2026 at 19:47

Vlad Fedorov (Hacker News):

On August 17, GitHub experienced an outage that lasted 7 hours and 47 minutes. It disrupted github.com, authentication, GitHub Actions, APIs, pull requests, issues, and Copilot, affecting developers and organizations around the world.

[…]

Neither outage was caused by a code or configuration change. Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. Since April, monthly commits have grown from 1.4 billion to 2.9 billion. That growth explains the pressure on our systems, but it does not excuse these outages.

[…]

As part of the reliability commitments we made earlier this year, we have focused on three priorities: adding capacity, improving efficiency, and removing architectural bottlenecks. We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.

Alejandro Forero Cuervo:

This is the wrong way to think about this because there’s no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not “a component didn’t have enough capacity (because of auto scaling failures)”, but rather “this complex system collapses (rather than degrade gracefully) when demand exceeds capacity”.

When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.

Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections. Most other large internet services have since copied them, afaik.

Previously:

Microsoft almost gave away the keys to everyone’s Azure Cosmos DBs

31 July 2026 at 14:16

Microsoft has had a narrow escape from total embarrassment: A security company uncovered a critical vulnerability that could have compromised all Azure Cosmos DB databases — both those of customers and Microsoft’s own.

Google subsidiary Wiz found a flaw in the database’s Gremlin API, usually used for storing and managing property graph data.

If bad actors had discovered it first, they could have exploited it to acquire what Wiz called the Cosmos Master Key, which would have enabled them to use the primary key of any Cosmos database, resulting in read and write access to any account. They would also have had access to a list of every database on the service, with identifiers such as subscription and tenant IDs.

Azure Cosmos DB is a NoSQL database that underpins Microsoft’s cloud services. It can be accessed through SDKs for framework such as Python, Node.js, Java, and .NET.

Wiz described how it discovered the vulnerability in a blog post. It disclosed details of the flaw to Microsoft in November 2025.

Microsoft deployed a hot fix within two days, but it took another eight months to re-engineer the infrastructure, removing the Cosmos Master Key and introducing new guardrails to Cosmos DB to prevent similar attacks.

It is not the first time Cosmos DB customers’ primary keys have been under threat: In 2021, Wiz found a flaw in data exploration tool Jupyter Notebook that could be exploited to access the database keys and other secrets.

This article first appeared on CSO.

Microsoft’s Web IQ aims to give enterprise AI agents real-time web intelligence

4 June 2026 at 19:13

For the past two years, enterprises have focused on grounding AI systems in internal documents, databases and knowledge repositories. Microsoft now contends that the next challenge is giving those systems reliable access to the outside world as they move into production.

At its ongoing annual Build conference, Microsoft unveiled Web IQ, a new suite of AI-native APIs designed to connect AI agents and applications to real-time information from across the web, including web pages, news, images and videos.

The goal is to help developers build more accurate and context-aware AI systems while reducing the complexity of integrating web search, retrieval and grounding capabilities into enterprise applications, the company wrote in a blog post.

The APIs already underpins grounding for Microsoft Copilot and ChatGPT, and unlike traditional search APIs are designed to retrieve highly relevant information while minimizing token consumption, helping reduce both inference costs and response latency, Microsoft said.

Reducing the cost and complexity of web grounding

That focus on reducing inference costs and response latency to deliver Web IQ’s search capabilities will be valuable for CIOs and developers, said Phil Fersht, chief analyst at HFS Research.

“Developers have typically stitched this together themselves using search APIs, web scraping, retrieval-augmented generation, vector databases, custom ranking logic, crawling tools and separate orchestration layers. That works, but it is messy, brittle and expensive to maintain,” he said.

“The hard part is not just finding a web page. It is retrieving the right evidence, ranking it, selecting useful passages, reducing token waste, respecting publisher controls and doing all this fast enough for multi-step agents,” he added.

An agent running in production will typically run at least five or ten retrieval steps and in such scenarios latency and cost can become “ugly,” said Moor Insights and Strategy principal analyst Mike Leone.

“I’ve watched plenty of teams handle this by gluing a search API onto a model and hoping the model can sort through whatever HTML comes back, which isn’t ideal,” he said.

Simplifying development of applications

Beyond the infrastructure and cost advantages, Web IQ could also simplify the AI application building process developers, said Stephanie Walter, practice lead of the AI stack at HyperFrame Research.

“The value of Web IQ is that Microsoft is trying to make web grounding a reusable, agent-native service rather than a bespoke integration,” Walter said.

That reusability, Fersht said, will matter to CIOs as most enterprises do not want every development team rebuilding its own web grounding stack: “They need a common capability that is governed, scalable, low-latency and economically efficient.”

Still, Microsoft is not entering an untapped market and there are several offerings already available, including OpenAI web search, Google grounding, Perplexity APIs, Bing Search APIs, Azure AI Search, vector databases, and custom RAG pipelines.

However, Microsoft might have an advantage with IQ, despite Google’s strength in search and web-scale knowledge, OpenAI’s tooling push, and AWS’ Bedrock-data-service-partner integrations combo, Fersht said.

“Microsoft’s differentiation is that it can combine Bing’s global web index with Azure AI, Copilot, Foundry, Microsoft 365 and enterprise developer channels,” he said.

The other advantage is the existence of existing IQ offerings, such as Work IQ, Fabric IQ, and Foundry IQ, all of which are targeted towards adding more business context for agentic systems, Fersht added.

But uptake, Walter said, will depend on whether these integrations translate into better production outcomes.

Web IQ is currently available in limited access for select Azure customers. Enterprises can request access to Web IQ via their Microsoft account team or submit a form.

Beyond direct API access, developers can also configure the model-agnostic Web IQ as an MCP tool within Foundry IQ, a Microsoft spokesperson said.

❌