While its flagship Dreamforce event was in full swing on Wednesday, Salesforce was triaging a roughly seven and a half hour-long service outage that disrupted user access and caused “severe delays” and intermittent errors. Some customers were also unable to submit new support cases.

The service outage hit at 3:50 a.m. EDT, impacting “multiple instances across all regions,” Salesforce reported. It was marked resolved right around 3 p.m. EDT, after several hours of monitoring to determine that fixes had been successful.

Salesforce initially pegged the issue to an “external dependency failure” impacting the legacy login server. A core system component experienced increased load, limiting its capacity to process requests. The company confirmed that there were no issues with third-party infrastructure.

Beyond the obvious embarrassment from the outage occurring during Dreamforce, an analyst said the incident highlights a cloud resilience problem, rather than purely a legacy one.

“Cloud does not eliminate architectural dependencies,” said Abbas Jaffery, a principal advisory director at Info-Tech Research Group. “It can sometimes make them less visible. And when the platform is your system of record, those hidden dependencies become an enterprise risk rather than simply a technology risk.”

Rolling restarts, persistent issues throughout the day

Salesforce began experiencing service issues at 3:50 a.m. EDT on September 16, and initial investigation determined that requests were stalling while waiting on responses from an internal login service that was using up available server resources.

Initially, Salesforce blocked application programming interface (API) endpoints and attempted rolling restarts to revive the service, then pushed out fixes region-by-region. At 7:20 a.m., some customers were seeing service return to normal, and Salesforce was working on a “code-level permanent fix.”

However, the rollout did not complete for a number of instances, and some automated fixes didn’t fully resolve the issue. For example, customers reported that scheduled jobs were not running as expected, even after service was restored. Salesforce manually restarted those ‌instances.

Salesforce later reported that the impact radius was “narrower than initially understood” and saw signs of recovery by around 11 a.m. EDT, with most customers coming back online.

A sub-set of Hyperforce instances were the last to be restored. Mitigations were in place across all instances by 11:39 a.m. EDT, and Salesforce continued to monitor the issue until marking the incident resolved at 2:59 p.m. EDT.

“We apologize for how this incident affected you and your business,” Salesforce posted on its incident blog. “We will undertake a full investigation of the incident, establishing the technical trigger, the underlying cause, and preventive action to avoid a repeat in the future.”

Creating a ‘temporal’ data problem

For customers for whom Salesforce is a system of record, several hours of authentication and service disruption can create a “temporal data problem,” Info-Tech’s Jaffery explained. “Events that should have happened at different points in time may occur later, fail altogether, or arrive out of sequence,” he said.

For instance, a customer interaction may occur through another channel while Salesforce is unavailable, but an integration, workflow, or scheduled process that normally records or propagates that event is unable to run.

This has several potential consequences, Jaffery said. Transactions and customer service processes are delayed; APIs and middleware may accumulate retries, timeouts, and queues. Records can become temporarily inconsistent, and scheduled jobs and workflows may be missed. Employees could lose visibility into customer history or case status, even when the underlying data has not been lost, creating a “data divergence.”

The first mistake would be to assume that just because users can log in, the incident is over, he noted. “Enterprises should move immediately into a reconciliation and integrity phase,” Jaffery advised. This means not only verifying interactive access, but APIs, integrations, scheduled jobs, queues, workflows, automation, authentication flows, and downstream systems.

Enterprises should ask what transactions failed, partially completed, or were duplicated during the outage? Which scheduled or asynchronous processes did not execute? Did integrations retry successfully, or did they create a backlog or retry storm? Are downstream systems now consistent with Salesforce?

Security teams should also validate authentication and session behavior, privileged access, integration credentials, and any emergency changes made during recovery, Jaffery explained. “The most important question is not simply ‘Is Salesforce back?’, but ‘What did the business expect to happen during the outage, and can we prove that it actually happened?,’” he said.

What to look for in post-incident reports

A credible post-incident review from Salesforce should establish a causal chain: The trigger, dependency failure, technical propagation, customer impact, detection, mitigation, recovery, and permanent corrective action, Jaffery said.

The company should be able to answer these questions, he said:

• What was the actual initiating failure and why did the failure propagate into the login path?
• Why could the affected dependency consume sufficient capacity to affect core services?
• Why didn’t isolation or failover prevent the impact?
• Why did initial remediation attempts fail and why did the subsequent rollout require additional intervention?
• What safeguards are being added to prevent recurrence?
• How will Salesforce demonstrate that the corrective action actually works under failure conditions?

Service restoration simply tells customers: “We got it working again,” he noted. But root cause analysis tells customers: “We understand why it failed, why our controls didn’t prevent it, and what has changed so that the same failure mode is less likely to recur.”

It’s not just about ‘legacy’ pieces in the stack

One architectural lesson is that a legacy component does not have to be large to be critical, Jaffery pointed out. An older authentication service can remain part of a modern stack, and therefore become a dependency for newer services.

“The component’s age matters less than its position in the dependency graph, its blast radius, and the quality of its isolation and failure handling,” he said, pointing to this incident’s progression: Requests stalled waiting on an internal login service due to increased resource consumption led to investigation into an external dependency failure, which in turn revealed impact on a legacy login server. Finally, Salesforce said, “core system components experienced increased load, which limited its capacity to process request”.

That is a classic resilience question, Jaffery pointed out: Can a failure in one dependency remain in that one dependency, or does it become a platform-wide failure?

Modernization should not be identified simply by how much old technology has been replaced, he noted, it should also measure dependency concentration, isolation, “graceful degradation,” recovery paths, and failure blast radius.

“For enterprise architects, that is the real takeaway,” he said.

Maybe driven by agentic AI, exacerbated by layoffs

At this point, there are no obvious signs that this was a security incident, noted David Shipley, CEO of Beauceron Security. “Right now, this bears all the hallmarks of an update gone horribly wrong.”

He pointed to an incident in December 2025 when Amazon’s internal AI coding agent, Kiro, caused a 13-hour AWS outage in a mainland China region, noting, “I’m not going to be shocked if we don’t see some kind of agent role in this kind of scale disaster.”

Significant Salesforce layoffs over the last few years could also have had a negative impact on the outage and recovery, he added. “Having it happen during Dreamforce had to be all kinds of hell, though, for their sales and customer support teams,” he said. “Pour one out for them as they work on rebuilding relationships, face-to-face.”

This article originally appeared on CIO.com.

AWS is betting that AI agents need a different interface as they move beyond answering prompts and start working autonomously in the background.

The company has open-sourced Pizza Bot, a self-hosted application that gives users an inbox for managing work delegated to AI agents, with separate threads for ongoing tasks and a queue for work that is completed or needs human input, rather than keeping the management of agents restricted inside a conventional chat window.

The rationale, according to AWS, is that background agents do not always need a user’s attention while they work, and an inbox model will let users hand off longer-running tasks, return to them later, and see which jobs are complete or require intervention.

Under the hood

That approach is reflected in how the inbox organizes work with the help of an “All” tab that contains the history of each task or conversation, including the agent’s messages and work performed, the “Unread” tab that flags completed work that users have yet to review, and an “Action” tab that surfaces tasks paused while waiting for user input or approval.

The inbox interface also has a panel named Activity that shows users how an agent handled a particular task along with the transcript, AWS wrote in a blog post introducing Pizza Bot.

AWS’ inbox-oriented rationale also extends to Pizza Bot’s architecture.

It uses LangChain’s Deep Agents as the harness and LangGraph as the stateful runtime, with a combination of the two allowing an agent to checkpoint its progress as it works, preserving its messages, tool activity and current state so a task can be paused and resumed rather than being tied to a live chat session, the hyperscaler wrote.

Pizza Bot’s server sits on top of that stack, connecting the agent runtime with the user interface, skills, MCP servers and the model provider, with developers able to choose from Anthropic, OpenAI, Google Gemini and Amazon Bedrock, as well as local models through Ollama, it added.

As for its out-of-the-box capabilities, the application comes with skills for working with files, browsing the web and delegating tasks to specialist agents. Developers can also add existing Agent Skills and MCP servers to give the agents access to other tools and services, the hyperscaler wrote.

Enterprise integration could remain a hurdle

However, analysts were not too convinced that Pizza Bot’s out-of-the-box skills and support for existing tools will make its implementation easier for enterprises.

While the out-of-the-box capabilities and added support for existing skills saves enterprise teams the time to build the application, it leaves out the integration work required to make it work, said Bhupendra Chopra, chief revenue officer at IT consulting firm Kanerika.

“Integration is where most of the money in an enterprise agent deployment goes. A sales or finance team gets value from an agent when it can read and update CRM, email and ERP, and each of those systems needs a connector someone has to build, secure and maintain,” Chopra pointed out.

Pizza Bot’s integration could be further complicated by the application’s lack of support or any SLA, according to Manoj Chandra Jha, principal analyst at Nord-IQ Research. That means the integration and operational burden ultimately falls on enterprises, who would be responsible for running, securing and maintaining the open source software themselves, Jha said.

Rise in enterprise productivity likely

But for enterprises willing to take on that integration work, the inbox-oriented approach could improve productivity.

“It’s a meaningful shift because it changes the economics of delegating work to agents. A chat interface requires a person’s attention throughout the task, while an inbox brings them in only when their judgment is needed, much like how executives delegate work to their teams,” Chopra said.

“Coding agents have already demonstrated this model, where an engineer assigns an issue and reviews the resulting pull request. Pizza Bot extends that approach to tasks such as meeting preparation and follow-ups,” Chopra added.

Further, the analyst pointed out that the inbox-oriented approach could give enterprise teams a clearer window into how scheduled tasks perform.

“For scheduled tasks, having the results delivered as threads gives users a way to track what happened in the background and spot failures that might otherwise go unnoticed,” Chopra said.

Out of sight, out of mind

There are risks with this approach, however: “The inbox model can make bad work less visible. When somebody is watching an agent in a chat window, they can see it going off the rails,” said Phil Fersht, CEO of HFS Research.

In contrast, when hundreds of tasks are running quietly in the background, users may not see an agent making mistakes until the task is complete or an exception requires their attention, potentially making problems harder to catch early, Fersht said.

That reduced visibility can also create approval fatigue, according to Chopra.

“An agent sending back dozens of threads with multiple approval requests can condition users to approve them without reading them closely,” Chopra said.

There is also a risk in the delay between an agent preparing an action and a user approving it: “Information that was current when the agent paused may no longer be valid hours later, potentially resulting in an outdated CRM update or a meeting invite for a slot that is no longer available,” he said. “An assumption that might be caught in a live chat can go unchallenged as the agent continues building on it and consuming model resources. Scheduled runs can also add costs without anyone watching them in real time.”

Those disadvantages can be countered by designing processes with fewer, better-timed approval points, and verifying that the underlying data is still current before allowing an action to execute, he said.

Adoption likely to start with technical teams

Such tradeoffs are likely to shape where Pizza Bot gains traction.

“Adoption will likely be bottom-up, with individual technologists and small platform teams drawn to Pizza Bot’s control and standards-based design,” Jha said. “Risk-averse industries, especially ones in the regulated sectors, and business users with limited technical expertise, however, are likely to approach the application more cautiously,” he added.

That could leave Pizza Bot occupying a more targeted role in the enterprise, as a way for technical teams to experiment with asynchronous agents and new ways of delegating work, rather than an immediate replacement for the governed interfaces and managed services enterprises typically use for business-critical processes, he said.

This article first appeared on InfoWorld.