While its flagship Dreamforce event was in full swing on Wednesday, Salesforce was triaging a roughly seven and a half hour-long service outage that disrupted user access and caused “severe delays” and intermittent errors. Some customers were also unable to submit new support cases.
The service outage hit at 3:50 a.m. EDT, impacting “multiple instances across all regions,” Salesforce reported. It was marked resolved right around 3 p.m. EDT, after several hours of monitoring to determine that fixes had been successful.
Salesforce initially pegged the issue to an “external dependency failure” impacting the legacy login server. A core system component experienced increased load, limiting its capacity to process requests. The company confirmed that there were no issues with third-party infrastructure.
Beyond the obvious embarrassment from the outage occurring during Dreamforce, an analyst said the incident highlights a cloud resilience problem, rather than purely a legacy one.
“Cloud does not eliminate architectural dependencies,” said Abbas Jaffery, a principal advisory director at Info-Tech Research Group. “It can sometimes make them less visible. And when the platform is your system of record, those hidden dependencies become an enterprise risk rather than simply a technology risk.”
Rolling restarts, persistent issues throughout the day
Salesforce began experiencing service issues at 3:50 a.m. EDT on September 16, and initial investigation determined that requests were stalling while waiting on responses from an internal login service that was using up available server resources.
Initially, Salesforce blocked application programming interface (API) endpoints and attempted rolling restarts to revive the service, then pushed out fixes region-by-region. At 7:20 a.m., some customers were seeing service return to normal, and Salesforce was working on a “code-level permanent fix.”
However, the rollout did not complete for a number of instances, and some automated fixes didn’t fully resolve the issue. For example, customers reported that scheduled jobs were not running as expected, even after service was restored. Salesforce manually restarted those instances.
Salesforce later reported that the impact radius was “narrower than initially understood” and saw signs of recovery by around 11 a.m. EDT, with most customers coming back online.
A sub-set of Hyperforce instances were the last to be restored. Mitigations were in place across all instances by 11:39 a.m. EDT, and Salesforce continued to monitor the issue until marking the incident resolved at 2:59 p.m. EDT.
“We apologize for how this incident affected you and your business,” Salesforce posted on its incident blog. “We will undertake a full investigation of the incident, establishing the technical trigger, the underlying cause, and preventive action to avoid a repeat in the future.”
Creating a ‘temporal’ data problem
For customers for whom Salesforce is a system of record, several hours of authentication and service disruption can create a “temporal data problem,” Info-Tech’s Jaffery explained. “Events that should have happened at different points in time may occur later, fail altogether, or arrive out of sequence,” he said.
For instance, a customer interaction may occur through another channel while Salesforce is unavailable, but an integration, workflow, or scheduled process that normally records or propagates that event is unable to run.
This has several potential consequences, Jaffery said. Transactions and customer service processes are delayed; APIs and middleware may accumulate retries, timeouts, and queues. Records can become temporarily inconsistent, and scheduled jobs and workflows may be missed. Employees could lose visibility into customer history or case status, even when the underlying data has not been lost, creating a “data divergence.”
The first mistake would be to assume that just because users can log in, the incident is over, he noted. “Enterprises should move immediately into a reconciliation and integrity phase,” Jaffery advised. This means not only verifying interactive access, but APIs, integrations, scheduled jobs, queues, workflows, automation, authentication flows, and downstream systems.
Enterprises should ask what transactions failed, partially completed, or were duplicated during the outage? Which scheduled or asynchronous processes did not execute? Did integrations retry successfully, or did they create a backlog or retry storm? Are downstream systems now consistent with Salesforce?
Security teams should also validate authentication and session behavior, privileged access, integration credentials, and any emergency changes made during recovery, Jaffery explained. “The most important question is not simply ‘Is Salesforce back?’, but ‘What did the business expect to happen during the outage, and can we prove that it actually happened?,’” he said.
What to look for in post-incident reports
A credible post-incident review from Salesforce should establish a causal chain: The trigger, dependency failure, technical propagation, customer impact, detection, mitigation, recovery, and permanent corrective action, Jaffery said.
The company should be able to answer these questions, he said:
- What was the actual initiating failure and why did the failure propagate into the login path?
- Why could the affected dependency consume sufficient capacity to affect core services?
- Why didn’t isolation or failover prevent the impact?
- Why did initial remediation attempts fail and why did the subsequent rollout require additional intervention?
- What safeguards are being added to prevent recurrence?
- How will Salesforce demonstrate that the corrective action actually works under failure conditions?
Service restoration simply tells customers: “We got it working again,” he noted. But root cause analysis tells customers: “We understand why it failed, why our controls didn’t prevent it, and what has changed so that the same failure mode is less likely to recur.”
It’s not just about ‘legacy’ pieces in the stack
One architectural lesson is that a legacy component does not have to be large to be critical, Jaffery pointed out. An older authentication service can remain part of a modern stack, and therefore become a dependency for newer services.
“The component’s age matters less than its position in the dependency graph, its blast radius, and the quality of its isolation and failure handling,” he said, pointing to this incident’s progression: Requests stalled waiting on an internal login service due to increased resource consumption led to investigation into an external dependency failure, which in turn revealed impact on a legacy login server. Finally, Salesforce said, “core system components experienced increased load, which limited its capacity to process request”.
That is a classic resilience question, Jaffery pointed out: Can a failure in one dependency remain in that one dependency, or does it become a platform-wide failure?
Modernization should not be identified simply by how much old technology has been replaced, he noted, it should also measure dependency concentration, isolation, “graceful degradation,” recovery paths, and failure blast radius.
“For enterprise architects, that is the real takeaway,” he said.
Maybe driven by agentic AI, exacerbated by layoffs
At this point, there are no obvious signs that this was a security incident, noted David Shipley, CEO of Beauceron Security. “Right now, this bears all the hallmarks of an update gone horribly wrong.”
He pointed to an incident in December 2025 when Amazon’s internal AI coding agent, Kiro, caused a 13-hour AWS outage in a mainland China region, noting, “I’m not going to be shocked if we don’t see some kind of agent role in this kind of scale disaster.”
Significant Salesforce layoffs over the last few years could also have had a negative impact on the outage and recovery, he added. “Having it happen during Dreamforce had to be all kinds of hell, though, for their sales and customer support teams,” he said. “Pour one out for them as they work on rebuilding relationships, face-to-face.”
This article originally appeared on CIO.com.
Login
