While its flagship Dreamforce event was in full swing on Wednesday, Salesforce was triaging a roughly seven and a half hour-long service outage that disrupted user access and caused “severe delays” and intermittent errors. Some customers were also unable to submit new support cases.

The service outage hit at 3:50 a.m. EDT, impacting “multiple instances across all regions,” Salesforce reported. It was marked resolved right around 3 p.m. EDT, after several hours of monitoring to determine that fixes had been successful.

Salesforce initially pegged the issue to an “external dependency failure” impacting the legacy login server. A core system component experienced increased load, limiting its capacity to process requests. The company confirmed that there were no issues with third-party infrastructure.

Beyond the obvious embarrassment from the outage occurring during Dreamforce, an analyst said the incident highlights a cloud resilience problem, rather than purely a legacy one.

“Cloud does not eliminate architectural dependencies,” said Abbas Jaffery, a principal advisory director at Info-Tech Research Group. “It can sometimes make them less visible. And when the platform is your system of record, those hidden dependencies become an enterprise risk rather than simply a technology risk.”

Rolling restarts, persistent issues throughout the day

Salesforce began experiencing service issues at 3:50 a.m. EDT on September 16, and initial investigation determined that requests were stalling while waiting on responses from an internal login service that was using up available server resources.

Initially, Salesforce blocked application programming interface (API) endpoints and attempted rolling restarts to revive the service, then pushed out fixes region-by-region. At 7:20 a.m., some customers were seeing service return to normal, and Salesforce was working on a “code-level permanent fix.”

However, the rollout did not complete for a number of instances, and some automated fixes didn’t fully resolve the issue. For example, customers reported that scheduled jobs were not running as expected, even after service was restored. Salesforce manually restarted those ‌instances.

Salesforce later reported that the impact radius was “narrower than initially understood” and saw signs of recovery by around 11 a.m. EDT, with most customers coming back online.

A sub-set of Hyperforce instances were the last to be restored. Mitigations were in place across all instances by 11:39 a.m. EDT, and Salesforce continued to monitor the issue until marking the incident resolved at 2:59 p.m. EDT.

“We apologize for how this incident affected you and your business,” Salesforce posted on its incident blog. “We will undertake a full investigation of the incident, establishing the technical trigger, the underlying cause, and preventive action to avoid a repeat in the future.”

Creating a ‘temporal’ data problem

For customers for whom Salesforce is a system of record, several hours of authentication and service disruption can create a “temporal data problem,” Info-Tech’s Jaffery explained. “Events that should have happened at different points in time may occur later, fail altogether, or arrive out of sequence,” he said.

For instance, a customer interaction may occur through another channel while Salesforce is unavailable, but an integration, workflow, or scheduled process that normally records or propagates that event is unable to run.

This has several potential consequences, Jaffery said. Transactions and customer service processes are delayed; APIs and middleware may accumulate retries, timeouts, and queues. Records can become temporarily inconsistent, and scheduled jobs and workflows may be missed. Employees could lose visibility into customer history or case status, even when the underlying data has not been lost, creating a “data divergence.”

The first mistake would be to assume that just because users can log in, the incident is over, he noted. “Enterprises should move immediately into a reconciliation and integrity phase,” Jaffery advised. This means not only verifying interactive access, but APIs, integrations, scheduled jobs, queues, workflows, automation, authentication flows, and downstream systems.

Enterprises should ask what transactions failed, partially completed, or were duplicated during the outage? Which scheduled or asynchronous processes did not execute? Did integrations retry successfully, or did they create a backlog or retry storm? Are downstream systems now consistent with Salesforce?

Security teams should also validate authentication and session behavior, privileged access, integration credentials, and any emergency changes made during recovery, Jaffery explained. “The most important question is not simply ‘Is Salesforce back?’, but ‘What did the business expect to happen during the outage, and can we prove that it actually happened?,’” he said.

What to look for in post-incident reports

A credible post-incident review from Salesforce should establish a causal chain: The trigger, dependency failure, technical propagation, customer impact, detection, mitigation, recovery, and permanent corrective action, Jaffery said.

The company should be able to answer these questions, he said:

• What was the actual initiating failure and why did the failure propagate into the login path?
• Why could the affected dependency consume sufficient capacity to affect core services?
• Why didn’t isolation or failover prevent the impact?
• Why did initial remediation attempts fail and why did the subsequent rollout require additional intervention?
• What safeguards are being added to prevent recurrence?
• How will Salesforce demonstrate that the corrective action actually works under failure conditions?

Service restoration simply tells customers: “We got it working again,” he noted. But root cause analysis tells customers: “We understand why it failed, why our controls didn’t prevent it, and what has changed so that the same failure mode is less likely to recur.”

It’s not just about ‘legacy’ pieces in the stack

One architectural lesson is that a legacy component does not have to be large to be critical, Jaffery pointed out. An older authentication service can remain part of a modern stack, and therefore become a dependency for newer services.

“The component’s age matters less than its position in the dependency graph, its blast radius, and the quality of its isolation and failure handling,” he said, pointing to this incident’s progression: Requests stalled waiting on an internal login service due to increased resource consumption led to investigation into an external dependency failure, which in turn revealed impact on a legacy login server. Finally, Salesforce said, “core system components experienced increased load, which limited its capacity to process request”.

That is a classic resilience question, Jaffery pointed out: Can a failure in one dependency remain in that one dependency, or does it become a platform-wide failure?

Modernization should not be identified simply by how much old technology has been replaced, he noted, it should also measure dependency concentration, isolation, “graceful degradation,” recovery paths, and failure blast radius.

“For enterprise architects, that is the real takeaway,” he said.

Maybe driven by agentic AI, exacerbated by layoffs

At this point, there are no obvious signs that this was a security incident, noted David Shipley, CEO of Beauceron Security. “Right now, this bears all the hallmarks of an update gone horribly wrong.”

He pointed to an incident in December 2025 when Amazon’s internal AI coding agent, Kiro, caused a 13-hour AWS outage in a mainland China region, noting, “I’m not going to be shocked if we don’t see some kind of agent role in this kind of scale disaster.”

Significant Salesforce layoffs over the last few years could also have had a negative impact on the outage and recovery, he added. “Having it happen during Dreamforce had to be all kinds of hell, though, for their sales and customer support teams,” he said. “Pour one out for them as they work on rebuilding relationships, face-to-face.”

This article originally appeared on CIO.com.

Microsoft’s top lawyer argued Tuesday that legislators “must make secrecy [orders] the exception” in government subpoenas demanding information about LinkedIn users.

LinkedIn, which is owned by Microsoft, is fighting what it calls overly broad subpoena demands from the US government, which sometimes come with secrecy orders that prevent LinkedIn from alerting customers whose information is being requested.

The company is asking federal courts “to enforce meaningful limits on both the scope of government demands and the secrecy that can accompany them,” wrote Jon Palmer, Microsoft’s chief legal officer, in a Tuesday blog post. “We recognize law enforcement’s important role in protecting public safety and investigating crime, and sometimes that does need to be done covertly. At the same time, customers and users deserve meaningful limits and independent oversight through an adversarial process.”

He pointed out: “People and organizations increasingly entrust their most sensitive information to online services. If providers cannot challenge demands they know are overbroad—or if courts may silence them without a rigorous, adversarial review—the safeguards the law requires will be weakened precisely when they are most needed.”

A tricky issue

The issue is a tricky one. Law enforcement often use this type of subpoena as an investigative tool, seeking those who are engaged in illegal activities. The theoretical justification for secrecy is to avoid alerting the investigative target to make it less likely the suspect will try to destroy evidence or flee the jurisdiction.

Government lawyers are supposed to only make secrecy requests when absolutely essential. Microsoft is suggesting that courts and congress need to step in to curtail blanket government efforts. 

“The Fourth Amendment protects the right to be free from unreasonable searches and seizures. That right applies to papers kept in a desk and it also applies when personal and business records are stored online,” Palmer wrote. “Online service providers, like LinkedIn and Microsoft, also have a First Amendment right to speak to their customers when the government obtains an order to search their private information. Secrecy may sometimes be justified, but it should be tailored to demonstrated needs and subject to meaningful review.”

He added: “The government must seek only relevant information, justify secrecy with specific evidence and infringe on speech to the least extent possible.” In his post, he pointed to a recent legislative effort in the US House of Representatives that might mitigate the issue if it ends up becoming law. 

“On August 31, the House passed legislation to rein in secret surveillance and strengthen notice protections when the government seeks data held by technology providers,” he wrote. “The reforms would place clearer limits on secrecy orders, require greater accountability, and help ensure that secrecy is the exception – not the rule. The Senate should act promptly to send these historic reforms to the President.”

LinkedIn privacy battles

LinkedIn itself is currently fighting litigation that accuses it of directly violating the privacy rights of its customers, and a federal judge this month dismissed another similar case, but gave plaintiffs permission to refile, with a caveat.

“Given LinkedIn’s further arguments that users voluntarily download browser extensions, which by their nature intentionally expose data to websites, it seems unlikely that the plaintiffs will ever be able to allege a privacy violation, much less prevail at the end of the day,” US District Court Judge Vince Chhabria wrote. “But in an abundance of caution, dismissal is with leave to amend.”

But, he added, if the amended complaint isn’t filed within 14 days, “dismissal will be with prejudice.”

Privacy now a ‘data stewardship obligation’

Jeff Valdes, a director at Acceligence, noted, “there is definitely some irony here.”

“If Microsoft wants customers to view it as a steward of their privacy when the government comes asking for their information, customers are naturally going to apply that same standard to how Microsoft and LinkedIn collect, use, protect, and disclose information themselves,” he said. “Privacy is difficult to compartmentalize. You cannot have one philosophy of customer privacy for government access, another for product design, and another for your own commercial data practices without eventually creating a credibility problem.”

Mike Wilkes, enterprise CISO at Aikido Security, agreed, pointing out, “without meaningful limits, judicial scrutiny, and an expiration mechanism, a temporary investigative necessity starts looking a lot like a permanent architecture for invisible surveillance. The individual may never have an opportunity to challenge the scope of the request, because they may never even know the request existed until prosecutors show up with an indictment.”

That, he said, “is why Microsoft’s argument matters, despite the obvious irony of LinkedIn simultaneously defending itself against privacy claims from its own users.”

But Ryan O’Leary, an IDC research director, offered a different perspective.

“Microsoft makes no bones about using the data contained within its own systems for its own purposes. Both things can be true: Microsoft can fight for the privacy of its platform while still not necessarily respecting the privacy rights of its end users,” O’Leary noted. “This seems to come down to protecting its own proprietary data sets, not some altruistic privacy crusade.”

 At the same time, Valdes pointed out, Palmer’s post highlights how deeply privacy has become a top-tier enterprise IT priority.

“Privacy is rapidly becoming a much broader data stewardship obligation,” he said. “Companies holding sensitive information increasingly have to think simultaneously about government requests, third-party access, their own collection practices, AI use, data retention and what they tell customers about all of it. If you want to be trusted as the custodian of the world’s data, customers are going to judge how you protect that data in every direction.”

However, Wilkes noted, “Microsoft does not need to be a perfect privacy saint to be right about this particular problem.”

This article originally appeared on CSOonline.

The fight against AI slop is real, even in photography, which is why Apple introduced its Reference Image tech alongside its new iPhones. This shifts image verification to the moment of capture, creating a privacy-preserving digital negative that could help newsrooms, businesses, and individuals prove that images are real. Apple’s Security Research team just shared a white paper explaining how it works.

What is Apple Reference Image?

Introduced alongside the new iPhone 18 Pro series, Apple Reference Image is designed as a process to help prove the authenticity of images captured using that phone. Like so much in Apple, it’s a combined hardware/software feature that means when an image is captured, the camera also captures unalterable Reference Image data, with help from Private Cloud Compute, Apple’s cloud intelligence system.

That help means Apple thinks it now offers a system that protects image provenance to the extent of providing quantum-secure defense. The need to protect digital assets such as images from quantum-based abuse will become increasingly important as more state and state-adjacent adversaries seek out new ways to undermine security and do harm. Apple quite clearly considers that looming threat to be real; in 2024 it introduced encryption to protect iMessage against future attacks using quantum computers.

How do we use Reference Image?

Apple is reaching out with this tech, including providing APIs for iPhones, iPads, and Macs that third-party developers can use to build support for Reference Image into their apps. 

Otherwise, Reference Images can be viewed in the Photos app beside the main image, acting like a digital negative to enable visual comparison of both images. The idea is that the existence of this digital negative acts as a tool to prevent people from passing off edited or AI-altered images as being genuine pictorial narratives of events.

This will be useful in multiple situations. In the media, a news desk may turn to the feature to verify if an image is genuine or fake, while in the enterprise the tool provides a useful proof point when assessing images that relate to any workflow — that picture taken in the stock room may now be more useful as proof that stock is in hand, while a car hire firm may find it easier to prove damage during a rental period by verifying the pic. The main thing it does is help you identify changes in an image, whether those changes were wrought by human or AI.

Why Apple built this protection

Apple’s newly published white paper explains how it works and how the system is built to endure against interference, particularly as AI images are becoming increasingly convincing. After all, as digital enters most camera workflows at the sensor, proof of algorithmic activity isn’t necessarily enough to show an image has been altered out of its photorealistic beginning. It must be said that at this point we only have Apple’s word for the effectiveness of the solution, but I’m inclined to think it will be borne out.

Enter Apple Reference Image, which acts as a “chain of trust,” one the Apple team says is stronger and more enduring than existing approaches based on the Coalition for Content Provenance and Authenticity (C2PA) standard. Apple believes the C2PA approach remains too vulnerable because it is deployed after image capture and also creates a privacy risk by attaching images to public identity. You don’t have to look too far for instances in which photojournalists have been targeted by hostile forces once their identity has been realized, and the C2PA system arguably adds to that risk.

“Other industry solutions require a photographer or institution to vouch for an image using their own credentials,” the white paper explains. “We are concerned this puts some photographers, such as those operating in conflict zones, in a difficult position; it should not be necessary to forgo anonymity in order to prove image authenticity.”

Apple’s system is designed to kick in as images are captured, when a photographer can choose to create a securely timestamped reference image that accurately reflects what was captured by the iPhone’s camera sensor. The creation process leans into the powerful iPhone processor and also Apple’s Private Cloud Compute.

A little insight into how it works

As the image is captured, the iPhone cryptographically signs pixel data immediately after capture, which shows the image to be an exact depiction of what the camera sees. The system also captures relevant metadata, including use of the Secure Enclave on iPhone to ensure some values.

That means that the Reference Image now carries deep insight into the original, down to an unchangeable record of metadata. Not only can you see the image matches, but you can ensure the metadata is also the same. Furthermore, the image is also timestamped with a value taken from Apple’s own cryptographic timestamp service. Pull it all together and the digital negative is packed with verifiable, trusted data that can be contrasted with an image you want to check. 

There’s much more to this system, which Apple has built to be resilient to compromise while also protecting the privacy of the photographer. Instead of requiring the photographer act as the point of proof, Apple’s own Private Cloud Compute servers play the role by cryptographic signing of the image. 

“The result is a verification model that offers photographers, newsrooms, and everyday users renewed confidence that an image they’re viewing is a photograph actually captured by a camera,” the company said. It is, I think, also interesting to consider how Apple’s newly introduced tool for cryptographic proof could potentially be applied elsewhere.

The caveat is that the protection only extends to the main camera of the iPhone 18 Pro family and is not retrospective. It won’t be available in China on launch, while in Europe you can develop and view reference images but not create them.

Now please subscribe to my daily, human-curated Apple-related news headline feed at The Core, or follow me on BlueSky, LinkedIn, or Mastodon.