A new section in the iOS 27 privacy policy shows that Apple has reversed its stance on using customers’ data to train its AI models. Previously, the company assured customers that it had no need for customer data, and would figure out how to make Siri AI work well without it.
Apple maintains that it “takes steps to ensure that personal data is not stored or used by Apple.” But as Heisepoints out, and Apple has said in the terms of service for a few years now, the audio material may be listened to by Apple employees after anonymization.
[…]
It also admits that data collected may be reviewed by “review personnel.” It is unclear if “review personnel” are Apple employees or a third-party company contracted to read conversation transcripts, as it has historically been.
Starting today, when a user enables Siri AI, they will see a setup screen asking whether they want to opt in to “improve Siri and Apple Intelligence,” with two options: “Share Audio and Text” or “Not Now.”
[…]
What’s important to note here is that nothing is changing about the type of data Apple collects if you opt in to this setting. What’s changing is that Apple will now use the data to train and refine its Foundation Models, as well as Siri and Dictation like before.
One of the key advantages of MacWhisper, as it utilizes the Whisper technology, is its superior accuracy in transcriptions compared to other services I have used. I was an early adopter of Rev.com and eventually switched to Otter.ai, for which I paid approximately $100/year. However, I have since canceled my subscription.
[…]
After comparing the audio transcripts from MacWhisper and Otter, I found that while Otter’s formatting is superior, there is hardly any difference in the transcriptions. I started with the free model, which was sufficient for my needs, and eventually upgraded to the large models. Nowadays, I primarily use MacWhisper to transcribe my voice memos, which I then import into Lex.Page.
Whisper, which is made by OpenAI, has the distinct advantage of allowing you to transcribe audio in almost any language you may ever need to. For the time being, Otter can only transcribe English. The benefits of better accuracy and accurate punctuation that the service already holds over Siri hold up across other languages as well. If you’re looking to transcribe non-English audio accurately, Whisper is definitely the best option available on Mac.
By taking advantage of OpenAI’s Whisper framework, Audio Hijack can now take any audio it’s recording and generate a text transcript. While this process was possible before—see Jason’s lengthy post about his workflow—it required a lot of fiddling and several different tools to accomplish; it’s certainly a lot easier to simply drop Audio Hijack’s block into your existing session. If you record multiple sources, it can even appropriately label each one—though if you record multiple people via one input, say the other end of a Zoom call, it can’t distinguish between the various participants.
Apple’s new SpeechAnalyzer is the most accurate on-device speech engine we tested. It beat every Whisper model we ship, including Whisper Small, on both the clean and the noisy half of LibriSpeech, while running roughly three times faster than Small. And the API it replaces, SFSpeechRecognizer, came last on clean speech: behind even Whisper Tiny, a 40MB model.
[…]
The new API cuts word error rate by 3.5 to 4x on the same audio: from 9.02% to 2.12% on clean speech, and from 16.25% to 4.56% on noisy speech. There is no accuracy trade-off to weigh; the new API wins everywhere we measured, and it produces punctuated, cased text where the legacy engine’s output is rougher.
[…]
All five engines ran comfortably faster than real time: between roughly 12x and 40x on the M2 Pro, meaning an hour of audio transcribes in about 1.5 to 5 minutes on-device. SpeechAnalyzer was about 3x faster than Whisper Small per second of audio while beating it on accuracy.
SpeechAnalyzer’s ReportingOption.fastResults reads like a knob you’d reach for when you want to trade power for latency. That framing is what cost me the afternoon.
[…]
If a recording matters — and in a notes app, a lost meeting is the worst thing that can happen to a user — then the container you write while recording has to be PCM in CAF. Not m4a, and not AAC-in-CAF either.
[…]
AVAudioEngine’s inputNode silently ignores your device selection
[…]
I wanted summarization to run on-device. Everything else in Nib’s transcription path does, the audio never leaves the machine, and it would have been a much cleaner story to tell.
But 4,096 tokens for the Foundation Models isn’t enough.
At the heart of this architecture is our third generation of Apple Foundation Models (AFM), a family of five foundation models custom-built in collaboration with Google. These span from on-device models to server-based models running on Private Cloud Compute.
Apple Foundation Models are built to unlock a wide range of helpful experiences for our users, like an entirely new Siri and intelligent tools that make everyday apps smarter and more useful.
“The amount of the Google Assistant we use is none,” Federighi said, explaining that Apple uses none of the Gemini models deployed to Google’s customers, none of Google’s client-side code, and no Google Search infrastructure as the knowledge backbone.
The cloud models hosted on Private Cloud Compute are interesting: they’re not “Gemini” models; they’re Apple Foundation Models trained using proprietary data via RL, which were then “refined” with data from Google’s “frontier” models.
[…]
I’ve been covering the AFM family of models for a while now, and AFM Core Advanced is one of the most interesting on-device models I’ve read about in a while, especially in the context of model size for mobile devices and built-in multimodality with support for text, images, and audio. I’m very keen to play around with this model and understand how it holds up in practice. I wonder if the new CLI (!) for AFM will let you test this one.
It’s very cool that Apple shipped a 20B parameter on-device.
You can’t put 20B parameters in RAM at any reasonable precision. To make it work they are using pretty exotic architecture by today’s standards.
A small model predicts from the query (or prompt) which experts to load from Nand into RAM. The key distinction from a typical MoE is that you do this once per query and then generate all the tokens with the same experts (instead of switching the experts for every token).
FoundationModels is one of the highlights of this WWDC; I think it’s now finally what we hoped we were getting two years ago, and thought we were getting last year (just the models themselves were barely up for it then). I finally feel like I can build features with it, though I really don’t like the Sword of Damocles Apple is holding above Private Cloud Compute and how it can permanently lock you out of using the models forever across all your apps if you hit a download threshold. Poison pill
These strict limits don’t seem to be getting as much attention as they should. It’s nice that for small developers who meet the above criteria, access to PCC has no cost. But there’s no way (yet?) to buy your way out of these limits.
[…]
The “fewer than 2 million first-time app downloads from any of their apps” restriction is particularly notable. It’s not 2 million installations for apps that are using PCC, but 2 million downloads for any app the developer has ever released.
Is there a real time count somewhere of total lifetime downloads for a developer account, so one can plan if they at risk of the switch tripping? Reminds me of the small business qualifications. I had to try to figure out if an employers developer account was going to exceed the revenue threshold for the year. I was left trying to determine it from sales reports, not to mention sales date versus pay out date confusion. I think I botched the calculation in the end.
Claude for Foundation Models is a Swift package that makes Claude available as a server-side language model in Apple’s Foundation Models framework. The package conforms Claude to the framework’s LanguageModel protocol, so you drive it with the same LanguageModelSession API you use for Apple’s on-device model: respond(to:), streaming, guided generation, and tool calling all work the same way.
Requests go directly from your app to the Claude API; Apple is not in the request path and does not see prompts or responses. Usage is billed to your Anthropic account at standard API pricing.
Apple is literally stopping me from building a brand new app using Private Cloud Compute because I made a successful app a few years ago.
The new app might be a success, or it might be a complete flop. Nobody knows yet.
I would understand it if Apple were to limit developers once an app reaches a certain number of downloads or active users, but that’s not what’s happening here.
Instead, they’re blocking developers before they’ve even had a chance to build and ship the app.
Siri AI is an entirely new version of Siri deeply integrated into iPhone, iPad, Mac, Apple Watch, and Apple Vision Pro. It can draw on personal context understanding to search across messages, emails, photos, and more, and get things done across apps with even more systemwide app actions. Additionally, Siri AI can answer questions related to the content on a user’s screen or go out to the web to get up-to-date information using broad world knowledge and generate a helpful answer. A dedicated Siri app allows users to revisit a past conversation or kick off a new one — all in one place — and uses iCloud to privately sync conversational history across a user’s products.
Powered by the next generation of Apple Intelligence, Siri AI is a completely reimagined version of Siri that is more helpful, more capable, and more intelligent. With detailed, engaging responses and natural back-and-forth conversation, Siri AI helps users get more done than ever.
[…]
Siri has been rebuilt from the ground up with powerful AI at its core. It takes full advantage of the bold new architecture for Apple Intelligence, including the next generation of Apple Foundation Models that run on device and on servers using Private Cloud Compute. When Private Cloud Compute is handling users’ requests, their personal data is not stored nor made accessible to Apple or anyone else. Outside experts can continue to verify this privacy promise at any time. Additionally, Siri AI uses the system orchestrator to tap into core capabilities like the Spotlight index and App Toolbox, which work entirely on device to keep users in control of their data.
[…]
For products that support Apple’s most advanced on-device model ever, Siri AI offers even more expressive voices, as well as a major boost in accuracy with systemwide dictation. Users have the ability to customize the expressiveness and pace of Siri’s voice so it’s just right for them. Dictation now captures what users say as polished text with greater precision, automatically handling capitalization, punctuation, and formatting as they speak. With improved speech understanding, users can speak naturally and trust that their words will appear clearly, accurately, and as intended.
I remain uninterested in these more advanced Siri features. They matter in that Apple needs to show that they’re not “behind” and that they can actually deliver on what they announce and advertise. But I don’t find the use cases compelling and wouldn’t trust the results.
If you really care about a particular concert, would you believe Siri about how the tickets work and that it will auto-create a reminder at the right time? Why should I believe that it can figure out whether a particular irregularly shaped object can fit inside an irregularly shaped container when said shapes are probably not even published on the Web? Can it really dig into my e-mail to find the right flight info? An answer without being able to see the sources is not useful. And I certainly don’t want it to pick for itself which vacation photos to send out.
I just want the basic features—from a decade or so ago, like music controls, reminders, and access to my purchases—to work quickly, reliably, and offline to the extent possible. I did not hear anything to suggest that this was a focus. Instead, everyone is celebrating how slow the demos were because that proves they weren’t fake.
The other interesting thing to note from the keynote: watching the demos, there is ZERO doubt these are real, recorded demonstrations on live devices. There will be no confusion about what is real or fake like there was at WWDC 2024 and the initial attempt at the new Siri.
And, if your standards are the state of the art in AI circa June 2024, when Apple took their first crack at answering the question, they did quite well.
[…]
What’s fascinating about this specific demo is that it also showed just how far behind Apple is.
[…]
Still, the bubble was strong enough to convince OpenAI to make the exact same mistake Dropbox did: the company somehow convinced itself that it could make enough money selling subscriptions to consumers; Anthropic, meanwhile, realized that it was enterprises who were willing to pay for AI’s massive productivity benefits, even as OpenAI failed to capitalize on their consumer market penetration by refusing to build an advertising product.
This is a long-winded way of saying that I don’t think that Apple’s agentic shortcomings are a big deal, at least for now. Agents help you do work and be more productive, and consumers don’t want to work or care about being productive. What they do want to do is watch short-form video, and an iPhone is simply much better at that than any other device ever will be; in that context, Siri being good enough is enough, and it appears that Apple crossed that bar.
They need to execute, they need to prove this can scale, and most of all, they need to get third-party apps on board with App Intents and App Schemas. But it seems like they’re doing all of that. This is not a done deal but it is very realistic.
Siri doesn’t have access to the filesystem or tool-calling, so it’s completely unable to do the kinds of tasks you might easily do with Codex or Claude. It’s a good start, but the state of the art has already dramatically shifted over the past year. The question is will Siri move in this direction, or stick to providing answers from the web?
Will Apple introduce its own distinct Codex-style agentic programming model that can do all this, and work with Xcode? You would hope so…
The revamped version of Siri that is officially named “Siri AI” will be available to test in the iOS 27, iPadOS 27, macOS 27, and visionOS 27 developer betas starting today, and it is coming to the Apple Watch in a future watchOS 27 beta. “Siri AI” will also be included in the public betas of each platform that are launching in July.
“Siri AI” requires a device that is compatible with Apple Intelligence.
Apple senior vice president of software engineering Craig Federighi has explained why the company launched a standalone Siri app in iOS 27, after previously characterizing a dedicated chatbot as contrary to its Apple Intelligence strategy.
[…]
“We see Siri not as a separate chatbot, just an unintegrated place you go and chit-chat, but rather as an integral, conversational tool that you use in the moment, deeply integrated into your experience.”
…And playlists, contacts, settings and preferences. Peel off the agentic slop bandwagon and focus on core performance, smarter caching, and resilience under adverse network conditions.
Though this is a test conducted on the very first version of Siri A.I. from a fixed point in Calgary, it seems to work quite well. So far, it is better than any version of Siri Apple has released yet and, as you can see above, it is almost as good as the original 2010 demo, before Apple acquired the company.
On the one hand, if Siri has indeed become this reliable at understanding queries like those demoed so far, it’s indeed impressive. On the other hand, I can’t help but ask myself, once again, whether people at Apple even know how regular people actually use their phones and devices.
Apple executives downplay and obfuscate the role of Google in Siri AI. The Siri AI announcement didn’t even mention Google or Gemini once! Does anyone actually believe Apple’s BS? The argument seems to be that since Google made a bespoke implementation for Apple, Google is not really involved, and it’s all Apple. Ridiculous.
It’s really odd to me that Apple is getting so much credit for Google’s work. Siri now “works” to the extent it does because Siri is to a very large extent Gemini, and Apple has wrapped that in a bow.
Apple didn’t “nail” anything. Apple failed at its own internal efforts and was forced to outsource.
This is the advantage of being a platform: you can fail and still win.
Apple’s Siri team, led by Craig Federighi, held a post-WWDC keynote tech talk with members of the press this afternoon to talk through iOS 27 and the new Siri AI.
During the talk, Federighi shared more details about Apple’s collaboration with Google.
Apple looks like it will use some combination of Google Gemini models, combined with Google’s Confidential Inference and Apple’s own Private Cloud Compute for private hosting. These systems will process both your queries and evaluate private data from your devices.
[…]
Apple has since “expanded” PCC to encompass Google’s hardware as well. I will confess that I find the details of the new “expanded” PCC just a bit vague. It sounds a lot like Apple is primarily going to rely on Google’s existing confidential compute (running in Google datacenters) to process this data, but they’re bolting on a new layer of technical security to control which models are actually running. In any case: security experts can argue about whether this is good enough to keep Cozy Bear away from your data. What I will grant is that it’s probably good enough to keep Google and Apple from accessing your stuff, which is what most people are worried about in the first place.
So why am I so nervous?
[…]
So far we’ve discussed two adversaries: the misaligned designers of private agents (such as search operators), and the possibility of remote prompt injection attacks. But of course, in any discussion of technical privacy systems we need to talk about the last elephant in the room: your government.
Apple Intelligence on-device processing remains Apple’s most convincing argument in artificial intelligence. Rivals may lead in public model rankings, chatbot adoption and the speed of releasing experimental features, but Apple is building AI inside devices people already use for their most personal work.
That distinction changes the definition of winning.
Apple does not need every iPhone owner to open a separate chatbot app. It can place intelligence inside Messages, Mail, Photos, Notes, Shortcuts, Siri and third-party apps. Local models can operate on private information without automatically transferring that information to an external server.
The strategy appeared slow while the AI market measured progress through chatbot launches and model size. It looks more competitive as the industry confronts cloud costs, privacy concerns, unreliable connectivity and the difficulty of turning general-purpose AI into useful daily actions.
In his video “Apple Lost the AI Race,” Marques Brownlee captures the contradiction in one concise line:
“Apple lost the AI race. Actually, Apple won the AI race.”
The apparent reversal describes Apple’s current position well. Apple may have lost the first race for public perception. It is increasingly prepared for the longer contest over where AI runs, how it reaches users and who controls the personal data required to make it useful.
Apple Intelligence On-Device Changes the AI Relationship
Most popular generative AI products began as cloud services. A user sends a request to a remote model, the provider processes it in a data center and the answer returns through an app or browser.
That structure supports enormous models but creates dependence on internet access, server capacity and continuing infrastructure expense. It also means personal prompts leave the device, even when the provider uses strong security and privacy policies.
Apple Intelligence on-device begins from the opposite direction. The iPhone, iPad or Mac first attempts to handle the request locally using Apple Foundation Models optimized for Apple silicon.
That local model does not need to know everything available on the internet. It needs to understand the user’s request, work with relevant personal context and complete common tasks efficiently.
The difference is similar to owning a capable computer rather than renting every calculation from a distant server. Cloud systems remain useful for difficult workloads, but they no longer need to participate in every interaction.
For Apple, local processing is also a product advantage. The company controls the chip, operating system, model architecture, memory system and apps. Each layer can be designed around the others rather than assembled from unrelated services.
Image Credit: Apple Inc.
Privacy Becomes a Technical Feature
Technology companies frequently describe privacy as a policy. Apple is trying to make it part of the AI architecture.
When a request runs entirely on an iPhone, the associated data does not need to leave the device. A model can work with messages, appointments, photos or app content without uploading that personal context to a conventional cloud account.
That structure is especially relevant to Siri AI. A truly personal assistant needs access to information that users may never place willingly inside an ordinary chatbot: private conversations, travel plans, health-related details, work documents and relationships between people.
An assistant becomes more useful as it gains context. It also becomes more invasive when the same context is stored, analyzed or used for model training outside the device.
Apple Intelligence on-device allows Apple to expand personal context while maintaining a stronger boundary around it. The assistant can understand information locally, perform an action and discard temporary processing data without building a permanent advertising or training profile.
This is difficult for competitors whose businesses depend on cloud services, large-scale data collection or engagement inside separate AI products. Apple’s hardware business gives it different incentives.
Private Cloud Compute Extends the Model
Not every AI task can fit within the power and memory limits of an iPhone. Apple uses Private Cloud Compute when a request requires a larger model.
Private Cloud Compute is designed to extend the privacy architecture of Apple devices into the cloud. User information is processed only for the request, is not stored and is not made available to Apple for unrelated purposes.
Independent researchers can inspect software images used by the system, creating an unusual level of outside visibility into the cloud environment. Apple has also expanded the architecture while maintaining control over the software and privacy protections used for processing.
This creates a layered system rather than a simple choice between local and cloud AI.
Apple Intelligence first determines whether the device can handle the request. More complex work can move to Private Cloud Compute. External models may be offered separately when the user chooses to involve them.
The user does not need to understand the technical path behind every answer. The system chooses the smallest and most private environment capable of completing the task.
Apple Silicon Was an AI Investment
Apple’s control of its processors has become one of its largest advantages in the AI era.
The Neural Engine first appeared years before generative AI became a consumer obsession. It supported Face ID, photography, speech recognition and other machine-learning tasks. Apple continued expanding that hardware across the iPhone, iPad and Mac.
Those investments now provide a large installed base of devices designed to run machine-learning workloads locally. Recent Apple chips combine CPU, GPU, Neural Engine and unified memory in one architecture that can move information efficiently between different types of processing.
The same design helps foundation models run without requiring the power consumption or separate memory systems associated with conventional data-center hardware.
Apple also controls the operating systems that schedule those workloads. A model can be optimized for the exact processors, memory limits and energy behavior of supported devices.
Competitors can build excellent models, but they often need to support hardware from several manufacturers with inconsistent capabilities. Apple can decide that an AI feature requires a specific generation of silicon and design the complete experience around it.
Developers Gain a Local Model
The Foundation Models framework extends Apple Intelligence on-device beyond Apple’s own apps. Developers can use the system model through native Swift tools rather than shipping a separate large model or paying for every cloud request.
An app can generate structured content, interpret natural language or use tool calling while keeping supported processing on the device. Apple handles model delivery and system integration.
This can reduce operating costs for developers. A cloud-based AI feature creates an ongoing expense each time a user submits a request. Local inference uses hardware the customer already owns.
It can also improve reliability. Features may remain available without an internet connection and respond without waiting for a remote service.
The framework encourages a quieter form of AI adoption. Apps do not need to add a prominent chatbot tab or redesign themselves around a conversational interface. Intelligence can appear inside existing workflows as a useful capability.
That is where Apple’s strategy may reach scale. Millions of apps can incorporate small AI functions that users experience as normal software behavior rather than as visits to an AI destination.
Image Credit: Apple Inc.
The Installed Base Is Apple’s Distribution Advantage
The AI race is often discussed as a contest between models. Consumer adoption also depends on distribution.
Apple can deliver new intelligence features through operating-system updates across the iPhone, iPad, Mac and Vision Pro. Users do not need to create another account, choose a provider or transfer their personal information into a new service.
Siri already occupies the system interface. Apple Intelligence can work across first-party apps. Developers can connect their own features through Apple frameworks.
The scale is difficult to reproduce. A model provider may have stronger benchmark scores but still need users to download an app and consciously open it. Apple can introduce AI during actions people already perform: replying to a message, locating a photo, organizing a note or completing a shortcut.
This does not guarantee success. Apple must make Siri AI reliable, keep feature availability consistent and prove that local models can deliver enough capability to satisfy users accustomed to powerful cloud assistants.
The distribution advantage only becomes meaningful when the experience works.
Winning Without Building the Loudest Chatbot
Apple has made mistakes during its AI transition. Siri improvements arrived later than promised, regional regulations delayed features and competitors established powerful consumer brands before Apple completed its new system.
Those setbacks explain why the claim that Apple lost the AI race became persuasive.
They do not settle the outcome.
The first stage rewarded companies that released chatbots quickly and demonstrated increasingly large models. The next stage may reward systems that can integrate AI into ordinary life without demanding constant cloud access, high subscription costs or broad collection of personal information.
Apple Intelligence on-device is built for that stage. It uses hardware already in the user’s hand, protects more personal context locally and gives developers access to intelligence without requiring every interaction to become a server transaction.
Apple does not need to win every benchmark or release the most talkative assistant. It needs AI to become a reliable part of the devices people already trust.
Brownlee’s title works because both statements can be true. Apple lost the visible sprint that defined the early generative AI market. By concentrating on local processing, silicon, privacy and operating-system integration, it may already be positioning itself to win the race that follows.
Login
