OpenAI has paused new sign-ups and upgrades to its $200 ChatGPT Pro tier, citing a surge in demand for its Astra capability that is placing pressure on system capacity, according to company statements and an executive post on X.

“To make sure our current users have an incredible experience and continued access to Astra, we are going to pause subscriptions to our $200 Pro plan,” OpenAI member of technical staff Thibault Sottiaux wrote in a post on X. “These put the most strain on our systems and we wanted to take the smallest step that allows us to continue giving the broadest access possible.”

The company separately confirmed the move in its help documentation, stating that “as of September 10, 2026, we’re temporarily pausing new sign-ups and upgrades to the ChatGPT Pro $200 plan (Pro 20X).” The pause applies to users across Free, Go, Plus, and Pro $100 tiers seeking to upgrade, while “existing ChatGPT Pro $200 subscriptions … are not affected by this pause,” it said.

Sottiaux added that “there is no impact to existing accounts and we are working on adding more capacity as fast as we can,” pointing to ongoing efforts to scale infrastructure in response to demand.

Users who cancel or downgrade during the pause will not be able to re-subscribe to the $200 tier until the restriction is lifted, according to the company’s help page. The $100 Pro tier remains available, with lower usage limits. Promotions tied to the $200 tier are also paused, the page read.

Astra demand drives capacity decisions

The decision is directly tied to the uptake of Astra, which OpenAI has positioned as a more advanced capability within its platform. The scale of that demand, according to Sottiaux, has been atypical.

“Demand for Astra is really unprecedented,” he wrote in the post. “We’re pulling all the levers possible to sustain the demand, but I’ve not seen anything like it until now and we went through very steep growth before.”

He added that “priority will always be to keep excellent service for existing users,” noting that the company “might have to pause new Pro subscriptions for a bit if this continues.”

The pause comes amid broader rollout challenges tied to Astra. OpenAI Chief Executive Sam Altman recently described the launch as “messy” after some paying users were unable to access the model immediately, reflecting the operational complexity of deploying high-demand AI systems at scale.

Capacity constraints surface for enterprise workloads

The pause, analysts said, reflects how demand for high-intensity AI workloads is intersecting with infrastructure limits, particularly at the highest usage tiers.

“It signals that frontier capacity is still rationed,” said Bhupendra Chopra, chief revenue officer at Kanerika. “One week after launching Astra, OpenAI paused new sign-ups and upgrades to the $200 Pro tier, the plan with the heaviest Astra usage limits, while the $100 Pro tier, the API, Business and Enterprise all stayed open. That tells you the priority order.”

“Consumer power users are the release valve. Enterprise contracts are what the vendor protects,” Chopra said. “For CIOs the lesson is that a model being announced and a model being available to your workloads at the volume you need are two different events.”

Chopra said the move reflects how access to advanced AI capabilities is being managed.

“We are already there,” he said. “Rate limits, usage caps, queueing and now sign-up pauses are all forms of capacity gating, and every frontier vendor uses some of them.”

“A subscription buys you a place in the line. A fixed slice of compute comes only from a contract that says so,” he added.

Planning for constrained supply

For enterprise IT leaders, Chopra said model capacity should be treated as a dependency.

“Treat model capacity like any other supply chain dependency,” he said. “Ask the vendor three questions before you sign. What throughput is contractually committed, what happens to my workloads when demand spikes, and how quickly can I fail over to another model.”

“If a workload is production-critical, it should run on an enterprise agreement with committed capacity… and it should have a second model tested and ready,” he said.

A recurring pattern

Chopra said the pause reflects a broader pattern seen in recent launches.

“This is at least the third time in two years OpenAI has throttled or paused access at launch,” he said. “Each launch outruns capacity, capacity catches up, and the next model outruns it again.”

“OpenAI calls this pause temporary, and it will lift once capacity catches up,” he added. “The structural condition… will persist as long as model capability keeps outpacing data center buildout.” OpenAI has not provided a timeline for when new sign-ups to the $200 Pro tier will resume, stating only that the restriction is temporary.

The article originally appeared on CIO.com.

Every AI feature you’re building is based on a price that isn’t real. Current AI services are heavily subsidized as model providers seek to expand their market share, and subsidies inevitably end. Just think back to the early days of Lyft and Uber, when a single ride often cost less than taking public transit. As the costs associated with building on cheap AI infrastructure slowly start to reflect reality, the architecture decisions you made to get to market quickly may be the ones hemming you in.

This isn’t unique to AI. IT leaders have always had to weigh speed-to-value against the cost of dependency. What’s different now is how fast those dependencies are being built, and how little visibility most teams have into them.

Across IT, scalability is contingent on portability. And architecting with portability in mind is the best way to minimize lock-in as AI pricing models evolve, giving you leverage in future contract negotiations, and a fighting chance of switching platforms if the costs of one vendor get too high.

The challenge is that AI providers are incentivized to make leaving as painful as possible, which they achieve through deep integrations and proprietary features. You can control some of this through architectural decisions. Other parts are structural dependencies you should at least be aware of going in.

Swapping platforms will never be painless, but it’s less painful than being locked in when your vendor decides it’s time to recoup its losses, as every AI provider eventually will.

The part you control

The foundation for portability is the abstraction layer, an internal service your applications call instead of hitting Claude, GPT, or Gemini directly. It uses a request format defined by you, and translates that to whatever the vendor behind it expects. If you need to change vendors later, it’s closer to a configuration change than a code rewrite.

You don’t need to build this from scratch; open-source gateways like LiteLLM or Portkey already handle the routing and interface. Scoped to what you actually need, it’s a small project that one or two engineers can complete in a few weeks.

From there, audit the parts of your codebase that call a vendor’s API directly and route them through your proxy layer one at a time (have fun with that). Skip the features you’re not using yet, like fine-tuning or batch APIs, and focus on the 80% case. Think: text in, text out, and maybe tool calls.

The parts you don’t control

The API call is the easy part to abstract. Other dependencies may be tougher to account for given the pressure to deploy quickly, but you should at least be aware of them.

Prompts get tuned to a specific model’s characteristics, and those instructions won’t necessarily make sense to another model. If you do switch vendors, expect to re-tune each prompt. Fine-tuning a model will box you in, unless you use an open-weight model that you operate internally.

Vectors generated by one provider’s model won’t be compatible with another’s, so switching means re-embedding your knowledge base. Tool calling and structured output conventions differ by vendor, so code written around one model’s schema will need rewriting for another. Caching and batching mechanics carry their own vendor-specific rules and cost implications that don’t map across providers.

In short, code portability is cheap. Behavioral portability is the real cost, and no gateway tool solves that for you.

Build with your eyes open

Most teams don’t realize how locked into a model they’re becoming until they’re forced to move, which is the worst time to find out. How many parts of your codebase import a vendor SDK directly, versus calling a shared internal function?

If you want a real answer instead of a guess, pick one workflow that isn’t business-critical and try porting it to a different vendor. See what breaks and how long it takes to fix. That will uncover your dependencies better than a checklist.

The bottom line is that you should treat AI providers like any other infrastructure vendor where concentration risk matters. Hold AI providers to the same standard you would apply to any IT or technology vendor. There will likely always be some level of dependency, but you should know where the hooks are and choose them deliberately. That way, you’re prepared when AI price hikes inevitably arrive.

—

New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.

Adobe on Wednesday unveiled a range of updates to Acrobat, including the ability to turn PDF documents into interactive visual reports and podcasts. And a new Knowledge Base feature connects Adobe’s Acrobat AI assistant to document sources such as Microsoft SharePoint and Google Drive.

It’s the biggest Acrobat update in several years, Adobe said, and highlights the application’s evolution from PDF reader to what the company now describes as an AI-powered document productivity and creativity platform. 

Acrobat users open more than 400 billion PDFs in the app each year, Adobe said, claiming 650 million monthly active users as of last year. 

Among the new additions is the ability to turn dense PDFs into interactive, visual documents and presentation slides. To create these, Acrobat’s AI assistant scans the selected file, asks questions about the user’s intent, then generates documents charts, images and navigation.

It’s also possible to turn documents and web links into podcast-style audio summaries — similar to capabilities in Google’s NotebookLLM and Microsoft Copilot Notebook. A new Stylize feature can enhance plain-looking documents, applying a template to turn them into “professional deliverables” such as CVs and invoices, Adobe said.

The audio and visual generation features are all available in paid Acrobat plans (Acrobat Studio, Acrobat Express and Acrobat AI Assistant). 

Another addition, Acrobat Knowledge Base, lets customers connect the Acrobat AI assistant to a wider range of file types, enabling users to ask questions about information held in documents stored in Microsoft SharePoint and Google Drive, for example.  

Acrobat’s Knowledge Base “transforms curated document collections — policies, playbooks, product information, pricing guidance, research — into cited answers available from Slack, Microsoft Teams, Acrobat, and other places employees already work,” Abhigyan Modi, senior vice president for Adobe Document Cloud, said in a blog post. 

Adobe also added an Analyzer tool designed to retrieve data from large volumes of documents. This could involve analyzing thousands of contracts or invoices to identify information such as renewal dates and revenue clauses.

That turns information once trapped in files into data that can inform finance, procurement, compliance, and the systems that support them, said Modi.

Analyzer is the more interesting of the two releases, said Mike Leone, vice president and principal analyst at Moor Insights & Strategy. “Knowledge Base is a search product and search products get judged feature against feature. Analyzer is closer to a data product, because that extraction work is what companies currently pay people and pipelines to do,” he said. 

“What I’d want Adobe to answer is whether that data can leave Acrobat and land where the rest of the company’s data lives. If it can, this starts competing for real data budget. If it stays inside the app, it’s a very good Acrobat feature.” 

Access to Analyzer and Knowledge Base requires an Acrobat Studio for Enterprise subscription. (Adobe doesn’t publish pricing.) 

In addition, Knowledge Base includes a credit-based model that places some restrictions on usage, with licensed users allowed up to 1,000 queries a month. Other actions, such as document uploads, queries from Slack and Microsoft Teams, and queries by employees without an Acrobat Studio license, draw varying amounts of Acrobat Studio “shared credits.” Adobe didn’t say how much additional credits cost once a yearly limit is reached.

 More information about Adobe’s Knowledge Base credit usage policy is available on Adobe’s website. 

Analyzer also includes a credit system that limits document ingestion and attribute extraction.

For enterprise customers, usage-based pricing can add complexity when managing employee access.

“The issue for a broad deployment is that headcount barely predicts usage,” said Leone. “One person pointing it at 10,000 contracts will burn more credits than a hundred people asking the occasional question. Pooling the credits across the whole organization is the right design, because usage on something like this is rarely spread evenly.

“One of the big challenges is that people could very well start rationing themselves because they don’t know what a question costs,” he said. “Or worse, a couple folks use all the credits and everyone else is out of luck. The last thing any organization wants is to pay for a knowledge tool that people won’t use or can’t use.”

A slew of recent studies indicate that AI isn’t dramatically displacing workers. At the same time, there were no indicators that the fast-moving technology is creating a lot of jobs, even as AI skills remain in high demand.

Stanford University researchers in an August study found that any AI-related job declines were concentrated among young workers aged 22–25. But for older and more experienced workers that wasn’t the case.

“Claims of economy-wide AI-driven job losses are not visible in payroll data through June 2026,” the researchers said, citing ADP payroll information. Stanford partnered with ADP to conduct the research.

The decline for young workers is more often related to the automation of some work tasks by AI, as opposed to its use to complement work. In contrast, AI is more often used by experienced workers to augment what they do. 

According to the Stanford report, workers between 22 and 25 and employed in jobs highly exposed to disruption by AI are falling behind their older counterparts. Their employment levels in June were about 19% below the same age group in less-exposed occupations. That’s compared to where they would be if both groups had kept pace with each other.

“Experienced workers show no comparable gap,” the researchers said, adding that “this is consistent with recent reports of a worsening job market for entry-level workers.”

Employment of younger workers in AI-exposed jobs fell about 11% from late 2022, while the same age group in less-exposed jobs actually grew about 10%. “For older age groups, we find much less marked differences in employment growth across AI exposure quintiles,” the researchers said.

Stanford, which created a lab last year to study AI’s impact amid heightened fears of the technology’s economic effects, said its numbers were based on available indicators.

AI’s adverse impact on young workers has been well documented, with many of them laid off in the initial wave of AI automation. AI has also suppressed wages for entry-level workers.

AI was cited for 116,175 job cuts in 2026, but that number is declining. The technology  was blamed for 3,462 cuts in August, according to data from Challenger, Gray and Christmas.

It’s the “lowest monthly total since December 2025 when 142 cuts were attributed to AI,” the company wrote in a blog entry. The research firm also noted aggressive hiring plans for companies in 2027.

AI is adding value to companies in different ways, and humans are an important part of that process, said Michael Chui, a senior fellow at QuantumBlack, AI at McKinsey. “The night shift is real now,” he said. “People are sending off their agents to write code and checking in the morning.”

MIT late last year established an AI labor index to determine how agents are affecting the workforce. The school’s Project Iceberg looks at how the coordination and interaction with agentic AI changes how work is done.

Despite the disruptions, demand for AI-related talent continues to rise, tech industry consortium CompTIA said last week. About 320,000 job listings in August required AI-related capabilities, a 4.5% increase from July.

AI-related hiring is outperforming the broader tech market, with demand for AI skills up 96% year-over-year, said Kye Mitchell, head of Experis North America, which is part of ManpowerGroup.

“We’re seeing real growth…in marketing management and project management roles, suggesting employers are looking beyond the people building AI to the people applying it,” Mitchell said.

More broadly, the US economy added 162,000 jobs in August, according to US Department of Labor figures released last week. CompTIA noted that tech employment across all sectors rose by 86,000 workers in August.

But within the tech sector, 14,700 positions were cut, CompTIA said.

Databricks on Wednesday introduced Adaptive Instructed-Retriever, a new retrieval model designed to improve enterprise AI search by taking additional steps for complex queries while stopping early on simpler ones in order to help its customers balance answer quality, latency, and cost.

The new model builds on research behind Databricks’ earlier Instructed-Retriever-1 but takes a different approach to handling complex queries.

While Instructed-Retriever-1 uses parallel, single-step search and incorporates enterprise data schemas and custom instructions to improve retrieval, Adaptive Instructed-Retriever can combine parallel retrieval with sequential, multi-step search when additional evidence gathering is needed.

That distinction matters because applying multi-step search to every query can increase latency and the computational resources required for retrieval, while limiting every query to a single search step can hurt results for complex, multi-hop questions.

Adaptive Instructed-Retriever, the company said, is designed to take additional steps only when they are likely to improve retrieval quality, allowing it to return earlier on simpler requests.

To achieve that capability, Databricks trained the model using synthetic enterprise retrieval environments and an agentic data synthesis process, reusing training data from Instructed-Retriever-1 while adding synthetic multi-hop questions designed to benefit from multiple search steps.

It then used online reinforcement learning to train the model to weigh the potential improvement in retrieval quality against the cost of additional search steps, rewarding high-performing search trajectories while penalizing steps that did not produce corresponding gains.

That training approach, the company said, can produce multiple versions, or checkpoints, of the model with different quality-latency trade-offs, allowing one trained with a lower penalty for additional search steps to spend more time improving results and another with a higher penalty to limit search steps and reduce latency.

For customer enterprises deploying the model in production, these versions or checkpoints would allow retrieval performance to be matched more closely to the requirements of different applications rather than applying the same search strategy across all workloads, it added.

Adaptive retrieval could help control AI search costs

That flexibility could become increasingly valuable as enterprises move AI agents and other applications into production, where improving answer quality often requires additional inference, compute and retrieval resources that can raise the cost of running them at scale, analysts say.

“Repeated searches can improve answers to complex questions but add latency and compute costs, while a single search can be faster and cheaper but fall short on harder queries. The ability to determine when additional search is worthwhile could therefore help enterprises avoid spending resources without improving answer quality,” said Ashish Chaturvedi, executive research leader at HFS Research.

For CIOs, that ability to limit unnecessary search could eventually offer greater predictability over compute consumption and costs.

“The problem with agentic AI at scale is that consumption is hard to forecast, agents searching and re-searching create compounding, unpredictable cost and latency, and finance teams hate these variable bills. Knowing your agents will search within a defined ceiling, and that you can set that ceiling per workload, is what makes agentic search safe to run at scale rather than a runaway meter,” Chaturvedi noted.

The economics could become even more compelling, Chaturvedi added, if the specialized model can deliver the claimed retrieval quality of larger general-purpose models with lower latency.

Adaptive Instructed-Retriever, according to Databricks, matched or exceeded the retrieval quality of Claude Sonnet 5, GPT-5.6 Luna and DeepSeek-V4-Flash in its internal evaluations while completing requests in 5.8 seconds, or more than twice as fast as those models.

That difference could matter economically because retrieval is a high-frequency operation for data agents, Chaturvedi said, potentially allowing enterprises to avoid applying the cost and latency of larger models to every retrieval request.

Developers could shed some retrieval orchestration

There are advantages for developers too, especially in terms of reducing orchestration-level work.

Traditionally, developers handling complex, multi-hop questions often have to design the control flow for multiple search rounds themselves, including deciding when to refine a query, search again, or stop, but Adaptive Instructed-Retriever’s learned search policy shifts those decisions into the model, potentially reducing the amount of bespoke retrieval orchestration developers need to build and maintain, Chaturvedi said.

That, in turn, could make sophisticated retrieval capabilities more accessible to teams that lack the resources, expertise, and time to build a search agent from scratch, echoed Stephanie Walter, practice leader of AI stack at HyperFRAME Research.

More flexibility brings new trade-offs

However, there are trade-offs that developers should consider, analysts pointed out.

While the model’s checkpoints can help prevent individual queries from consuming an unpredictable amount of compute, teams will still need to test and monitor different settings to determine which one of these delivers the best balance of cost, latency and retrieval quality for each application or use case, Walter said.

That, in turn, adds another layer of performance and cost optimizations developers need to manage, Chaturvedi echoed.

More so because being able to choose the right model checkpoint means working through real business questions and documenting where answers are coming from in your database or other knowledge bases, said Mike Leone, principal analyst at Moor Strategy and Insights.

“Very few enterprise development teams have built such a curated list. In practice, that means most teams are likely to simply select a setting once and leave it unchanged,” Leone added.

Even with the right checkpoint in place, however, Adaptive Instructed-Retriever does not address every challenge that can affect retrieval quality. Teams would still need to prepare and organize their data, enforce permissions, evaluate retrieval quality, and verify final answers, Walter said.

Echoing Walter, Leone added that a smarter search policy cannot compensate for poorly structured underlying data or a lack of agreement over what key terms mean, meaning it could still retrieve the wrong answer even if it does so more efficiently.

There are also caveats that CIOs should take note of.

Adaptive Instructed-Retriever’s performance and quality claims, Chaturvedi said, are based on Databricks’ own benchmarks against competitors selected by itself, meaning that these claims will need to be tested against an enterprise’s actual workloads rather than accepted at face value.

Even if those results hold up, however, CIOs will still need to decide whether the potential gains in retrieval performance, latency, and cost justify adding a specialized model to their AI stack.

Rather, using a larger general-purpose model for both retrieval and reasoning could offer a simpler alternative by reducing the number of models enterprises need to deploy and manage, Chaturvedi said. The economic case for Adaptive Instructed-Retriever for CIOs, therefore, will depend on whether its claimed performance holds up on real enterprise workloads and whether the potential savings outweigh the additional complexity of using a specialized model, Chaturvedi added.

AI note-taking applications are increasingly used by workers to record meetings, generate conversation summaries and suggest post-meeting action items. 

Along with the promised productivity benefits — enabling users to focus on meetings rather than actively taking notes — the emergence of these AI tools has raised privacy questions, particularly around obtaining prior consent by meeting participants for their use. Those concerns, in turn, have prompted a spate of lawsuits against software vendors that sell AI notetaking tools.

In some ways, the underlying questions are not new: It’s long been possible to record a phone call with a dictaphone, and laws around surreptitious recording have been around for decades. Yet widespread access to AI notetakers via desktop or smartphone apps means that it’s easier than ever to record a conversation for future reference. 

Among the thorny questions arising from the technology’s use: what happens to conversation data sent to a software vendor’s servers for processing? Are the recording and transcript used to train those vendors’ AI models, for example, or create biometric voiceprints? Those practices are among the issues being considered in US courts. 

Otter, which claims to have 35 million users, was subject to a class-action complaint in a federal court in California last year. That complaint accused Otter of recording individuals without consent and using their voices to train its speech recognition AI tools. Last month, a judge rejected Otter’s attempt to dismiss the main claims, though the scope of the case was narrowed.

A lawsuit filed against another vendor, Fireflies, in an Illinois court late last year, alleges the company collects and stores biometric voiceprints without user consent, in violation of the Illinois Biometric Information Privacy Act (BIPA). Fireflies claims to have more than 20 million individuals and 1 million organizations as customers.

Earlier this year, a class-action complaint in a Washington court claimed that a live transcription feature in Microsoft’s Teams collaboration application also violates BIPA by collecting biometric data without consent. 

And most recently, a complaint alleged that Granola, a well-funded startup, designed its product to be used without the knowledge of all meeting participants, in violation of the Electronic Communications Privacy Act (ECPA). That complaint also claims Granola trains its AI models on conversation data without consent. 

None of these cases has yet been resolved, and it’s unclear whether any of the vendors broke the law in the design and delivery of their products and services. But the lawsuits highlight considerations for businesses that allow the use of AI notetakers, both in terms of prior consent for recordings and understanding how conversation data is handled.

More broadly, the lawsuits raise questions about how existing privacy and consent rules apply to new technologies that make it easier to record others, whether through AI note-taking apps, smartglasses, or other recording devices.

Computerworld spoke with Brian McGinnis, partner at law firm Barnes & Thornburg and a founding member and co-chair of the firm’s Data Security and Privacy Law practice group, about the focus of the lawsuits, potential outcomes, and how businesses can deploy AI notetaking apps safely.

Several cases have already been brought against popular AI note-taking apps. What are some of the main commonalities between these? Which laws are the vendors accused of breaching? “The common allegation is that these companies capture communications of people who did not agree to the recording or receive adequate notice. Some of the lawsuits also allege that meeting data is used to train AI models and that consent cannot meaningfully be withdrawn once the data has been processed.

“There are various federal and state claims. You’ve got the Electronic Communications Privacy Act, a federal wiretapping statute. As a general matter, the Electronic Communications Privacy Act permits an interception when one party consents, subject to statutory exceptions and questions about whether the technology constitutes an unlawful interception or third-party eavesdropping. In other words, if you, as the user, provide consent, you can be on a meeting with 20 other people and it’s deemed to be sufficient; you don’t necessarily need to get the consent of other people. 

“But California and a minority of other states are what we call ‘two-party consent’ states, meaning each individual on the call has to give consent; it’s not sufficient for you as the person who turns the notetaker on to provide the consent — you also have to get the consent of others that are being recorded. There are state laws around that.

“There’s a law called CIPA, the California Invasion of Privacy Act, that’s being utilized in this context.  We see a lot of suits being brought under that law — it’s a wiretapping statute designed for telephone wiretapping that’s now being applied to the internet.”

What about the use of biometric data? “A growing number of states regulate biometric data, with Illinois’ BIPA being particularly prominent because it provides a private right of action. This means an individual can sue a company for violation of the law.  

“That law covers biometric information. With an audio recording or recording of ‘dumb’ video that isn’t running any algorithms, you’re not necessarily collecting any biometrics. But when you start identifying people, you’re recording things like faceprints or voiceprints, which are in the definition of biometric information within the statute. Now you’re not only collecting personal information, but also biometric information, which is considered very sensitive and much more highly regulated. 

“Then you’ve got the private right of action that goes against it. Part of the argument here is that recordings taken by the AI notetakers can be used to produce some form of biometric information, such as a voiceprint.  

“We’ve seen a lot of cases, and a lot of changes in industry as a result of this law. Shutterfly had a famous case about scanning for people’s faces and things like that in Illinois that changed the photo storage and processing industry a little bit. 

“A lot of companies stay out of Illinois to avoid this law specifically. But a customer organization might not know exactly who’s in a meeting and where a person is located at the time of the meeting. So, you need to either follow that law and get individual consent, or stay out of states with biometric laws if you want to use some of these tools. 

“It’s just a further challenge for these applications if the goal is to be used as much as possible with as little detection as possible.”

What are some of the potential outcomes for these cases? “I think it’d be unlikely to get an outright ban, absent passing some kind of new law that says these tools are per se illegal for use. It’s much more likely the outcome will require some changes and controls over the way that they get used. The clearest case would be some kind of a pop-up notice: ‘Hey, this meeting’s being recorded, here’s who it is, here’s their privacy policy, here’s their terms of service – do you consent to it?’ And getting opt-in consent from anybody who wants to be recorded. 

“The notion that only one person in the meeting has to say it’s okay and you can just automatically record everybody else, I think that’s probably at risk, and could potentially be replaced with a standard that requires everybody to consent before it’s considered legal.

“But the newer —and more interesting — wave of these is the Granola case, where part of its marketing is that people aren’t aware that it’s there. The Granola complaint alleges that the product was designed to operate without alerting other participants. It’s possible that kind of activity could result in a decision that would ultimately ban it outright. In other words, that it’s illegal to utilize these tools if you aren’t providing notice to everybody and/or aren’t getting consent from everyone on the call. That’s a possible outcome that we could see.

“To me, that’s interesting when you think beyond AI note-taking and into the broader world, with conversations around, like, Meta Glasses, or any of these kinds of AI wearables. Granola in particular has an Apple Watch app, and there are other wearable or physical devices — there’s one [Plaud Note] that sticks on the back of your phone and is essentially an always-on recording device. 

“With these devices, you’re going from an online meeting where you can provide notice and there’s a structure to obtain consent, to walking down the sidewalk and recording people and casual conversations. Maybe it’s somebody you’re having a conversation with, maybe it’s somebody at the table next to you in the coffee shop that you have no relationship with whatsoever — what are the laws around consent and notice in those cases, where there’s no digital interface to put that up in front of people? Do you have to go around with a pad of paper and a pen and get people to sign consent to use these things? Do you have to physically tell them?

“Those are the more interesting conversations that this line of cases is just at the beginning of helping us get some answers on. In other words; how do you get consent? How do you provide notice in a world where we don’t have documents or screens in front of us to easily handle that? Those are interesting questions that the law will have to figure out here.”

These tools are increasingly used in the workplace. How can businesses be sure they deploy the technologies safely? “We’re getting a lot of questions from our clients about this topic, because they’re really unsure and uncertain of how to handle these tools.

“Obviously, there’s a push for broad use of AI within their companies, for productivity increases within their company. People like the tools; they want to be able to use them. But then we also hear a lot of stories about ‘I jumped on a meeting; I didn’t even know it was being recorded, then I got an email afterward with a transcript of it,’ and ‘half of what it recorded wasn’t actually what I said, or it was interpreted incorrectly’ — things like that. So there’s a lot of consternation amongst our clients about the people within their organizations using it. 

“I can’t tell clients definitively that they’re legal as they are; there are ways to use this legally and safely that aren’t going to get your organization sued, but you probably have to do some things that are beyond what’s provided ‘out of the box’ by the software providers. 

“With Granola, for example, my understanding is that certain notice features may not be enabled by default. You can turn on video or audio watermarking, and you can turn on the notice that pops up in these things, but I think it ships without those features enabled. That puts the responsibility on the individual user to determine and then implement their own legal privacy and legal compliance mechanisms using their settings.

“You really have to play to the most stringent state’s law. I would advise a company that, to decrease your chances of getting in trouble for use of these tools, you need to obtain consent of all parties on the call. You can do that verbally, as well; written is even better. 

“Then it’s about having an internal AI note-taking policy. Think of a BYOD policy, which all these companies have, or an AI usage policy — this could be part of that policy, or a standalone policy: ‘Here’s how our organization thinks about these tools:  you can only use these approved tools and, if you’re going to use them, you have to turn on these features. You have to get consent from everyone. Here are limits on what you can do with the output of those transcripts or recordings.’ 

“If you set all that up and do the compliance and governance work, I think you can use these tools and most likely stay on the right side of the law. Certainly, the law doesn’t prevent consenting adults from consenting to the use of these kinds of tools.  

“But the further you get away from that written consent standard, the more problematic it becomes. If you just want to go to a notice standard and not obtain actual opt-in consent, or, even worse, if you want to try and do this without anybody knowing, that potentially could get challenged.”

By now, you probably know my stance on Gemini and other generative AI gobbledegook — right?

In short: It’s an insanely powerful form of technology (obviously). But it’s also wildly inaccurate and ineffective as an all-purpose answer engine, and by cramming it into every last nook and cranny, Google is doing a serious disservice to its users — and to itself.

That being said, generative AI can be incredibly useful. The key is simply to frame it as a purpose-specific tool for the right type of limited task as opposed to treating it as the end-all answer for everything, as most tech players seem overly eager to do these days.

Here in the land of Android, I’m always looking for purpose-specific ways Gemini can make our lives easier. Over time, I’ve found those ideas often aren’t the big headline-making features Google and other tech companies market but rather random little off-the-beaten-path possibilities that rarely get emphasized.

Today, I’ve got a new Gemini Android advantage for you to explore — another one that I’ve never seen advertised anywhere but that works brilliantly well and solves a long-standing limitation of our modern mobile gizmos.

It’ll take you all of a minute to master and cost you precisely $0 to pull off. Ready?

[Get next-level knowledge in your inbox with my free Android Intelligence newsletter — one useful new thing to try every Friday, straight from me to ye.]

Gemini’s custom sound advantage

This Gemini-provided power-up very much follows the path of thinking about AI as a limited, purpose-specific tool for a narrow task — the exact sort of area where AI excels.

I won’t keep you waiting: Gemini, as I discovered whilst poking around within it recently, is really good at generating unique, custom ringtones and notification sounds based on your personal preferences and specifications.

It almost seems obvious, once you think about it. But I sure hadn’t thought about it up until now — and I suspect most other folks haven’t, either.

The advantage here goes beyond just superficial silliness, too: By creating your own custom ringtones and notification noises, you can know exactly what event your phone is alerting you to within a split second of hearing the associated sound — without having to rely on the same limited and often underwhelming pool of default options that everyone else uses (or, worse yet, having to lean on shady, ad-ridden “ringtone apps” to find mediocre alternatives). That means you’ll know within a heartbeat when your boss or an especially important client is calling or if an incoming call is a random unknown number.

On the notification front, you can create a specific, memorable sound to accompany a new message in an important work-related Slack channel and a different distinctive sound for a high-priority email. They’ll all be sounds that you make and identity with and that no one else in the world uses. That’s pretty powerful.

And creating these custom sounds couldn’t be much easier, either, once you realize it’s possible:

First, just fire up Gemini on your phone (or on any device you’re using) and tell it what you want. So, for instance…

• Create a soft, subtle ringtone that’s reminiscent of an old office phone sound.
• Create a ringtone that sounds like an 8-bit version of The Final Countdown.
• Create a one-second high-pitched notification sound that brings to mind the Super Mario Bros theme song.
• Create a short notification sound that mimics the Back to the Future time circuits noise.
• Create a notification sound that sounds like a robot saying “ALERT, ALERT!”

The only limit is your imagination. And if you’re not feeling especially imaginative, you can even ask Gemini to give you a list of ideas for distinctive, memorable ringtones or notification sounds based on geeky nostalgia, 80s pop-metal, or whatever it is that tickles your fancy.

Once you’ve sent a request, Gemini will — somewhat confusingly — spit back a big honkin’ block of HTML code. This is because, for reasons unknown, the system can’t or won’t just provide a sound file directly. Go figure.

But from there, it’s just one more quick step to get the file you need: Click or tap the copy icon in the upper-right corner of that code block — the icon that looks kinda like two stacked, rounded rectangles — to copy the entire chunk of code onto your system clipboard.

Then open up your browser and go to the website JSFiddle.net. It’s a well-known, long-standing tool for running code like HTML right in your browser and seeing the result, and it works entirely in the browser and without any downloads or special permissions required.

Tap or click into the “HTML” field, then paste in the code from your clipboard (by long-pressing any open space, on Android, and selecting “Paste” from the menu that appears). Click or tap the “Run” button, and — ta-da: You’ll see a player to listen to your custom Gemini-generated sound and a button to download it.

Provided you like what you hear, hit the button to download the file, then head into the Sound section of your Android system settings to save it and make it available for applying as any kind of alert or ringtone. The exact steps for doing that will vary from one type of Android device to another, but on a Pixel or another phone that follows Google’s core Android interface, you’ll tap on either “Ringtone” or “Notification,” then tap “My Sounds” and tap the plus icon to import your own custom file from your phone’s local storage.

On a Samsung device, you’ll start the same way but look for the plus icon within the “Ringtone” section of the settings, then tap the “Folders” tab to find your locally downloaded files. Samsung doesn’t make it easy to add in your own custom notification sounds, annoyingly, so what you’ll have to do is open up the Google Files app, find the file you downloaded in your “Downloads” folder, then tap the three-dot icon alongside it to copy it to the folder called “Notifications.” Once the file is in that folder, it should show up as a notification noise option within the Samsung Android settings.

And remember: You can use Android’s notification channels to set specific sounds to be used for different types of alerts, even within a single app — and you can use the Google Contacts app to set custom ringtones for different people, too.

With Gemini as your personal composer, you’ll never be stuck with generic overused sounds again — and you’ll always know exactly what’s happening from the first note of whatever noise you hear.

Keep the experience-enhancing wisdom coming with my free Android Intelligence newsletter. One new useful trick in your inbox every Friday!

We spent the last decade building entire finops departments just to decipher what the cloud bill was saying. Just when we figured out how to stop leaving idle compute instances running, generative AI introduced an infinitely more opaque, faster-moving layer of spend.

AI bill shock has spread like a slow moving hurricane across the industry. Aside from the spike in large language model (LLM) costs, the deeper architectural problem is attribution: where exactly is the money going, and exactly what value is it delivering to the business?

When API calls are wrapped in layers of automated agents and prompt templates, your application becomes a black box that consumes capital to spawn tokens. If a features engine is burning thousands of dollars a month just to have an LLM output pleasantries or process low-value data, that isn’t innovation, it’s an uncovered manhole, a gaping liability. Architectural maturity means treating tokens like any other constrained resource.

Fortunately, there are a few good levers we can pull to control AI spend. Here are the five key tools for taming the beast.

Model routing

Don’t use a nail gun if a thumbtack will suffice.

The most powerful models like the latest Claude Opus are some of the most sophisticated software systems ever built. They are capable of handling extremely complex and subtle use cases. They are also voracious beasts when it comes to compute (and therefore dollar) consumption. They are often overkill.

We naturally start with the most powerful model we have when prototyping and sketching out APIs and requirements. That’s because we are in the “get it working” mode and we don’t want to be tangling with model limitations. But for production, it is absolutely essential to dial in on what is the least powerful model we can use.

Sometimes we can address this by tuning the model choice and discovering what will work manually. But in larger enterprise architectures, we can move to a conditional routing layer. Frameworks like RouteLLM or Semantic Router can dynamically direct simple classification, basic text parsing, and user intent detection to fast and cheap utility models like GPT-4o mini or Claude 3 Haiku.

You can also use advanced AI gateways for routing and spend allocation. At the infrastructure level, wrapping this logic in a dedicated AI API gateway (such as Kong, Cloudflare AI Gateway, or Portkey) gives you the ultimate control. A mature gateway allows you to implement “cascade routing” to automatically fall back to a cheaper or open-source model if your primary LLM hits rate limits or latency spikes. More importantly, an AI API gateway centralizes your telemetry, stripping away the black box so you can enforce hard token budgets per tenant or per microservice. You are essentially building a load balancer for cognitive tasks.

Of course, AI gateways also introduce complexity and cost themselves, so these factors must be balanced into the equation. But the bottom line is, there is a right-sized model for your project. You can save a bundle by routing requests to a good-enough, Goldilocks model.

A side note: Another trend in model routing is “repatriation of compute.” That entails running models (often open-source models) on local hardware, instead of renting generative AI endpoints on the cloud.  This implies all the trade-offs and benefits of other on-prem options.

Semantic caching

In conventional web apps, caching is a solved problem. You put Redis in front of your database, map an exact query to a cached payload, and you’re done. But LLMs break traditional caching because human language is infinitely variable. “How do I reset my password?” and “I forgot my login info” are entirely different strings, but communicate identical intents.

If you rely on overly precise matching, your hit rate will be effectively zero, and you will pay the full token price for every phrasing permutation.

Enter semantic caching. It’s like regex for meaning. Instead of hashing the raw text string, you run the incoming prompt through a fast embedding model and do a similarity search against previously answered prompts. You define a confidence threshold (say, 0.92), and if it is met, you serve the cached response (bypassing the LLM and its cost entirely).

The architectural effect is twofold. First, a semantic cache hit reduces your inference cost for that query to exactly zero. Second, it drops response latency from seconds to milliseconds. You can implement this using purpose-built frameworks like GPTCache, or, if you prefer a leaner infrastructure, by using pgvector inside your existing PostgreSQL database to store and match the embeddings.

However, just like dynamic routing, semantic caching demands a rigorous architectural calculus. You are trading generation costs for embeddings and lookup costs. To check the cache, you still have to tokenize the prompt, call a cheap model (like text-embedding-3-small), and execute a vector search.

You also introduce the very real danger of semantic flattening. Tuning the similarity threshold is a delicate art. Set it too low, and your application starts serving general, recycled answers to nuanced user questions. For open-ended, creative, generative AI applications, semantic caching is practically useless. But for retrieval-augmented generation (RAG) implementations, customer support bots, and internal knowledge bases where users ask the same 20 questions a thousand different ways, it is the most effective cost-reduction lever you can pull.

Prompt caching

Whereas semantic caching stores the response to a given intent, prompt caching stores the data needed to contextualize the question. When the user enters a prompt, it is sent to the generative AI endpoint. But instead of your application having to repeatedly gather and send the massive contextual info needed to frame the prompt—from a RAG pipeline, a database, or other source—that information is already pre-loaded in the context cache. The model simply applies the new question to the cached data, slashing both your latency and your input costs.

The core idea behind prompt caching is that the AI engine itself retains the relevant contextual data for incoming prompts. Tokens cached this way get a massive discount (50% to 90%) compared to raw input tokens. But the details of how it is handled vary between providers.

For OpenAI, prompt caching is automatic, but requires attention to how your app uses the endpoints in order to leverage them. OpenAI enforces a 1,024-token minimum threshold. More importantly, it requires an exact, byte-for-byte prefix match starting from token zero. If you inject a dynamic variable (like a timestamp or a user ID) at the very top of your system instructions, you instantly invalidate the cache and pay full price for every token that follows.

Some other providers, like Anthropic Claude and Google Gemini, use explicit caching. You must deliberately engineer the cache into your architecture, either by hitting a dedicated API endpoint to upload the static document (the context data) with a defined time to live, or by injecting strict cache control breakpoints into your JSON payload. Explicit caching often requires paying a “write premium” up front, but it gives you the control to guarantee your heavy RAG data stays alive in memory for hours rather than minutes.

Prompt discipline

Just because a model can consume gargantuan amounts of tokens, doesn’t mean it should.

With models now capable of ingesting one million to two million tokens in a single prompt, there is a temptation to brute-force use cases into submission by dumping the entire PDF, the entire codebase, or the last three years of logs into the payload and let the LLM sort it out.

This is a finops blunder. You are paying for every single token you send. Furthermore, from an engineering standpoint, stuffing the context window runs into the very real “muddy middle” problem (aka “context rot”), where the model loses resolution into the middle of the content. As the payload grows, performance and accuracy suffer.

The cure is a strict RAG diet. Naive RAG implementations might pull the top 20 results from a vector database and blindly concatenate them into the prompt. Architectural maturity requires a more ruthless filtering process before those tokens ever reach the expensive LLM.

This is where reranking becomes key. Reranking takes the compiled prompt context and makes sure it is efficiently structured before it hits the LLM. Instead of sending raw search results directly to your primary model, you introduce a tiny, highly-efficient cross-encoder (like Cohere Rerank or an open-source BGE model). You cast a wide net in your cheap vector database to fetch 50 potential matches, then pass them to the reranker to score their actual relevance against the user’s specific prompt. The reranker ruthlessly trims the fat, allowing you to pass only the top three or four hyper-relevant chunks to the expensive frontier model.

You are trading a massive, expensive LLM context payload for a tiny, cheap reranking step. The architectural math here is almost always a resounding win. While a cross-encoder might add 100 milliseconds of latency to the pipeline, it routinely slashes prompt token counts by 80% or more, while simultaneously increasing the accuracy of the final generation. In the generative AI spend playbook, giving your LLM less to read is one of the highest-leverage moves you can make.

Response constraints

By default, LLMs are polite, conversational, and verbose. If you ask an API for data, it desperately wants to start with “Certainly! Here is the JSON you requested,” and end with “Please let me know if you need anything else!”

In a web interface, this is charming. In an application’s API architecture, it is a leak in the finops hull. Generation tokens (output) are almost universally priced 3x to 5x higher than prompt tokens (input). Every pleasantry is burning your most expensive commodity.

Taming LLM responses requires strict API-level enforcement. First, max_tokens must be treated as a hard circuit breaker to prevent runaway generation loops, not as a formatting tool. (Relying on max_tokens to shorten answers often just results in truncated, unparseable JSON.) Second, use stop sequences (stop_words). If your model only needs to output a SQL query, setting a stop sequence for the markdown closing tag (“`) or the word “Explanation:” forces the model to sever the connection the moment the computational work is done.

Modern APIs also offer structured outputs (or strict JSON mode), which forces the model to adhere to a rigid schema you define. By combining this with zero-tolerance system prompts (“Output ONLY valid JSON. Omit all conversational text and pleasantries. Failure to do so will break the system”) you force the LLM to behave like a traditional API endpoint rather than a chatbot.

The architectural balance here is to avoid degrading the model’s reasoning ability by strictly limiting its input. LLMs do not have internal working memory, they “think” by iterating on token output. If you force a model to solve a complex logic problem and output a single boolean true or false without giving it space to reason first, its accuracy will plummet. The finops goal isn’t really to force minimal words; it is to ensure that every expensive output token is performing actual computational work (like chain of thought reasoning).

For more on how to manage AI back ends, take a look at this article.

How to effectively consume generative AI to satisfy user needs is the first battle. The second is keeping the finances from undermining that hard work. The five keys discussed here should give you all the power you need to make sure the engineering and the cost controls are working together.