As AI shifts from output-generating large language models (LLMs) to armies of agents taking actions on their own, there is a growing threat that failures could affect system reliability.

Temporal, a Bellevue, WA firm founded in 2019, hopes to solve that problem by stabilizing AI and long-running computing processes through “durable execution,” a technology that reliably resurrects failed computing processes on other hosts. If a machine crashes in the middle of agentic AI transactions, the company resurrects the function on a different host so it can continue exactly where it left off. Temporal says its durable execution process comes with a 100% durability guarantee.

The company was solving these kinds of distributed-systems problems long before the generative AI (genAI) rush, and today it powers infrastructure from Coinbase to Airbnb to OpenAI.

For IT decision-makers, Temporal’s offerings promise to bring reliability to AI operations — especially in regulated industries. Co-founder Samar Abbas spoke recently with Computerworld about his company’s technology, AI reliability, execution and what IT leaders need to keep in mind when evaluating and deploying agentic AI.

How did Temporal come to be a backbone powering AI system? “OpenAI recently announced we power many of their products underneath the cover — that’s where Temporal comes in.

“A couple of years ago, as LLMs got smarter, use cases were about generating outputs. Now, the industry is moving into agentic solutions, where AI isn’t just generating outputs, it’s taking actions. The longer these systems run, the more tools they invoke and the more value they create. 

“They’re evolving into a whole army of agents coordinating to get complex tasks done. This starts to look like distributed systems. We become that execution authority, the reliability backbone for those agentic systems to get their tasks done from A to Z.”

What is Temporal doing in the backend to keep them stable? “It’s a new paradigm for building applications, we call that durable execution. If you write a function and a failure happens, you lose all of your state. When a machine crashes, we maintain the state. We seamlessly resurrect that function on a different host and continue executing exactly where it left off, without you as a developer writing a single line of code. 

“It’s not just a cache, it’s an operating system guaranteeing execution in the presence of failure. We provide 100% durability guarantees behind it. People can use it for mission-critical use cases like transcribing doctor conversations, where even losing one [word] has big implications.”

Did this challenge exist before AI? “My co-founder and I have been at it pretty much our entire careers, spanning almost three decades. We’ve been solving this problem of distributed systems, at Amazon, Microsoft and then Uber, before starting Temporal. We are definitely pre-AI. 

“As more workloads move to cloud and become distributed, it introduces this class of failures. This is the fifth time I’m building this system. I built simple workflow, then [a] durable task framework at Azure, then Cadence at Uber, now Temporal. The company is roughly seven years old, but the code base is over 10 years old, because we started at Uber and it’s been running in production ever since.”

And now you tackle that same problem at AI scale? “The funny thing is, especially with AI, that same problem is now at such a massive scale, it’s gone on steroids, with agentic systems coming online and doing real work. We’ve been powering every Coinbase transaction, every Snap story, every Airbnb booking, every Yum Brands order —  KFC, Pizza Hut, Taco Bell.

“A couple of years ago, it looked like we’d over-invested in the scale and reliability of the platform. Then AI showed up in such a big way that now I’m asking the team: how do we scale another 10x, 100x, even 1000x? These agents are hitting the market in a way we couldn’t have imagined.”

Can you give a concrete example of that 100% guarantee? “When you’re moving money — debit or credit — imagine a failure happens after you’ve debited an account. Temporal guarantees execution of the entire function from start to finish, in the presence of all sorts of failures. At the end of the day, we’re in the business of selling reliability. 

“Developers today spend 80% of their time building resiliency into their systems by working with low-level primitives like queues, databases, retry mechanisms and durable timers, and stitching them together. What we provide eliminates that. And that reliability is exactly what’s becoming critical as enterprises move agents into production.”

Why should CIOs and other IT leaders care about AI reliability? “These agents are now real, no longer prototypes or [proofs of concept]. They’re doing meaningful work in those organizations. Reliability and durability weren’t on the radar for CIOs 12 months ago. Now, it’s the blocker. Especially in regulated industries, if a transaction gets left in the middle, it has real regulatory concerns. 

“That’s what’s holding them back from transitioning into the AI stage. It comes up in pretty much every conversation, in the first 10 minutes I’m talking to a CIO.”

CW: Where do the hardest problems sit today? “Security, governance, identity, authorization, authentication. This is where majority of the problems exist, and the ecosystem is so immature. Even things like MCP, I see organizations struggling to implement MCPs to build these agents. This is an area ripe for innovation and a lot of work, including here at Temporal.”

What skills should developers and CIOs build for the AI age? “The biggest skill now is that developers need to be more product focused. Product-minded engineers are the ones who will thrive, because the whole software development flow, writing code by hand, debugging by hand, testing it, is going to look completely different. The job of an engineer is now building the system that builds the system. 

“Two things are happening in this AI world. First, a lot more engineers are coming into the fold, because companies like Replit or Lovable are making software development approachable for people who haven’t been professionally trained. Second, there’s going to be a lot more applications written than ever before. The problem space is shifting toward how we run all those applications safely and reliably, with all the guardrails. That’s where the industry is headed.”

France’s OVHcloud is moving beyond cloud infrastructure into frontier AI model development, a shift that could test whether Europe can produce another serious alternative to US and Chinese AI systems.

The company, one of Europe’s leading homegrown cloud providers, plans to train a family of models from scratch and aims to open-source them once they meet its performance targets, CEO Octave Klaba told Reuters.

The move would put OVHcloud in closer comparison with Mistral AI, the Paris-based model developer that has become Europe’s most visible challenger to US AI labs.

Klaba said the economics of building advanced AI models have changed, with improvements in chips, training methods, and synthetic data reducing the cost of a project that may once have required about $1.15 billion (€1 billion) to now cost less than $230 million (€200 million).

Reuters reported that OVHcloud said one of its models has completed pre-training on Jupiter, the Germany-based EuroHPC supercomputer described as Europe’s fastest and its first exascale system, though the company has not yet disclosed detailed performance benchmarks.

This comes as European governments and enterprises are increasingly having to assess AI infrastructure through the lens of data governance and continuity of access, rather than performance alone.

Those concerns were sharpened this month after Anthropic said a US government export-control directive required it to suspend access to its Fable 5 and Mythos 5 models by foreign nationals inside and outside the US.

Training is only the opening cost

OVHcloud’s lower cost estimate does not capture the full cost of becoming a frontier AI model provider, said Neil Shah, vice president for research and partner at Counterpoint Research.

The $230 million (€200 million) figure likely refers mainly to the initial training run, Shah said. Once trained, however, models require continued investment because they can become depreciating assets if they are not improved with fresh data.

OVHcloud would also need to spend on fine-tuning, post-training, sovereign infrastructure, storage, security, distribution, and enterprise support. It would also need enough scale to make model serving economically viable against established AI providers such as Google and Anthropic.

“Model is seen as a depreciating asset if it is not consistently trained and kept fresh with the data,” Shah said.

That makes OVHcloud’s plan a test not only of technical capability, but also of policy support and economic viability. If the company falls short, enterprises may be reluctant to shift workloads away from more established models.

The lower training cost could still give OVHcloud a credible starting point, said Charlie Dai, principal analyst at Forrester.

The budget range can be enough to produce a credible frontier model as efficiency gains reduce the cost of entry, Dai said. But enterprise competitiveness will depend on sustained capabilities beyond training, including inference efficiency, data pipelines, evaluation frameworks, and ecosystem reach.

Buyers need proof

OVHcloud’s plan remains an expression of intent rather than demonstrated capability, said Sanchit Vir Gogia, chief analyst at Greyhound Research, pointing to the absence of published benchmarks and other details.

“$200 million now buys a serious training run,” Gogia said. “It does not buy a serious enterprise AI franchise.”

Gogia said questions around sovereignty also extend to the infrastructure used to train the model, noting that pre-training was run on Jupiter rather than on infrastructure owned or controlled by OVHcloud.

The system is a publicly owned European supercomputer in Germany that runs on American silicon, Gogia said, adding that this shows how partial European AI sovereignty remains.

CIOs will need evidence that the models can be supported in production, governed effectively, audited when needed, and exited without major disruption.

Gogia said a European-owned model could reduce some dependence on US and Chinese providers, but would not remove jurisdictional risk. “Sovereignty does not abolish the off switch,” he said. “It changes whose hand rests upon it.”

OVHcloud’s move into model development could also alter the lock-in risks enterprises need to assess, Gogia said. Customers may be able to move cloud infrastructure later, but find it harder to shift AI workloads once applications and processes are built around a provider’s models and governance tools.

The article originally appeared on ComputerWorld.

France’s OVHcloud is moving beyond cloud infrastructure into frontier AI model development, a shift that could test whether Europe can produce another serious alternative to US and Chinese AI systems.

The company, one of Europe’s leading homegrown cloud providers, plans to train a family of models from scratch and aims to open-source them once they meet its performance targets, CEO Octave Klaba told Reuters.

The move would put OVHcloud in closer comparison with Mistral AI, the Paris-based model developer that has become Europe’s most visible challenger to US AI labs.

Klaba said the economics of building advanced AI models have changed, with improvements in chips, training methods, and synthetic data reducing the cost of a project that may once have required about $1.15 billion (€1 billion) to now cost less than $230 million (€200 million).

Reuters reported that OVHcloud said one of its models has completed pre-training on Jupiter, the Germany-based EuroHPC supercomputer described as Europe’s fastest and its first exascale system, though the company has not yet disclosed detailed performance benchmarks.

This comes as European governments and enterprises are increasingly having to assess AI infrastructure through the lens of data governance and continuity of access, rather than performance alone.

Those concerns were sharpened this month after Anthropic said a US government export-control directive required it to suspend access to its Fable 5 and Mythos 5 models by foreign nationals inside and outside the US.

Training is only the opening cost

OVHcloud’s lower cost estimate does not capture the full cost of becoming a frontier AI model provider, said Neil Shah, vice president for research and partner at Counterpoint Research.

The $230 million (€200 million) figure likely refers mainly to the initial training run, Shah said. Once trained, however, models require continued investment because they can become depreciating assets if they are not improved with fresh data.

OVHcloud would also need to spend on fine-tuning, post-training, sovereign infrastructure, storage, security, distribution, and enterprise support. It would also need enough scale to make model serving economically viable against established AI providers such as Google and Anthropic.

“Model is seen as a depreciating asset if it is not consistently trained and kept fresh with the data,” Shah said.

That makes OVHcloud’s plan a test not only of technical capability, but also of policy support and economic viability. If the company falls short, enterprises may be reluctant to shift workloads away from more established models.

The lower training cost could still give OVHcloud a credible starting point, said Charlie Dai, principal analyst at Forrester.

The budget range can be enough to produce a credible frontier model as efficiency gains reduce the cost of entry, Dai said. But enterprise competitiveness will depend on sustained capabilities beyond training, including inference efficiency, data pipelines, evaluation frameworks, and ecosystem reach.

Buyers need proof

OVHcloud’s plan remains an expression of intent rather than demonstrated capability, said Sanchit Vir Gogia, chief analyst at Greyhound Research, pointing to the absence of published benchmarks and other details.

“$200 million now buys a serious training run,” Gogia said. “It does not buy a serious enterprise AI franchise.”

Gogia said questions around sovereignty also extend to the infrastructure used to train the model, noting that pre-training was run on Jupiter rather than on infrastructure owned or controlled by OVHcloud.

The system is a publicly owned European supercomputer in Germany that runs on American silicon, Gogia said, adding that this shows how partial European AI sovereignty remains.

CIOs will need evidence that the models can be supported in production, governed effectively, audited when needed, and exited without major disruption.

Gogia said a European-owned model could reduce some dependence on US and Chinese providers, but would not remove jurisdictional risk. “Sovereignty does not abolish the off switch,” he said. “It changes whose hand rests upon it.”

OVHcloud’s move into model development could also alter the lock-in risks enterprises need to assess, Gogia said. Customers may be able to move cloud infrastructure later, but find it harder to shift AI workloads once applications and processes are built around a provider’s models and governance tools.

In hardware, when you ship something broken, the consequences are severe and often irreversible. That’s the world I worked in for years, in verification roles at Mellanox and later at Alibaba. The stakes forced the industry to build a rigorous verification culture. You proved designs worked before they left the building.

In software, verification disciplines look like CI/CD pipelines, static analysis, canary deployments, and observability. But those systems were built around code written at human speed, with human comprehension baked into the process. AI code generation has broken that assumption. The writing process can no longer be trusted to carry institutional knowledge and judgment into the codebase. The industry is being pushed toward the kind of rigorous verification culture that hardware engineers have practiced for decades.

Enterprises are generating code faster than at any point in history. Google recently disclosed that 75% of the company’s new code is now AI-generated. Meta has set internal targets requiring most of its engineers to generate the majority of their committed code with AI tools by mid-2026. The velocity gains are significant. But a growing body of evidence suggests the industry is accumulating a new form of technical debt, one that is less visible than the traditional kind and harder to unwind. It’s also preventable, and the organizations that get ahead of it will have a meaningful advantage over those that don’t.

Understand what makes this debt different

The standard narrative frames this as AI writing bad code. That’s not quite right. The more precise problem is cognitive debt: the loss of understanding of how and why software was built the way it was.

When a human writes code, something else happens alongside the typing. They simulate edge cases, reason through dependencies, and make judgment calls grounded in organizational context, including the business requirements behind a feature, the best practices the team has established, and the reasoning behind past architectural choices. That cognitive loop is how institutional knowledge gets built. When AI writes the code, you can get output that is syntactically correct, passes CI, ships cleanly, and leaves no one holding the mental model. The code works until something changes or breaks, and then the team is excavating a black box.

This is distinct from traditional technical debt, which is messy code. Cognitive debt is invisible code that functions but that nobody truly owns. And it compounds faster, because the same velocity that makes AI generation attractive is what prevents anyone from stopping to build the understanding that maintainability requires.

GitClear’s analysis of 211 million changed lines of code across major repositories found that during 2024, duplicate code blocks of five or more lines increased eightfold, while refactoring dropped from 25% to under 10% of all code changes. Refactoring is the slow, unglamorous work that keeps codebases healthy, and developers are doing far less of it. Google’s 2024 DORA report found that a 25% increase in AI adoption correlates with a 7.2% decrease in delivery stability. DORA analysts note that the root cause isn’t flawed code per se; AI inflates batch sizes, and larger changesets have always been riskier to ship.

These findings aren’t indictments of AI-assisted development. They’re diagnostics, and they point toward a specific set of fixes.

Close the context gap first

In a survey of 609 developers we conducted last year, 65% said AI misses relevant context during critical tasks like refactoring, writing tests, or reviewing code. Context is the primary driver of AI code quality, and it’s where most enterprise organizations are underinvesting.

When an AI tool generates code without access to your organization’s architectural decisions, historical pull requests, security policies, or existing module patterns, you get solutions that are locally correct but globally incoherent. Closing that gap requires context engineering: ensuring the tools and agents you use have access, at the right moment, to the right organizational knowledge, and the judgment to determine what is actually relevant for a given task. A retrieval system that surfaces too much irrelevant context can degrade output quality as readily as one that surfaces too little. The specific tooling matters less than the discipline. Context infrastructure needs to be actively maintained, not indexed once and forgotten.

Build this infrastructure before you scale AI generation. Retrofitting is significantly harder. Treat it the way you treat your CI pipeline, as a prerequisite for safe production deployment.

Consider what happens when a team has built this context infrastructure well. Their code review tooling knows about a deprecated internal API, because that deprecation decision lives in months of past pull request discussions that have been indexed and surfaced. When generated code references the old API, the review flags it. Without that context layer, the same mistake gets waved through every time. That’s the kind of institutional knowledge that evaporates when humans stop writing every line of code, and that you have to actively work to preserve.

Build a verification layer that matches your generation velocity

Almost all of the investment in AI-assisted development has gone into generation. Very little has gone into verification. That imbalance is where the tech debt accumulates.

I think of these as the blue team and the red team. The blue team covers code generation, autocomplete, and agentic coding. It’s getting the headlines, the budgets, and the product launches. The red team covers integrity checks, behavior coverage, and alignment with organizational standards. In most organizations, it’s an afterthought. A CI pipeline catches obvious failures. A code review might happen, but reviewers are overwhelmed by the volume of AI-generated output and cannot meaningfully evaluate all of it. The result is code with a veneer of having been reviewed without anyone having actually understood it.

The Crowdstrike outage of 2024 is worth keeping in mind here. AI didn’t generate the problematic code, but the incident illustrated what happens when a single software error propagates through production systems without sufficient verification. That exposure multiplies when code is being generated faster than humans can understand it.

A real verification layer means automated analysis that evaluates whether generated code aligns with your organization’s best practices, architectural standards, and compliance requirements. It means test coverage that reflects intended behavior, not only the happy path the AI chose to generate tests for. And it means traceability: a connection between the requirement and the implementation, so that six months from now, someone can understand what the code does and why it exists.

The numbers support investment here. In the same developer survey, teams that integrated AI into their code review workflow saw quality improvements in 81% of cases, compared to 55% for comparable teams without it. 

Make ownership non-negotiable

Every piece of AI-generated code in production needs an accountable human who understands it well enough to maintain it. This is harder than it sounds, and it’s where most organizations are falling short.

The same velocity that makes AI generation attractive also creates pressure to skip the slow work of genuine comprehension. A developer reviews a 500-line pull request that an AI generated in three minutes and faces a real choice: spend two hours actually understanding it, or approve it because it looks right, passes the test, and “LGTM” (looks good to me). 

Real ownership means slowing down generation velocity enough to allow for meaningful review, and being explicit with your team that this is the right trade-off. When that doesn’t happen, you’ve started building your next legacy system.

What to do this quarter

The good news is that none of this requires a multi-year transformation. The structural problems are real, but they have concrete solutions, and engineering leaders can make meaningful progress on all three fronts without waiting for the next budget cycle.

• Audit your context infrastructure. If your AI tools are generating code without access to your organization’s architectural decisions, deprecated APIs, and security policies, fix that before expanding generation velocity further. The quality of context determines the quality of output.
• Invest in the red team. Establish an automated code quality and governance layer that evaluates generated code against your specific organizational standards, beyond functional correctness. This is a distinct investment from your code generation tooling and needs to be treated as such.
• Establish ownership explicitly. Every AI-generated feature in production should have a human owner who genuinely understands it. Make that a formal requirement.

The organizations that get this right will find that AI generation becomes far more reliable once it has a verification layer underneath it. The ones that don’t will keep shipping faster while understanding their systems less, until the accumulated debt forces a reckoning. That’s a solvable problem. The question is whether you solve it now or later.

—

New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.

Because generative AI (genAI) tools and services have become so ubiquitous (and popular), the costs of using them are going through the roof — leading to an insatiable appetite for tokens.

Tokens represent a common way to measure and price AI use. Much like letters and words in English, large language models (LLMs) grasp a sentence or query by breaking words into tokens.

With the AI explosion well under way, tokens are now “the fundamental units of data our models process, many representing a problem being solved,” according to Google CEO Sundar Pichai. (Google, by the way, processes about 3.2 quadrillion tokens a month.)

But as the price of all those tokens adds up, business and IT execs are looking for ways to cut costs while keeping corporate productivity up. Uncontrolled token use has already landed one company with an unexpected $500 million AI bill. 

There are a number of ways companies can rein in the price of AI at the model, infrastructure, silicon, and business levels. Here’s a look at how some of those savings might actually be achieved.

Switch to lower-cost models

One way of potentially saving money is by re-routing AI work to a cheaper model, Pichai said. At Google that would Gemini 3.5 Flash. It delivers “frontier-level capabilities at less than half the price of comparable frontier models.

“If companies use a mix of [Gemini 3.5] Flash and other frontier models, they could save a lot of money,” Pichai said.

Those kinds of models provide cheaper tokens, with reasoning that’s good enough for many users — if not as strong as mainstream Gemini 3.5 — to deliver useful results.

“There is sometimes overkill with the [LLMs],” said Deepak Seth, senior director analyst at Gartner. “I don’t always need a large language model which has been trained on the works of Charles Dickens and Shakespeare and Harry Potter.”

Hyperframe Research principal analyst Steven Dickens can’t stop using Amazon’s Quick, which costs $20 a month, for personal tasks. “It is great personal ROI as it has not only made tasks faster, but unlocked tasks I would never have even attempted previously,” Dickens said.

Don’t forget the hardware and software part of the equation

The token crisis isn’t new, said Dheeraj Pandey, CEO of DevRev, who likens what’s going on now in the AI market to the disruptions that emerged with the arrival of cloud computing and virtualization years ago. 

“We let chaos reign and then we had to rein in the chaos,” Pandey said. “The word that people started using was server consolidation and virtualization.”

The answer to the token problem, he said, is the same: “Anything in systems can be solved with caching and indirection.”

DevRev, for example, is building a memory layer between AI agents and primary data sources, such as Salesforce or ERP records; that can cut token load and make data movement more efficient. The layer holds a knowledge graph with answers to common agent questions and runs on cheaper CPUs, avoiding more costly GPU cycles.

Sending agents straight at systems like ServiceNow and Salesforce “will burn a lot more tokens. It’s also not precise. And finally, it’s not safe enough where I can roll it back in case an agent has committed a mistake,” Pandey said.

Network automation firm NetBrain uses a different method: It uses conventional computing to map a network’s layout then feeds only key information to models for planning and reasoning, where AI excels. “So you don’t have to spend all the tokens,” said Netbrain CTO Song Pang.

Focus on prompt efficiency

Staffing firm ManpowerGroup has found that prompt efficiency can be an effective tool for improving token use, both internally and externally for clients.

For example, users accessing its internal labor-market tool initially needed 10 follow-up questions to drill into a query. A year later, more efficient use of prompts has brought that number down to an average of four, said Max Leaming, head of data science and AI solutions at ManpowerGroup. 

“They’re using fewer tokens and they’re simply more efficient,” he said. “And that in large part has to do with your ability to prompt efficiently.”

Go local

New AI hardware that generates free tokens at home could ease some of the cost crisis.

At GTC Taipei earlier this month, Nvidia and Microsoft unveiled RTX Spark, an agentic AI desktop PC that runs agents and 120-billion-parameter models locally on Windows. The goal is “to deliver unmetered intelligence to every home and every desk with Windows,” Microsoft CEO Satya Nadella said in a statement.

Some companies are looking to reduce cloud AI costs by putting their own hardware in data centers, with vendors such as HPE and Dell providing servers installed in independent facilities. (On-premise AI is gaining ground amid sovereign AI and geopolitical concerns, including the recent conflict in the Middle East, where large data centers were struck with missiles.)

“There are local, region-specific and multiple vendor AI solutions. All of those things can help mitigate the risk. But they’re not going to eliminate it,” said Max Goss, senior director analyst at Gartner.

Use forward-deployed engineers

Reducing token costs is something that may fall to forward-deployed engineers (FDEs) in customer environments, said Taimur Rashid, managing director of AWS’s Generative AI Innovation Center.

“I expect these teams to be able to architect systems that have those cost requirements in mind, whether it’s use a different model or a different use case that doesn’t increase the per-token cost,” Rashid said.

Companies may spend heavily on token consumption, “but if you’re generating revenue, as long as the economics work out, then you’re at peace,” Rashid said.

The use of FDEs is gaining ground as IT decision-makers look to both rollout successful AI deployments while also keeping an eye on costs.

Change the measure of success from tokens to outcomes

Even with the current emphasis on reducing token use to save money, the metrics used to measure AI success are likely to shift, Gartner’s Seth said. At some point, token-based pricing will move more toward an outcome-based model, where the unit of value is outcomes, not fragments of words.

“Some companies are moving towards outcome-based pricing,” Seth said. “When people start realizing the real cost of tokens, then companies will start looking at token efficiency.”

Adobe’s Firefly Graph is now available to Creative Cloud customers, offering a node-based workflow tool designed to help business create content at scale with generative AI (genAI). 

With Firefly Graph, users can connect multiple tools in visual workflow, with each “node” performing a specific task before passing its output to the next node. This gives creative professionals more control over generated outputs, according to Adobe, and makes it easier to try out ideas by swapping, adjusting or adding components.

For example, a user could start with a text prompt box that connects to a node that generates an image using an AI model from Adobe or third-parties such as Google and OpenAI. Further along the chain, the user could add nodes to remove a background or upscale an image, for instance, before producing an image, video or other asset ready for use.

Changing one aspect, such as adding a reference image or adapting the text prompt, would change the final output.

It’s an approach similar to node-based workflow tools such as ComfyUI — a startup valued at $500 million which claims more than 4 million users. Others include Weavy, acquired by Figma last year for a reported $200 million. 

With so many AI tools available to creative professionals, workflows can get complex and hard to replicate, said Elliot Sedegah, director for strategy and product marketing at Adobe. Firefly Graph provides access to more than 300 different node types, including images, video editing and AI generation tools across Adobe’s portfolio and third-party tools. 

“Whether you’re working at a mom-and-pop shop or a larger enterprise, you’re looking for consistency and then bringing that into a workflow so that you’re not hopping in and out of different tools,” he said. “Putting all that together takes massive amount of time, and sometimes it’s very difficult to even know what you did.”

Once created, workflows can be shared across an organization as repeatable processes for other individuals or teams to use. “Think of that rock star creative that you have and the recipes they create: those are now canonized as workflows, as assets, that the rest of the organization can take and reuse over and over again,” said Sedegah.

In addition, while creative professionals are needed to created high quality assets, reusable workflows can be put into the hands of broader teams to create content for large audiences, said Sedegah.

Firefly Graph addresses a challenge that most large creative organizations face, said Lisa Gately, principal analyst at Forrester — namely that their best creative workflows “live inside the heads of a few experts.

“Teams can generate images and video with AI, but reproducing the exact sequences of creative decisions, model selections, edits, and refinements that lead to a high-quality result is difficult and inconsistent. Firefly Graph turns those workflows into reusable assets,” she said.

While other node-based workflows aim to address similar problems, Adobe’s pitch is that Firefly Graph provides customers with the benefit of integration into its product suite. 

“Firefly is a full, broader AI creative studio, not just a node-based tool, so [Firefly Graph] is a part of a bigger picture,” said Sedegah. “The strength is having everything in one place with the tools that people know.” 

“Where Adobe differentiates is in enterprise integration,” said Gately, with Adobe connecting Firefly Graph to a range of other Adobe tools. Those include Creative Cloud applications; Firefly Boards for ideation; and Firefly Creative Production. 

“The workflow becomes part of a broader content supply chain instead of a standalone creation tool,” she said. ”Organizations committed to other tools are unlikely to migrate for a node-based canvas — making a change is about the broader content supply chain.”

Firefly Graph is available now to Adobe Creative Cloud for Enterprise subscribers. Creative Cloud for Enterprise is licensed by seat, requiring custom, volume-based enterprise agreements. Creative Cloud for Enterprise customers receive credits for Firefly Graph. Creative Cloud for teams customers can sign up for a public beta online. Creative Cloud pricing starts at $99.99 per license each month.

This article was updated on June 19 with pricing information from Adobe.

Microsoft has introduced usage-based billing for Copilot Cowork, which is now generally available. 

Microsoft unveiled Copilot Cowork in March, pitching it as an AI agent that’s capable of independently performing long-running, multi-step tasks — even when a user’s computer is off. 

It’s built on the same technology that underpins Anthropic’s Claude Cowork. Unlike Claude Cowork, which can interact directly with files and applications on a user’s computer, Copilot Cowork runs in Microsoft’s cloud environment and acts on documents held in a customer’s Microsoft 365 tenant.

On Tuesday, Microsoft unveiled pricing details for Copilot Cowork, which involves usage-based billing in addition to a Microsoft 365 Copilot license ($30 per user each month for large enterprises before discounts, and $20 for Microsoft 365 Copilot for Business). 

The usage-based pricing is calculated from four components, according to Microsoft: “model use, context retrieval, tool calls, and runtime.” 

In practice, this means more intensive tasks that draw on multiple sources, use “deep reasoning” and generate two or more outputs will lead to higher costs — denoted in Copilot Credits. 

There are two payment options: pay as you go — priced at 1 cent per credit — and P3, where customers commit to usage volume in advance and receive a discount.

Cowork is turned off by default; IT admins can decide when to make it available and which employees get access. Admins can also impose spending limits at the tenant, group, and user level, and receive notifications when spending reaches a certain level.

Users will also be able to see how much each tasks costs in credits (available “soon after” the general availability launch, Microsoft said). 

Copilot Cowork customers can select from multiple AI models, including Anthropic’s Opus 4.8 and Sonnet 4.6, while those enrolled on the Frontier program can access OpenAI’s GPT 5.5 and Microsoft’s own Cowork 1 model.

Microsoft is considering hosting a version of DeepSeek’s open source models. “We are actively exploring a range of options that meet those requirements, and DeepSeek is one of several models under consideration. We’ll share more specifics on the underlying model closer to release,” said a Microsoft spokesperson.

Microsoft also announced new integrations with third-party apps such as Miro and Monday.com, with more — such as Adobe, Box, and Canva — coming soon.

This article was updated on June 19 to add Microsoft’s comment on DeepSeek.

Most generative AI tools know less about R than languages like JavaScript and Python, thanks to how much training data is available for each. However, with a little extra setup, you can give a large language model (LLM) the knowledge it needs to improve its R skills.

Here are 10 ways to help generative AI write R code like a pro.

Use a coding agent

AI coding agents have more power, flexibility, and coding-focused tools than general-purpose chatbots.

Anthropic’s Claude Code and OpenAI’s Codex agents have versions that run in a terminal, IDE extensions, desktop and mobile apps, and other integrations.

R users may also be interested in Posit’s Posit Assistant, which is designed for data analysis in both R and Python. It needs less setup for R than general-purpose coding agents, and it has more built-in knowledge about data science, R package development, and Shiny apps. Plus, it can read objects in your R and Python environments by default, which can be useful in some situations (although perhaps not if you’re working with sensitive data). There are versions of Posit Assistant for RStudio, the Positron IDE, and the terminal. Setup info is here.

Note: Posit Assistant is not the same as the older Positron Assistant that was demo’d at last year’s posit::conf(). The older assistant will eventually be superceded by Posit Assistant. Posit Assistant comes pre-installed with RStudio but not pre-activated, so you don’t need to use it unless you want to.

Google’s original entry in this space, Gemini CLI, is being retired in favor of a new “Antigravity CLI” starting June 18.

There are other IDE platforms with built-in agents, including Cursor and Windsurf, and terminal + subscription coding agents such as Warp, which was open-sourced in April. However, I won’t be covering those here.

You may be limited at work by what your employer allows you to use, but you can still experiment on your own with public data — and then lobby for a new agent if a different one better meets your needs. Whatever you choose, though, moving to one of these from a web-based chatbot should produce better code.

“The shift from chatbot to agent is the most important change in how people use AI since ChatGPT launched,” says Ethan Mollick, co-director of the Wharton School’s Generative AI Labs at the University of Pennsylvania.

Set up CLAUDE.md, AGENTS.md, or GEMINI.md knowledge files

All the major coding agents look for knowledge files to load each time you start a session, and you can easily edit those files. The files can include things like your proficiency in various languages and how you like to document projects. For example, I tell my LLMs that I’d like a README.md in every project with info on how to use the project and technical info on how the project was coded.

Most coding tools let you have both an overarching main file that’s loaded into every session, and an additional file per project.

If you’re adding Claude or Codex to an existing code base, the /init slash command will create one of those .md project files after scanning and analyzing the code. You can also ask an LLM to interview you to create or add to one of those files.

Claude Code looks for a user-level main CLAUDE.md file at ~/.claude/CLAUDE.md and then a project-level file at ./CLAUDE.md or ./.claude/CLAUDE.md. The /memory slash command shows where those files are located. Codex uses AGENTS.md files. Antigravity docs say it looks for a main GEMINI.md file plus GEMINI.md and/or AGENTS.md files in your working directory.

Positron Assistant is set up to read AGENT.md, AGENTS.md, POSITRON.md, CLAUDE.md, GEMINI.md, and LLMS.txt in a workspace root directory, according to the docs. See the Positron docs for more on custom instructions.

Use agent skills

Agent skills can turn an LLM with questionable R knowledge into an expert that understand how you want it to write code. You can develop your own skills, work with your LLM to create skills, or download skills written by others. Claude even has a skill to help you build skills. This is one of the best ways to stop having to repeat the same instructions over and over. Instead, the LLM will “remember” what’s in your skills files by loading them when they’re needed.

What’s the difference between the knowledge .md files mentioned above and a skill? I like to think of CLAUDE.md or AGENTS.md files as important background info that’s always available, while skills are more specific instructions that are useful at certain times. Or, as Claude Opus 4.7 put it: “Skills are for triggered workflows (‘when user asks X, do Y’). A standing rule that applies to every project isn’t trigger-based; it’s a directive I should always follow.” Standing rules should go in CLAUDE.md.

“Create a skill when you keep pasting the same instructions, checklist, or multi-step procedure into chat, or when a section of CLAUDE.md has grown into a procedure rather than a fact,” according to Anthropic’s Claude Code docs. “Unlike CLAUDE.md content, a skill’s body loads only when it’s used, so long reference material costs almost nothing until you need it.” Only skill descriptions are loaded at the start of each session, so the LLM knows what skills are available.

Anthropic originally created skills for Claude in October 2025 and then published the concept as an open standard a couple of months later. Most coding agents now use them.

Anthropic has an easy-to-folllow guide to creating custom skills here if you want to try your hand at building your own.

Posit Assistant comes with several R-related skills built in and allows users to add more. You’ll find them here.

If you’re not using Posit Assistant, you can add Posit-developed skills to another coding agent. Posit published a GitHub repository with skills for creating R packages, Shiny apps, and Quarto documents; for creating and resolving GitHub pull requests; for open-source R and Python package releases; for code reviews (R, Python, JavaScript/TypeScript, SQL); and for creating architectural documentation for a codebase. Browse them all here.

Read and edit the skills you download

Skills are just a folder with one or more markdown text files that have a structured YAML header. That folder can include optional resources and scripts, too. Skill instruction files are easy to read and edit, so you can make sure they do what you want. Tweak them as desired.

For example, Claude helped me write my own customized R skill. Claude leans toward tidyverse packages, but there are other packages I like a lot, too, such data.table. I made sure the R skill knows my package preferences, and under what circumstances I prefer each.

If one of your preferred procedures changes, remember to update the associated skills. Or, ask an LLM to update it for you. Keeping skills up to date will make your coding agent more useful.

Keep your CLAUDE.md or AGENTS.md files updated, too. Maybe you started off with a CLAUDE.md file that said you’re an experienced Python programmer who is just learning R. Three months from now, you might want to update your R skill level. Or, perhaps you told an AGENTS.md file that you like thoroughly documented code, but now you’d prefer a lighter touch. You may want to have your coding agent go over those files with you from time to time and ask you about the main points and whether they need updating.

Use the btw R package and its MCP server

It’s frustrating when an LLM writes code based on an ancient R package version in its training data, or when it doesn’t know about a relatively recent package at all. btw solves this problem by letting your coding agent access info about all the R packages installed on your system. That means it can write code based on your specific R environment. Plus, btw lets an LLM access variables in your current R environment via a Model Context Protocol (MCP) server.

MCP servers are a standardized way for LLMs to access external data — in this case, your running R session (external doesn’t have to mean cloud). Like skills, MCP was created by Anthropic as an open standard and has since been adopted by most major AI platforms. The btw R package includes an MCP server, which you can install for Claude Code by running the following code in a terminal window (not R console).

claude mcp add -s "user" r-btw -- Rscript -e "btw::btw_mcp_server()"

btw comes with a lot of tools. It’s not a great idea to overload your LLM with tools it may never use. If you just want to register btw tools for accessing your R session and looking up package documentation, you can run a command like this instead:

claude mcp add -s "user" r-btw -- Rscript -e "btw::btw_mcp_server(tools = btw::btw_tools('btw_tool_run_r', 'docs', 'env'))"

You only need to do this once.

To connect Claude Code to your open R session, you also need to run btw::btw_mcp_session() in every new R session. (I sometimes forget this part until I’m puzzled why Claude can’t read my R variables or even see what packages I have installed.)

You can see a demo of btw in this Posit video that compares Claude Code and Posit Assistant.

You don’t need to do this setup for Posit Assistant, since tools for accessing your R session and variables are built in.

Use plan mode for your projects

I cannot stress this enough. Just as humans benefit from having a plan before starting to implement a project, LLMs benefit from working through a plan before beginning to generate code. Plus, it can be useful at times to “brainstorm” with an LLM about structures and features, ask it for alternatives, or even have another LLM review a plan before implementing it.

Anthropic advises starting with plan mode. “Letting Claude jump straight to coding can produce code that solves the wrong problem. Use plan mode to separate exploration from execution,” they wrote in a best practices doc.

Claude, Codex, Gemini, and Posit Assistant all have a plan mode in their CLI versions that can be activated with the /plan slash command.

Make sure your coding agent learns from its mistakes

This is one of the best tips I read first from Joe Amditis, associate director of operations at the Center for Cooperative Media at Montclair State University: If your coding agent makes a mistake and you work together to fix it, make sure to save that memory so it doesn’t make the same error again.

Your agent may have an auto memory system that’s supposed to do some of this (I’ve used Claude Code’s), but you can instruct any coding agent to keep a separate lessons learned file for each project, too, or ask it what’s the best way to ensure it doesn’t make the same error again. This isn’t a guarantee, any more than you can be assured an LLM will follow all the rules in a CLAUDE.md or AGENTS.md file or all the steps in a skill. But it has definitely cut down on my agents making the same errors over and over.

Have your agent write tests and do code reviews

Tests and code reviews done by a coding agent aren’t a substitute for a human checking that software works. Review an LLM’s results yourself any time a task is important! However, agent-generated tests are still a good way to head off some problems and cut down the time you spend as your LLM’s debugging partner. If you don’t know where to start to help your agent know how to test and review code, see Posit’s testing-r-packages and critical-code-reviewer skills. Your agent may also have built-in code review skills, such as Claude Code’s /code-review slash command.

Code review skills are available from many sources. For example, Sentry has a code review skill in its repository. Aimed specifically at its project’s needs, it can be a useful sample for similar skills. And superpowers is a popular set of general-purpose programming skills that’s billed as “a complete software development methodology for your coding agents, built on top of a set of composable skills and some initial instructions that make sure your agent uses them.” I don’t use superpowers, but a lot of other people clearly do given its more than 227,000 GitHub stars and 20,000 forks.

Another tip: Several experts suggest using a different LLM — maybe even from a different provider — to conduct a code review, since each model has its own strengths and weaknesses.

Don’t forget general prompting good habits

Whatever programming language you’re using, being clear will improve your outputs. Keep your requests targeted, and don’t expect the LLM to read your mind.

“Codex handles complex work better when you break it into smaller, focused steps,” OpenAI says in its developer docs. “Smaller tasks are easier for Codex to test and for you to review. If you’re not sure how to split a task up, ask Codex to propose a plan.”

“Take time to make your prompts as relevant as possible, just as you would when helping a new teammate scope a task,” Google advises in its Five Best Practices for Using AI Coding Assistants. “Consider what details you need to share for a person to succeed, and provide all those details to your AI tool.”

And, don’t load up your context window to the LLM’s published limit. Performance often degrades as you get close to the maximum token limit.

Take advantage of open LLMs, particularly if you have budget and token limits

Open-weight models — especially ones small enough to run on a desktop computer — may never rival frontier LLMs like Claude, GPT, or Gemini. But for a lot of R coding, they don’t need to.

To see how well an LLM can write R code, it should have the same kind of harness that Claude Code, Codex, or Posit Assistant gives larger LLMs. “The harness is the product,” argues Mallory Mejias at Sidecar, an AI education company. “The model is the engine inside it — important, but increasingly interchangeable.” I wouldn’t go that far, since I still find the LLM to be a critical piece of this equation, and models differ. But the same LLM will perform differently depending on the tools and context it has.

Posit recently added Google’s open-source Gemma 4 26B to Assistant’s existing options of larger, closed-source LLMs. “Up until this point, models of this size — small enough to run comfortably on high-end consumer laptops — were on our radar but not yet capable enough to drive an agent harness like Posit Assistant,” Posit senior software engineer Simon Couch wrote on the Posit blog. “This has changed in the last few months with releases like Gemma 4.” However, to use it in Posit Assistant in RStudio, you still need to run it through a Posit AI subscription, not locally. It uses 1/10 the budget that the same session with Claude Sonnet would consume.

Several projects can use Claude Code to run local LLMs. Ollama, open-source software for running LLMs locally, does this with the terminal command ollama launch claude --model , such as:

ollama launch claude --model gemma4:26b

The makers of Unsloth, an open-source framework for running and training models, say running local LLMs inside Claude Code can be very slow due to a cache invalidation issue. They say this can be fixed by adding "CLAUDE_CODE_ATTRIBUTION_HEADER" : "0" to ~/.claude/settings.json under “env”.

Unsloth can also use Claude Code to run local models, but it’s somewhat more involved to install and set up than Ollama. See this Unsloth tutorial.

Open-source coding agent Open Code runs LLMs from OpenAI, Anthropic, Google, Kimi, Alibaba’s Qwen, and others. I haven’t warmed up to it yet compared with commercially backed CLI tools I’ve tried, but it has 174,000 GitHub stars so clearly a lot of fans.

AI consultant Thomas Wiegold, who switched from Claude to Open Code, said “OpenCode covers everything I need for my daily workflow. It’s fast, the provider flexibility is genuinely useful rather than theoretical, and the TUI is better for extended sessions…. That said, I’m not going to pretend it’s all smooth sailing.” He noted some stability issues and a remote code execution vulnerability earlier this year.

Wiegold’s suggestion? “Install both. Try others. Stay flexible.”

In summary

Large language models remain imperfect and unpredictable tools, but they’re improving rapidly — as are the harnesses around them. Even frontier LLMs in commercial coding agents can ignore instructions at times and otherwise behave in unpleasantly surprising ways. However, you’ll vastly improve your chances of generating quality code if you use coding agents, take time to set them up with quality instructions, and remember good prompting techniques. Good luck!

—

Coding agent info

Claude Code – Anthropic’s coding agent is available in the terminal and as an IDE extension, a desktop app, mobile apps, Slack app, and a cloud version that can work directly in your repos on GitHub. Arguably the first breakthrough coding agent, it now includes an elegant remote-control option where you can start a session on your desktop and then continue it on a phone or tablet. While designed for Claude LLMs, you can use it to run local LLMs via tools like Ollama or Unsloth Studio. When using Claude models, it tends to offer less usage than other vendors’ options. Anthropic has posted the terminal version’s code on GitHub, with usage governed by Anthropic’s commercial terms of service. You can use Claude Code via API pay per use or as part of a Claude subscription.

Codex – OpenAI’s answer to Claude Code is available via terminal, app, IDE extension, or cloud, with integrations for GitHub, Slack, and Linear. The CLI tool is open source under an Apache license, available on GitHub, and works via API pay per use or as part of a ChatGPT subscription.

Posit Assistant – The Posit (formerly RStudio) coding agent is designed for data work in R and Python, although Posit senior software engineer Simon Couch told me it would do fine for programming work for other uses and in other languages. It is available for the RStudio IDE (via a Posit AI subscription in RStudio), Positron IDE, and as a stand-alone CLI terminal app, although the CLI tool currently doesn’t feel as full-featured as the IDE integrations (that may change). As of this writing, you can also use API keys from Anthropic, OpenAI, and Snowflake Cortex and a GitHub Copilot account in Positron. Posit Assistant is being updated quite frequenty, Couch told me. If you tested it a month or two ago, it has more capabilities now.

Antigravity CLI – Google’s terminal-based tool will be replacing the Gemini CLI for unpaid and Google One users this month. There’s also an Antigravity IDE, SDK, and platform for orchestrating multiple agents. One advantage of Antigravity: Google offers a free plan with access to Gemini 3.5 Flash, Gemini 3.1 Pro, Gemini 3 Flash, Claude Sonnet 4.6, Claude Opus 4.6, and gpt-oss-120b, and what it describes as “generous” but undefined weekly rate limits based on “the degree we have capacity.”

OpenAI has signed a partnership agreement with Visa that allows the company’s AI agents to use the payment card for e-commerce transactions. The agreements lets users shop for everything from groceries and diapers to airline tickets without having to manually enter a lot of information.

“As AI agents become active participants in the economy, Visa’s focus is on ensuring that transactions are reliable, secure, and seamless,” Visa Chief Product and Strategy Officer Jack Forestell said in a statement, according to AP.

The pact means AI agents can complete purchases on a user’s behalf at virtually any merchant that accepts Visa. Details about the financial terms of the agreement, or whether specific transaction fees will apply, were not immediately detailed.

Developers are caught between the joy — or pressure — of using agents to ship 10x faster today and the dread of how they will maintain that code tomorrow. The gap between “vibe” code and code that can be deployed to millions of users is vast and easy to underestimate. Closing the gap requires care, expertise, and effort, with the payoff coming later. Agents are able to complete increasingly complex programming tasks but without the quality we need. What’s missing, and how can we fill the gap?

Why agent-generated code degrades: the bloat problem

Enterprise code has to clear three bars: it must be maintainable, reliable, and secure. Out-of-the-box AI agents can miss all three. Let’s focus on the biggest and most visible maintainability issue, which is bloat: redundant validation, defensive checks that cannot fire, near-duplicate functions, dead code that nothing removes. A None check on a parameter typed as dict. A try/except around a call that never throws. Two functions, identical except for the negation in their return statement.

Bloat varies dramatically by model. Sonar’s LLM Leaderboard runs every frontier model through 4,400+ Java tasks and analyses the code generated. To complete the benchmark, GPT-5.4 High generated 1,159,000 lines of code at an 81.05% pass rate, while Claude Opus 4.7 Thinking generated only 336,000 lines of code to return a better than 82.52% pass rate. Different models generate dramatically different code to achieve similar outcomes.

Bloat is not just messy. Carnegie Mellon researchers studied 807 open-source projects that had adopted Cursor, matched against 1,380 controls, measured by SonarQube. A short-term velocity gain disappeared by month three, while static analysis warnings rose 30% and code complexity rose 41% — both persistent. The harder it became to change the codebase and the more bugs it contained, the more the velocity was dragged down. Any experienced developer will know how this goes: quality problems compound until the code feels impossible to change and the only option is the dreaded rewrite.

Three forces produce bloat once a model is in use:

1. Agents do not feel the maintenance burden. Armin Ronacher, the creator of Flask, made the point on the Pragmatic Engineer podcast in late April. Humans feel the cost of bad code over time, and as Ronacher put it, “if the pain gets too big, you as a human are incentivized to fix the cause of your pain” — so we refactor. Agents do not. They obliviously extend bad structure indefinitely. A senior engineer’s job is to say no to unnecessary abstraction. The agent has no equivalent reflex.
2. Training rewards apparent completeness. Pretraining corpora are full of explanatory material — Stack Overflow answers, tutorials, README snippets — deliberately self-contained and verbose. Post-training compounds the effect: human raters prefer outputs that look thorough, so models learn that “comprehensive” reads as better. When uncertain which edge case matters, the safe move is to handle all of them. Each guard is locally defensible. The aggregate is bloat.
3. Iterative generation has no deletion pressure. Agents add but rarely delete. Removing dead code does not make any test go green, so superseded functions accumulate alongside their replacements. SlopCodeBench, a March 2026 benchmark across 11 coding models, found rising structural complexity in 80% of trajectories and rising verbosity in 89.8%. Agents continue to patch bad code, treating every task as if it’s their last.

AC/DC: the loop that compensates

What closes the gap is a loop around each iteration of agent work. The agent does what it is good at — generating code — and our job is to wrap that with three steps the agent cannot reliably do on its own. At Sonar we call this the Agent Centric Development Cycle, or AC/DC: guide, verify, solve.

Guide

Many teams overcorrect on context. They paste the style guide, three years of architectural decisions, and the entire onboarding doc into the agent’s instructions and expect output to improve. ETH Zurich researchers tested this and found the opposite: large context files often reduced task success against no context at all, and added 20% or more to inference cost.

Keep agent-facing context short — under 200 lines is a useful heuristic — and restrict it to fundamentals that can’t easily be inferred from the code: naming conventions, architectural invariants, what has been tried and failed. However, this will only get you so far, so make sure to provide specific context for each task. If you have architectural expectations, don’t expect the agent to guess them. Software architecture tools can be used to provide additional context in the guide phase.

Task shape matters too. Break the work into steps and agree on a plan; ask the agent to provide three solutions and evaluate the impact on quality of each. There is no perfect software architecture, and you understand the trade-offs in your codebase best, so think critically about the changes before they happen. Without this, the agent will confidently pick an option, seemingly at random, and the further it goes the harder it is to “unpick.” If you want to test this, ask three instances of your preferred agent to complete a task that involves some polymorphism and watch each one confidently suggest a different solution.

Verify

The most expensive verification mistake is doing it last. Reviewing 200-line pull requests (PRs) after the agent is done is the dynamic behind the Faros/DORA figures Addy Osmani highlighted: 98% more PRs merged in high-adoption teams, review times up 91%. Verification inside the loop is different. Unit test runs, static analysis, and security scanners produce output the agent can act on. This is where AI-native tooling belongs: purpose-built for the agent to invoke, not just for humans to consult through a UI.

Human reviewers cannot keep up. When agents merge twice as many PRs per week and each one takes nearly twice as long to review, doubling the review staff still leaves you behind. Automated verification is the only response that scales. Fast feedback has always been a fundamental tenet of good software engineering. Feeding it directly back to the agent protects the developer from simple mistakes and leaves them headroom to work on the harder ones.

Solve

If verification happens within the agentic loop, the agent can fix any issues whilst the code is being generated without expensive remediation steps. Static analysis tools can guide the agent on how to resolve the issue quickly. Some cases need human judgment — a None check at a system boundary may document a real precondition. But most of the work is mechanical. Automate the obvious fixes and let engineers spend their attention on the cases that are not.

The investment that compounds

Better models will keep arriving. They may not change the mechanism of bloat or the dynamics of compounding decay. The loop is what does — bounded tasks, sharp context, in-loop verification, and a deliberate “solve” step to clear bloat before it accumulates.

The same logic governs how autonomy should expand. Reduce human interventions only when the agent’s guide, verify, and solve cycle is making them redundant. Our biases can sting us here: an agent’s ability to write code can lead us to agree with it more than we should. Don’t trust blindly; wait for the evidence.

The teams that will be shipping enterprise-quality code with AI agents in 18 months are not the ones running the “best model.” They are the ones treating workflow as the engineering investment, with the seriousness once given to build systems and CI. The model is the tool, the workflow is the discipline. That is where the durable advantage compounds.

—

New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.

Hot AI companies can’t stop talking about forward-deployed engineers (FDEs), which are now very much in vogue. 

FDEs, in case you haven’t heard, are hired by companies looking (hoping?) to successfully deploy AI tools and services. It’s one of the hotter professions in a world still trying to understand the impact of AI on careers.

So, what exactly are FDEs — are they techy lone rangers like the ones OpenAI, Google and Microsoft are hiring? Turns out it’s not so much about individual engineers who swoop in to design and roll out AI deployments; it’s more about a team of engineers working together at customer sites.

At least, that’s the view at Amazon Web Services (AWS).

In fact, according to Taimur Rashid, managing director of the AWS Generative AI Innovation Center, the FDE concept pre-dates the current generative AI (genAI) gold rush. The same kinds of engineering teams were needed for the earlier machine-learning and cloud eras to help companies with deployments.

Rashid recently talked about how AWS sees FDEs as a profession in a conversation with Computerworld. And he weighed in on the desired job skills the company seeks in this increasingly AI-centric era.

What is an FDE? “We view it as a team. It’s a cross-functional team that has engineers, scientists, strategists, and folks that can piece technology and business together. In some cases, we do have to have security engineers in there, too. 

“I see them as anesthesiologists. They have to prep so many things, monitor things throughout. We see ourselves as a frontier deployment team helping customers adopt all forms of AI, whether it’s genAI, agentic AI, even emerging trends like physical AI. We’re helping these companies become frontier themselves.”

How does an FDE engagement begin, and how is it structured? “Where we see the forward deployed model is when customers come in — for example, we have our executive briefing center in Seattle and in Arlington, VA. When customers share what they’re trying to do, very quickly a customer’s like, “What’s the quickest way I can go and build something with you?” 

“We’ll forward deploy our people in, we’ll embed them in your business and we’ll go through these 45-day sprints that we typically design. Through those successive sprints, we’re building stuff together, we’re proving value, and then they can expand that to a much broader engagement.

Where do FDEs actually sit? Client side, internally, or in-between? “It’s mixed, and it largely depends on what the customer’s preference is. We’ve seen models where the customer has been very adamant that, ‘We want your teams with us in our business.’  In those cases, we forward deploy the majority of the teams on site. 

“We have models where customers are fine with you being wherever you’re based, as long as you’re still embedded virtually. And then there’s a hybrid. We deploy anywhere from five to seven people. Sometimes, the baseline is actually three.”

Will cost savings be the job of an FDE, or someone else on the team? “I expect these teams to be able to architect systems that have those cost requirements in mind, whether it’s use a different model for a different use case that doesn’t increase the per-token cost…or think about ways where you can use semantic caching. I personally think you may have a high spend in token consumption, but if you’re generating revenue, as long as the economics work out, then you’re at peace.”

What challenges have you gone through deploying FDEs in the real world? “One of the challenges worth highlighting is when customers get really excited about us forward deploying resources, what they end up realizing is [that] they’re not set up to absorb that right away. They realize they have to go through … process-related things, security access — all those operational things. 

“One very good example is the Commonwealth Bank of Australia. They said: “Prioritization’s a big thing, and if you forward deploy and you’re 100% dedicated, how do we ensure that our teams are also equally 100% dedicated?’ When you’re sitting in your office, you’re distracted by your day-to-day. So they said, ‘Why don’t we create a neutral ground in Seattle? You fly your people, we’ll fly our people. We’ll give them three weeks of dedicated time so they have no distractions.’”

Have you gone into projects where they want AI but have no security or governance ready? “I’ve been through this before, certainly. We do see customers that have security processes and capabilities, but it’s not as tight as it should be in the age of AI. Governance is the biggest area where customers right now have the biggest gap. I’m talking governance around agents. In the past two months, almost 100% of the conversation around agents is not about capability. It’s all about governance. 

“That is a big area right now where forward deploy teams are helping with governance education, and building the scaffolding for that.”

How does software engineering fit into the FDE model? “One of the greatest learnings is that as we forward deploy resources and get customers to take AI and integrate it into their systems, the knowledge of software engineering is so important. Today, a customer can use one of the Claude models and scan their code base and look at vulnerabilities. 

“The tough part is not assessing those vulnerabilities, it’s remediating [them]. Remediating [them] requires software engineering experience, because you have got to merge code, test it, deploy it. We largely see that the frontier software development teams are smaller and they’re managing agents that are doing various tasks across the software development lifecycle.”

AWS has many models at your scale, open source, closed — it’s more complex than what other AI vendors offer.  How do you nail down the talent? “It’s massive, and when you look at not only scale, it’s the complexity of the stack. We take an approach where we fundamentally do three important things: No. 1, we want to ensure people understand concepts; they have to understand pre-training, post-training, reinforcement, fine-tuning. 

“Secondly, we make sure that our teams are well versed in their first-party services. The third thing is that by design, AWS has always been about choice. We say, ‘Let’s do 80-20 here. What is 20% of those specialties that we need to have, which can cover 80% of what most customers are trying to do?’”

What skills should software developers learn to move into FDE work — the top three or four things? “We look at three categories. First category is entirely functional. Are they more engineering specific? Are they science specific? Are they security specific? Our litmus test is not only knowledge of the function, but the actual hands-on work that they can do with it. Secondly is around domain. I focus on what is their domain understanding across the whole AI lifecycle? The third thing is cultural. We are looking for folks that are okay at dealing with ambiguity, being good at stakeholder management, and having that startup mindset. 

“This AI transformation’s not for the faint of heart.”

There’s a rush for FDEs from OpenAI, Google, and others. What’s different about what you look for? “I don’t know entirely what the others are looking for, but I’ll tell you what we have been looking for in the past. When we hired solution architects, it was about systems level understanding. But what I see more and more is hands-on experience, cultural mindset, all these things are very important. If I had to pick one thing that is going to be very important in the talent that we either upskill or future talent that we hire, it has to be the application of AI towards software engineering and system integration tasks.

You’re upskilling AWS talent as well? “We do. You will see some publications coming out in the next couple of weeks around how do we do this across our software teams and how does that translate to customer-facing roles.”

We’ve all heard the mantra from the quants in the business community: you can’t manage what you can’t measure. And if that’s true for human intelligence, it should be true for the artificial kind too.

How do we measure agents and large language models (LLMs)? We’re just beginning to come up with statistical metrics. Here are several of the most common metrics that designers and users toss about when they’re evaluating a model.

[ See also: 27 questions to ask before choosing an LLM ]

Time to first token

How long does it take to generate the first token? For real-time applications with time constraints, faster responses can be essential. It’s well-known that people hate waiting even a few milliseconds. The teams that develop user interfaces learned decades ago that it’s important for the software to respond quickly when a human is waiting for an answer. Even a few seconds of delay mean that the human will wander off to another window to check some email or place some bet on a prediction market. Time to first token is a good measure for models that will be working directly with the fickle human intelligences and their latent attention deficit disorder.

Time per output token

Take the total time it takes to respond and divide by the total number of tokens. The time to first token measures how long it takes to start a response and this measures the average speed as the model through all of the tokens. In basic LLMs, this value is generally fairly constant. Once the prefill is done and the LLM enters the decode phase, the output tokens usually appear at a constant stream. When the output is long enough, the startup time to first token is amortized away. In some of the more complicated architectures with loops for planning or gathering data from various tools, the average speed can vary as the model shifts in and out of making agentic decisions.

Tokens per second

This is just the reciprocal of the average time per token. Sometimes it is reported separately for different stages in the pipeline.

Throughput (requests per minute)

If a system supports more than a single user, tracking the number of different requests makes sense. These throughput numbers can be quite useful for measuring the power of some of the newer pipelines that are more efficient when they’re answering multiple prompts at the same time.

Error rate

Not every request gets an answer. The error rate tracks how often rate limits, timeouts, or model “refusals” get in the way. Better accounting tracks each independently because the number of failures in each category can be very different.

Token efficiency

Not all work tokens are visible and not all tokens are part of the final outcome. This measures how much work is done to produce the final result. As models become more complex or agentic and the pipelines become more sophisticated, the efficiency tends to drop. Agentic reasoning and strategic planning typically require more tokens that don’t appear in the final answer. This is generally a measure of how expensive a model might be to run.

Tail latency

It’s all well and good to measure the average time to answer, but in some cases a few very slow responses can really color people’s judgement. Some applications require good performance all of the time. Would you want to ride in an autonomous car that gets steering instructions very quickly “on average” instead of always? What if that’s only 99% of the time? Tail latency uses a mixture of queuing theory and detailed measurements to track the worst moments in the long tail of the latency graph. It’s useful when even occasional delays are problematic. 

Total cost of ownership

Projects that use an API or buy output from providers just look at the cost per 1M tokens. They’re effectively renters. The groups that are buying GPUs and paying for electricity, though, will add up these costs and other indirect costs like depreciation and maintenance to come up with a number that estimates how much the tokens really cost to produce. This value will depend upon demand and utilization rates—that is, on how many users are sending in prompts and how efficiently the model fits in a particular GPU and its RAM.

Parameters

Many models have numbers in their name followed by a B. This is meant to roughly capture the number of parameters, or the number of variables the model uses to generate outputs from inputs. The number “70B” means that there are about 70 billion parameters in the model. This is a good estimate for the complexity of the model and the size of the training set that has been stuffed into it. Generally bigger numbers mean a larger amount of information is hiding inside the model. It often means that it will take a bigger GPU with more RAM to generate an answer with it. It’s not a very precise number, though, because there are many other areas of the architecture that can influence whether the model can generate the answer you want inside your budget. There continue to be advances and it’s not uncommon for someone to claim that a new model with X parameters is better than an old model with 2X or 3X parameters.

Hallucination rate

While everyone wants LLMs to generate accurate output, measuring it can be difficult because deciding what’s accurate is sometimes complicated. One approach is to ask the LLM to summarize a document. Then another model evaluates how well the summary matches the original. While this may not catch all subtle slips, it will capture enough of the worst departures from reality. Some researchers have built complex test sets with curated answers. The LLMs that deliver the expected answers get the highest scores. Some common benchmarks are TruthfulQA, HaluEval, QAFactEval, and Vectara’s Hallucination Evaluation Model (HHEM).

Toxicity and bias scores

If measuring accuracy is difficult, building a metric to detect toxic or biased output is even more challenging because the definitions can be so protean. Still, some teams have built LLMs that key on particular concepts or word choices. They can detect some of the most obvious red flags that could generate political trouble. Some well-known versions include Granica Screen and Perspective API.

PII leakage

One of the biggest fears is that LLMs will somehow absorb information that may be considered personal and private. Some of the simplest measures can be as simple as regular expressions that look for the sixteen digit numbers used for credit card transactions. Many of the model builders work on eliminating personally identifiable information (PII) from the training set before beginning.

Tool-calling accuracy

As models grow more complex and agentic, they often gain access to various tools or Model Context Protocol (MCP) gateways that can help them find the best answers. Not all models take advantage of this help. The tool-calling accuracy scores count how often the models choose the best tool for the job. One particular example of this measurement is BFCL (Berkeley Function Calling Leaderboard).

Prompt sensitivity

The value captures how small changes in the language of the prompt induces the model to produce different results. It’s like a derivative from calculus class, although it’s generally computed experimentally using some collection of test prompts. There are a number of different approaches that depend upon different types of changes. Some test sets are built with small rephrasing of the request that are semantically the same. Others mix together different ways of specifying the problem, some with examples, say, and some without. Some specific examples include PromptSE and ProSA.

Semantic similarity and conciseness

Some metrics evaluate the answers by comparing them to a set of gold standard answers. This often involves feeding them to a vector embedding model and searching a retrieval-augmented generation (RAG) database for similar answers. This can track how concise or fluffy the answers might be as well as looking for how much variability might be introduced through changing parameters like the temperature. One common example is the BERTScore.

Grounding score

Many systems that combine an LLM with a vector search tool for RAG measure the effectiveness of the combination with a benchmark like the grounding score. The LLM is presented with extra data from the vector search and the benchmark measures how closely it follows this extra information. That is, how much of the answer comes from the provided source documents and how much is synthesized using the data in its training set. Some examples include RAGAS, TruLens, ARES (Automated RAG Evaluation System), RGB (Retrieval-Augmented Generation Benchmark), HaluEval, and HalluHard. A similar concept is called “context adherence,” “context precision,” “context recall,” or “faithfulness.”

Model variability

Most LLMs fold in a certain amount of random entropy, and this amount is often controlled by a parameter called the “temperature.” The model variability is a measure of how much the answers will change between runs. Some applications like chatbots require a certain amount of variability because the randomness adds a bit of “life” to the answers. Other applications like those in mission-critical areas like law or medicine will undermine confidence if the answers vary.

Format compliance rate

In some roles, LLMs are asked to produce data in strict formats like JSON or CSV. This is often important if the data will be fed into some pipeline for further processing or storage. The format compliance rate tests a number of common formats and measures how often the LLM returns semantically correct data. Agentic systems that glue together multiple LLMs and other tools rely heavily on LLMs with good scores on this benchmark.

Instruction following

Some prompts include very specific instructions and the adherence can be measured empirically. For example, some prompts will ask the LLM to produce exactly 300 words or a poem in rhyming couplets. These tests use a collection of sample prompts that ask for answers that can be easily measured. Some specific examples include IFEval, FollowBench, and the BFCL (Berkeley Function Calling Leaderboard), a value that is mentioned above in the section on tool usage.   

Subgoal success rate

As agentic models become more common, it’s helpful to track how well the model performs on each of the various parts of the agent’s strategic plan. All of the metrics here can be broken down and tracked for each of the subgoals.

Plan stability

Agentic models start with a plan. Some of them are smart enough to abandon the plan or at least adjust it as the work evolves. Plan stability measures how often the plans are adjusted. A high rate of adjustment could mean that the agent is a bad planner or just flexible or maybe both.

Self-correction score

Some agents are able to dive deeper and recognize their mistakes. The self-correction score measures how often the model will make a mistake and then recognize it, either on its own or after being prompted with the question, “Are you really sure?”

Jailbreak resistance

Some users try to find clever ways to lure the LLM into tossing aside any restrictions on topics or answers. In the past, some LLMs could be fooled by being told the answer was part of a play or a work of fiction. So discussing forbidden subjects wasn’t a problem because it was all pretend. Newer models have more elaborate defenses. Measures of the ability to resist deception include JailbreakBench, AgentHarm, and Tele-AI-Safety. 

Prompt injection vulnerability

Sometimes untrusted data from extra sources or skills may include malicious instructions that can exploit the LLM. Benchmarks such as Skill-Inject and SPIKEE (Simple Prompt Injection Kit for Evaluation and Exploitation) work with known attack vectors and measure how susceptible a model is to targeted prompt injection attacks.

Copyright infringement score

Some LLMs can regurgitate the data in their training corpus in a way that seems like plagiarism or copyright infringement. This can be an issue when the training material wasn’t carefully licensed. The copyright infringement score measures how often the LLM may parrot the training material a bit too closely. Tools for defending against this include CopyrightCatcher and DE-COP.

RULER

How well can a model extract information from the entire context? NIAH (needle-in-a haystack) benchmarks measure how well a model can retrieve small, crucial bits of information from long contexts. RULER takes NIAH tests further with the ability to vary the types and quantities of needles, the size of the haystack, and the complexity of the task.

GSM8K 

The developers of GSM8K (Grade School Math 8K) set out to benchmark an LLM’s ability to tackle multistep mathematical problems, so they gathered 8,500 problems that are common in grade school math classes. While the focus is explicitly on solving math homework problems, the benchmark also measures the ability to construct reasoning chains.

GPQA

The Graduate-Level Google-Proof Q&A is composed of hundreds of hard questions that might normally be answered by humans in graduate school, generally in science. To make the benchmark harder, the researchers focused on questions that non-experts often get wrong. The term “Google-proof” means that the benchmark includes questions that can’t be easily answered by asking a search engine.

MMLU-Pro

The MMLU-Pro benchmark builds on the Massive Multitask Language Understanding dataset to test a model’s understanding of a broad set of scientific knowledge. It includes more than 12,000 questions about general scientific fields like biology, chemistry, economics, and law. 

MBPP

Google created MBPP (Mostly Basic Python Problems) to evaluate how well a model was solving coding questions. Each problem comes with a statement, a gold standard solution, and several similar test cases. The number of accurate answers to these questions is a good measure of how well the model will solve many of the simpler Python coding problems presented by users.

SWE-bench

This collection of several thousand software engineering challenges evaluates how well a model solves programming problems. The developers created it by selecting a number of issues and corresponding pull-requests from a dozen or so Python projects. After some limitations appeared, the creators expanded the set by creating SWE-Bench+, SWE Bench Verified, and SWE-Bench Pro.

LMSYS Chatbot Arena

Instead of creating a fixed set of test prompts, the Large Model Systems Organization’s Chatbot Arena is a dynamic system that feeds the same prompt to different models and then asks humans to pick the best results. These head-to-head contests produce an Elo-like rating that is similar to the one used to score chess players.

Price

The rest of these metrics are useful, but as the real estate agents say, the three most important numbers on a property listing are price, price, and price. The cost is a bit less important for measuring AIs, but only a bit. Price can make a huge difference between a project being profitable and a moneysink. When the cost for each inference is a tad too high, it’s impossible to make it up with volume.

The key caveat is that a cheaper model isn’t a good idea if it generates answers that are filled with hallucinations or worse. The quality of the answers can differ greatly, and saving a few pennies can be a mistake. To make matters more complicated, there’s an explosion in different styles and approaches. Sometimes it makes sense to pay a bit more for a model that delivers answers with the right vibe.