We spent the last decade building entire finops departments just to decipher what the cloud bill was saying. Just when we figured out how to stop leaving idle compute instances running, generative AI introduced an infinitely more opaque, faster-moving layer of spend.

AI bill shock has spread like a slow moving hurricane across the industry. Aside from the spike in large language model (LLM) costs, the deeper architectural problem is attribution: where exactly is the money going, and exactly what value is it delivering to the business?

When API calls are wrapped in layers of automated agents and prompt templates, your application becomes a black box that consumes capital to spawn tokens. If a features engine is burning thousands of dollars a month just to have an LLM output pleasantries or process low-value data, that isn’t innovation, it’s an uncovered manhole, a gaping liability. Architectural maturity means treating tokens like any other constrained resource.

Fortunately, there are a few good levers we can pull to control AI spend. Here are the five key tools for taming the beast.

Model routing

Don’t use a nail gun if a thumbtack will suffice.

The most powerful models like the latest Claude Opus are some of the most sophisticated software systems ever built. They are capable of handling extremely complex and subtle use cases. They are also voracious beasts when it comes to compute (and therefore dollar) consumption. They are often overkill.

We naturally start with the most powerful model we have when prototyping and sketching out APIs and requirements. That’s because we are in the “get it working” mode and we don’t want to be tangling with model limitations. But for production, it is absolutely essential to dial in on what is the least powerful model we can use.

Sometimes we can address this by tuning the model choice and discovering what will work manually. But in larger enterprise architectures, we can move to a conditional routing layer. Frameworks like RouteLLM or Semantic Router can dynamically direct simple classification, basic text parsing, and user intent detection to fast and cheap utility models like GPT-4o mini or Claude 3 Haiku.

You can also use advanced AI gateways for routing and spend allocation. At the infrastructure level, wrapping this logic in a dedicated AI API gateway (such as Kong, Cloudflare AI Gateway, or Portkey) gives you the ultimate control. A mature gateway allows you to implement “cascade routing” to automatically fall back to a cheaper or open-source model if your primary LLM hits rate limits or latency spikes. More importantly, an AI API gateway centralizes your telemetry, stripping away the black box so you can enforce hard token budgets per tenant or per microservice. You are essentially building a load balancer for cognitive tasks.

Of course, AI gateways also introduce complexity and cost themselves, so these factors must be balanced into the equation. But the bottom line is, there is a right-sized model for your project. You can save a bundle by routing requests to a good-enough, Goldilocks model.

A side note: Another trend in model routing is “repatriation of compute.” That entails running models (often open-source models) on local hardware, instead of renting generative AI endpoints on the cloud.  This implies all the trade-offs and benefits of other on-prem options.

Semantic caching

In conventional web apps, caching is a solved problem. You put Redis in front of your database, map an exact query to a cached payload, and you’re done. But LLMs break traditional caching because human language is infinitely variable. “How do I reset my password?” and “I forgot my login info” are entirely different strings, but communicate identical intents.

If you rely on overly precise matching, your hit rate will be effectively zero, and you will pay the full token price for every phrasing permutation.

Enter semantic caching. It’s like regex for meaning. Instead of hashing the raw text string, you run the incoming prompt through a fast embedding model and do a similarity search against previously answered prompts. You define a confidence threshold (say, 0.92), and if it is met, you serve the cached response (bypassing the LLM and its cost entirely).

The architectural effect is twofold. First, a semantic cache hit reduces your inference cost for that query to exactly zero. Second, it drops response latency from seconds to milliseconds. You can implement this using purpose-built frameworks like GPTCache, or, if you prefer a leaner infrastructure, by using pgvector inside your existing PostgreSQL database to store and match the embeddings.

However, just like dynamic routing, semantic caching demands a rigorous architectural calculus. You are trading generation costs for embeddings and lookup costs. To check the cache, you still have to tokenize the prompt, call a cheap model (like text-embedding-3-small), and execute a vector search.

You also introduce the very real danger of semantic flattening. Tuning the similarity threshold is a delicate art. Set it too low, and your application starts serving general, recycled answers to nuanced user questions. For open-ended, creative, generative AI applications, semantic caching is practically useless. But for retrieval-augmented generation (RAG) implementations, customer support bots, and internal knowledge bases where users ask the same 20 questions a thousand different ways, it is the most effective cost-reduction lever you can pull.

Prompt caching

Whereas semantic caching stores the response to a given intent, prompt caching stores the data needed to contextualize the question. When the user enters a prompt, it is sent to the generative AI endpoint. But instead of your application having to repeatedly gather and send the massive contextual info needed to frame the prompt—from a RAG pipeline, a database, or other source—that information is already pre-loaded in the context cache. The model simply applies the new question to the cached data, slashing both your latency and your input costs.

The core idea behind prompt caching is that the AI engine itself retains the relevant contextual data for incoming prompts. Tokens cached this way get a massive discount (50% to 90%) compared to raw input tokens. But the details of how it is handled vary between providers.

For OpenAI, prompt caching is automatic, but requires attention to how your app uses the endpoints in order to leverage them. OpenAI enforces a 1,024-token minimum threshold. More importantly, it requires an exact, byte-for-byte prefix match starting from token zero. If you inject a dynamic variable (like a timestamp or a user ID) at the very top of your system instructions, you instantly invalidate the cache and pay full price for every token that follows.

Some other providers, like Anthropic Claude and Google Gemini, use explicit caching. You must deliberately engineer the cache into your architecture, either by hitting a dedicated API endpoint to upload the static document (the context data) with a defined time to live, or by injecting strict cache control breakpoints into your JSON payload. Explicit caching often requires paying a “write premium” up front, but it gives you the control to guarantee your heavy RAG data stays alive in memory for hours rather than minutes.

Prompt discipline

Just because a model can consume gargantuan amounts of tokens, doesn’t mean it should.

With models now capable of ingesting one million to two million tokens in a single prompt, there is a temptation to brute-force use cases into submission by dumping the entire PDF, the entire codebase, or the last three years of logs into the payload and let the LLM sort it out.

This is a finops blunder. You are paying for every single token you send. Furthermore, from an engineering standpoint, stuffing the context window runs into the very real “muddy middle” problem (aka “context rot”), where the model loses resolution into the middle of the content. As the payload grows, performance and accuracy suffer.

The cure is a strict RAG diet. Naive RAG implementations might pull the top 20 results from a vector database and blindly concatenate them into the prompt. Architectural maturity requires a more ruthless filtering process before those tokens ever reach the expensive LLM.

This is where reranking becomes key. Reranking takes the compiled prompt context and makes sure it is efficiently structured before it hits the LLM. Instead of sending raw search results directly to your primary model, you introduce a tiny, highly-efficient cross-encoder (like Cohere Rerank or an open-source BGE model). You cast a wide net in your cheap vector database to fetch 50 potential matches, then pass them to the reranker to score their actual relevance against the user’s specific prompt. The reranker ruthlessly trims the fat, allowing you to pass only the top three or four hyper-relevant chunks to the expensive frontier model.

You are trading a massive, expensive LLM context payload for a tiny, cheap reranking step. The architectural math here is almost always a resounding win. While a cross-encoder might add 100 milliseconds of latency to the pipeline, it routinely slashes prompt token counts by 80% or more, while simultaneously increasing the accuracy of the final generation. In the generative AI spend playbook, giving your LLM less to read is one of the highest-leverage moves you can make.

Response constraints

By default, LLMs are polite, conversational, and verbose. If you ask an API for data, it desperately wants to start with “Certainly! Here is the JSON you requested,” and end with “Please let me know if you need anything else!”

In a web interface, this is charming. In an application’s API architecture, it is a leak in the finops hull. Generation tokens (output) are almost universally priced 3x to 5x higher than prompt tokens (input). Every pleasantry is burning your most expensive commodity.

Taming LLM responses requires strict API-level enforcement. First, max_tokens must be treated as a hard circuit breaker to prevent runaway generation loops, not as a formatting tool. (Relying on max_tokens to shorten answers often just results in truncated, unparseable JSON.) Second, use stop sequences (stop_words). If your model only needs to output a SQL query, setting a stop sequence for the markdown closing tag (“`) or the word “Explanation:” forces the model to sever the connection the moment the computational work is done.

Modern APIs also offer structured outputs (or strict JSON mode), which forces the model to adhere to a rigid schema you define. By combining this with zero-tolerance system prompts (“Output ONLY valid JSON. Omit all conversational text and pleasantries. Failure to do so will break the system”) you force the LLM to behave like a traditional API endpoint rather than a chatbot.

The architectural balance here is to avoid degrading the model’s reasoning ability by strictly limiting its input. LLMs do not have internal working memory, they “think” by iterating on token output. If you force a model to solve a complex logic problem and output a single boolean true or false without giving it space to reason first, its accuracy will plummet. The finops goal isn’t really to force minimal words; it is to ensure that every expensive output token is performing actual computational work (like chain of thought reasoning).

For more on how to manage AI back ends, take a look at this article.

How to effectively consume generative AI to satisfy user needs is the first battle. The second is keeping the finances from undermining that hard work. The five keys discussed here should give you all the power you need to make sure the engineering and the cost controls are working together.

Nvidia CEO Jensen Huang argues it’s the “most illogical thing in the world” to believe AI will kill SaaS, and says the markets “got it wrong” in sending SaaS stocks tumbling. Why “illogical”? Because, as analyst Benedict Evans stresses, the fact that everyone can now easily spin up code actually doesn’t solve any SaaS problems, because creating code and tools is the easy part: “The hard part is knowing that you need a tool for this in the first place and then knowing what the tool should do.”

If that’s true, then what Shopify is doing with Sidekick is potentially revolutionary by embracing and extending the traditional SaaS model.

In December, Shopify introduced custom app generation through Sidekick. A merchant (Shopify customer) describes a tool, and Sidekick writes the code, using Shopify’s interface components and connecting to its Admin API. It’s a cool option and has been very popular. Shopify says merchants created almost 4,000 custom apps in the first three weeks following the release.

This isn’t surprising: Shopify is delivering the customization customers have long wanted from enterprise SaaS while ensuring they still enjoy the comfort of ongoing support and platform stability.

I suspect we’ll see more enterprise SaaS like this, despite silly predictions that AI would kill SaaS. Cheaper code gives customers a way to fix the things they dislike about an application without replacing everything they do like. For a vendor willing to accommodate that, AI could make its product considerably more useful and, hence, much “stickier.”

The feature that never ships

Cloudflare’s Jeremy Morrell discusses this in an excellent essay on extensible software. “In the past year, your users have suddenly acquired the ability to speak code into existence,” he writes, allowing for “markets of one.” His argument is that software should let customers put that ability to work, building additions around an established core.

Morrell has a commercial interest here, which he discloses: He works at Cloudflare and thinks its infrastructure is well suited to running those additions. He’s not wrong. But it’s the general opportunity to match customizability with stability that matters so much.

There are really (really) good reasons why vendors haven’t allowed customizability of their core products. Enterprise software companies limit deep customization to prevent high maintenance costs, security risks, and broken system updates. Oh, and to prevent their products from becoming convoluted bloatware. No matter how much you may want feature X, that feature would likely be irrelevant to almost everyone else, so adding it would complicate the product for the majority. No one wants that.

Customers have long paid consultants to customize software or they’ve hired developers to do it themselves (all while taking care not to violate their vendor licenses). Morrell points to Salesforce, which has spent years letting businesses build their own logic into its platform. In other words, AI didn’t invent extensibility. No, AI just makes it incredibly cheap. There’s a meaningful opportunity here for customers who can describe the change they need but have never been able to get it built. Some will save money on software or customizations they previously bought, while others will finally get software that nobody could profitably sell them.

Following the money

This change, however welcome for enterprises, won’t be comfortable for developers selling a narrow reporting tool or a small workflow improvement. Their customers may decide that an LLM-generated extension is good enough. An AI-built customization needn’t match every feature of the commercial product. It only has to do the job that the customer was paying for.

The platform underneath that extension has a different calculation. Every useful tool built against its data and APIs gives the customer another reason to stay. AI can make an individual feature less valuable to sell while making the product that hosts it more valuable to own. (Yet another reason to tsk-tsk foolish eulogies for SaaS companies.)

Ultimately, this should benefit anyone who both knows the business and can build more efficiently for that business. Benedict Evans makes this point in his newsletter, arguing that cheaper coding doesn’t solve the problem of discovering what software ought to do or persuading an organization to use it. A useful idea that spans several departments still needs those departments to agree. Even excellent code can’t sign off on a new way of running accounts payable.

Evans also describes how improvised tools eventually become important enough to need formal ownership and support. Anyone betting that companies will happily maintain everything they can generate should spend some time with the person who inherited the last departmental spreadsheet. Silicon Valley folks with little knowledge of real-world enterprises might think enterprises will vibe code all their software now, but they won’t. That’s not how software works because it’s not how people work.

This cuts both ways for established software vendors. Customers may continue to need their expertise while becoming less willing to accept a product’s limitations. A vendor that makes customization easy can benefit from that impatience, while one that blocks it has given the customer a new reason to investigate alternatives. The difficult decisions will come when customization interferes with what the vendor already sells.

I wrote in 2022 about Microsoft’s complicated relationship with open source, and how a company’s broad commitment to openness can collide with a division’s revenue targets. The same tension applies here. A CEO can be excited about letting customers build anything. The executive responsible for selling the premium reporting package may be less enthusiastic. Resolving that disagreement will determine what customers actually get to build, however capable the AI becomes.

Getting comfortable

Customers have their own calculations to make. An application shaped around the way your company works can be wonderful, but it can also be a pain to leave.

There are limits to what Shopify’s generated apps can do today. They are admin tools, not storefront or checkout extensions. Shopify also tells merchants to test them before installation because they can change customer-facing data. A generated app can be wrong, and running it inside a familiar product doesn’t relieve the merchant of checking its work.

Shopify lets merchants inspect and edit the generated code and restore an earlier version. That’s useful, but code written for Shopify’s Admin API still depends on Shopify. Having the source doesn’t give you another company ready to run it unchanged.

AI might eventually make moving those extensions easier, too. But recreating a screen is one thing; reproducing the behavior it relies on throughout a business is a considerably bigger undertaking. That includes the assumptions about records and transactions that the custom code gets to take for granted while it runs on the original platform.

None of this makes customization a bad deal. Customers can gain more control over their daily work while, somewhat ironically, becoming more dependent on a supplier. For SaaS vendors, that’s a good commercial reason to let customers finish the bits their developers will never get around to building, even when it costs them an add-on sale (and potentially some support headaches, depending on how they’re architected for extensibility). The customer is doing work that makes the underlying product more valuable.

AI, in short, has the potential to make SaaS dramatically more valuable, not extinct.

Microsoft has released Visual Studio Code 1.136, an update to the popular programming editor that previews an Agent Merge feature and introduces two experimental features for agentic coding.

VS Code 1.136 was released September 2 and can be downloaded from code.visualstudio.com.

Agent Merge, a preview feature, helps take a pull request across the finish line, Microsoft said. Agent Merge asks an agent to address review feedback, fix failed checks and merge conflicts, and rerun workflows until the pull request is ready to merge. To use Agent Merge, turn on the setting chat.agentMerge.enabled. Then you can enable Agent Merge for a session by running “Enable Agent Merge for Active Session” from the Agents window or selecting the “Agent Merge” button in the title bar.

VS Code 1.136 also introduces multi-root workspaces, an experimental feature that allows developers to work with GitHub Copilot and Claude agents across multiple project folders in the editor window chat view. Agent hooks remain scoped to a single workspace folder, Microsoft said. If hooks are detected in multiple folders, VS Code prompts the user to select the primary folder from which to load them.

Another experimental feature in VS Code 1.136 is chat backgrounds in the Agents window. This allows users to personalize the Agents window with a theme-aware pattern of built-in VS Code icons or an image of their own. Also new for chat are enterprise controls for dictation data. The new controls allow administrators to manage the dictation model and language-model transcript cleanup through enterprise policies. Microsoft said it was possible to require on-device transcription and disable language-model cleanup, so dictation remains available while preventing dictation data from being sent to cloud transcription or a Copilot model.

OpenAI launched GPT-6 Astra on Thursday, disclosing that the new flagship model has crossed the “Critical” threshold for cybersecurity risk under its Preparedness Framework, a classification the company said triggers additional deployment restrictions.

“GPT‑6 Astra is rolling out today to a limited set of organizations and over the coming days will become available to all ChatGPT Plus, Pro, Business, and Enterprise users, as well as through the OpenAI API and AWS,” OpenAI said in a statement.

Enterprise administrators must manually enable Astra for their workspace, since access is off by default at launch, according to the company.

Developers can access Astra in the API as gpt-6-astra or through Amazon Bedrock, OpenAI said, priced at $10 per million input tokens and $50 per million output tokens. Pro, Business, and Enterprise users also get a variant called Astra Pro, and the company said Astra supports Zero Data Retention for eligible API customers.

Company claims perfect score on exploit benchmark

OpenAI said it tested Astra without production safeguards on ExploitBench, and that the model scored 100%, up from 78.5% for predecessor GPT-5.6 Sol. On ExploitGym, a broader exploit-development benchmark, the company said Astra reached a 42.4% success rate against 30.3% for Sol, while using fewer output tokens.

“Its ability to identify and develop zero-day exploits can help defenders find and patch weaknesses, but it also creates a need for stronger safeguards,” OpenAI said in the blog post.

Recommended for you

OpenAI also tested Astra on vulnerabilities disclosed in the three months before launch, to check whether it could find flaws on its own rather than recalling old exploits from training data. The model found two new zero-day vulnerabilities during that test, OpenAI said, and it is now disclosing both to the software makers involved.

Sanchit Vir Gogia, chief analyst at Greyhound Research, said the Critical label is a disclosure event rather than a capability event.

“Astra’s capability did not change between 10 August, when OpenAI said Critical capability could not be ruled out, and September 1, when it said the threshold was met,” Gogia noted. “The testing changed. The model did not.”

That inverts the obvious enterprise response, he said.

“Astra is now the only frontier model whose cyber capability an enterprise actually knows, because it is the only one measured against a published threshold, while every unlabelled model already sitting behind enterprise credentials has never been measured that way and will not be until its vendor chooses to measure it,” Gogia pointed out. “Those models are not safer.”

OpenAI said the public version of Astra will refuse advanced offensive tasks such as generating proof-of-concept exploits, though it plans to loosen those restrictions for vetted defenders through a program called OpenAI Daybreak in the coming weeks.

The launch follows OpenAI’s rollout of GPT-5.6 Sol, which the company said scored 73.5% on ExploitBench at launch, and comes months after Anthropic’s Fable and Mythos models were briefly pulled from export markets over similar concerns.

Governance shifts from the model to the harness around it

Gogia said the bigger shift is that reasoning now translates into state change, since a wrong chatbot answer is an information problem while a wrong agent action inside a customer-record system is an operating event.

“The governance unit therefore moves off the model,” he said, arguing the relevant question is no longer which model is approved, but how much damage a given identity can do before a control intervenes.

Amit Kumar Jena, head of AI development at Kanerika said the visibility problem is concrete: when an agent acts through a user interface, systems of record log the action as a person, so an agent that updates 400 ERP rows shows up as a service account making 400 updates, with no record of which instruction or model version produced them.

“You lose granularity inside the exact system a regulator or auditor will ask to see,” Jena added.

OpenAI said it built a new evaluation, informed by an incident involving Hugging Face, to test whether a model given an impossible task would exceed its authorized scope.

“Compared to GPT‑5.6 Sol, which without production safeguards went beyond the authorized target 48% of the time, GPT‑6 Astra did this in 0% of cases,” the statement added.

Gogia said the more uncomfortable finding is that Astra behaves better and watches worse: OpenAI reports decreased chain-of-thought monitorability against Sol, with Astra less likely to reveal incriminating reasoning, and its monitoring covers OpenAI’s own external deployment but nothing published extends that telemetry to customers. “OpenAI being able to monitor Astra does not mean an enterprise can audit Astra,” Gogia said.

The article originally appeared on CSO.

Meta is upgrading its Muse Spark family with a new model aimed at long-horizon coding tasks and agentic software development while keeping its standard API pricing unchanged, giving development teams access to improved capabilities without paying more per token.

The new Muse Spark 1.3 model claims to complete coding tasks using about 25% fewer tokens and 20% fewer tool calls than its predecessor, potentially giving development teams more efficient AI-assisted coding at the same per-token price, the company wrote in a blog post.

The efficiency gains come from Muse Spark 1.3 taking fewer turns when they aren’t needed and producing less verbose responses, while improving its ability to sustain longer-running tasks by generating its own context, correcting gaps in its plan, following complex instructions, managing multiple workflows and recognizing its limitations, it added.

Productivity and cost benefits

For developers, the combination of improved capabilities and unchanged pricing could boost productivity while encouraging teams to use AI for a broader range of coding tasks, said Pareekh Jain, principal analyst at Pareekh Consulting.

“Since it is better at understanding complex prompts and following instructions, developers can get better results with fewer retries and potentially use less AI for the same tasks, while also making it easier to extend the model to a broader range of coding and software development workloads with reduced human supervision due to its improved reliability,” Jain said.

These workloads could include understanding large codebases, debugging, changing multiple files, using developer tools, and testing code, Jain added.

That, in turn, could potentially reduce inference costs related to AI-assisted software development for CIOs.

“CIOs potentially get more work for the same token price if Muse Spark 1.3 completes tasks with fewer tokens, tool calls and retries,” Jain further said.

Lower cost per task may not translate into lower overall AI spending

However, that reduction in inference costs may not translate into lower overall AI spending.

Primarily because, most enterprise teams will use the more capable model for longer running and more complex tasks, offsetting the reduction in inference costs, said Abhishek Satapathy, principal analyst at Avasant.

That scenario is more likely as enterprises tend to increase their use of a resource when efficiency improvements make it cheaper to consume, a dynamic reflected in the Jevons Paradox, Jain said.

Enterprises should therefore look at the cost of successfully completing a task rather than the model’s per-token price, Jain added.

Efficiency claims remain disputed

Meta’s benchmarks of the new model, which were conducted internally, also come with caveats when viewed against independent evaluations.

While Meta claims that Muse Spark 1.3 uses fewer tokens than its predecessor, independent evaluations have produced different results, raising questions about how consistently those efficiency gains will translate across workloads.

Artificial Analysis, an AI benchmarking platform, found that Muse Spark 1.3 generated about 100 million tokens while completing its Intelligence Index evaluation, compared with a 71 million-token median across the models it evaluated.

While these results do not directly replicate Meta’s internal coding evaluation, which measures token use on specific coding tasks, they highlight how model efficiency can vary significantly depending on the workload and evaluation method, said Manoj Chandra Jha, principal analyst at Nord-IQ Research.

“The differing results reinforce the need for enterprises to test the model against their own workloads before budgeting for expected efficiency gains,” Jha added.

Eesel AI, an AI productivity platform, also cautioned users against Meta’s claims about Muse Spark 1.3’s performance.

“Meta’s scorecard compares 1.3’s max mode (still gated behind safety testing at launch) against 1.2’s xhigh mode, so part of the jump is a reasoning-tier change, not a clean generational leap,” it wrote in a blog post.

That makes it harder for enterprises to determine how much of the reported performance gain comes from improvements in 1.3 itself and how much comes from the higher reasoning configuration used in Meta’s comparison, Jha said.

Redditors have also expressed skepticism over Meta’s benchmarks, terming it “classic benchmaxxing”, the practice of optimizing a model or workload to score high on a benchmark rather than a real-world use case.

However, analysts remained optimistic about the new model’s value and its adoption.

“This is Meta playing the value card as the new model undercuts Anthropic and OpenAI’s flagship pricing by 4-to-6x, while still producing results that are competitive with more expensive models,” said Jha.

That should potentially position it as a “strong” alternative for enterprises running complex software development workloads at scale, echoed Satapathy.

One of the problems facing those who develop agents today, especially when building agents that run on edge systems with a mix of local and cloud AI, is trust: ensuring that the code that runs on users’ PCs doesn’t access private data or call unexpected APIs. How can we keep systems that don’t have predictable operations under control, when it’s that very unpredictability we’re relying on, as they orchestrate calls across local and remote systems, building and running workflows on the fly.

If you worked on agents the first time around, back in the 1990s, when they were tools designed to manage and run long workflows for users, you’ll recognize the problem. After all, the security risks associated with delivering arbitrary code to PCs and servers were what blocked those first agents from wide adoption (that and the complexity of building and sharing the necessary runtimes).

Securing modern agents

Computer systems have evolved considerably over those decades, especially with the arrival of hardware support for virtualization, a technology that Microsoft has built into much of its modern platform. Hardware virtualization is key to technologies like secure containers, to Windows’ own internal isolation model for critical OS components, and to cross-platform tooling like the Windows Subsystem for Linux. New tools like the WSL container platform have begun to extend that virtualization model with SDKs that allow programs to launch and control the virtualization life cycle.

Virtualization also underpins Microsoft’s approach to building a trustworthy framework for agent operations, allowing them to operate in a secure manner by isolating their operations from the operating system as a whole, locking down access to functionality and running them in secure containers and microVMs. With agents working autonomously, it’s essential to keep them isolated from desktop systems, giving them access only to the services they need — as examples of badly configured agents with overly broad prompts deleting files have shown.

Developers need tools to manage that isolation, allowing them to develop code in a permissive environment, before shipping locked-down applications that operate with policy-based restrictions. This approach needs tooling to build and manage those restrictions, providing a framework based on managed, isolated containers. This is where Microsoft Execution Containers (MXC) come into play.

What are Microsoft Execution Containers?

Intended to be cross-platform, MXC offers Windows, macOS, and Linux controls, working with more than one virtualization framework. The tooling is designed to offer a common abstraction layer, so you only need to build policies once before delivering them to all supported operating systems and virtualization environments. By using policies to control payloads, MXC can provide different levels of control for different workloads. An agent that’s designed to provide personalized search will need fewer guardrails than an OpenClaw-like system that’s being set up as an agentic personal assistant with full access to your data and applications.

An early adopter of MXC is GitHub Copilot. Here a single command in any CLI session (once you have enabled experimental mode) switches you into sandboxed mode with limited access to your file system (locking it down to your PATH, your working directory, Windows temp folders, and your profile), with outbound-only connectivity. There is a config option that allows you to open more directories if you need wider access, but the out-the-box guardrails should help reduce risks to your code from out-of-control agents.

Getting started with MXC

MXC is an open-source project, written in Rust, with the project hosted on GitHub. It’s configured using JSON and provided with a set of TypeScript SDKs that help manage the sandbox life cycle, which is defined as a set of steps: provision, start, execute, stop, de-provision. Sandboxes can be run as “one-shot” (a single execution) or as part of a long workflow to support the local elements of a business process.

Getting started requires a Rust tool chain and a recent build of Node.js, along with npm. Although regular releases do include binaries, it’s still best to clone the MXC repository and build it yourself. (Release source code is available, but working with the repository keeps you up-to-date with the latest changes.) The latest release, MXC 0.8.0, offers the most up-to-date set of policies, which are essential for managing the sandbox guardrails used to corral agents. It also improves networking support, with better security and improved tooling for delivering policies across different sandbox environments.

It’s a good idea to spend some time understanding the MXC policy schema. This is where you learn what MXC can do and what it can’t. It’s important to understand that MXC cannot provide more isolation than you get from a locked-down virtual environment, and that it can’t switch virtualization endpoints on the fly. The schema documentation shows the principles that are used by MXC’s development team. For example, they indicate that policies are an expression of intent, that the enforcement mechanisms are implemented in the tooling, and that the default position on any policy is “deny.” Thus, users need only define the policies they want to have a more permissive approach, as having no policy definition is by default an opt out.

Further, the tooling is designed to be cross-platform. You need only define policies once, and the Windows, macOS, and Linux versions of MXC will implement them on a target sandbox. How they are enforced will differ among the platforms and sandboxes, but whatever MXC does, it is driven by user intent. Putting the user requirements first is key, as the purpose of MXC is to provide a way for users to trust agents and to know that agents won’t run amuck and destroy data.

Using MXC to understand agent behavior

If you’re using MXC to develop agents and you want to tune the rules it uses, you can run MXC in audit and learning modes. This allows operations that would otherwise be blocked; instead of denying the operations, it logs them along with other operations, allowing you to tune your policies to actual agent behavior. Of course, it’s a good idea to run audit and learning modes on a virtual machine, or on a scratch test install, to ensure that your development systems are not impacted by an uncontrolled agent.

Once you have written a JSON policy, you can either use it either to spawn and run a sandbox directly or to generate a sandbox-specific configuration file that can be used with multiple sandboxes. If you take the latter approach, you can edit the generated configuration JSON and use it to launch sandboxes programmatically. It’s important to note that whichever approach you take, MXC lets you avoid working with kernel-level tools to manage hypervisors and their associated networking.

Interestingly, one of the supported isolation environments is Windows Sandbox (WSB). While this lightweight desktop sandbox is often used as a development tool, it can be configured to launch with managed access to defined selections of the host Windows file system. Although the applications you want to run in a WSB sandbox must be installed explicitly, it’s easy to imagine an MXC-managed agent using Windows Sandbox with a set of scripts to install a pre-defined set of applications, with mapped read-only access to a subset of the host’s files. The agent could then use its own APIs and local Model Context Protocol (MCP) servers to work with local data in the background of a user’s own sessions, treating the agent as a user’s “shadow.”

Locking down microVMs with MXC and Hyperlight

Finally, there’s even scope for working with microVMs that host single-process applications and microkernel-like environments, using technologies like Hyperlight to host agents and other high-risk applications, with MXC providing additional isolation. This approach allows a low-risk agent orchestrator to launch higher-risk agents as needed, without compromising overall security.

By using microVMs, you’re able to manage resources more effectively, running a dense network of secure agents with the same resources as another VM might use for one agent. It’s an approach that allows you to conserve resources by spinning up and tearing down microVMs as needed, something that will be useful in constrained edge environments.

Microsoft also is using MXC in conjunction with its cloud-based virtual PC environments, improving isolation and protection in its Windows 365 for Agents Cloud PCs. Here you can pre-configure virtual environments where agents can operate without affecting users’ systems, by working with cloud replicas of local systems that can be reset in the event of sandbox escapes or other failures.

MXC is still a very young project, and the final release is likely to differ considerably from the current preview release, as both the code and its supported sandboxes evolve. For now, though, MXC is well worth investigating as a way of securing agents that run on edge endpoints as well as agents that run in your own workflows, using tools like the ones built into GitHub Copilot.

First AI gave us code completion, predicting the lines of code, functions, and boilerplate we needed based on what we’d already typed. Then came code suggestions and code generation, where AI writes new code or improves existing code based on a natural language prompt.

Today, coding agents can look at your code, suggest improvements, compile and test the improved code, and iterate on that to come up with something better, faster, and safer. They can plan, write, test, review, and debug code, and chat with you about all of the above. They can tackle complex development workflows entirely on their own.

What will AI do for developers next? There is no better place to find out than TechCrunch Disrupt, TechCrunch’s annual showcase of technology startups and tech innovation. Sign up now for TechCrunch Disrupt 2026 and receive a 25% discount by using promo code INFOWORLD25.

TechCrunch Disrupt 2026 takes place October 13-15 at Moscone West Convention Center in San Francisco, where you’ll rub shoulders with startup founders, investors, AI experts, software engineers, and all manner of technology builders and innovators. Over three jam-packed days, you’ll be treated to 200+ conference sessions across six stages, led by 250+ technology leaders.

You’ll learn what’s next in AI and tech, and how AI agents and generative AI are changing software engineering, developer tools, enterprise software, and information security. You’ll learn how AI is reshaping SaaS and cloud infrastructure, how companies are building and deploying AI systems, and how AI is driving innovation in finance, healthcare, manufacturing, and other industries.

You’ll even learn how Anthropic, the company behind Claude Code, uses fleets of coding agents in its own software engineering work. This session, by Anthropic technical staff members Steve Androulakis and Sachin Malhotra, will provide you with practical patterns for delegating agents, reviewing their output, and recovering when they get things wrong.

The star-studded lineup for TechCrunch Disrupt 2026 features technology leaders from Amazon, Anthropic, Atlassian, Databricks, Flock, Glean, Google, NVIDIA, Okta, OpenAI, Replit, Runware, Together AI, and many other companies on the cutting edge of AI and software tech. Don’t miss your opportunity to connect with peers, exchange ideas, and explore the latest innovations at this premier event.

Register for TechCrunch Disrupt 2026 today. Use promo code INFOWORLD25 and save 25%.

Discount code is valid for Attendee, Founder, and Investor passes.