Most teams building AI agents are pouring effort into better memory, longer context windows, conversation summaries, retrieval systems that pull up anything the agent has ever seen. These are useful tools. But they do not solve the problem of knowing whether an action actually completed. Memory tells an agent what happened. It doesn’t tell the agent what’s true right now. And for any agent that takes real action — refunding money, changing an order, updating an account — that distinction is the whole ballgame.

Memory can remember a request without knowing the outcome

Picture a customer service agent that approves a refund. The refund actually goes through, but the confirmation response times out before the agent hears back. From the agent’s point of view, all it has is a record of a request it made, not proof of what happened next. If the agent trusts its own memory, it sees an unfinished task and tries again. The reasoning is sound. The second refund is still wrong.

More conversation history doesn’t fix this. What the agent needs is a record it can trust: the request ID, the current status, the confirmed outcome. Memory can tell you what the customer asked for. It can’t tell you what another system actually did. This isn’t a new problem; recommendation systems ran into a version of it years before agents existed.

A prediction is not the same as current intent

I spent several years building recommendation and personalization systems, and one thing became clear fast: more history doesn’t automatically mean a better decision. A recommender looks at what someone bought, browsed, or listened to, finds a pattern and predicts what they’ll want next. These are all informed guesses rather than facts. I saw this play out around election season. A customer buys several books about a candidate or a political topic and the system, correctly, learns that as an interest. Weeks later, the election is over and the customer has moved on, but their purchase history hasn’t changed. The purchase history is still accurate. It just no longer reflects the customer’s current intent. So, customer data changes with each interaction, so recommendations need to respond to new information.

Freshness in a large-scale personalization system

Early in a real-time personalization platform I worked on, one of the hardest problems wasn’t traffic volume; it was freshness. A customer’s inferred interests could lag well behind their actual recent behavior, so the system would keep confidently recommending things that no longer matched what the person wanted. We rebuilt the pipeline so that signal updates reflected recent behavior almost immediately instead of with a long delay. That gap between what the system knows and what’s currently true is small when you’re recommending a book. It’s a much bigger problem once you move from recommending content to an agent that’s authorized to act on someone’s behalf. Consider a customer who has just started a trial and is eligible for a $0 first-item offer. Personalization should show that available benefit instead of another invitation to start a trial. A fresh eligibility signal lets it change the next offer. Unlike inferred interests, trial eligibility is a recorded fact.

Stale data can produce a bad recommendation. Stale data acted on by an autonomous agent can produce a real-world mistake. Agents need more than memory; they need a reliable state.

Reliable state tells the agent what’s true right now

Reliable state is a verified record of a specific operational fact. In the refund example, the conversation records what the customer requested. The payment system records whether the refund happened. The order system records the order’s status, while current policy and permission records determine whether the action is allowed. Each fact has its own authoritative source. State can itself become stale, so the agent must check the relevant source rather than rely on its memory of an earlier answer.

This matters even after the “yes/no” check passes. A customer might withdraw permission mid-process. An order might fall outside the window where changes are allowed. And even when every check passes, the action itself can still fail partway through, so the agent also needs a safe way to retry without duplicating the damage. The action system must enforce current permissions and other preconditions when it executes, because they can change after an earlier check.

Designing a privacy-request architecture

In privacy and compliance architecture work I’ve done, a single customer request, like asking to delete their data, can trigger a long chain of downstream effects across many internal systems. The challenge wasn’t any one system doing its job correctly; it was making sure there was one authoritative, centralized place that recorded whether the overall request had actually been completed, rather than each system independently assuming it had. That’s the same principle an agent needs: a system of record it defers to, rather than trusting its own memory of what it asked for.

A few categories of tooling exist specifically to solve different slices of this “what’s true right now” problem:

• Business state/workflow databases, Postgres, DynamoDB, Spanner, Cosmos DB.
• Workflow orchestration, Temporal, AWS Step Functions, Azure Durable Functions.
• Agent state, tools like LangGraph checkpoints.
• Session/cache state, Redis, Memcached.
• Event history, Kafka and similar streaming systems.
• Permission state, Open Policy Agent, Amazon Verified Permissions.

None of these replace the others. A production agent usually needs several of them working together, which raises the next question: what happens when an action only partly succeeds?

Safe actions need a receipt and a recovery plan

Every important action an agent takes should carry a unique identifier, like a receipt number. If the response gets lost, the agent must reuse that ID for the same intended operation. The receiving system must recognize the ID and prevent the operation from being applied twice, within its supported deduplication window. This is how an idempotent API makes retries safe; attaching an ID alone is not enough. A preliminary status check cannot prevent two concurrent requests from executing the same action. The AWS Builders’ Library explains this design in Making retries safe with idempotent APIs.

Teams also need reconciliation, regularly comparing what the agent believes happened against what the system of record actually shows, the same way you’d check a receipt against a bank statement. When they disagree, someone needs a clear next step: recheck, reverse the action, or escalate to a human. None of this works without clear ownership.

Reconciliation at high transaction volume

In large-scale commerce infrastructure work, we treated “did this transaction actually complete” as something that could never be answered from memory or assumption, at high volume, even a small rate of unconfirmed or duplicated actions adds up to real customer impact. We built reconciliation directly into the system so that what one part of the pipeline believed had happened was continuously checked against what was actually recorded, with idempotency built in to prevent retries from creating duplicate transactions. The lesson generalizes directly to agents: never let an action retry blindly without checking a source of truth first.

Before you give an agent more autonomy, decide who owns each critical fact and who’s responsible for cleanup when something breaks, because something eventually will.

Monitor the decision, not just the model

All of this changes what a team should actually be watching. Latency, cost and model accuracy matter, but none of those metrics will tell you that an agent acted on a stale permission or announced a refund that never happened. For each consequential decision, trace the information used, its freshness, the rules checked, the attempted action and the confirmed result. What you actually want to be able to trace is:

• what information the agent used
• when that information was last updated
• which rules were checked before acting
• what action was actually attempted
• what result was recorded afterward

As a leader, the job isn’t just watching model quality; it’s making sure every piece of information an agent acts on has a clear source and a way to verify it’s current. Agents are genuinely good at interpreting messy, ambiguous situations and deciding what to do. But they shouldn’t be the record of truth themselves. That job still belongs to the systems built to hold it.

Memory tells an agent what happened. Reliable state tells it what’s true now, and whether it’s safe to act.

Artificial intelligence does not advance at the same pace across industries. It presses forward in some directions while lagging behind in others.

Spend time with today’s most advanced AI applications, and this contrast becomes obvious. In software development, AI is quickly becoming ubiquitous. It writes production-ready code, explains obscure libraries, and iterates at a pace human teams have difficulty matching.

But place that same AI model inside a complex customer support workflow or ask it to reason through a nuanced clinical scenario, and the cracks begin to show. Multi-step reasoning falters. Context gets lost. Performance drops in ways that can seem inconsistent with the model’s strengths elsewhere.

These AI models are often similar. They run on similar hardware and are often trained in similar ways. So why the mismatch in performance across tasks? The simplest explanation is also the most overlooked: data.

Software engineering benefits from an immense, structured, and highly visible digital record. Code is written in standardized languages, benefits from robust documentation, is reviewed in public forums, and is discussed at scale. That ecosystem has generated a robust and massively useful pool of training material.

Other fields often do not. For example, healthcare data is scattered across institutions, wrapped in privacy constraints, expressed in multiple modalities, and rarely ready out-of-the-box for AI training. Enterprise workflows are captured in internal systems that were never designed for training AI. Multilingual speech data varies widely in quality and representation.

This imbalance creates what I describe as “the data gap.” This is the distance between what models are capable of in theory and what they can achieve in practice, because the right data does not yet exist in usable form. Closing this data gap may be the most important—and least glamorous—challenge in AI today.

The missing pillar of AI progress

Three forces are driving the recent advances in AI: the models, the chips, and the data.

On the AI models, the field has invested heavily. Major research organizations employ thousands of researchers and scientists who are actively refining architectures, training techniques, and evaluation methods. Breakthroughs are measured in benchmark scores, conference papers, and model performance on human tasks. On the computing chips, the investment has been equally intense. Hardware manufacturers and infrastructure providers are pouring billions of dollars into building and supporting data centers that deliver faster results via large-scale training.

Yet data has not received the same institutional focus for AI development. Conversations with researchers at frontier AI labs share a similar frustration that today’s model capabilities in key use cases, such as healthcare, are limited less by architectural imagination and more by the availability of high-quality, domain-specific data. The bottleneck is not always a lack of ideas, but a lack of reliable inputs.

We are long past scraping the internet for useful data, and this path does not scale. Progress depends on building and curating datasets that reflect the complexity of true lived experience and organizational processes. That work requires both scientific rigor and research specialization in the field of data for AI.

The dataset behind every leap

The history of AI reinforces a consistent lesson: major leaps in model capability follow major leaps in the availability of quality data. From early vision systems that relied on clearly labeled images to today’s language models trained on massive text collections, each major leap has depended on access to more high-quality data.

Architectural innovation alone is rarely enough. The value of these new approaches only emerges when paired with large, structured, and representative datasets that reveal what the models can actually do in practice. Whether in vision or language, progress has depended on the painstaking work of collecting, organizing, and validating the underlying data.

Large language models illustrate this clearly. Their emergence was not just the result of better training techniques, but of access to an unprecedented volume of data. The models did not generate that data. They relied on it. That pattern raises a pressing question for the present: who is building the next generation of foundational datasets?

Across domains ranging from healthcare to audio to agentic task performance, there is no widely accepted blueprint. What constitutes a gold-standard dataset for training an AI agent to handle complex enterprise tasks? What does a clinically meaningful evaluation look like for a model that will assist in medical decision-making? How should multilingual speech data be curated to ensure broad representation and reliable performance?

These are not simple sourcing problems. They are fundamental research challenges that need to be solved.

When data is treated as a commodity

Too often, consequential data decisions are handled like procurement exercises. An organization requests “medical conversations” or “wildlife scenes,” and the request is routed to internal procurement or data sourcing teams, or to external data vendors, who assemble data that appears to match the description. The implicit assumption is that data is interchangeable, that one dataset is as good as another so long as it meets a basic specification.

Actual application suggests otherwise. Seemingly small choices about factors such as inclusion criteria, annotation standards, filtering rules, and validation protocols can dramatically alter downstream performance. Data design shapes model behavior as much as architecture does.

Three structural issues compound the problem:

• Capacity: There are relatively few specialized teams dedicated to building domain-specific datasets at the highest level of rigor. Talent and funding have gravitated toward model development and hardware innovation. Data work often operates in the background, even though it underpins both.
• Design: Constructing a dataset is a distinct discipline from designing a neural network. It requires expertise in experimental design, domain knowledge, and statistical validation. Expecting model researchers to simultaneously shoulder the full burden of data research, while also training and evaluating the models, overlooks the complexity of the upstream task.
• Translation: The researchers who are requesting specific data sources to improve the models are often not the same people responsible for sourcing that data. As a result, nuances and research-backed expertise can often be missing or diluted as requests pass through layers of procurement and vendor relationships. The result can be data that meets the specification sheet, but in fact fails to advance model performance.

The rise of annotation providers and reinforcement learning services has addressed part of the need. Rating model outputs, labeling text, and evaluating structured information are essential for many optimization tasks. But these activities generate data that is carefully constructed for specific, bounded purposes.

The frontier challenges in AI require more. They demand datasets derived from real human activity and organic organizational processes. Such data is complex, multimodal, and sensitive. It is rarely AI-ready by default. And converting it into reliable training and evaluation material is a scientific undertaking.

The need for scientific rigor for the AI data layer

If high-quality data is a central bottleneck, then scientific rigor is part of the solution. Just as leading model-builders have dedicated research labs and hardware has dedicated development ecosystems, the data layer for AI requires focused, scientifically-grounded institutions.

This means engaging directly with core questions like dataset design, evaluation methodology, and quality control. The conversation cannot end at volume; it must address data structure, representativeness, and expert validation.

Dataset construction must be approached as experimental design. Protocols must be documented and validated. Evaluation frameworks must test whether the dataset truly reflects the intended applications.

The field also requires standards and benchmarks that reflect real-world complexity, not simplified proxies. In healthcare, for instance, evaluating a system intended for clinical assistance with generic question-and-answer tests is insufficient. Real-world clinical environments involve multimodal inputs and contextual judgment. Benchmarks must reflect that reality if they are to function as meaningful gates before deployment.

Quality measurement is another crucial frontier. Finance relies on standardized metrics such as credit scores to assess risk. AI lacks an equivalent for datasets and benchmarks. Developing clear methodology to quantify dataset quality and evaluation reliability brings clarity to model assessment.

The criteria for evaluating a multilingual audio library will differ from those of a multimodal oncology dataset. Yet the underlying principle remains constant. Better models require better-defined, better-measured data.

The risks of getting it wrong

As AI systems move closer to high-stakes deployment, weak data practices carry tangible risks.

Benchmarks cannot be created with the same data that is used for training—that’s giving the test answers to the model ahead of time. Scaling data volume without prioritizing data quality and selection diminishes model performance gains, and can even bias against or omit underrepresented populations. These are methodological challenges, and ones that must be solved.

The rigor required at the data layer may not attract headlines. It does not typically lend itself to dramatic product launches. Yet the data layer for AI is foundational to trust, safety, and sustained progress for all AI progress.

An ecosystem for the data era

No single organization can resolve the data gap alone. What is needed is an ecosystem of AI data labs and research groups, each focused on different domains and challenges but united by a commitment to scientific discipline. These institutions would collaborate with model researchers and domain experts who would tackle challenges such as dataset contamination, factuality, groundedness, de-identification, international representation, and bias. They would design benchmarks that mirror real-world complexity rather than simplified abstractions.

AI’s trajectory will not be determined solely by larger models or faster chips. It will be shaped by the datasets we construct, the standards we adopt, and the rigor we apply at the foundation. The uneven frontier we see today reflects an uneven data landscape. Bridging the gap requires deliberate, research-driven dataset design.

If we want AI systems capable of operating reliably in clinical contexts, navigating enterprise workflows, and functioning responsibly across languages and cultures, we must treat data for AI as a first-class scientific endeavor.

AI models have their research labs. AI chip-builders have their fabrication plants. AI data needs institutions of equal seriousness and ambition.

—

New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.