Postgres just turned 30. Over those three decades, I’ve watched it pick up hot standby replication, logical replication, asynchronous IO, just-in-time compilation, and dozens of other features built for enterprise workloads. Every one of them solved a real problem and raised the floor on what a DBA needs to know just to do the job well.

I don’t think most people outside this world fully register how much that floor has risen. In the early days, improvements were small and mostly about stability. Now, 30 years in, the focus is squarely on enterprise functionality, and that functionality comes with complexity attached. The more you build, the more there is to understand and manage, and that increases every year.

Hiring more people doesn’t solve this

Most organizations treat a scaling database problem as a staffing problem. Hire enough DBAs and you’re covered. I think that’s the wrong way to look at it entirely.

It’s not far off from the old “too many cooks spoil the broth” problem. You don’t make better soup by adding cooks. You make better soup with a better chef. A 100-terabyte database with serious performance issues doesn’t get fixed by putting 10 DBAs on it instead of one. It gets fixed by someone who has seen that failure before and knows exactly where to look.

Experienced DBAs are expensive and hard to find, and the problem is only getting worse. A few years ago, it was IoT driving demand for time series expertise at scale. Now it’s AI, and organizations are building enormous databases to support retrieval pipelines and large language model (LLM) training, often without the experience on hand to run them safely.

Where human error comes from

Humans are unpredictable. We make assumptions. Sometimes, we think the process is getting in the way, so we take a shortcut around it. And we make mistakes because that’s simply human nature.

I don’t think that’s something we’re ever going to fully engineer away as long as a person is in the loop, and honestly, we want a person in the loop making the real decisions. So it becomes a balancing act. You have to understand the mistakes people tend to make, build the discipline to follow good processes consistently, and still know when a situation calls for judgment rather than a checklist.

Why AI helps more than people expect

I think this is where AI has earned its place. It’s not as a replacement for the DBA, but as something that follows a process more consistently than a person under pressure tends to.

Modern AI systems still have problems with hallucination and getting the full picture right. However, with the right guardrails, auditing, and tracking in place, they are able to follow the process very well. It is much easier for me to say, “Forget it, I know a shortcut, I’ll just push this through,” than it is for a well-constrained model to deviate from its instructions. As long as the instructions are clearly defined, LLMs tend to stick to them.

Another advantage worth naming is if an LLM is doing the diagnostic work, you can put much stronger gating around its ability to actually make changes. You can let it follow a tight, well-defined process to diagnose a problem while still requiring a human to approve anything that touches the live system, whether that’s creating an index, dropping one, or changing a configuration value.

That combination gets you to a solution far faster than the old way, using the model’s training plus whatever supplemental knowledge you feed it through retrieval-augmented generation (RAG), while keeping a person at the one point where it actually matters. Did the model get the diagnosis right? Does the proposed fix line up with the actual problem? That’s the question a human needs to answer before anything gets applied.

The industry hasn’t figured out the limits yet

I am not sure we are far enough along the path as an industry to really know what we’re getting wrong. I am working through these questions myself right now, looking at autonomous agents for Postgres operations and where the real risk is.

What’s clear to me already is that strong guardrails matter in two places, not one. You need guardrails on the prompts so the model verifies its findings and doesn’t hallucinate into a bad recommendation. And you also need guardrails on what the model is actually allowed to do. Letting an agent create an index carries a manageable risk. Letting it truncate your orders table does not. Auditing matters just as much; when something does go wrong, you need to see exactly what happened and undo it. The fallback needs to stay in place until the industry broadly trusts these systems more than it trusts a human.

Picture a load spike that hits overnight, the kind where batch processing that normally runs fine suddenly does not. The old way means spending hours trawling logs, index stats, table stats, and query performance before you land on a likely cause and you can apply a fix.

Put an LLM on that same problem and the hours-long process compresses into minutes. It’s the same productivity shift we’ve already seen in software development, where an experienced user with the right skills and guardrails can get LLM output that matches or beats a junior developer’s work. The quality comes down to this process: Structure the work, define a clear spec, build the implementation plan, then verify each stage before moving on.

The same discipline applies to database operations. The worst thing you can do is rush in and say to the model, “This happened, go fix it.” You need to guide the tools properly to get good results from them. When it’s done right, it works.

Where this goes from here

A few things need to be true for this to scale across the enterprise. Frontier models already understand Postgres well, but they don’t know everything, so you need to feed them supplemental knowledge alongside that training. Full documentation matters here, so do internal runbooks. The process has to require the model to actually consult that material rather than coast on what it already knows.

I expect the same kind of structured skills that are common in AI-assisted development, the step-by-step approach to planning and verification, to show up specifically for Postgres operations. That will make the job easier for DBAs, and easier is what matters.

We’re probably not far from agents that sit and watch a Postgres instance around the clock and rarely need a human to step in. The slower part is trust. For example, look at how long it took people to actually believe AI-written code was any good. I would guess it is the same pattern here, but I don’t doubt for a second that we will get there.

In the meantime, you do not need to hand an LLM the keys to your database tomorrow. Start smaller and begin with read access to your monitoring data. It’s a low-risk place to start.

An agent watching constantly can catch the kind of slow, compounding anomaly that threshold-based alerting tends to miss, a gradual rise in IO alongside a gradual rise in memory, and flag it before it becomes the kind of outage that costs real money every hour it runs.

That’s not a small thing. The DBA shortage is not going away on its own, and waiting for the industry to fully trust autonomous agents before using AI at all is the same as falling behind. The teams that figure out where the human belongs in this process are the ones who’ll be ready when the rest of the industry catches up.

—

New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.

Microsoft is betting that the integration of HorizonDB, the cloud-native PostgreSQL alternative it is developing, with Azure will attract more enterprise AI and agentic workloads to its cloud services.

Enterprises may not be willing to take that bet.

It’s been nine months since Microsoft unveiled HorizonDB, but the service remains in public preview with no announced general availability date. Why put AI projects on hold waiting for HorizonDB to arrive, when AWS, Google, Databricks, Snowflake, and others already have production-ready PostgreSQL services positioned for the same AI workloads that Microsoft says it is building HorizonDB to handle?

AWS has had the longest head start. Aurora PostgreSQL became generally available in 2017 and has since evolved from a cloud-native PostgreSQL database into an AI-ready service with vector search and integrations with Amazon Bedrock. Similarly, Google’s AlloyDB, which followed in 2022, now includes AlloyDB AI with vector search, embeddings and model interaction for generative AI and agentic applications.

Databricks and Snowflake, too, have their own platform-centric services in the form of Lakebase, which became generally available on AWS and Azure this year, and Snowflake Postgres, which was made generally available in February 2026.

As the latecomer, when Microsoft pitched HorizonDB at Ignite in November 2025 it talked up its new architectural approach to cloud-native PostgreSQL, built around disaggregated compute and storage and a database-as-log design. The hyperscaler also positioned native vector search and deep integration with Foundry and Fabric as key differentiators for AI-heavy workloads.

No reason to wait

Those architectural differences may not be compelling enough for CIOs to wait for HorizonDB to become generally available, though.

“Most enterprises with urgent needs will not wait. A long preview window creates uncertainty around SLAs, pricing, operational maturity, and roadmap confidence,” said David Linthicum, an independent cloud consultant.

And, said Stephanie Walter, practice lead of AI stack at Hyperframe Research, enterprises cannot build mission-critical production plans around an undefined GA date, regional footprint or support commitment.

Given the difficulty of unwinding a poor database choice, enterprises will approach unknown quantities with caution.

“Database platforms eventually become sticky control points. Once the database is connected to the rest of the application, analytics, AI, and governance stack, switching becomes a business transformation rather than just an infrastructure swap,” said Michael Ni, principal analyst at Constellation Research.

In the case of a cloud database, there’s also the unwelcome possibility of “huge egress fees” in case of change, said Bradley Shimmin, lead of the data and analytics practice at The Futurum Group.

All that uncertainty is likely to lead enterprises to restrict HorizonDB to experimental use cases for now, Shimmin added.

Performance anxiety

Analysts also questioned whether HorizonDB’s technical differences will show up in performance benchmarks.

Microsoft has said HorizonDB can deliver up to three times the throughput of open-source PostgreSQL, but makes no comparisons with rival offerings such as Aurora or AlloyDB that it will compete with, Walter said.

The bigger question, according to Igor Ikonnikov, advisory fellow at Info-Tech Research Group, is whether those performance advantages, still largely on paper, translate into a meaningful difference in production.

“A database with a better compute benchmark can still be more expensive once resilience and ecosystem costs are included,” Ikonnikov said.

The economics also point to another HorizonDB limitation, particularly for workloads that are not continuously running, said Advait Patel, senior site reliability engineer at Broadcom.

HorizonDB currently uses provisioned compute rather than a serverless, scale-to-zero model, meaning customers continue to incur compute charges while an instance is provisioned, even if its workload is intermittent or idle, Patel said.

There are developer considerations too.

HorizonDB’s PostgreSQL compatibility does not necessarily mean every existing PostgreSQL application will move cleanly as in its current form the database supports only an approved set of PostgreSQL extensions rather than arbitrary ones, Walter said.

Who should wait?

For enterprises already deeply invested in Microsoft’s Azure ecosystem, those limitations may not be enough to rule out waiting for HorizonDB, Patel said: The chance to integrate the database with AI services and the wider Microsoft stack may outweigh immediate availability, he added.

That calculus also reflects how enterprises typically make database decisions in the first place: not by comparing databases in isolation, but by weighing how well they fit into the broader technology stack, including the cloud platform they have standardized on, Ikonnikov said.

For Azure shops, the choice may therefore be less about moving an existing workload away from Aurora or AlloyDB and more about whether a new Azure workload should start on Azure Database for PostgreSQL today or wait for HorizonDB when it becomes available, he said.

That may be an open question for some enterprises, said Devin Pratt, research director at IDC. “Plenty of organizations are still mid-decision, not locked in,” he said.

Microsoft finally offers a timeframe

Microsoft still won’t say exactly when HorizonDB will launch, with Shireesh Thota, corporate vice president for Azure Databases at Microsoft, saying only, “General availability for Azure HorizonDB is currently targeted for the second half of 2026.”

That narrows it down to a period of a little over four months, including Microsoft’s FabCon and Ignite conferences — an eternity in AI.

Separating transactional databases from analytical systems was, until recently, considered good architecture. Now, as enterprises adopt AI agents that continuously read, reason over, and act on business data, data warehouse and database vendors are increasingly deciding that separation has become a liability.

Just weeks after Databricks unveiled its Lakehouse Transaction and Analytical Processing (LTAP) offering based on Neon Postgres to bring operational (OLTP) and analytical (OLAP) processing closer together, EnterpriseDB (EDB) has introduced converged analytics capabilities for its managed EDB Postgres AI database service with the same intent.

Both vendors are responding to the same pressure of enabling AI agents for enterprises to operate on fresh operational data without waiting for pipelines and replicas, but EDB argues its approach starts from a fundamentally different place.

“Databricks is building from the lakehouse outward, trying to pull transactional capability in through Lakebase,” said Max Romanenko, chief engineering officer at EDB, while “we’re building from the operational layer with Postgres, which is where enterprises already run their most critical workloads, and expanding from there.”

In contrast to Databricks’ lakehouse-centric LTAP, EDB keeps Postgres as the operational source of truth and uses Apache Iceberg as a shared catalog layer connecting Postgres with ClickHouse, WarehousePG, and Spark compute engines, Romanenko said.

In this way, operational data remains in Postgres while historical and tiered data is stored in Iceberg-managed object storage, allowing analytical engines to query the same data through a common catalog without requiring separate copies or ETL pipelines, he said.

That architectural distinction matters to EDB, according to Romanenko, because the vendor is targeting enterprises that want AI and analytics capabilities without moving sensitive data into a cloud-managed platform: “For us, it’s always been about the data sitting on infrastructure the customer owns and controls.”

Focus on data sovereignty and predictable economics

EDB’s promotion of control “will resonate with CIOs focusing on sovereignty, regulated data, and hybrid deployment,” said Stephanie Walter, practice leader of AI stack at HyperFrame Research. “This should enable them to run AI and analytics closer to the data, on infrastructure that their enterprise controls, without creating yet another proprietary data estate.”

For Ashish Chaturvedi, leader of executive research at HFS Research, EDB’s approach in converged analytics will offer more predictable costs than Databricks LTAP for CIOs already struggling to manage their analytics and AI budgets.

EDB’s per-core pricing model can make costs easier to forecast than consumption-based cloud data platforms, where query volumes, AI workloads, and data processing demands can cause bills to fluctuate, Chaturvedi said.

But predictable bills are not necessarily lower bills, warned, Igor Ikonnikov, advisory fellow at Info-Tech Research Group. “The hardware requirements for high-speed operational data processing are higher and relatively more expensive compared to cheap lakehouse storage,” he said.

EDB’s architecture could also simplify data governance by reducing the number of platforms enterprises need to manage. Since operational, analytical, and AI workloads can access data through a common Postgres-Iceberg foundation, enterprises may be able to avoid deploying and governing multiple specialized data stores, and so have fewer systems to license and secure, according to Devin Pratt, research director at IDC.

Reducing architectural tax for engineering teams

EDB’s converged analytics could also simplify operations for developers and data engineering teams.

Its architecture reduces the number of systems developers must integrate and maintain, while eliminating much of the pipeline work traditionally required to move data between transactional and analytical systems, according to Walter.

And, said Pratt, “Zero-ETL means far less plumbing to build and break, so engineers spend their time creating value.”

EDB and Databricks are not the only ones pursuing converged analytics to support agentic systems and other applications needing immediate access to operational data, historical context, and governance controls.

Snowflake has been expanding support for operational workloads by embracing open table formats,and Microsoft has combined transactional and analytical services under a broader data architecture via its Fabric platform.

Evolution of autonomous databases?

Converged analytics, though, was only one part of EDB’s update to its Postgres AI platform.

It has also made generally available what it calls an “agentic database” feature, designed to automate routine database administration tasks.

The system continuously monitors hundreds of operational and performance metrics, detects anomalies, recommends corrective actions, and, where enterprise policies permit, can automatically apply fixes, the company said.

These automated agents can help enterprises optimize and tune their databases up to 10 times faster, it said.

Walter remained skeptical: “It is more an evolution of autonomous database concepts than a wholly new category. Oracle and other database vendors have offered autonomous database capabilities for years.” Where EDB can differentiate itself, she said, is in extending those autonomous capabilities with AI-driven reasoning, automated remediation, and governance controls that allow enterprises to determine how much authority the system receives.

I’ve seen a lot of promising AI prototypes fall apart after launch. And it’s rarely because the model was bad. More often, the problem starts much earlier; teams treat the data layer like something they can figure out later.

They’ll spend weeks fine-tuning prompts, testing models and debating evaluation scores, then throw together the retrieval pipeline over a weekend and move on. At first, everything looks great in demos. But a few months later, the system gives outdated answers; the embeddings no longer match the source documents, and nobody fully understands what changed.

What started as an impressive prototype slowly becomes difficult to trust in production. The teams that avoid this tend to realize one thing early: Embedding pipelines are fundamentally a data engineering problem, not an entirely new AI discipline. It’s still ETL (Extract, Load, Transform) at its core, but with embeddings and vector stores as the destination instead of a warehouse.

Once you start looking at it that way, a lot of things become clearer. Problems like versioning, data freshness, lineage and retries stop feeling “AI-specific.” They’re data infrastructure problems we’ve already spent years learning how to solve.

Why do we need embedding pipelines?

Large language models are extraordinary reasoners trapped inside a time capsule. When training ends, the model’s knowledge is sealed. It does not know what your team decided in last quarter’s strategy review. It has never read the support ticket that came in this morning. It cannot find the clause buried on page 47 of your master service agreement. It’s brilliant, but blind to anything specific to your organization.

Layer on top of that a hard context window limit, a ceiling on how much text the model can process in a single interaction, and you have a clear problem: you cannot just hand it everything you own.

The answer the industry converged on is retrieval-augmented generation, or RAG. Instead of stuffing everything into the context window, you build a retrieval layer that fetches only the most relevant pieces of information at the moment a question is asked and passes just those to the model. That retrieval layer is powered by a vector database, and the process that populates it, which is taking raw documents and transforming them into searchable semantic representations, is what I mean when I say embedding pipeline.

Every team building an internal AI assistant, a smarter enterprise search tool, an automated customer support agent or a document Q&A system needs one. The question is not whether to build it. The question is whether you build it like a prototype or like infrastructure.

How an embedding pipeline works

An embedding pipeline has three stages: ingestion, chunking and indexing. Here is what each one means and how I relate them to a typical ETL process.

Ingestion is extraction

Getting your raw content, PDFs, wiki pages, Word documents, database records, transcripts, out of wherever it lives and into the pipeline. This is ETL’s extract stage, almost verbatim.

I see teams cut corners here more than anywhere else, and it’s often where production systems first start to fail. A document gets updated, but the pipeline doesn’t pick it up. A file gets deleted, but its chunks remain in the index, still returning outdated answers months later. And because there’s no obvious error, no one reports it.

The fix is Change Data Capture (CDC). This maintains a manifest of every document you have ingested, a content hash and a timestamp. On each run, we compare sources against that manifest, re-ingest what changed, delete what is gone and treat your document the way you would treat any source table you are syncing incrementally.

Chunking is transformation

Once your documents are in the pipeline, you cannot embed them whole. A 30-page technical report is too long to represent meaningfully as a single vector, and even if it were not, returning the entire report in response to a narrow question would bury the model in irrelevant context.

Chunking is the process of breaking each document into smaller pieces that are focused enough to embed accurately and retrieve precisely. This is ETL’s transform stage, and it deserves the same level of design discipline.

The most common mistake I see is treating chunk size as a default configuration option rather than a product decision. It is not. The right chunk size depends entirely on the nature of your content and the nature of your queries. Dense technical documentation needs finer granularity than a collection of FAQs. A legal contract with clause-level logic needs different treatment than a set of onboarding emails. What works for one document will actively degrade retrieval quality on another.

My strong preference is to treat your chunking configuration as a versioned pipeline parameter, not hardcoded logic. When you change it, and trust me, you will. You need to re-chunk in a controlled, observable way, compare retrieval quality before and after, and roll back if it degrades. That is just good transform-layer hygiene. It is no different from versioning a data cleaning rule or a field mapping.

Indexing is the load

The final stage is where chunked text gets converted into vectors and stored in a vector database where it can be searched by semantic similarity rather than keyword match.

In the conversion step, embedding is handled by a model specifically trained to turn text into dense numerical representations that encode meaning. Two chunks expressing the same idea in different words will produce vectors that cluster close together in that mathematical space. Two chunks discussing entirely different topics will sit far apart. When a user asks a question, the system embeds that question the same way, finds the chunks whose vectors are nearest, and returns them as context for the model to reason over.

That is a genuinely new capability. But the discipline around indexing is not. One data engineering principle I keep coming back to is versioning.

In embedding pipelines, every chunk in your index should be tagged with the embedding model name and version used to generate it, this is non-negotiable. Embedding models evolve, and vectors produced by different versions are not comparable in a reliable way. You cannot safely search across them as if they are interchangeable.

This exact problem shows up when teams upgrade embedding models mid-pipeline without a proper migration plan. You end up mixing vectors from different generations in the same index, and retrieval starts to degrade in ways that are hard to detect. The system just quietly begins returning subtly wrong answers.

I treat an embedding model upgrade the same way I treat a schema migration: Plan it explicitly, execute it in full and validate retrieval quality on a representative query set. The stakes are the same as any breaking change to your data model.

Pipeline observability is not optional

Once an embedding pipeline is running in production, the question shifts from “did it run” to “did it run correctly.” That distinction matters more here than in most pipelines, because failures are rarely loud because the index looks fine, queries return without errors and the system quietly surfaces wrong answers until someone notices the AI has stopped being useful.

The same observability discipline that makes any data pipeline trustworthy applies directly here. Once you treat embedding pipelines as production systems, you stop thinking in isolated steps and start thinking in signals. For example, chunk counts per document become a simple but powerful health check, a sudden drop is usually not a model issue, but a sign of broken ingestion or upstream parsing failures.

You also need a “golden set” of queries with known-good outputs. This runs after every pipeline change, much like data quality checks after a transformation. This is how you catch regressions that don’t show up as explicit failures.

On top of that, you can track lineage: Which embedding model version produced which chunks, and when each document was last ingested. That makes it possible to trace retrieval issues back to specific changes instead of guessing.

And finally, freshness becomes a first-class signal. If documents start going stale beyond an acceptable threshold, that should surface in monitoring long before users experience degraded results.

The metric that ties it all together is retrieval quality over time. Treat it like any other pipeline SLA, measured, tracked and owned.

Summary

Embedding pipelines definitely come with a lot of new language, new tools and a genuinely different capability in the semantic layer. But the funny thing is, the principles that actually make them reliable in production are not new at all.

We have versioning, freshness, quality checks and monitoring. These are problems data engineering has already spent years solving.

The real work is taking that same discipline and applying it to a pipeline that just happens to output vectors instead of rows in a table. Once you start seeing it that way, a lot of the chaos around AI systems becomes much easier to reason about.

That’s the difference between building a cool AI demo and building something people can actually depend on. One is a prototype, whereas the other is infrastructure.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?