Separating transactional databases from analytical systems was, until recently, considered good architecture. Now, as enterprises adopt AI agents that continuously read, reason over, and act on business data, data warehouse and database vendors are increasingly deciding that separation has become a liability.

Just weeks after Databricks unveiled its Lakehouse Transaction and Analytical Processing (LTAP) offering based on Neon Postgres to bring operational (OLTP) and analytical (OLAP) processing closer together, EnterpriseDB (EDB) has introduced converged analytics capabilities for its managed EDB Postgres AI database service with the same intent.

Both vendors are responding to the same pressure of enabling AI agents for enterprises to operate on fresh operational data without waiting for pipelines and replicas, but EDB argues its approach starts from a fundamentally different place.

“Databricks is building from the lakehouse outward, trying to pull transactional capability in through Lakebase,” said Max Romanenko, chief engineering officer at EDB, while “we’re building from the operational layer with Postgres, which is where enterprises already run their most critical workloads, and expanding from there.”

In contrast to Databricks’ lakehouse-centric LTAP, EDB keeps Postgres as the operational source of truth and uses Apache Iceberg as a shared catalog layer connecting Postgres with ClickHouse, WarehousePG, and Spark compute engines, Romanenko said.

In this way, operational data remains in Postgres while historical and tiered data is stored in Iceberg-managed object storage, allowing analytical engines to query the same data through a common catalog without requiring separate copies or ETL pipelines, he said.

That architectural distinction matters to EDB, according to Romanenko, because the vendor is targeting enterprises that want AI and analytics capabilities without moving sensitive data into a cloud-managed platform: “For us, it’s always been about the data sitting on infrastructure the customer owns and controls.”

Focus on data sovereignty and predictable economics

EDB’s promotion of control “will resonate with CIOs focusing on sovereignty, regulated data, and hybrid deployment,” said Stephanie Walter, practice leader of AI stack at HyperFrame Research. “This should enable them to run AI and analytics closer to the data, on infrastructure that their enterprise controls, without creating yet another proprietary data estate.”

For Ashish Chaturvedi, leader of executive research at HFS Research, EDB’s approach in converged analytics will offer more predictable costs than Databricks LTAP for CIOs already struggling to manage their analytics and AI budgets.

EDB’s per-core pricing model can make costs easier to forecast than consumption-based cloud data platforms, where query volumes, AI workloads, and data processing demands can cause bills to fluctuate, Chaturvedi said.

But predictable bills are not necessarily lower bills, warned, Igor Ikonnikov, advisory fellow at Info-Tech Research Group. “The hardware requirements for high-speed operational data processing are higher and relatively more expensive compared to cheap lakehouse storage,” he said.

EDB’s architecture could also simplify data governance by reducing the number of platforms enterprises need to manage. Since operational, analytical, and AI workloads can access data through a common Postgres-Iceberg foundation, enterprises may be able to avoid deploying and governing multiple specialized data stores, and so have fewer systems to license and secure, according to Devin Pratt, research director at IDC.

Reducing architectural tax for engineering teams

EDB’s converged analytics could also simplify operations for developers and data engineering teams.

Its architecture reduces the number of systems developers must integrate and maintain, while eliminating much of the pipeline work traditionally required to move data between transactional and analytical systems, according to Walter.

And, said Pratt, “Zero-ETL means far less plumbing to build and break, so engineers spend their time creating value.”

EDB and Databricks are not the only ones pursuing converged analytics to support agentic systems and other applications needing immediate access to operational data, historical context, and governance controls.

Snowflake has been expanding support for operational workloads by embracing open table formats,and Microsoft has combined transactional and analytical services under a broader data architecture via its Fabric platform.

Evolution of autonomous databases?

Converged analytics, though, was only one part of EDB’s update to its Postgres AI platform.

It has also made generally available what it calls an “agentic database” feature, designed to automate routine database administration tasks.

The system continuously monitors hundreds of operational and performance metrics, detects anomalies, recommends corrective actions, and, where enterprise policies permit, can automatically apply fixes, the company said.

These automated agents can help enterprises optimize and tune their databases up to 10 times faster, it said.

Walter remained skeptical: “It is more an evolution of autonomous database concepts than a wholly new category. Oracle and other database vendors have offered autonomous database capabilities for years.” Where EDB can differentiate itself, she said, is in extending those autonomous capabilities with AI-driven reasoning, automated remediation, and governance controls that allow enterprises to determine how much authority the system receives.

I’ve seen a lot of promising AI prototypes fall apart after launch. And it’s rarely because the model was bad. More often, the problem starts much earlier; teams treat the data layer like something they can figure out later.

They’ll spend weeks fine-tuning prompts, testing models and debating evaluation scores, then throw together the retrieval pipeline over a weekend and move on. At first, everything looks great in demos. But a few months later, the system gives outdated answers; the embeddings no longer match the source documents, and nobody fully understands what changed.

What started as an impressive prototype slowly becomes difficult to trust in production. The teams that avoid this tend to realize one thing early: Embedding pipelines are fundamentally a data engineering problem, not an entirely new AI discipline. It’s still ETL (Extract, Load, Transform) at its core, but with embeddings and vector stores as the destination instead of a warehouse.

Once you start looking at it that way, a lot of things become clearer. Problems like versioning, data freshness, lineage and retries stop feeling “AI-specific.” They’re data infrastructure problems we’ve already spent years learning how to solve.

Why do we need embedding pipelines?

Large language models are extraordinary reasoners trapped inside a time capsule. When training ends, the model’s knowledge is sealed. It does not know what your team decided in last quarter’s strategy review. It has never read the support ticket that came in this morning. It cannot find the clause buried on page 47 of your master service agreement. It’s brilliant, but blind to anything specific to your organization.

Layer on top of that a hard context window limit, a ceiling on how much text the model can process in a single interaction, and you have a clear problem: you cannot just hand it everything you own.

The answer the industry converged on is retrieval-augmented generation, or RAG. Instead of stuffing everything into the context window, you build a retrieval layer that fetches only the most relevant pieces of information at the moment a question is asked and passes just those to the model. That retrieval layer is powered by a vector database, and the process that populates it, which is taking raw documents and transforming them into searchable semantic representations, is what I mean when I say embedding pipeline.

Every team building an internal AI assistant, a smarter enterprise search tool, an automated customer support agent or a document Q&A system needs one. The question is not whether to build it. The question is whether you build it like a prototype or like infrastructure.

How an embedding pipeline works

An embedding pipeline has three stages: ingestion, chunking and indexing. Here is what each one means and how I relate them to a typical ETL process.

Ingestion is extraction

Getting your raw content, PDFs, wiki pages, Word documents, database records, transcripts, out of wherever it lives and into the pipeline. This is ETL’s extract stage, almost verbatim.

I see teams cut corners here more than anywhere else, and it’s often where production systems first start to fail. A document gets updated, but the pipeline doesn’t pick it up. A file gets deleted, but its chunks remain in the index, still returning outdated answers months later. And because there’s no obvious error, no one reports it.

The fix is Change Data Capture (CDC). This maintains a manifest of every document you have ingested, a content hash and a timestamp. On each run, we compare sources against that manifest, re-ingest what changed, delete what is gone and treat your document the way you would treat any source table you are syncing incrementally.

Chunking is transformation

Once your documents are in the pipeline, you cannot embed them whole. A 30-page technical report is too long to represent meaningfully as a single vector, and even if it were not, returning the entire report in response to a narrow question would bury the model in irrelevant context.

Chunking is the process of breaking each document into smaller pieces that are focused enough to embed accurately and retrieve precisely. This is ETL’s transform stage, and it deserves the same level of design discipline.

The most common mistake I see is treating chunk size as a default configuration option rather than a product decision. It is not. The right chunk size depends entirely on the nature of your content and the nature of your queries. Dense technical documentation needs finer granularity than a collection of FAQs. A legal contract with clause-level logic needs different treatment than a set of onboarding emails. What works for one document will actively degrade retrieval quality on another.

My strong preference is to treat your chunking configuration as a versioned pipeline parameter, not hardcoded logic. When you change it, and trust me, you will. You need to re-chunk in a controlled, observable way, compare retrieval quality before and after, and roll back if it degrades. That is just good transform-layer hygiene. It is no different from versioning a data cleaning rule or a field mapping.

Indexing is the load

The final stage is where chunked text gets converted into vectors and stored in a vector database where it can be searched by semantic similarity rather than keyword match.

In the conversion step, embedding is handled by a model specifically trained to turn text into dense numerical representations that encode meaning. Two chunks expressing the same idea in different words will produce vectors that cluster close together in that mathematical space. Two chunks discussing entirely different topics will sit far apart. When a user asks a question, the system embeds that question the same way, finds the chunks whose vectors are nearest, and returns them as context for the model to reason over.

That is a genuinely new capability. But the discipline around indexing is not. One data engineering principle I keep coming back to is versioning.

In embedding pipelines, every chunk in your index should be tagged with the embedding model name and version used to generate it, this is non-negotiable. Embedding models evolve, and vectors produced by different versions are not comparable in a reliable way. You cannot safely search across them as if they are interchangeable.

This exact problem shows up when teams upgrade embedding models mid-pipeline without a proper migration plan. You end up mixing vectors from different generations in the same index, and retrieval starts to degrade in ways that are hard to detect. The system just quietly begins returning subtly wrong answers.

I treat an embedding model upgrade the same way I treat a schema migration: Plan it explicitly, execute it in full and validate retrieval quality on a representative query set. The stakes are the same as any breaking change to your data model.

Pipeline observability is not optional

Once an embedding pipeline is running in production, the question shifts from “did it run” to “did it run correctly.” That distinction matters more here than in most pipelines, because failures are rarely loud because the index looks fine, queries return without errors and the system quietly surfaces wrong answers until someone notices the AI has stopped being useful.

The same observability discipline that makes any data pipeline trustworthy applies directly here. Once you treat embedding pipelines as production systems, you stop thinking in isolated steps and start thinking in signals. For example, chunk counts per document become a simple but powerful health check, a sudden drop is usually not a model issue, but a sign of broken ingestion or upstream parsing failures.

You also need a “golden set” of queries with known-good outputs. This runs after every pipeline change, much like data quality checks after a transformation. This is how you catch regressions that don’t show up as explicit failures.

On top of that, you can track lineage: Which embedding model version produced which chunks, and when each document was last ingested. That makes it possible to trace retrieval issues back to specific changes instead of guessing.

And finally, freshness becomes a first-class signal. If documents start going stale beyond an acceptable threshold, that should surface in monitoring long before users experience degraded results.

The metric that ties it all together is retrieval quality over time. Treat it like any other pipeline SLA, measured, tracked and owned.

Summary

Embedding pipelines definitely come with a lot of new language, new tools and a genuinely different capability in the semantic layer. But the funny thing is, the principles that actually make them reliable in production are not new at all.

We have versioning, freshness, quality checks and monitoring. These are problems data engineering has already spent years solving.

The real work is taking that same discipline and applying it to a pipeline that just happens to output vectors instead of rows in a table. Once you start seeing it that way, a lot of the chaos around AI systems becomes much easier to reason about.

That’s the difference between building a cool AI demo and building something people can actually depend on. One is a prototype, whereas the other is infrastructure.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?