A year and a half ago I sat in a conference in Silicon Valley and watched an AI agent write production-quality code in real time. I knew our engineering model was going to change, and within a few weeks my team was past the question of whether to adopt AI. The real debate was how far we were willing to go in redesigning how we build legal software around it.

Since then, output per engineer across our R&D organization has roughly tripled. We ship nearly twice as many releases per quarter, deployments have grown from 82 to more than 155, and customer reported defects are down 65% per million lines of code across the last 18 months. We track all of this with DORA metrics, cycle time, pull requests merged per developer, and lines changed per developer against a fixed baseline, and the improvement holds up on every measure.

Getting there had less to do with the tools than with how the organization works. Every CTO I talk to is running some kind of AI coding pilot and seeing decent results: faster prototyping, better test coverage, quicker first drafts. Almost none of them have changed the development life cycle itself, and that is where the real gains were for us.

We spent the past 18 months rebuilding our product development life cycle around AI. Requirements, development, testing, security, deployment, and governance all changed, some beyond recognition. We got plenty wrong along the way, and a few of the lessons would have saved us months if someone had handed them to me at the start.

The biggest gains came from removing handoffs

We assumed most of the productivity gain would come from AI writing code faster. It didn’t. The biggest gains came from getting rid of the handoffs between stages.

In the old model, a feature passed through at least four handoffs on its way out the door: product to development, development to QA, QA to security, and security to deployment operations. Each of those queues cost us time and context. A feature could be code complete in a day and then sit for two weeks waiting on teams that each had their own backlog and their own priorities.

So, we restructured. A team now carries a feature end to end, and AI agents handle the rote work at each stage. A feature that used to take 15 days to go from spec to a working pull request now takes about four hours.

If you take one thing from this article, measure the total time from idea to deployment rather than lines of code generated, and go find where the work sits idle. That is where the gains are hiding.

Governance accelerated adoption

Most engineering organizations add governance after the innovation has proven itself. We flipped that order. We build software for lawyers, where a wrong answer on high-stakes work can cost a client dearly, so moving fast and cleaning up later was never on the table.

Before we scaled past the first pilot teams, we set standards for code review, security scanning, and quality on anything AI produced. Confidence scoring lets routine approvals pass through automatically while anything below threshold goes to a human. Security and quality gates run inside the pipeline itself rather than as a checkpoint at the end.

What we didn’t expect was how much this sped up adoption. Engineers trusted the system, so nobody had to wonder whether code from an agent would pass review, introduce a vulnerability, or get them blamed when the model got something wrong. Eighteen months ago, about 3% of our pull requests had any AI assistance. Today it’s 68%, every engineer in the organization uses AI, and vulnerability density has fallen 76% over the same period. Governance came first and adoption followed, which is the opposite of the sequence most people assume.

Pick your tools and get good at them

A better model or agent ships almost every week, and early on we burned real energy evaluating them. At some point we stopped and standardized on a small set of coding agents.

Going deep paid off more than staying current. We learned the strengths and failure modes of our coding agents and tuned our workflows around how they actually behave, which matters more than whatever the next release promises. Teams that chase every new model spend all their time evaluating, and every switch resets the muscle memory and the trust they built with the last tool. They end up with a lot of pilots and not much change.

Automate the entire life cycle

The obvious place to point AI coding tools is at the coding. Some of our biggest wins came from the edges of the life cycle.

We built or deployed dedicated agents for every stage: requirements, sprint planning, code generation, test authorship, security scanning, deployment, and site reliability. Each one owns a single job and gets measured on a single outcome. What makes it work is a centralized knowledge repository that product and engineering maintain together, a curated and structured knowledge base with retrieval-augmented generation on top. Every agent pulls product context, domain rules, and prior decisions from that same source of truth, so what comes out of one stage is clean input for the next. Without it we would have a set of fast agents that lose the thread at every handoff.

Requirements definition was the most dramatic early win. It used to take weeks. We built an agent that grills a raw product idea the way a tough stakeholder would, pushing on gaps until the spec is ready for development, and that phase now finishes in an afternoon.

Testing moved even faster as quality engineers shifted their focus to optimizing and improving the agents that build tests. AI now generates 99% of our new tests, up from 47% a year ago, with more than 39,000 AI-developed tests in the suite, a number we could never have staffed our way to. Each test is generated directly against the code change it covers and checked against the functional requirements defined in the product requirements document, so growth in volume doesn’t come at the cost of traceability. Engineers spend their time reviewing, refining, and hardening what the system produces rather than writing first drafts, and that shift, from authoring tests to curating and strengthening them, is a big part of why quality improved while we sped up.

We started with one area of the product development life cycle, got it right, then expanded. I would recommend that sequence to anyone. The key is to develop a model that can be expanded to the rest of the phases after experimenting on what works rather than trying to roll it out to every area and having to rebuild the framework later.  

Design for the velocity you’re about to create

We set a public and very specific goal: double R&D productivity in 12 months. It was uncomfortable, and there were stretches where I wasn’t sure we would make it. But a number that aggressive forced everyone to accept that incremental improvement wasn’t going to get us there, and we had to take apart old ways of working while the technology underneath was still maturing. We hit the goal inside the year, and output has kept climbing ever since.

What I underestimated was how fast the bottleneck would move downstream. Once engineering sped up, the constraint moved to go to market. Every release still needs documentation, enablement content, briefings for customer success, and customers who are ready for what’s coming, and for a while we were shipping faster than any of that could keep up. We now treat go to market as its own automated phase of the life cycle, with the same agent-driven approach we applied to code and testing, and a simple rule that nothing ships until the rest of the organization can support it. If I were starting over, I would build that capacity from day one instead of finding out about it the hard way.

The real transformation is organizational

The tools matter, but everyone can buy the same tools. What we actually did over these 18 months was redesign the system around AI: the way work flows between teams, the governance that lets people trust the output, the small set of tools we committed to, and a target aggressive enough to force all of it. Any one of those alone would have produced another interesting pilot. Together they produced a different engineering organization.

None of this is finished. The models keep improving and the life cycle will keep changing with them. The organizational groundwork is what lets us absorb each improvement as it arrives, and for any technology leader still stuck at the pilot stage, that groundwork is where I would start.

—

New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.

The consulting firms told us this would be different. They arrived with slick slide decks, expensive vendor packages, and promises to revolutionize how we do business. Their message was clear: Digital transformation was no longer optional; it was existential. Companies that failed to adapt would be left behind and crushed by more agile competitors with cloud-native architecture and data-driven strategies.

That was back in 2020, when Boston Consulting Group released research showing that 70% of digital transformations failed to meet their objectives. The consulting industry rushed to publish its frameworks, maturity models, and proprietary methodologies. Enterprise IT organizations spent millions on these programs, hoping to finally bridge the gap between technological capability and business value.

Several years on, I can say that little has changed. If anything, the issue has worsened.

The damage that was done

Most digital transformations didn’t just fail to deliver ROI; they actively damaged the organizations that undertook them. Technical debt accumulated as companies deployed monolithic cloud platforms that were neither modular nor scalable. Business misalignment became endemic as transformations focused on technology for its own sake rather than on solving actual business problems. Organizations found themselves locked into expensive vendor relationships that failed to deliver on their promises.

I’ve spoken with dozens of enterprise CIOs and CTOs over the past several years, and the stories are remarkably consistent. Transformations that were supposed to take six months stretched to two years. Budgets estimated at a few million exploded to six, eight, even twelve times the original projection. The consulting firms had excellent PowerPoint skills, but when it came to implementation, the results were consistently disappointing.

The damage wasn’t just financial. Organizations lost confidence in their IT capabilities. Talented people left because they were tired of being managed by consultants who couldn’t code their way out of a paper bag. Business stakeholders grew cynical about technology initiatives, having been burned too often by projects that promised transformation but delivered chaos.

Why the consulting models failed

Let’s be direct about what happened. The consulting industry discovered that digital transformation was highly profitable. They could sell their methodologies, vendor partnerships, and “best practices” frameworks, and charge premium fees for implementation support. The problem was that these frameworks were largely one-size-fits-all approaches disguised as bespoke solutions.

A major global bank doesn’t have the same requirements as a regional healthcare provider, yet both were sold the same transformation playbook. The consulting firms had invested too heavily in their methodologies to admit that the real answer might be to start with business requirements and work backward toward technology. Instead, they assembled prepackaged solutions optimized for vendor partnerships rather than clients’ actual needs.

The talent problem was equally severe. The consulting firms could certainly talk about digital transformation; they had excellent slides and compelling narratives. But when it came to implementation, the skill gaps were glaring. These organizations assembled solutions by following playbooks rather than understanding how the technology components actually fit together. The result was systems that were overengineered, overprovisioned, and ultimately underperforming.

Cloud computing was supposed to be the great enabler, but many organizations misunderstood and misapplied it. Organizations lifted and shifted their existing problems along with their existing architecture. Monolithic applications running on expensive virtual machines became the new normal, with organizations paying cloud costs that far exceeded their on-premises expenses, yet achieving none of the scalability benefits the cloud was supposed to deliver.

What we still need to learn

As we grapple with the next wave of technology transformation—this time centered on artificial intelligence—the lessons from digital transformation failures should be front and center. The first and most important lesson is brutally simple: Focus on business needs first. I know this sounds obvious, but I guarantee you that every failed digital transformation project lost sight of its core business requirements. They built technology for technology’s sake without understanding the problems they were trying to solve.

Too many transformations focused on individual business units rather than enterprisewide strategies. The result was a collection of systems that couldn’t work together cohesively. Isolated solutions felt like progress but actually made integration harder and created new dependencies that hadn’t existed before.

Organizations also need to invest in serious training for their people. I’m not talking about sending IT staff to vendor certification programs that teach them to pass tests on proprietary platforms. I mean fundamental architectural education that helps people understand how to assemble technology effectively.

We face a massive shortage of AI, cloud, and security architects, as well as other technical leaders who can design solutions that work. Rather than addressing this gap, organizations are placing unqualified people in these roles and hoping for the best. It’s past time to invest in education.

The case for simplicity

Here’s something that also should be obvious but clearly isn’t: You can solve the same business problem with $1 million or $20 million, depending on the sophistication of the IT organization building the solution. Both designs will work reasonably well, but one solution costs 20 times more because it is overbuilt and overengineered.

Organizations are adding AI to everything under the sun, even when simpler solutions would work just as well. They’re overprovisioning cloud resources because they lack the architectural discipline to rightsize their infrastructure. They’re building systems designed for scale they’ll never reach, complexity they don’t need, and features no one requested.

The path forward requires a minimalist approach. Build systems that are good enough for their intended purpose. Focus on requirements rather than capabilities. Resist the temptation to adopt every new technology just because it’s available.

Digital transformation didn’t fail because the technology was flawed. It failed because we forgot that technology exists to serve the business, not the other way around. To succeed with AI, we need to remember that fundamental truth. And remember the consulting industry’s tendency to sell us expensive solutions to problems we don’t have.

Meta won’t judge employees by how much they use AI when it comes to annual performance reviews, despite early efforts to drive AI adoption focusing on so-called token maxing.

The company has told employees that it “will not use AI adoption dashboards or token counts to evaluate impact,” according to  a report by The Information.

The announcement came in an internal memo from executives Maher Saba and Santosh Janardhan, which said that instead of measuring AI usage, “managers should look at output quality, velocity, problem complexity and scope taken on.”

This marks a culture change for Meta, where engineers had previously competed to consume the most AI tokens, displaying their scores on a leaderboard. Meta then discovered that its employees were being diverted from regular work because they were using AI to carry out additional tasks to boost their scores.

Amazon had similar results when it implemented a leaderboard to track AI use; it also found that some employees were trying to game the system by using AI to complete unnecessary tasks, and it has now deleted it.

The company, however, does also monitor employees’ use of AI for training purposes, in a program introduced in April, but this is information was not used to measure employee performance.

Meta had already started to look askance at the concept of using AI metrics as a tool to assess employees. Earlier this year, Chief Technology Officer Andrew Bosworth told employees in a memo that “nobody should be using AI tools just for the sake of using them,” adding that “token usage alone is not a measure of impact of any kind.”