My current model-selection strategy is embarrassingly simple… and wrong: I choose the most capable (expensive) model because I’m worried the cheaper one might get something wrong. That makes me spend too much, of course, which is why I (like you, perhaps?) continue to struggle with one of the most fundamental challenges in AI: How can I know when a cheaper model is sufficient for a task? Or, really, which model should I use at all?
I asked a friend, Leo Zheng, who leads marketing for Fireworks AI, an AI infrastructure company that runs and improves open-weight models. Surely it was his job to know? His answer surprised me. I thought the answer would come down to models, but it doesn’t. Fireworks, he said, wants to “enable every company to own the continual learning loop within their four walls.” The idea is to abstract away model updates while companies feed the system new signals as customer behavior changes.
That’s when it clicked. I was asking how to automate model choice, but the harder problem is building a feedback loop that tells a company what worked. Model choice then becomes simply one important action the system can take, not the be-all and end-all decision a developer must get right in advance. Arguably, the more important component is integrating enterprise data into that continual learning loop that Zheng describes.
I’ve made that argument before, and I was right. Sort of. Proprietary data doesn’t automatically guarantee AI success. Neither does dumping that data into a retrieval-augmented generation (RAG) system. Your competitors can do the same. The more durable advantage comes from outcome data: traces that connect what the system saw, what it did, and whether the business result was acceptable. Those traces become valuable when a company uses them to improve the system and carry the learning to the next model.
Make the data earn its keep
Public benchmarks are useful, but they only tell you how a model performed on someone else’s test. They can’t tell a bank whether a fraud alert was correct, inform a retailer whether an AI agent resolved a return, or tell a software company whether AI-generated code survived review. Enterprises already have pieces of those answers buried in support transcripts, accepted patches, abandoned carts, etc. The trick is turning that mess into something a system can learn from.
Think about a software company. It already collects product analytics to improve its product. AI can extend that flywheel. Application signals can improve the system powering the product. How? By capturing user intent, engagement, and preferences.
This isn’t foolproof, however. A fast learning loop built on a weak proxy is simply a fast way to make the system worse. For example, suppose Toni in customer support accepts a suggested response. Was it good? Maybe. But what if Toni was rushing (as she is prone to do) and the customer reopened the ticket the next day? It’s not enough to act on inputs alone; a company needs to connect the suggestion to the eventual outcome or risk rewarding the wrong behavior.
Acceptable or ideal outcomes must be defined by the enterprise. A platform like Fireworks can optimize through learning loops, but only if desired outcomes are defined. This changes the cost calculation. After all, the cheapest model by token can be the most expensive model by outcome if people must review, retry, and repair its work. A company can be hemorrhaging money on AI failures while congratulating itself for lowering inference costs.
Open weights get useful
None of this requires open weights, of course. A company can own its evaluations, improve prompts and retrieval, and route among closed APIs and open models without changing a single parameter. For many workloads, that will be enough.
This generally won’t start with reinforcement learning, either. It starts by picking one workflow with an observable outcome and logging the request, model, cost, answer, and what happened next. Do that for a quarter and patterns should emerge. A cheaper model will be good enough for some work, while a proprietary frontier model will earn its price elsewhere. However, in a third category, every model may fail in the same repeatable way, and it’s this last category that makes the serious case for changing the weights.
A few years ago, I argued that open source succeeds when it makes developers’ lives easier, not when it wins an ideological purity contest. The same applies to open weights. The importance of open weights isn’t that they’re a cheaper facsimile of whatever OpenAI and Anthropic are shipping. Instead, they matter because they shift leverage away from the models and toward users with proprietary outcome data. They let enterprises turn what they learn into something they can shape, keep, and carry forward as models change.
Open weights turn model renters into model owners. That’s a very big deal.
For Cursor’s Composer 2, the company started with the open-weight Kimi K2.5 model, then specialized it through continued pretraining and reinforcement learning in environments built to resemble Cursor itself. The interesting thing isn’t that Cursor picked the “right” model, but rather that Cursor can use what it learns from developers to keep making that model more “right” for them. In this case, Fireworks ran the reinforcement learning inference, while Cursor supplied what only Cursor could: its product environment and definition of good. Cursor subsequently built Composer 2.5 on the same Kimi K2.5 checkpoint and says 85% of its compute went into additional training and reinforcement learning.
The base model was the starting point, not the finished product.
This is why open weights matter. They turn the model from a finished product into raw material. The foundation labs have broad knowledge, far more than any normal company could hope to collect. What they don’t know is whether your patch shipped, your fraud was stopped, your customer stayed, or Toni’s response made matters worse. You do.
RAG lets a model consult your data. A learning loop lets the system learn from it. One helps answer today’s question, but the other helps the company answer tomorrow’s question better.
Control must be convenient
Raw weights alone won’t deliver that control. Open source didn’t take over the enterprise because CIOs wanted tarballs. It won because Red Hat, and especially cloud providers, turned theoretical control into everyday convenience. Open-weight AI needs the same translation.
Enterprises want the convenience of a closed API without surrendering control over the intelligence their data creates. That’s the real Fireworks opportunity. Not faster tokens, but a shorter distance between learning something about your customers and making the product smarter because of it. Fireworks CEO Lin Qiao calls this “application-specific intelligence,” and that’s exactly right. Enterprises don’t need to outbuild OpenAI or Anthropic. Even if they wanted to, they couldn’t. Instead, they need to turn what they know about their customers and workflows into intelligence those labs can’t supply.
Fireworks isn’t alone. Together AI also spans training, evaluation, and serving, while Baseten now talks about connecting production feedback to model training. This is becoming a category, not simply a Fireworks feature. Fireworks’ particular bet is that the entire learning loop should belong to the customer.
So, back to the question I was asking Zheng: How do I pick the right model in advance? By now it’s clear: I can’t. Neither can you, and that’s fine, because it’s not really the right question.
Once the enterprise owns the test, it no longer has to trust a vendor’s benchmark or brand. New models must prove themselves against the work that matters to that company, then be promoted, specialized, or discarded accordingly. Model choice becomes a consequence of learning, not an act of faith. In effect, the models audition for the enterprise, not the other way around.
No one can predict whether OpenAI, Anthropic, Google, or the open-weight community will lead six months from now. The sensible response is to stop making that prediction central to the strategy. The enterprise’s own data should decide which model to pick. That’s the bet I find compelling in Fireworks: Make open-weight models as easy to improve as APIs are to call, so enterprises can stop treating their data as context for somebody else’s model and start using it to build intelligence they control.
Login
