The Quiet Migration Happening Inside AI Infrastructure Teams
A growing number of ML engineering teams are quietly moving their model serving workloads off Modal and onto Baseten – not because of a dramatic failure, but because of a slow accumulation of friction that compounds at scale.
Why Baseten Is Gaining Ground
Baseten has been building model serving infrastructure since 2019, which makes it older than most of its direct competitors in the current wave of AI tooling. That history matters. The company spent years working with production ML teams before the generative AI boom arrived and suddenly made model serving a mainstream concern. By the time teams started deploying large language models at scale, Baseten already had battle-tested abstractions around cold start latency, GPU scheduling, and inference optimization that newer entrants were still drafting on whiteboards.
The platform’s core offering is Truss, an open-source model packaging framework that lets engineers define their serving environment in code rather than through a dashboard. This approach appeals directly to teams with strong DevOps instincts – the kind of teams that want their infrastructure to live in version control and behave predictably across environments. For many ML engineers, this is a default expectation, not a premium feature. Modal’s more abstracted serverless model can feel opaque when something goes wrong, which is precisely when engineers want full visibility.
Baseten has also put significant engineering effort into GPU warm pools and request batching, two areas that directly affect inference cost and latency at the margins where real production workloads live. Cold starts on serverless GPU platforms are a real operational headache – a model that takes 30 to 60 seconds to load from cold is unusable in most real-time applications. Baseten’s architecture prioritizes keeping models warm in ways that Modal’s fully serverless design makes harder to guarantee without significant configuration work.
The economics are worth examining directly. Modal bills per-second of GPU compute, which sounds efficient on paper but can produce unpredictable invoices when traffic patterns are spiky or when cold starts eat compute without serving requests. Baseten’s pricing model gives teams more direct control over resource allocation, which makes cost forecasting easier. For a startup running tight margins on inference costs, predictability can matter as much as the rate itself.
Where Modal Still Holds and Where It Doesn’t
Modal is genuinely good at what it was designed for: fast, flexible serverless compute for Python workloads that need GPU access without infrastructure management. For researchers, solo developers, and teams running batch jobs or experimental pipelines, Modal’s developer experience is difficult to match. The platform makes it almost trivially easy to spin up a GPU-backed function with a decorator and a few lines of Python. That frictionless entry point is real, and it explains why Modal has accumulated a passionate user base in research-adjacent environments.
The tension emerges when teams move from prototyping to production. Modal’s design philosophy optimizes for flexibility and speed of iteration, which are exactly what you want when you’re exploring model architectures or building internal tools. But production serving has different requirements: consistent latency, graceful traffic handling, detailed observability, and clear cost controls. These are areas where Baseten has invested more deeply, and where the gap between the two platforms becomes visible under load.
Observability is a specific pain point worth naming. Baseten offers more granular request-level logging and performance tracing out of the box. For a team debugging a latency spike in production, the difference between a platform that shows you exactly what happened at the request level and one that gives you aggregate metrics is the difference between a 20-minute fix and a 4-hour investigation. This is not a theoretical difference – it’s the kind of thing that shapes which platform gets renewed at the next billing cycle.
Modal has responded to some of these concerns with product updates and expanded documentation, and its team moves quickly. The platform’s community is active and vocal, which accelerates bug fixes and feature requests in ways that can outpace larger, slower-moving infrastructure companies. Baseten, by contrast, tends toward a more enterprise-oriented sales motion, with dedicated support and onboarding that resonates with teams inside funded startups and mid-size companies but can feel like overkill for a two-person research lab.
The teams switching to Baseten are not, in most cases, teams that had a bad experience with Modal. They are teams whose requirements changed. A company that started with five model endpoints running occasional batch inference now runs fifty endpoints handling real-time traffic from paying customers. The platform that fit the first scenario does not automatically fit the second, and Baseten has positioned itself squarely in the second scenario’s territory. This pattern mirrors what has happened in other developer infrastructure categories – where newer, more focused tools gain traction not by winning head-to-head evaluations but by being the obvious choice at a specific stage of a company’s growth.
What This Means for the Broader Inference Market
The Baseten-versus-Modal dynamic is a preview of a larger sorting that is happening across the model serving market. As AI workloads mature from experiments into revenue-generating products, the infrastructure requirements get more specific, not less. Teams are moving past “can this run my model” and toward “can this run my model reliably, cheaply, and observably under unpredictable traffic.” That question has a different set of answers depending on which platform you ask.
Baseten’s bet is that the majority of AI-native companies will eventually need what it’s built – dedicated inference infrastructure with production-grade controls. The open question is whether Modal, or any of the other entrants in this space, will close the production-readiness gap fast enough to retain teams as their workloads scale. Right now, the migration pattern is real, and the teams making the switch are not going back.
