An open source tool that spreads AI inference workloads across GPU clusters has reached version 1.0. The release means the project, called llm-d, is now considered ready for production use.
The project is run by the Cloud Native Computing Foundation. Its backers include IBM, Red Hat, Google and Nvidia.
Supporters say the tool could let more companies host AI inference, including cloud providers beyond the three largest. It aims to get as much performance as possible from scarce and costly GPU hardware.
How llm-d Works
llm-d is a cluster-level orchestrator. It automatically routes workloads to server endpoints to make efficient use of available machines.
It also tries to have AI workloads pull data from local caches as often as possible. This is called a high cache-hit ratio, and it lowers the token cost of running AI at scale.
The project was founded in 2025. Red Hat already packages llm-d with the vLLM inference engine in its Red Hat AI product, and Google runs both tools in its Google Kubernetes Engine service.
Version 1.0 adds multi-tenancy support to keep service quality steady when several AI models share a cluster. It also brings upgrades without disruption and a stable, hardened API.
A feature called agent-aware serving helps AI agents save tokens. It does this by using a local context cache instead of recomputing outputs.
Early Users Report Lower Costs
Cloud provider DigitalOcean uses its own forked version of llm-d for its Managed Agents and Serverless Inference services. Salman Paracha, senior vice president of AI Engineering, said hosting those workloads would be too costly without it.
“We don’t think we can get to distributed scale without llm-d,” Paracha said. He said spreading token workloads helps the company reach a higher cache-hit ratio and better performance.
IBM Research used llm-d to run AI agents for a recent quantum computing project. It compared a cheaper open-weight model, z.AI’s GLM-5.3, with Anthropic’s Claude Opus 5 on the same system.
Running agents for 24 hours on Opus cost $300, while the same period on GLM-5.3 cost $26, according to IBM Research chief scientist Ruchir Puri. Researchers wrote that llm-d improved the cheaper model enough to keep up with the frontier model.
Jason Andersen, an analyst at Moor Insights & Strategy, said most enterprises will likely get llm-d packaged from a cloud provider or software vendor. He said Google’s support for scheduling TPU workloads next to Nvidia GPUs could be an early step toward moving work across hardware.
Paracha and Puri doubted that moving workloads between chip types is practical today. Paracha said model kernels are closely tied to specific hardware from makers such as Google, Nvidia and AMD.
Andersen said CoreWeave offers llm-d as an option but has not made it a core part of its products. Nvidia has contributed parts of its Dynamo framework, though the two tools share many features.
“They’re just basically saying, ‘Hey, if you deploy it on us and use it, we’re cool with it,’” Andersen said of vendor support.
CoreWeave and Nvidia are among the founding contributors. AMD, Cohere, JetBrains, Mistral, Roblox and DigitalOcean publicly endorsed version 1.0 this week.
