Sachi Desai

When AI developer and ops teams first adopt large models on Kubernetes, the focus is usually on getting something to run. Initially, the inference service may respond correctly, latency is acceptable, and a proof of concept becomes a production endpoint. It can feel like the hard part is over. As adoption grows, however, operations start to change.

Over the past year, AI inferencing has become significantly more resource-intensive due to the exponential growth in the size and capabilities of large language models (LLMs). These models are not only larger but also more capable, powering a wide range of applications from advanced reasoning and instruction-following to highly specialized, domain-specific tasks.

Generative AI is booming as industries and innovators look for ways to transform digital experiences. From AI chatbots to language translation tools, people are interacting with a common set of AI/ML models, known as language models in everyday scenarios. As a result, new language models have developed rapidly in size, performance and use cases in recent years.