Jesse Williams

Introduction Over the past few years, large language models (LLMs) have transformed how we build intelligent applications. From chatbots to code assistants, these models are used to power production systems across industries. But while training LLMs has become more accessible, deploying them at scale remains a challenge.

Running machine learning on Kubernetes has evolved from experimental curiosity to production necessity. But with hundreds of tools claiming to solve ML (machine learning) deployment, which ones should you consider? This guide cuts through the noise, presenting the essential open source tools that real teams use to build, package, deploy, and monitor ML models on Kubernetes. Most of these tools are fairly well known, however, I tried to incorporate a few emerging and lesser known tools.