Running large language models on Kubernetes usually means dealing with GPU infrastructure, model serving, scheduling, networking, and operational complexity.
Instead of calling a hosted LLM API, we will deploy an open source model directly on Azure Kubernetes Service (AKS), let KAITO provision the required GPU infrastructure, expose the model through an inference service, and send a real prompt to it.
Azure Kubernetes Service provides another approach through the AI Toolchain Operator add on, based on the open source Kubernetes AI Toolchain Operator (KAITO) project.
KAITO allows us to describe an AI workload through a Kubernetes Workspace resource. From there, the operator can coordinate the GPU compute, model deployment, inference runtime, and Kubernetes resources required to serve the model.
In this guide, we will walk through the steps required to deploy Microsoft Phi 4 mini-instruct on AKS using:
- Azure Kubernetes Service
- AI Toolchain Operator / KAITO
- NVIDIA A100 GPU compute
- Phi 4 mini instruct
- vLLM
- An OpenAI compatible inference API
Microsoft’s AKS documentation describes KAITO as a managed add on for deploying and operating open source LLM workloads on Kubernetes, with capabilities including vLLM integration, prompt formatting, streaming responses, and OpenAI compatible APIs.
The target architecture is:
Azure Subscription → AKS → KAITO → Workspace → GPU Node Pool → Phi-4-mini + vLLM → OpenAI-compatible API → Application
Why Use KAITO on AKS?
Without an AI operator, running an LLM on Kubernetes can require manually coordinating several components:
- GPU node pools
- GPU scheduling
- NVIDIA configuration
- Model containers
- Model downloads
- Inference servers
- Kubernetes Services
- Scaling
- Resource scheduling
Tej Pratap
Ewald Hollestelle
rsujiraja
Sathish Veerapandian
EzCloudInfo 