Small Models, Big Agents: Fine-Tuning and Serving SLMs on Kubernetes
LLM, Kubernetes, KEDA, Karpenter, AI
Key Takeaways
- Match the model to the task: which agent steps suit small CPU models, and which genuinely need a larger LLM.
- Deploy efficient SLM inference on CPUs with llama.cpp.
- Build an agent whose routing, tool selection, argument extraction, and self-critique run on small CPU models, handing off to a larger model for open-ended reasoning when that's the right call.
- Train task-specific CPU specialists (classification, scoring, extraction, embedding) for each decision in the pipeline.
- Give agents safe, read-only tools so answers stay grounded in live cluster state.
- Right-size infrastructure with KEDA and Karpenter to balance cost and performance.
- Apply "right tool for the right task" thinking to production agentic AI.
Target Audience
- Platform Engineers
- Kubernetes Engineers
- MLOps Engineers
- AI/ML Engineers
- Cloud Architects
- DevOps Engineers
- Technical Leaders responsible for AI platforms
Requirements for Attendees
Technical Requirements
- Basic understanding of Kubernetes concepts.
- Familiarity with containers and cloud-native applications.
- General knowledge of Generative AI, LLMs, and AI agents.
Equipment Requirements
Laptop required.
Experience Level
Intermediate.
Duration
2–3 hours.