OpenShift

KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat

Aug 27, 2026 · 21:48

Yuchen Fama and Ashish Kamra of Red Hat argue that public inference benchmarks hide the chaotic reality of agentic workloads, and show how KV cache-aware routing plus prefill/decode disaggregation in the open source llm-d framework tackles it. Red Hat's traces show agentic sessions running from a few turns to 3,000, cache hit rates over 90%, and input-output token ratios past 100:1, making a 10x cost gap between cached and uncached tokens. A live demo shows routing reusing cache on the same pod cutting time from 3 seconds to 1, while a fresh system prompt pays the full 3 again. For prefill/decode disaggregation, they report P99 inter-token latency dropping from roughly 900 milliseconds to about 100 across 16 H100s serving gpt-oss, but note it only wins in the middle concurrency band and requires RDMA or RoCE to move KV caches. Their closing case study runs GLM 5.2 on H200s with three prefill workers to one decode, achieving 4x faster time to first token and 60% more requests.

Lobster Trap: OpenClaw in Containers from Local to K8s and Back — Sally Ann O'Malley, Red Hat

May 22, 2026 · 21:56

Sally Ann O'Malley of Red Hat argues that running OpenClaw in containers with Podman and Kubernetes delivers secure, portable, and reproducible AI agent setups. She uses Podman secrets and OpenClaw's secret ref feature to manage API keys, ensuring secrets stay out of logs and configs. O'Malley demonstrates a local installer that spins up an OpenClaw container in two seconds and lifts the same workload to Kubernetes. She cites an Nvidia team of 10 engineers each running their own OpenClaw in Kubernetes for model evals, claiming it replaced the work of six people. Her vision is a team-standard containerized OpenClaw baseline with company-approved MCP servers and skills, enabling reproducible onboarding and personalization across an organization.