KVCache.AI

We've open-sourced AgentENV in collaboration with kvcache-ai. AgentENV is a distributed system for running agent environments at scale. Its components power agentic RL training for Kimi K3, with fast snapshot, resume, and fork support for large-scale parallel agent workflows. Explore on GitHub: github.com/kvcache-ai/Age…

SGLang day-0 speed on Kimi K3: 423 tok/s (measured on gsm8k), plus RL support ready in Miles @radixark! How the largest open-source model runs this fast: we natively implemented and deeply optimized K3’s new architecture with fused KDA decode kernels, DP attention, DSpark, PD disagg, and KDA-aware prefix caching. We've passed Kimi Vendor Verifier and are ready for production! Thanks to @Kimi_Moonshot, @nvidia, @AMD, @KVCache_AI, @modal, and @baseten for building this with us, and to @googlecloud, @nebiustf, @fal, @digitalocean, @runpod, @DeepInfra and @gmi_cloud for serving K3 on SGLang. Blog, cookbook, benchmarks in the comments. P.S. This demo video? Kimi K3 made it itself. Play the game 👇

🚀 Mooncake v0.3.12 is out! This release brings major upgrades across Mooncake Store, Transfer Engine, Expert Parallelism, and platform support. A huge thank you to our amazing community: we're thrilled to welcome 53 new contributors in this release! ❤️ ✨ Highlights: • Distributed SSD-backed KV cache pooling • Smarter transfer scheduling with intent- & policy-based routing and deadline-aware QoS • Broader platform support: TPU/PJRT, HPE Slingshot/CXI, AMD HIP/RDMA, and Sunrise • Published wheel packages for AWS EFA and MUSA • Official Docker Hub images • Many performance optimizations and reliability fixes throughout the project #Mooncake #AIInfrastructure #LLM #OpenSource

NVIDIA inference software keeps driving down token costs, long after AI infrastructure is deployed. ⚡ In just one month on NVIDIA Blackwell, software optimizations improved DeepSeek V4 performance by up to 5×, reducing token costs to roughly one-fifth of previous levels. NVIDIA's integrated inference software stack compounds improvements across runtimes, kernels, networking, and hardware, delivering up to 20× higher throughput on the same GPU. Co-designed with NVIDIA GPUs, CPUs, networking, and systems, and powered by CUDA-native open source frameworks, NVIDIA's inference software stack ensures new model breakthroughs and optimizations run on NVIDIA from day zero, and keep improving throughput and lowering cost after deployment. See how @Baseten, @Cognition, @DeepInfra, @togethercompute, and @Cursor_ai are turning continuous software innovation into lower cost per token: nvda.ws/4eRT43m

Introducing GLM-5.2: Frontier Intelligence, Open Weights - Significant improvements in coding and agentic tasks - Strong long-horizon capabilities with a 1M context window - Two levels of reasoning effort: GLM-5.2 (max) pushes the limits, while GLM-5.2 (high) strikes a strong balance between performance and token efficiency - MIT-licensed open weights - Same API pricing as GLM-5.1 Tech Blog: z.ai/blog/glm-5.2 Weights: huggingface.co/zai-org/GLM-5.2 API: docs.z.ai/guides/llm/glm… Coding Plan: z.ai/subscribe Chat: chat.z.ai

🚀 We just launched KV Cache Analyzer by KVCache.AI! 📊 Analyze KV cache hit rates and estimate prefill throughput speedup under different cache budgets and eviction policies. 🧪 Use preset traces or your own local traces, choose the model and parameters, and see how KV cache reuse improves LLM inference performance across different settings. 👉 Try it now: kvcache.ai/tools/kv-cache…