Quantization's Hidden Tax, Cliff Tokens, Smarter Memory
Three new arXiv papers reveal hidden costs in quantized reasoning models, single-token failure triggers, and a new framework that cuts agent memory errors by up to 79%.
Articles Tagged "Inference"
Best LLM Gateways 2026 - LiteLLM, Portkey, and 5 More
A hands-on comparison of seven LLM gateway and routing tools - LiteLLM, Portkey, Helicone, OpenRouter, Martian, Cloudflare AI Gateway, and Bifrost.
DiffusionGemma 26B Review: 4x Faster, Real Tradeoffs
Google DeepMind's DiffusionGemma generates 1,000+ tokens per second through parallel diffusion, trading 5-19 benchmark points against Gemma 4 for speed and unique bidirectional generation capabilities.
Amazon Eyes Third-Party Trainium Sales to Rival Nvidia
Amazon CEO Andy Jassy hints the company will sell Trainium3 racks directly to outside data centers, citing a potential $50B revenue run rate and sold-out chip supply.
Baseten Closes $1.5B Round as Open-Source AI Matures
Baseten's $1.5B raise at a $13B valuation signals a structural shift as open-source models displace closed APIs in enterprise AI.
Voxtral TTS
Mistral's first open-weight text-to-speech model: 4B parameters, 70ms latency, voice cloning from 3 seconds of audio, and a 68.4% win rate over ElevenLabs Flash v2.5 in blind tests.
Faster Agents, Skewed Evals, and Brand Bias in LLMs
Three new papers: agents that compile runs into 8-13x faster state machines, benchmark scores that shift with compute budget, and big brands monopolizing LLM recommendations.
AMD Instinct MI450 - 2nm, 432 GB HBM4, 40 PFLOPS
AMD's CDNA 5 accelerator on TSMC 2nm with 432 GB HBM4 memory - the GPU behind OpenAI's 1GW deployment and Oracle's 50,000-chip supercluster.
Google DiffusionGemma: Parallel LLM Hits 1,100 t/s
Google DeepMind open-sources DiffusionGemma, a 26B MoE model that generates 256 tokens per denoising pass instead of one at a time, reaching 1,100 tokens per second on a single H100.
DiffusionGemma 26B
DiffusionGemma 26B is Google DeepMind's open-weight discrete diffusion language model that generates 256 tokens in parallel, reaching 1,100+ tokens/sec on H100 - roughly 4x faster than autoregressive models of the same size.
Ministral 3 8B
Mistral AI's mid-tier open-weight edge model - 8B parameters, 256K context, Apache 2.0 license, built for agentic pipelines and cost-sensitive production workloads.
MiniMax M3 Makes 1M Context Viable With Sparse Attention
MiniMax M3 uses sparse attention to cut long-context inference cost 20x, topping GPT-5.5 on coding benchmarks at a fraction of the price.
