Speculative decoding is a proven, mathematically lossless way to cut LLM latency and raise throughput 2-3x. This whitepaper covers how it works, variant comparison, framework support, and a production deployment reference architecture.

In this white paper, you'll discover:

• What Is Speculative Decoding: Speculative decoding is an inference acceleration technique that lets a large language model produce text faster without changing its output distribution.
• The Latency Problem in Autoregressive LLM Inference: Large language models generate text autoregressively: each token is produced one at a time, and every new token depends on all tokens that came before it.
• How Speculative Decoding Works: Speculative decoding runs two models in concert: a draft model and the target model.
• Draft Model Strategies and Selection: The choice of draft model is, in the words of practitioners at General Compute, "the most consequential decision" when deploying speculative decoding.
• Acceptance, Verification, and Losslessness: The mathematical guarantee that makes speculative decoding safe for production is losslessness.

Download the full white paper below for the complete analysis, architecture detail, and implementation guidance.