KV Caching in Large Language Models (LLMs)
Master the mechanics of KV caching to optimize Large Language Model inference. This course covers the architectural theory, memory footprint calculations, PyTorch implementation, and advanced optimizations like GQA and PagedAttention.
The Autoregressive Bottleneck & Inference Phases Explain the computational bottleneck and quadratic time complexity of naive autoregressive text generation. Distinguish clearly between the compute-bound 'prefill' phase and the memory-bound 'decode' phase. The KV Cache Mechanism Describe the mechanism of KV caching and how it reduces the time complexity of the decode phase from quadratic to linear. Explain how caching shifts the primary hardware bottleneck from compute (FLOPs) to memory bandwidth. Calculating KV Cache Memory Footprint Formulate the mathematical equation for KV cache memory size based on layers, heads, dimension, and context length. Calculate the memory footprint of a KV cache for a given model architecture and context length. Implementing KV Cache in PyTorch Implement a basic KV cache mechanism during the inference loop using PyTorch. Manipulate and update the `past_key_values` tensor accurately across generation steps. Advanced Optimizations: MQA, GQA, and PagedAttention Compare Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) as architectural methods to reduce KV cache size. Explain how vLLM and PagedAttention solve memory fragmentation issues in KV caching.