Archive
All articles
23 articles across Silicon, System, and Algorithm
Context Window Management for Multi-Agentic Platforms
Eight strategies ordered by leverage-per-complexity — from prompt caching to bandit-scored tool selection — with the economics, code, and latency budget for each.
The KV Cache: A Definitive Engineering Analysis
Naive autoregressive inference recomputes the entire attention matrix at every decoding step. KV caching stores the immutable K and V projections instead, converting per-step O(t·d) cost to O(d) exactly — no approximation.
Flash Attention: A Complete Architectural Autopsy
From O(N²) HBM catastrophe to Hopper-native warp-specialized pipelines: every trick, every number, every tradeoff decoded. Standard attention is not bottlenecked by arithmetic — it is bottlenecked by memory bandwidth.
PagedAttention: How vLLM Borrowed Virtual Memory to Unlock GPU Serving at Scale
Static KV cache pre-allocation wastes 60–80% of GPU memory under production traffic patterns. PagedAttention applies OS-style virtual memory paging to the KV cache, enabling 3× higher concurrency on the same hardware.
Continuous Batching: The Scheduling Insight That Made LLM Serving Actually Work
Static batching blocks an entire batch until its longest sequence finishes. Moving the scheduling boundary from request to iteration frees GPU slots at every decode step, taking utilization from 20–35% to 70–85%.
The Missing Decision Guide: Parallelism Strategies in LLM Training and Inference
NVLink delivers 9–18× more bandwidth than InfiniBand NDR. Every parallelism decision is a negotiation with that ratio — this is the decision tree from hardware topology and model architecture to a concrete TP/PP/SP/DP configuration.
NCCL Collective Operations: The Architecture of Distributed Gradient Truth
AllReduce, ReduceScatter, AllGather, and Broadcast are the atomic grammar of distributed training, with derivable cost models. AllReduce decomposes exactly into ReduceScatter + AllGather — the identity that powers ZeRO and FSDP.
The Brutal Economics of LLM Inference
TTFT, TPOT, throughput, and cost per million tokens are the four numbers that determine whether your LLM infrastructure makes money. Prefill is compute-bound; decode is memory-bandwidth-bound by a factor of 295.
CUDA Streams and Kernel Concurrency: The Overlap Engine Behind ZeRO-3 and FSDP
A CUDA stream is a FIFO queue of GPU operations executing asynchronous to the CPU — the complete definition. Understanding it is the prerequisite for overlapping AllReduce communication with compute in ZeRO-3 and FSDP.
Writing Custom GPU Kernels in Triton: A Hands-On Guide for ML Engineers
CUDA forces simultaneous management of threads, warps, shared memory, and register pressure. Triton lets you write per-block logic in Python; the compiler handles vectorization, coalescing, and warp scheduling. Fused softmax end-to-end.
ZeRO vs FSDP: A Rigorous Dissection for Distributed Training Engineers
AdamW training consumes 16 bytes per parameter — 12 of them optimizer states. ZeRO and FSDP shard that triad differently: this is the quantitative account of what each does to your memory, communication bus, and iteration time.
Positional Encoding in Transformers: A First-Principles Engineering Treatise
Self-attention is blind to token order by design. From sinusoidal foundations through RoPE, ALiBi, and YaRN, every positional encoding makes a different tradeoff between in-context position and out-of-distribution length generalization.
Attention's Memory Problem Has Three Solutions: MQA, GQA, and MLA
At 32K context and batch 32 in FP16, Llama-3-70B's KV cache exceeds 137 GB — more than an H100's HBM. MQA, GQA, and MLA compress that cache architecturally, each making a different quality-memory tradeoff.
Speculative Decoding: The Physics of Stolen Time in Autoregressive Inference
A large model verifies tokens in roughly the same wall-clock time it generates one. Speculative decoding weaponizes this asymmetry: measurable, provable speedup without changing the probability distribution the target model assigns.
Beyond Vanilla Speculative Decoding: Medusa, Eagle, and Lookahead
Vanilla speculative decoding requires a separately trained and versioned draft model — one that must be realigned every time the target is fine-tuned. Medusa, Eagle, and Lookahead each eliminate that operational burden differently.
Mixture-of-Experts Internals: A Systems Engineer's Field Manual
MoE decouples total parameters from per-token compute — DeepSeek-V3 uses 37B of 671B parameters per token, an 18:1 ratio dense models cannot match. The router, AllToAll dispatch, and aux-loss design are where that leverage is won or lost.
The Inference Engineer's Definitive Guide to Quantization
LLM decode is memory-bandwidth-bound: loading 140 GB of FP16 weights takes 42ms at H100 bandwidth regardless of batch size. Cutting to INT4 quadruples your throughput ceiling and eliminates multi-GPU tensor parallelism on 70B models.
The LoRA Family: A Mathematically Precise Guide to Parameter-Efficient Fine-Tuning
Fine-tuning weight updates concentrate in a low-rank subspace of the d×d parameter matrix. LoRA exploits this to reduce trainable parameters by 10,000× per layer, from the foundational algebra through QLoRA and DoRA.
The Modern Alignment Landscape: A Sharp Comparative Analysis for ML Engineers
RLHF requires four models in memory simultaneously and is notoriously hyperparameter-sensitive. DPO eliminates the reward model algebraically. ORPO eliminates the reference model. KTO works with binary feedback alone.
Engineering the Infinite Context Window: A Systems-Level Guide to 128K+ Token Models
Extending to 1M tokens hits three physical walls: quadratic attention compute, linear KV cache memory (327 GB for Llama-3-70B at 1M tokens), and positional encoding extrapolation. Each demands a different class of solution.
H100 vs H200 vs B200: TCO for Inference Infrastructure
Beyond the spec sheet: deriving actual cost per million tokens for each generation, accounting for memory capacity, bandwidth, rack power, and cooling — the numbers that determine your infrastructure decision.
Intra-node vs Inter-node Interconnects in Distributed Training
NVLink, NVSwitch, InfiniBand, and RoCE — the bandwidth and latency numbers that determine whether your distributed training job scales or stalls.
GPU Memory Hierarchy and Kernel Performance
Why memory bandwidth — not FLOPs — is the binding constraint for most LLM workloads, and how H100's five-level hierarchy determines what your kernels can actually achieve.
