Deep technical writing on LLM, GPU, and ML systems internals — decoded from silicon to system to algorithm.
The statistics that make a watermark detectable, the entropy ceiling that bounds it, the attacks that cost under $50, and the compliance arithmetic that made three frontier labs ship it anyway.
Eight strategies ordered by leverage-per-complexity — from prompt caching to bandit-scored tool selection — with the economics, code, and latency budget for each.
Naive autoregressive inference recomputes the entire attention matrix at every decoding step. KV caching stores the immutable K and V projections instead, converting per-step O(t·d) cost to O(d) exactly — no approximation.
From O(N²) HBM catastrophe to Hopper-native warp-specialized pipelines: every trick, every number, every tradeoff decoded. Standard attention is not bottlenecked by arithmetic — it is bottlenecked by memory bandwidth.
Static KV cache pre-allocation wastes 60–80% of GPU memory under production traffic patterns. PagedAttention applies OS-style virtual memory paging to the KV cache, enabling 3× higher concurrency on the same hardware.
Static batching blocks an entire batch until its longest sequence finishes. Moving the scheduling boundary from request to iteration frees GPU slots at every decode step, taking utilization from 20–35% to 70–85%.
