What actually happens when you press Send?
The model is a deterministic numerical program up to the sampling step. The surrounding chat service supplies conversation state, turns text into tensors, schedules GPU work, and streams the sampled tokens back.
A response is not computed “all at once.”
The prompt is processed in a broad parallel phase called prefill. After that, the model enters a narrow loop: predict one token, append it, update cached state, and predict the next. Within one sequence, token n+1 depends on token n; across many independent sequences, the same iteration can run together.
Request lifecycle
The service serializes role-tagged messages, policies, tool results, and the newest user message into one model input.
All input tokens pass through the transformer. The service builds a per-layer cache of keys and values.
The final hidden vector is projected to one score per vocabulary token: the logits.
A sampler chooses a token. That token becomes the input for the next decode iteration.
Generation ends on an end token, a length limit, a stop sequence, a tool call, or a server-side policy.
Where does a follow-up question go?
The model itself does not magically retain a private conversation object. Conceptually, the service replays the relevant transcript on each turn. Providers may reuse cached prefixes so they do not recompute identical history, but the semantics are the same: prior messages are part of the next input.
“Memory” has several meanings. Chat history is application state. The context window is the token sequence visible to the current request. The KV cache is temporary numerical state used to accelerate that request. Long-term user memory, when present, is a separate retrieval feature that inserts selected facts into a later prompt.
Tokens, role wrappers, embeddings, and the context window
Transformers operate on fixed-width vectors, not strings. A tokenizer maps text to IDs; an embedding table maps each ID to a learned vector; position information lets the model distinguish order.
A toy tokenizer
Real tokenizers are model-specific. This toy view only demonstrates that tokens are often subwords—not one token per word.
Embedding lookup ≈ array indexing
Position and context
The same token needs a different representation at position 5 than at position 500. Modern models commonly encode relative position through mechanisms such as rotary position embeddings.
The context window is the maximum token span the model can use in one request. Conversation history, retrieved documents, tool outputs, and generated tokens all consume it. When the window fills, the application must truncate, summarize, or otherwise manage old context.
A transformer block is two big ideas with skip connections
A decoder-only language model stacks many near-identical blocks. Each block first lets tokens read relevant earlier tokens, then applies a learned nonlinear transformation to each token position.
A vector per token carries the model’s evolving representation. Normalization stabilizes the scale before a sublayer.
Each position computes weighted reads from itself and earlier positions. Future positions are masked.
Large matrix multiplies transform each token independently. In an MoE model, a router selects a few expert FFNs.
Attention in one equation
The current token’s representation is projected into one or more query vectors.
Every visible token advertises features that queries can match.
The normalized similarity scores mix the value vectors into a contextual read.
The FFN is where batching shines
A typical FFN expands a vector to a larger intermediate width, applies a nonlinearity, then projects back. Those learned matrices are enormous and reused for every token.
For one token, the GPU reads a huge matrix to transform one row. For many tokens, it can reuse that matrix across many rows in a single matrix-matrix operation. That reuse is central to the article’s batching argument.
Explore a toy causal attention map
Click a query token. Its heatmap row is highlighted, and the bars show that same row in detail. Darker cells and longer bars mean a larger attention weight. The values are hand-crafted for teaching, not from a trained model.
—
Select a token to inspect which earlier positions it reads.
Why “multi-head”? The model computes several attention patterns in parallel. Different heads can specialize in different relationships—nearby syntax, long-range references, delimiters, code structure, or other learned features—then concatenate their results.
Prefill once; decode one token at a time
The first response token comes from the final prompt position. Every later token requires another pass through all transformer layers, but cached attention state means prior tokens do not need to be fully recomputed.
Step through a toy generation
Temperature reshapes the candidate distribution. “Next token” chooses the highest-probability candidate so the demo stays reproducible.
Compare the two phases
Move the sliders. The units are conceptual: the point is parallel width versus sequential depth.
Prefill
parallel over prompt positionsFor each layer, many prompt token positions can be processed together under a causal mask. Large prompts form large matrix operations and often use GPU compute efficiently.
Decode
sequential within one sequenceOnly the newest token position is processed for each active sequence. Output step 12 cannot begin until step 11 has been selected.
Three latency metrics matter. Time to first token (TTFT) includes queueing and prefill. Time per output token (TPOT) describes the decode cadence. End-to-end latency combines both. A model can feel slow to start but fast once streaming if prefill or admission queueing dominates.
The KV cache turns recomputation into memory
At every transformer layer, prior tokens already produced attention keys and values. Saving them lets the next token attend to history without rerunning all earlier tokens through that layer.
Without a cache
To generate token 1, process the prompt. To generate token 2, process the prompt plus token 1 again. Then process everything again for token 3. The token-wise work repeats in a growing triangle, while full attention recomputation can grow even faster.
With a KV cache
Process the prompt once. For each new token, compute only its new query, key, value, and FFN activations; read prior keys and values from memory; append the new K/V entries.
KV memory calculator
A simplified estimator for a grouped-query attention model. Real architectures and compression schemes vary.
Naive contiguous reservationred = unused
Paged, on-demand allocationgreen = allocated pages
PagedAttention borrows from virtual memory. The logical KV sequence is split into fixed-size blocks that can live in non-contiguous physical memory. This reduces fragmentation, enables on-demand growth, and can share blocks for common prefixes. It does not make the underlying K/V information free; it manages it more efficiently.
Batch independent decode steps so the GPU sees a matrix, not a trickle
One user can only offer one next-token position at a time. A busy inference service can collect the newest token positions from many users, stack them as rows, and run the same model weights over all rows together.
From matrix-vector-ish work to a large GEMM
Move the batch size. The visualization caps visible rows, but the dimensions and metrics use the selected value.
same weights for every row
Think “deliver a huge library once, serve many readers.” Model weights are large and mostly constant. With batch size 1, expensive memory traffic and kernel launch overhead serve one token row. With a larger batch, the same loaded weights participate in many multiply-accumulates before being evicted.
The throughput–latency tradeoff
This is a deliberately simple queueing toy, not a hardware benchmark. It makes the direction of the tradeoff visible.
Static batching versus continuous batching
Each column is one scheduler iteration. P = prefill, Dn = the nth generated token, · = an occupied but idle slot.
What modern schedulers improve
Iteration-level / in-flight batching revisits the active set after each decode step. Finished requests leave; newly arrived requests enter; prompt chunks and decode tokens can sometimes share a token budget. This avoids waiting for an entire old batch to finish.
What they cannot remove
Each sequence still has a dependency chain. KV cache still consumes memory. Large weight matrices still need to be read. Communication still happens in distributed models. A scheduler can pack work more cleverly, but it cannot abolish the underlying physics.
Important refinement to the article’s simplification: modern systems do not universally require every sequence to have the exact same length. They can pack variable-length sequences, use paged/ragged attention kernels, or batch the parameter-heavy linear operations while handling attention selectively. The core tradeoff remains: more concurrent token rows generally improve throughput, while queueing, memory, and fairness constraints shape latency.
Giant models are distributed programs
If the weights do not fit on one accelerator, every token step becomes a coordinated computation across devices. The model can be split within layers, across layers, or across experts—each introducing communication and synchronization.
Three common parallelism axes
They are frequently combined in a 2D or 3D deployment.
Split one layer’s matrices across GPUs
Each device computes a shard of the same matrix operation. Partial results must be combined, often with all-reduce or all-gather collectives.
Columns or rows of attention / FFN weights
Runs concurrently on the same token batch
Produces a partial activation
Synchronizes before the next operation
Split consecutive layers into stages
A microbatch flows from early layers to later layers. Multiple microbatches overlap so each stage has something to do.
Embedding and early representation building
Receives activations from stage 1
Processes a later microbatch concurrently
Final normalization and output head
Place different experts on different GPUs
The router dispatches token rows to the devices that own their selected experts, then gathers the outputs. This creates all-to-all communication.
Receives tokens routed to this expert group
Grouped GEMMs process local token buckets
Load balance determines utilization
Outputs return to original token order
Pipeline warmup, steady state, and drain
More microbatches amortize the empty diagonals, but require more concurrent work and state.
A “pipeline bubble” is idle capacity caused by dependencies. Later stages wait during warmup; earlier stages wait during drain. If too few microbatches are in flight, the empty regions dominate. Larger waves or smarter schedules improve utilization, but may raise queueing, memory use, and latency.
MoE saves active compute, but creates a routing-and-batching problem
Instead of applying the same dense FFN to every token, an MoE layer contains many expert FFNs and a router picks a small subset for each token. The total parameter count can be huge while active parameters per token stay much smaller.
Dense FFN
Easy to batch: every token row uses the same matrices. One large batch becomes one or a few large GEMMs.
MoE FFN
Harder to batch: the global batch fractures into many smaller per-expert batches, plus dispatch and gather communication.
Route a toy batch across 16 experts
Each token selects two experts. The first 12 tokens are drawn; the histogram includes the full selected batch.
Sparse compute, enormous distributed state
Why high concurrency helps
With only one or a few token rows, most experts receive no work and selected experts receive tiny matrices. With a large global batch, tokens can be grouped into larger per-expert GEMMs, communication is amortized, and expert devices are more consistently occupied.
Why the architecture also contains countermeasures
DeepSeek-V3 uses load-balancing strategies and Multi-head Latent Attention (MLA), which compresses the cached attention state. A smaller KV footprint allows more active sequences, which in turn makes larger batches easier to sustain.
The article’s thesis, stated precisely: a sparse giant model can have attractive cost per token at high aggregate load while being inefficient for one personal sequence, even on enough hardware to hold it. The issue is not that a single request is impossible; it is that weight movement, communication, pipeline stages, and fragmented expert work are poorly amortized.
Efficiency tricks target different bottlenecks
There is no single “make inference fast” switch. A production stack manages compute, memory bandwidth, KV capacity, fragmentation, communication, scheduling, and tail-latency objectives simultaneously.
Continuous batching
Admit and retire requests at iteration boundaries instead of freezing a batch until all sequences finish.
Paged KV cache
Allocate fixed-size KV blocks on demand; reduce fragmentation and enable prefix sharing.
Chunked prefill
Split long prompts into chunks so a huge prefill does not stall ongoing decode work.
Prefix caching
Reuse KV state for identical prompt prefixes, such as shared system prompts or documents.
Quantization
Store weights and sometimes KV data in fewer bits, reducing memory footprint and bandwidth.
Fusion & efficient attention
Combine operations and minimize intermediate memory traffic; tile attention to use fast on-chip memory.
Speculative decoding
A smaller draft process proposes several tokens; the large model verifies them in parallel.
Overlap communication
Run compute while collectives or expert dispatch happen, hiding part of network latency.
Compressed KV / GQA / MLA
Reduce the key-value state stored per token so more sequences fit in memory.
Optimize the metric you actually care about
Maximum raw throughput can be a bad product setting if tail latency becomes unacceptable. Production schedulers usually enforce service-level objectives, fairness, and admission control rather than simply chasing the largest possible batch.
From transformer math to the article’s economics
The serving behavior follows from a small set of dependencies: one sequence decodes serially, model weights are huge and reusable, KV state grows with context, and giant sparse models distribute work across many devices.
Offers one new token row per decode iteration.
Reading and coordinating them for one row is inefficient.
Supply independent rows that can share the same weight pass.
Produce bigger GEMMs, fuller experts, and fewer pipeline bubbles.
Higher fleet throughput, but queueing and memory can raise latency.
The mental model to keep
A chat turn is a token sequence. Follow-ups work because relevant previous messages are included again or their prefix computation is reused.
Prefill is wide; decode is deep. Prompt positions can run in parallel, but generated tokens form a dependency chain.
The KV cache is a time–memory trade. It avoids recomputing prior attention state while consuming memory proportional to active context.
Batching happens across independent sequences. It turns many skinny operations into larger matrix operations and amortizes weight movement.
MoE fractures the batch. Tokens must be regrouped by expert, so a large global batch is needed to create healthy per-expert batches.
A giant model is also a distributed system. Placement, collectives, queues, memory managers, and tail-latency policies matter as much as the transformer equations.
Nuances worth remembering
Can a single user’s prompt tokens be batched?
Yes during prefill: all prompt positions can be processed in parallel under a causal mask. The article’s batching focus is the decode phase across different requests, because one sequence only produces one new position at a time.
Does batching make one token literally free?
No. The batch takes more computation and can take longer than batch size 1. The benefit is that total time grows much more slowly than the number of rows over a useful range, so aggregate throughput and cost per token improve.
Must batched sequences have equal lengths?
Not universally. Padding is one option, but modern serving engines also use packed inputs, per-sequence block tables, ragged kernels, and selective batching. Different lengths still complicate memory access and scheduling.
Why can local latency still be good with enough hardware?
Overprovisioning can minimize queueing and keep a single sequence responsive. The scale disadvantage is mainly utilization and cost efficiency: the deployment pays for many weights, devices, and communication links while serving very few token rows.
Where does model quality enter this story?
Training determines the learned weights and architecture. The inference system described here executes those weights efficiently. Quantization, caching, routing, and decoding optimizations can affect quality or exact outputs, but they do not create the underlying capabilities from scratch.
A compact answer to “why is DeepSeek cheap at scale but expensive for personal use?”
DeepSeek-V3 activates only part of its enormous parameter set for each token, which controls arithmetic per token. But all experts still need to live across a large distributed deployment, token rows must be routed and communicated, and a small request stream creates tiny per-expert batches and poor pipeline utilization. At hyperscale, many concurrent requests fill those expert buckets and pipeline stages, amortizing the infrastructure over far more tokens.
Primary references behind the explainer
The artifact is designed to stand on its own. These links provide the article’s framing and deeper technical details.
The article this explainer responds to: batching, throughput versus latency, MoE utilization, and pipeline bubbles.
The original transformer paper: scaled dot-product attention, multi-head attention, causal masking, and position-wise FFNs.
An influential serving-system design that changes the active batch every model iteration and selectively batches operations.
Virtual-memory-style KV block management that reduces fragmentation and enables larger effective batches.
Chunked prefill and scheduling techniques for the throughput–latency tradeoff and pipeline bubbles.
Current implementation-oriented documentation for packed requests, token budgets, KV cache choices, and iteration scheduling.
Architecture and deployment details: 671B total parameters, 37B active per token, MLA, DeepSeekMoE, expert routing, and load balancing.