▥ENGINE ROOM
Illustrative simulation
Under the hood / 001

A reply, in motion.

Three tricks that make language models fast.
Follow the memory, watch the guesses, meet the experts.

Preset prompt 01 / 04
Why does the sky turn orange at sunset?
Tokens generated
24 / 24

One step at a time

KV cache
12 MB

Stored keys + values

Draft acceptance
—

Verified by the target model

Expert routing
2 of 8 / token

Only selected experts activate

01Token ledger

● Run complete
#TOKENDRAFTKV / MBEXPERTS
The completed simulated response appears here.

02Memory that grows

━ KV cache
CACHE SIZE / MB
KV cache grows by 0.5 MB for every generated token.
0TOKEN INDEX →24

Remember, don’t recompute. Each new token adds keys and values to memory. Earlier tokens keep their cached attention state.

03A team of specialists

TOP-2 ROUTING

Sparse work, shared load. Two experts handle each token. Brighter cells mean more assignments in this run.

↗ KV cache

Keep the attention keys and values from previous tokens. The cache grows with the sequence, saving repeated computation.

⇢ Speculative decoding

A small draft model proposes tokens. The target model verifies them: green guesses survive; orange guesses are replaced with the target token shown in the ledger.

⁙ Mixture of experts

A router selects two of eight expert networks per token. The full model has many specialists, but each token uses only a few.