Fig. 35. §3 AI and Computing
Transformer
A transformer is a neural network first described in 2017, for translating text.
1Facts
- Shelf
- AI and Computing
- Kind
- Built from a recipe
- Made from
- Encoder–decoder key: ◉ attention · ◒ masked attention · + add & norm · » feed forward · ⠿ embedding · ∿ positional encoding · ╱ linear · ▁▃▇ softmax · N× repeated N times. HELLO WORLD goes in, and the prepared decoder example, given START HOLA, ends with MUNDO. Its words and attention strengths are stored.
- Sound
- Yes
- Licenses
- Own work (made in the page)
2How to play
Tap it to watch a prepared next-word example. Try the Encoder–decoder diagram in the Splash tab.
3Its controls
Table I. Every control on this Splash.
| Control | Kind | Range | Start |
|---|---|---|---|
| Predict the next word | Tap | ||
| Predict | Button | ||
| Diagram | List | 2 choices | Token flow |
| View | List | 2 choices | Poster (2D) |
4About it
A transformer is a neural network first described in 2017, for translating text. It splits text into tokens, and in each layer every token uses attention: it looks at all the others and weighs how much each one matters to it. Language models built this way predict the next token.
Tap it: arcs jump between THE CAT SAT ON, thicker where attention is stronger, one color for each of two heads; the tiles rise through the layers, and MAT drops in. The Encoder–decoder diagram in the Splash tab is the classic design. Its key: an eye is attention; a lidded eye, masked attention (it only looks back); +, add and norm; », feed forward; a vector, embedding; a wave, positional encoding; a slash, linear; bars, softmax; N×, repeated N times. HELLO WORLD goes in, and given START HOLA, the prepared example ends with MUNDO. The words and attention strengths are prepared; this view does not calculate attention or predict text.
5Details
- Splats
- Opens at 200,000 at Mid Detail, 120,002 at Low, and up to 720,000 at Max
6Is it right?
The classic encoder-decoder diagram names the main parts of the 2017 transformer correctly. The words and attention arcs are prepared examples; no attention matrix, translation, or next-word inference is calculated.
Correct.
The classic diagram follows the 2017 encoder-decoder architecture.[1]
The box labels include embedding, position, masked self-attention, encoder attention, add and norm, feed-forward, linear, and softmax.
Simplified.
The splash presents prepared next-word and translation examples.[1]
MAT and the MUNDO tile label are literal outputs, and arc strengths are stored. The How-to and About now say that these are prepared examples.
Simplified.
The first attention picture is schematic rather than a causal decoder calculation.[1]
Some arcs lead to later token positions; there is no QK product, mask, normalization, or learned attention value.
- [1]Attention Is All You Need, Vaswani and coauthors, 2017
- Simplified
- MAT and MUNDO are stored outputs. Arc strengths are fixed, not attention probabilities. The first picture permits links to later tokens, so it should not be read as causal decoder attention.
7Sources and credits
Made in Splashery. Its code is MIT.
