Transformers from first principles: build a tiny GPT, from counting letters to mixture of experts

This post goes with a deep-dive video on the LevelUP I.T. YouTube channel. It builds a transformer language model from nothing: first a model that only counts letters, then each fix that the counting model forces on us, until the result is a working tiny GPT. Once those pieces are in place, a model-card line such as "a sparse-attention mixture-of-experts model with a one-million-token context window, quantized to 4 bits" reads as plain engineering. Every worked example below can be checked with pencil and paper. Published figures (GPT-3, Qwen2.5-7B, Mixtral, DeepSeek-V3, the GPT-2 tokenizer) say where they come from.

The whole model in one picture

This is the map for the rest of the post. Text is cut into tokens. Each token becomes a vector (its embedding plus its position). The vectors pass through a stack of identical transformer blocks, where attention lets tokens exchange information and a feed-forward network processes each token on its own, with residual connections and layer normalization keeping the stack trainable. An output head turns the last vector into one score per vocabulary entry, softmax turns the scores into probabilities, one token is sampled, appended, and the whole thing runs again. The notes on the right are the spec-sheet words; each one is a part of this picture.

What you'll find here

• Why a language model is a loop of next-token guesses, and what it outputs at each step.
• A complete language model built from counting: vocabulary, probability distributions, sampling and temperature.
• The exact reason counting breaks, and why the fix is generalization, not a bigger table.
• Embeddings, cross-entropy loss and gradient descent, with the numbers worked out.
• Attention by hand: queries, keys and values, the scaled dot product, softmax, the weighted sum.
• Causal masking, multi-head attention and three ways to encode position.
• The feed-forward network, residual connections, layer norm, and the transformer block.
• Byte pair encoding, pre-training and post-training.
• Inference and the KV cache, sized exactly from a published model configuration.
• Parameters and memory, quantization worked by hand, long context and sparse attention, and mixture of experts.