Transforming tokens with a transformer

You’ll read the real blueprint of your model with Ollama, check its numbers with two small Python scripts, and then run a tiny transformer of your own. Use PowerShell as in Lesson 3. On macOS or Linux, use python3, and replace the PowerShell filter with grep.

1Read the model’s summary

Run this:

ollama show llama3.2:1b

Near the top, under Model, you’ll see architecture llama, parameters 1.2B, context length 131072 and embedding length 2048. You met this screen in Lesson 2. Now you’ll go one level deeper.

✅ Checkpoint: You can see embedding length 2048. That’s the width of the grid from the Engineer’s View.

2Read the architecture numbers

The -v flag prints everything Ollama knows about the model. It’s long, so filter it to the lines that start with llama.:

Run this:

(ollama show llama3.2:1b -v) -match "llama\." | Sort-Object

Among the lines you’ll see (there are a few more):

    llama.attention.head_count       32
    llama.attention.head_count_kv    8
    llama.block_count                16
    llama.context_length             131072
    llama.embedding_length           2048
    llama.feed_forward_length        8192
    llama.vocab_size                 128256

Match each line with the table in the Engineer’s View. 16 layers, 2,048 wide, 32 attention heads sharing 8 key/value heads, and a feed-forward step 8,192 wide. That’s the whole design of your model in seven numbers.

✅ Checkpoint: You found llama.block_count 16 and llama.embedding_length 2048.

3Look inside one layer

The same command also lists every table of numbers in the file, called tensors. Show just the ones in layer 0:

Run this:

(ollama show llama3.2:1b -v) -match "blk\.0\." | Sort-Object

You should see something like this:

    blk.0.attn_k.weight         Q8_0    [2048 512]
    blk.0.attn_norm.weight      F32     [2048]
    blk.0.attn_output.weight    Q8_0    [2048 2048]
    blk.0.attn_q.weight         Q8_0    [2048 2048]
    blk.0.attn_v.weight         Q8_0    [2048 512]
    blk.0.ffn_down.weight       Q8_0    [8192 2048]
    blk.0.ffn_gate.weight       Q8_0    [2048 8192]
    blk.0.ffn_norm.weight       F32     [2048]
    blk.0.ffn_up.weight         Q8_0    [2048 8192]

Nine tensors. attn_q, attn_k, attn_v and attn_output are attention. ffn_gate, ffn_up and ffn_down are feed-forward. The two norm tensors are the normalisation steps. Q8_0 is the 8-bit quantization from Lesson 2; the small norm tables are kept at full precision (F32).

Notice attn_k and attn_v are only 512 wide, not 2,048. That’s the 8 key/value heads × 64 numbers each.

It saves memory, and you’ll see why that matters in step 5. Now count the layers:

Run this:

((ollama show llama3.2:1b -v) -match "blk\.").Count

You should see this:

144

✅ Checkpoint: 144 tensors = 16 layers × 9 tensors each. Every layer has exactly the same shape.

4Count the 1.2 billion parameters yourself

Run this:

mkdir C:\ai-lab\transformer
cd C:\ai-lab\transformer
notepad params.py

Paste this whole block, then save:

# Numbers copied from: ollama show llama3.2:1b -v
vocab = 128256        # llama.vocab_size
width = 2048          # llama.embedding_length
layers = 16           # llama.block_count
heads = 32            # llama.attention.head_count
kv_heads = 8          # llama.attention.head_count_kv
ffn = 8192            # llama.feed_forward_length
head_dim = width // heads

embedding = vocab * width
attention = width * width * 2 + width * (kv_heads * head_dim) * 2   # q, o and k, v
feed_forward = width * ffn * 3                                      # gate, up, down
norms = width * 2

per_layer = attention + feed_forward + norms
total = embedding + layers * per_layer + width                      # + final norm

print(f"Embedding table:    {embedding:>13,}")
print(f"One layer:          {per_layer:>13,}")
print(f"All {layers} layers:      {layers * per_layer:>13,}")
print(f"Total parameters:   {total:>13,}")
print(f"Share in layers:    {layers * per_layer / total:.0%}")

Run this:

python params.py

The text version of the same output:

Embedding table:      262,668,288
One layer:             60,821,504
All 16 layers:        973,144,064
Total parameters:   1,235,814,400
Share in layers:    79%

1,235,814,400. That’s the “1.2B” that Ollama shows.

If you look for general.parameter_count in the -v output, you’ll find 1.235814432e+09, which is 32 more. Those 32 numbers are a small table used for word positions, which you’ll meet in Lesson 5.

Two things to notice. One layer is about 61 million parameters, and 79% of the model lives in the 16 layers. The feed-forward part is the biggest piece of each layer: 50 of its 61 million.

✅ Checkpoint: Your total is 1,235,814,400, worked out from seven numbers.

5Work out how much memory the context window needs

Remember the KV cache from the How it works section. For every token in the context, every layer keeps one key and one value per key/value head. Every number in this calculation comes from step 2 (the 64 is 2,048 ÷ 32 heads):

Run this:

notepad kv_cache.py

Paste this whole block, then save:

layers, kv_heads, head_dim = 16, 8, 64      # from ollama show llama3.2:1b -v
bytes_per_number = 2                        # 16-bit cache

per_token = 2 * layers * kv_heads * head_dim * bytes_per_number   # 2 = keys + values
print(f"Memory per token: {per_token:,} bytes ({per_token / 1024:.0f} KB)")

for context in (2048, 4096, 8192, 32768, 131072):
    print(f"Context {context:>7,} tokens -> {per_token * context / 1024**3:5.2f} GB")

Run this:

python kv_cache.py

You should see this:

Memory per token: 32,768 bytes (32 KB)
Context   2,048 tokens ->  0.06 GB
Context   4,096 tokens ->  0.12 GB
Context   8,192 tokens ->  0.25 GB
Context  32,768 tokens ->  1.00 GB
Context 131,072 tokens ->  4.00 GB

32 KB for every single token. A 4,096-token context needs about 0.12 GB, small next to the 1.3 GB model. The model’s full 131,072-token context needs 4 GB, three times the size of the model itself.

Lesson 2 warned you that a maximum context window makes RAM use jump. Now you know exactly why, and how to calculate it for any model.

✅ Checkpoint: You can explain why 131,072 tokens of context needs 4 GB of RAM on top of the model.

6Run a tiny transformer you can read

Your Llama model is too big to look inside. So here is a complete transformer in about 60 lines of Python, with the same parts: embedding, attention with a causal mask, feed-forward, residual connections, normalisation and an output layer.

It’s tiny: 10 words, 16 numbers wide, 2 layers. First install numpy:

Run this:

python -m pip install numpy

Run this:

notepad tiny_transformer.py

Paste this whole block, then save:

import numpy as np

rng = np.random.default_rng(7)

vocab = ["<start>", "the", "server", "is", "down", "up", "slow", "disk", "full", "."]
d_model, n_heads, n_layers, d_ff = 16, 4, 2, 64
head_dim = d_model // n_heads


def weights(rows, cols):
    return rng.normal(0, 1 / np.sqrt(rows), (rows, cols))


def rms_norm(x):
    return x / np.sqrt((x ** 2).mean(axis=-1, keepdims=True) + 1e-5)


def softmax(x):
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)


def attention(x, w):
    t = x.shape[0]
    q = (x @ w["q"]).reshape(t, n_heads, head_dim).transpose(1, 0, 2)
    k = (x @ w["k"]).reshape(t, n_heads, head_dim).transpose(1, 0, 2)
    v = (x @ w["v"]).reshape(t, n_heads, head_dim).transpose(1, 0, 2)
    scores = q @ k.transpose(0, 2, 1) / np.sqrt(head_dim)
    future = np.triu(np.ones((t, t), dtype=bool), 1)
    scores[:, future] = -np.inf          # a token may not look ahead
    mixed = softmax(scores) @ v
    return mixed.transpose(1, 0, 2).reshape(t, d_model) @ w["o"]


def feed_forward(x, w):
    return np.maximum(0, x @ w["up"]) @ w["down"]


def block(x, w):
    x = x + attention(rms_norm(x), w)    # mix information between tokens
    x = x + feed_forward(rms_norm(x), w) # think about each token on its own
    return x


embed = rng.normal(0, 1, (len(vocab), d_model))
layers = [{"q": weights(d_model, d_model), "k": weights(d_model, d_model),
           "v": weights(d_model, d_model), "o": weights(d_model, d_model),
           "up": weights(d_model, d_ff), "down": weights(d_ff, d_model)}
          for _ in range(n_layers)]

words = ["the", "server", "is"]
ids = [vocab.index(w) for w in words]
print("Tokens:", words, "-> IDs", ids)

x = embed[ids]
print("After embedding:", x.shape)
for n, w in enumerate(layers, start=1):
    x = block(x, w)
    print(f"After layer {n}:  ", x.shape)

logits = rms_norm(x) @ embed.T
print("Logits:         ", logits.shape)

probs = softmax(logits[-1])
top = np.argsort(probs)[::-1][:3]
print("Next-token guess:", " | ".join(f"{vocab[i]} {probs[i]:.1%}" for i in top))

Run this:

python tiny_transformer.py

You should see this:

Tokens: ['the', 'server', 'is'] -> IDs [1, 2, 3]
After embedding: (3, 16)
After layer 1:   (3, 16)
After layer 2:   (3, 16)
Logits:          (3, 10)
Next-token guess: is 99.7% | . 0.1% | down 0.1%

Look at the shapes. 3 tokens go in as a 3 × 16 grid. Every layer gives back a 3 × 16 grid.

Only the last step changes the shape: 10 scores per token, one for each word in the vocabulary. Llama has the same structure, just bigger: 2,048 instead of 16, 16 layers instead of 2, and 128,256 words instead of 10. Llama also adds position information and a richer feed-forward step, which this toy leaves out.

And the guess? The model is 99.7% sure the next word is “is” again. That’s not a bug.

The weights are random, so the layers’ notes mean nothing, and what’s left in the grid is mostly the last word it read. Training, from Lesson 2, is what changes these random numbers into useful ones. The architecture is only the empty building.

✅ Checkpoint: You see the same shape, (3, 16), after every layer, and a guess that only repeats the last word.