Initial vectors and positional encoding

You’ll explore 50,000 real word vectors, test the famous word maths, and then build both kinds of position code yourself. You only need Python and numpy from Lesson 4. Use PowerShell as before; on macOS or Linux, use python3.

1Load real word vectors

Run this:

mkdir C:\ai-lab\embeddings
cd C:\ai-lab\embeddings
notepad word_vectors.py

Paste this whole block, then save:

import gzip, os, urllib.request
import numpy as np

URL = ("https://github.com/RaRe-Technologies/gensim-data/releases/download/"
       "glove-wiki-gigaword-50/glove-wiki-gigaword-50.gz")
FILE = "glove-wiki-gigaword-50.gz"

if not os.path.exists(FILE):
    print("Downloading word vectors (66 MB, only the first time)...")
    urllib.request.urlretrieve(URL, FILE)

words, vectors = [], []
with gzip.open(FILE, "rt", encoding="utf-8") as f:
    next(f)                                  # first line is a header
    for line in f:
        parts = line.rstrip().split(" ")
        words.append(parts[0])
        vectors.append(np.array(parts[1:], dtype=np.float32))
        if len(words) == 50000:              # the 50,000 most common words
            break

vecs = np.array(vectors)
unit = vecs / np.linalg.norm(vecs, axis=1, keepdims=True)
index = {w: i for i, w in enumerate(words)}


def similarity(a, b):
    return float(unit[index[a]] @ unit[index[b]])


def nearest(vector, skip=(), n=5):
    scores = unit @ (vector / np.linalg.norm(vector))
    best = [i for i in np.argsort(-scores) if words[i] not in skip]
    return [(words[i], round(float(scores[i]), 2)) for i in best[:n]]


print("Loaded", len(words), "words, each with", vecs.shape[1], "numbers")
print("'server' starts with:", np.round(vecs[index["server"]][:6], 2))
print()
for a, b in [("server", "computer"), ("server", "network"), ("server", "banana")]:
    print(f"similarity({a}, {b}) = {similarity(a, b):.2f}")
print()
print("Closest to 'linux':", nearest(vecs[index["linux"]], skip={"linux"}))
print()
v = vecs[index["king"]] - vecs[index["man"]] + vecs[index["woman"]]
print("king - man + woman =", nearest(v, skip={"king", "man", "woman"}, n=3))

The script downloads GloVe, a free, public-domain set of word vectors from Stanford University, trained on Wikipedia and news text. It keeps the 50,000 most common words.

Run this:

python word_vectors.py

The text version of the same output:

Downloading word vectors (66 MB, only the first time)...
Loaded 50000 words, each with 50 numbers
'server' starts with: [ 0.77 -0.38  1.12  1.77  0.08 -0.89]

similarity(server, computer) = 0.68
similarity(server, network) = 0.56
similarity(server, banana) = 0.22

Closest to 'linux': [('unix', 0.85), ('open-source', 0.79), ('kernel', 0.78), ('os', 0.77), ('desktop', 0.76)]

king - man + woman = [('queen', 0.86), ('daughter', 0.77), ('prince', 0.76)]

Each word is 50 numbers. The numbers on their own mean nothing to you or me: 0.77, −0.38, 1.12. But compare two words and meaning appears.

“Server” is much closer to “computer” (0.68) than to “banana” (0.22). The nearest neighbours of “linux” are unix, open-source, kernel and os. Nobody told the model that; it learned it from how the words are used.

✅ Checkpoint: You see queen at the top of the king − man + woman line, with a score of 0.86.

2Do maths with meaning

That last line was an analogy: man is to king as woman is to …? Try a few more. Add these lines to the end of word_vectors.py:

Add this to the end of the file:

def analogy(a, b, c):
    """a is to b, as c is to ...?"""
    v = vecs[index[b]] - vecs[index[a]] + vecs[index[c]]
    return nearest(v, skip={a, b, c}, n=3)


print()
print("france -> paris, india ->", analogy("france", "paris", "india"))
print("walk -> walked, run ->", analogy("walk", "walked", "run"))
print("microsoft -> windows, apple ->", analogy("microsoft", "windows", "apple"))

Run this:

python word_vectors.py

At the end you’ll now also see:

france -> paris, india -> [('delhi', 0.89), ('bombay', 0.82), ('mumbai', 0.81)]
walk -> walked, run -> [('went', 0.79), ('ran', 0.79), ('drove', 0.79)]
microsoft -> windows, apple -> [('window', 0.73), ('boxes', 0.72), ('kitchen', 0.69)]

France is to Paris as India is to Delhi. Walk is to walked as run is to ran (tied with “went”). The grammar and the geography are both hiding in the numbers.

But the last one fails: Microsoft is to Windows as Apple is to … window, boxes, kitchen? The word-vector list has only one vector for “windows”, and it leans towards the glass kind.

A fixed vector can’t know which meaning you want. That’s exactly the limit of initial vectors, and the reason the transformer layers exist.

✅ Checkpoint: Delhi comes first for France → Paris, India → ?, and the Windows/Apple line gives household words.

3Make a seat number for every position

Run this:

notepad positions.py

Paste this whole block, then save:

import numpy as np

d_model = 16


def position_vector(pos):
    """Sine/cosine position code from the 2017 transformer paper."""
    i = np.arange(d_model // 2)
    angle = pos / (10000 ** (2 * i / d_model))
    v = np.zeros(d_model)
    v[0::2] = np.sin(angle)
    v[1::2] = np.cos(angle)
    return v


def similarity(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))


print("Position 0 :", np.round(position_vector(0)[:6], 2))
print("Position 1 :", np.round(position_vector(1)[:6], 2))
print("Position 2 :", np.round(position_vector(2)[:6], 2))
print()
for other in (11, 15, 30, 100):
    print(f"similarity(position 10, position {other:>3}) = {similarity(position_vector(10), position_vector(other)):.2f}")

Run this:

python positions.py

You should see this:

Position 0 : [0. 1. 0. 1. 0. 1.]
Position 1 : [0.84 0.54 0.31 0.95 0.1  1.  ]
Position 2 : [ 0.91 -0.42  0.59  0.81  0.2   0.98]

similarity(position 10, position  11) = 0.94
similarity(position 10, position  15) = 0.77
similarity(position 10, position  30) = 0.72
similarity(position 10, position 100) = 0.03

Position 0 is a simple pattern of 0s and 1s, and every position after it is different. Now look at the similarities: position 10 looks a lot like position 11 (0.94), less like 15 and 30, and almost nothing like 100 (0.03).

So the code doesn’t just label positions. It also tells the model which positions are close to each other.

✅ Checkpoint: Position 10 is most similar to position 11 and least similar to position 100.

4Rotate instead of adding, the RoPE way

Run this:

notepad rope.py

Paste this whole block, then save:

import numpy as np

rng = np.random.default_rng(3)
head_dim = 8
q = rng.normal(size=head_dim)          # what one token is looking for
k = rng.normal(size=head_dim)          # what another token offers


def rotate(v, pos):
    """RoPE: turn each pair of numbers by an angle that grows with the position."""
    i = np.arange(head_dim // 2)
    angle = pos / (10000 ** (2 * i / head_dim))
    x, y = v[0::2], v[1::2]
    out = np.empty_like(v)
    out[0::2] = x * np.cos(angle) - y * np.sin(angle)
    out[1::2] = x * np.sin(angle) + y * np.cos(angle)
    return out


def score(q_pos, k_pos):
    return rotate(q, q_pos) @ rotate(k, k_pos)


print(f"Word at 5 looks at word at 3:     {score(5, 3):.4f}")
print(f"Word at 105 looks at word at 103: {score(105, 103):.4f}")
print(f"Word at 5 looks at word at 25:    {score(5, 25):.4f}")
print(f"Length before and after rotating: {np.linalg.norm(q):.4f} {np.linalg.norm(rotate(q, 105)):.4f}")

Run this:

python rope.py

You should see this:

Word at 5 looks at word at 3:     11.2093
Word at 105 looks at word at 103: 11.2093
Word at 5 looks at word at 25:    -6.0356
Length before and after rotating: 3.9471 3.9471

Look at the first two lines. Positions 5 and 3 give exactly the same score as positions 105 and 103: 11.2093. RoPE only cares about the distance between the two words, which is 2 in both cases.

A gap of 20 gives a completely different score. And rotating never changes the length of the vector, so the word’s meaning isn’t damaged, only turned.

✅ Checkpoint: The 5→3 and 105→103 scores are identical, and the vector length stays 3.9471.

5Find RoPE in your own Llama model

Run this:

(ollama show llama3.2:1b -v) -match "rope" | Sort-Object

You should see something like this:

    llama.rope.dimension_count       64
    llama.rope.freq_base             500000
    rope_freqs.weight                F32     [32]

That’s the same idea as your rope.py, just bigger. dimension_count 64 means 64 numbers per head are rotated, as 32 pairs. freq_base 500000 sets how slowly the slowest pairs turn; your script used 10,000, like the 2017 paper. A bigger base makes the slow pairs turn more slowly, so positions stay distinguishable across 131,072 tokens. And rope_freqs.weight [32] is the 32-number table from Lesson 4: one stretch factor per pair.

✅ Checkpoint: You found llama.rope.freq_base 500000 and the 32-number rope_freqs table.