Predicting completions from prompts

You’ll start in Python with example scores, so every number is exact and you can see each step. Then you’ll switch to your real model in Ollama.

The Python scripts use only the standard library, so there’s nothing to install. Use PowerShell as before; on macOS or Linux, use python3 and nano, and run the Ollama steps in PowerShell 7 (pwsh), which is free for both.

1Turn scores into chances

Run this:

mkdir C:\ai-lab\sampling
cd C:\ai-lab\sampling
notepad next_word.py

Paste this whole block into next_word.py, then save:

import math

# Example scores (logits) for the word that comes after "The server is".
# A real model gives a score to every one of its 128,256 tokens.
SCORES = {
    "down": 6.0, "running": 5.6, "not": 5.1, "up": 4.6, "currently": 4.2,
    "offline": 3.8, "slow": 3.1, "back": 2.7, "banana": -1.5,
}
OTHERS = 128_247      # the rest of the vocabulary...
OTHER_SCORE = -15.0   # ...all with a very low score
TEMPERATURES = [0.2, 0.8, 1.5]


def chances(temperature):
    top = max(SCORES.values())
    e = {w: math.exp((s - top) / temperature) for w, s in SCORES.items()}
    e["(the 128,247 others)"] = OTHERS * math.exp((OTHER_SCORE - top) / temperature)
    total = sum(e.values())
    return {w: v / total for w, v in e.items()}


tables = [chances(t) for t in TEMPERATURES]
print('Next word after "The server is"\n')
print(f"{'word':<22}" + "".join(f"{'temp ' + str(t):>11}" for t in TEMPERATURES))
for w in tables[0]:
    print(f"{w:<22}" + "".join(f"{p[w] * 100:>10.1f}%" for p in tables))
print(f"{'total':<22}" + "".join(f"{sum(p.values()) * 100:>10.0f}%" for p in tables))

The scores are made up for this lesson, but they’re shaped like a real model’s: a few good guesses near the top, then a huge crowd of tokens with very low scores. The script runs softmax at three temperatures.

Run this:

python next_word.py

You should see this:

Next word after "The server is"

word                     temp 0.2   temp 0.8   temp 1.5
down                        87.1%      43.2%      27.7%
running                     11.8%      26.2%      21.2%
not                          1.0%      14.0%      15.2%
up                           0.1%       7.5%      10.9%
currently                    0.0%       4.5%       8.3%
offline                      0.0%       2.8%       6.4%
slow                         0.0%       1.2%       4.0%
back                         0.0%       0.7%       3.1%
banana                       0.0%       0.0%       0.2%
(the 128,247 others)         0.0%       0.0%       3.0%
total                        100%       100%       100%

Same scores, three very different jars. At 0.2, “down” is almost certain. At 0.8, Ollama’s default, it’s 43%.

At 1.5, the crowd of junk tokens wakes up and takes 3.0%. Look at “banana” too: it never gets a real chance, at any of these temperatures.

✅ Checkpoint: “down” falls from 87.1% to 27.7% as the temperature goes up, and the others row goes from 0.0% to 3.0%.

2Pick words the way Ollama does

Run this:

notepad sample.py

Paste this whole block, then save:

import math
import random

# The same example scores as next_word.py, plus the rest of the vocabulary
SCORES = {
    "down": 6.0, "running": 5.6, "not": 5.1, "up": 4.6, "currently": 4.2,
    "offline": 3.8, "slow": 3.1, "back": 2.7, "banana": -1.5,
}
for n in range(128_247):
    SCORES[f"#{n + 9}"] = -15.0      # junk tokens, named by number

# The settings Ollama uses unless you change them
TEMPERATURE = 0.8
TOP_K = 40        # keep only the 40 best tokens (0 = off)
TOP_P = 0.9       # keep the best tokens until they add up to 90% (1.0 = off)
MIN_P = 0.0       # drop tokens below this share of the best one's chance (0 = off)
SEED = 42         # same seed, same picks (None = different every run)
PICKS = 1000


def softmax(scores, temperature=1.0):
    top = max(scores.values())
    e = {w: math.exp((s - top) / temperature) for w, s in scores.items()}
    total = sum(e.values())
    return {w: v / total for w, v in e.items()}


# 1. Filters, in the same order as llama.cpp (the engine inside Ollama)
tokens = sorted(SCORES, key=SCORES.get, reverse=True)
if TOP_K > 0:
    tokens = tokens[:TOP_K]
p = softmax({t: SCORES[t] for t in tokens})
kept, added = [], 0.0
for t in tokens:
    kept.append(t)
    added += p[t]
    if added >= TOP_P:
        break
kept = [t for t in kept if p[t] >= MIN_P * p[kept[0]]]

# 2. Temperature, then pick
rng = random.Random(SEED)
if TEMPERATURE <= 0:
    final = {t: (1.0 if t == kept[0] else 0.0) for t in kept}   # always the best one
else:
    final = softmax({t: SCORES[t] for t in kept}, TEMPERATURE)
picks = rng.choices(list(final), weights=list(final.values()), k=PICKS)

print(f"temperature={TEMPERATURE} top_k={TOP_K} top_p={TOP_P} min_p={MIN_P} seed={SEED}")
print(f"Tokens still allowed: {len(kept):,} of {len(SCORES):,}\n")
print(f"{'token':<15}{'chance':>8}{'picked':>9}")
junk = [t for t in kept if t.startswith("#")]
for t in kept:
    if not t.startswith("#"):
        print(f"{t:<15}{final[t] * 100:>7.1f}%{picks.count(t):>9}")
if junk:
    junk_set = set(junk)
    print(f"{'junk tokens':<15}{sum(final[t] for t in junk) * 100:>7.1f}%"
          f"{sum(1 for t in picks if t in junk_set):>9}")
print("\nFirst 10 picks:", " ".join(picks[:10]))

This script builds a full 128,256-token vocabulary, runs the filters in llama.cpp’s order, applies the temperature, and then picks 1,000 times so you can count what comes out.

Run this:

python sample.py

The text version of the same output:

temperature=0.8 top_k=40 top_p=0.9 min_p=0.0 seed=42
Tokens still allowed: 5 of 128,256

token            chance   picked
down              45.2%      432
running           27.4%      285
not               14.7%      153
up                 7.9%       75
currently          4.8%       55

First 10 picks: running down down down not running up down down down

Out of 128,256 tokens, only 5 survive. Top-k cut it to 40, then top-p kept adding tokens until they covered 90%. Those 5 cover 92%, and every junk token is gone. “down” was picked 432 times out of 1,000, close to its 45.2% chance.

✅ Checkpoint: 5 of 128,256 tokens are allowed, and “down” is picked 432 times.

Now make it greedy. Change the temperature line:

Change this line:

TEMPERATURE = 0

Run this:

python sample.py

You should see this:

temperature=0 top_k=40 top_p=0.9 min_p=0.0 seed=42
Tokens still allowed: 5 of 128,256

token            chance   picked
down             100.0%     1000
running            0.0%        0
not                0.0%        0
up                 0.0%        0
currently          0.0%        0

First 10 picks: down down down down down down down down down down

All 1,000 picks are “down”. Temperature 0 means: skip the random draw, take the top token. This is what you want for answers that must be the same every time.

Change the temperature back to 0.8, and tighten top-p instead:

Change these two lines:

TEMPERATURE = 0.8
TOP_P = 0.5

Run this:

python sample.py

You should see this:

temperature=0.8 top_k=40 top_p=0.5 min_p=0.0 seed=42
Tokens still allowed: 2 of 128,256

token            chance   picked
down              62.2%      604
running           37.8%      396

First 10 picks: running down down down running running running down down down

With top-p at 0.5, only the two strongest words make it through. The answers are still varied, but only between safe choices.

✅ Checkpoint: At temperature 0 every pick is “down”. At top-p 0.5 only down and running are left.

3See what the seed does

Change TOP_P back to 0.9 and run the script twice. You get exactly the same counts and the same first 10 picks both times, because SEED = 42 starts the random generator at the same point. Now switch the seed off:

Change this line:

SEED = None

Run it twice:

python sample.py
python sample.py

You should see something like this (the counts change on every run):

PS C:\ai-lab\sampling> python sample.py
temperature=0.8 top_k=40 top_p=0.9 min_p=0.0 seed=None
Tokens still allowed: 5 of 128,256

token            chance   picked
down              45.2%      453
running           27.4%      284
not               14.7%      150
up                 7.9%       66
currently          4.8%       47

First 10 picks: not down running running down not running not running down

PS C:\ai-lab\sampling> python sample.py
temperature=0.8 top_k=40 top_p=0.9 min_p=0.0 seed=None
Tokens still allowed: 5 of 128,256

token            chance   picked
down              45.2%      437
running           27.4%      277
not               14.7%      151
up                 7.9%       98
currently          4.8%       37

First 10 picks: not down running running running down down down running not

The chances are identical, but the picks are different every time. That’s your laptop’s chat with Ollama: same model, same prompt, a new random draw each time. Put SEED = 42 back when you’re done.

✅ Checkpoint: With a seed, two runs match exactly. With SEED = None, the counts change but the chances don’t.

4Look inside your real model

Now ask llama3.2:1b for its real top five guesses. This uses Ollama’s logprobs option, which needs version 0.12.11 or newer. Check first:

Run this:

ollama --version

If it’s older, download the latest Windows installer from ollama.com and run it. Your models stay where they are. Then paste this into PowerShell as one block:

Run this:

$body = @{
  model = "llama3.2:1b"
  prompt = "The server is"
  raw = $true
  stream = $false
  logprobs = $true
  top_logprobs = 5
  options = @{ temperature = 0; num_predict = 1 }
} | ConvertTo-Json
$r = Invoke-RestMethod -Uri http://localhost:11434/api/generate -Method Post -Body $body -ContentType "application/json"
$r.logprobs[0].top_logprobs | Select-Object token, @{n="percent"; e={[math]::Round([math]::Exp($_.logprob) * 100, 1)}}

The text version (your tokens and percentages may be a little different):

token        percent
-----        -------
 down          11.4
 not            9.7
 currently      6.3
 running        5.8
 a              4.9

These are the model’s own chances before any temperature or filter. Notice how unsure it is: the favourite is only about 11%. A tiny model with a half sentence and no context has lots of reasonable ways to go on.

The space at the start of each token is real; Llama tokens carry their leading space (Lesson 3). Two settings matter here: raw = $true skips the chat template so the model just continues your text, and num_predict = 1 stops after one token.

Now check what it picked and why it stopped:

$r.response
$r.done_reason

You should see something like this:

 down
length

At temperature 0 it took the top token. And done_reason is length, because it hit the one-token limit, not because it had finished. Remember that word; you’ll need it in the Break It lab.

✅ Checkpoint: You can see five tokens with their percentages, and done_reason says length.

5Make Ollama repeat itself

Ask for a server name three times in a row, at the normal temperature of 0.8, but with a fixed seed:

Run this:

$body = @{
  model = "llama3.2:1b"
  prompt = "Suggest one name for a new backup server. Reply with the name only."
  stream = $false
  options = @{ temperature = 0.8; seed = 42 }
} | ConvertTo-Json
1..3 | ForEach-Object { (Invoke-RestMethod -Uri http://localhost:11434/api/generate -Method Post -Body $body -ContentType "application/json").response }

You should see something like this:

VaultGuard
VaultGuard
VaultGuard

Three identical answers. Now delete ; seed = 42 from the options line, so it reads options = @{ temperature = 0.8 }, and run the whole block again:

You should see something like this:

VaultGuard
BackupHive
SafeHarbor

Three different names. Finally, change it to options = @{ temperature = 0 } and run it once more. You get the same name three times again, this time without any seed, because greedy decoding has no random draw at all.

✅ Checkpoint: With seed 42 or temperature 0 the three answers match. Without either, they change.