Tokenization

You’ll use tiktoken, a free, open-source tokenizer library from OpenAI, to see the pieces and IDs for any text. Then you’ll use Ollama to count tokens the way your Llama model counts them.

Everything runs on your laptop. The steps use Windows PowerShell; on macOS or Linux, use python3 where you see python.

1Install Python

Download Python 3 from python.org and run the installer. On the first screen, tick Add python.exe to PATH before you click Install. Then open a new PowerShell window.

Run this:

python --version

You should see this:

Python 3.13.1

✅ Checkpoint: You see Python 3.10 or newer. Any 3.x version from the last few years works.

2Install tiktoken

Run this:

python -m pip install tiktoken

The last line should look like this (versions may differ):

Successfully installed ... tiktoken-0.14.0

✅ Checkpoint: No red error text, and the last line says Successfully installed.

3See the tokens in a sentence

Make a folder for this lesson and open Notepad in it:

Run this:

mkdir C:\ai-lab\tokens
cd C:\ai-lab\tokens
notepad tokens.py

Click Yes when Notepad asks to create the file. Paste this, save, and close Notepad:

Paste this whole block:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
text = input("Type a sentence: ")
ids = enc.encode(text)
print("Characters:", len(text), "| Words:", len(text.split()), "| Tokens:", len(ids))
for i in ids:
    print(f"{i:>7}  {enc.decode([i])!r}")

Run it and type the sentence when asked:

Run this:

python tokens.py

The text version of the same output:

Type a sentence: The server rebooted at 2 AM.
Characters: 28 | Words: 6 | Tokens: 9
    976  'The'
   6017  ' server'
  67186  ' reboot'
    295  'ed'
    540  ' at'
    220  ' '
     17  '2'
   7219  ' AM'
     13  '.'

The first run takes a few seconds longer because tiktoken downloads its vocabulary file (about 3.6 MB) once and caches it. Every run after that works offline.

✅ Checkpoint: 6 words became 9 tokens. rebooted was split into reboot and ed, and the space before the 2 is a token of its own.

4Try words that split in surprising ways

Run python tokens.py again for each line below. These are the results you should get:

Notice that “unbelievable” doesn’t split the way you’d split it at school. The pieces come from statistics, not grammar.

An IP address costs 9 tokens because every dot is a token of its own. And the last line shows that case, and even a leading space, change the ID completely.

✅ Checkpoint: You can explain why server (5282) and server (6017) have different IDs.

5Compare languages and tokenizers

Run this:

notepad langs.py

Paste this whole block, then save:

import tiktoken

new = tiktoken.get_encoding("o200k_base")   # newer tokenizer (GPT-4o family)
old = tiktoken.get_encoding("cl100k_base")  # older tokenizer (GPT-4, GPT-3.5)

samples = {
    "English": "Hello, how are you?",
    "Hindi": "नमस्ते, आप कैसे हैं?",
    "Telugu": "హలో, మీరు ఎలా ఉన్నారు?",
}
for lang, text in samples.items():
    print(f"{lang:<8} characters={len(text):>3}   new={len(new.encode(text)):>3}   old={len(old.encode(text)):>3}")

Run this:

python langs.py

You should see this:

English  characters= 19   new=  6   old=  6
Hindi    characters= 20   new=  9   old= 20
Telugu   characters= 22   new=  8   old= 36

The older tokenizer needs 36 tokens for the Telugu greeting; the newer one needs 8. Same text, 4.5 times the cost, just from changing the tokenizer. When someone quotes you “tokens per word”, always ask: with which tokenizer?

✅ Checkpoint: English costs about the same in both. Hindi and Telugu cost far less with the newer tokenizer.

6Count tokens in real files

At work you’ll rarely tokenize one sentence. You’ll want to know whether a whole document or log will fit. Create three files in the same folder:

Run this:

notepad count_file.py

Paste this into count_file.py:

import sys
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
text = open(sys.argv[1], encoding="utf-8").read()
tokens = len(enc.encode(text))
words = len(text.split())
print(f"{sys.argv[1]}: {words} words, {tokens} tokens, {tokens / words:.2f} tokens per word")

Run this:

notepad incident.txt

Paste this into incident.txt (a short incident report):

At 02:10 the monitoring team noticed that the payment service was slow. Users were waiting more than ten seconds for a page to load. The on-call engineer checked the dashboards and saw that one database server was using all of its memory. She restarted the reporting job that was running on the same server, and the memory came back down within a few minutes. Payments were normal again by 02:35. The root cause was a new report that loaded a full year of data into memory instead of reading it in small parts. The team has asked the developers to fix the report and has moved it to a separate server so it cannot affect payments again.

Run this:

notepad app.log

Paste this into app.log (five lines from the same incident):

2026-10-18T02:10:04.118Z ERROR [payment-api-7f9c] db-pool timeout after 5000ms host=10.20.4.17:5432 req_id=8c1e2f0a
2026-10-18T02:10:04.392Z WARN  [payment-api-7f9c] retry 1/3 for req_id=8c1e2f0a latency_ms=10412
2026-10-18T02:10:05.006Z ERROR [payment-api-2b11] db-pool timeout after 5000ms host=10.20.4.17:5432 req_id=91d07c3b
2026-10-18T02:10:05.771Z INFO  [report-job-01] loaded rows=48211934 mem_used=31.6GiB
2026-10-18T02:10:06.240Z ERROR [kernel] Out of memory: Killed process 22841 (postgres) total-vm:33554432kB

Run this:

python count_file.py incident.txt
python count_file.py app.log

You should see this:

incident.txt: 117 words, 132 tokens, 1.13 tokens per word
app.log: 43 words, 242 tokens, 5.63 tokens per word

Plain English costs a little over 1 token per word. The log costs more than 5 tokens per word, because timestamps, IDs, IP addresses and numbers break into many small pieces. If you plan a log-analysis tool using word counts, you’ll be wrong by a factor of five.

✅ Checkpoint: Your log file uses roughly five times more tokens per word than the English report.

7Count tokens the way your Llama model does

tiktoken shows OpenAI’s tokenizers. Your llama3.2:1b model has its own. Ollama can tell you how many tokens it read.

Run these two commands. Copy the Telugu text from this page:

Run this:

ollama run llama3.2:1b --verbose "Hello, how are you?"
ollama run llama3.2:1b --verbose "హలో, మీరు ఎలా ఉన్నారు?"

Look at the prompt eval count line. You might expect 6 tokens for the English greeting, but Ollama reports 31.

The extra 25 are the hidden chat template: the special tokens, plus a short system header that the Llama 3.2 template always adds. You pay for those on every single message.

Now subtract: 61 minus 31 is 30. The Telugu greeting costs 30 more tokens than the English one, so 36 tokens in total with Llama’s tokenizer. That matches the chart in the section above.

✅ Checkpoint: You found the hidden template cost (25 tokens) and the extra cost of Telugu (about 30 tokens) on your own machine.