Types of prompt

You’ll run one script, measure.py, with different settings. It sends each test to llama3.2:1b at temperature 0, checks the reply against the right answer, and prints a score with the average prompt tokens, answer tokens and time. There are two test sets: 8 messy log lines to rewrite in one fixed format, and 8 small IT maths questions.

The prompt token averages are exact, so yours will match. Replies, scores and times will vary a little. On macOS or Linux, use python3 and nano.

1Set up and run zero-shot

Run this:

mkdir C:\ai-lab\prompt-types
cd C:\ai-lab\prompt-types
notepad measure.py

Paste this whole block into measure.py, then save:

import json
import re
import time
import urllib.request

MODEL = "llama3.2:1b"
TASK = "logs"       # "logs" or "math"
STYLE = "zero"      # logs: "zero" or "few"    math: "zero" or "steps"
EXAMPLES = "good"   # few-shot examples: "good" or "biased"
ROLE = ""           # optional system message

LOGS = [  # (log line, expected host, expected level)
    ("Oct 19 09:14:02 web01 nginx: [error] upstream timed out while reading response header", "web01", "ERROR"),
    ("2026-10-19T09:15:40Z db02 postgres WARNING: checkpoints are occurring too frequently", "db02", "WARN"),
    ("[INFO] 19/10/2026 09:16 backup01 - nightly backup finished in 42 minutes", "backup01", "INFO"),
    ("fs01 kernel: EXT4-fs warning: disk usage at 91%", "fs01", "WARN"),
    ("09:18:11 mail01 postfix/smtpd: connect from unknown[203.0.113.7]", "mail01", "INFO"),
    ("ERR vpn01 tunnel to branch-office went down", "vpn01", "ERROR"),
    ("app03 | level=info | user login ok for user=ravi", "app03", "INFO"),
    ("Oct 19 09:21:55 dns01 named: error (network unreachable) resolving 'example.com'", "dns01", "ERROR"),
]
GOOD = [
    ("Oct 18 22:01:13 proxy01 squid: [warn] cache disk is 85% full", "proxy01 | WARN | cache disk is 85% full"),
    ("2026-10-18T22:05:00Z auth02 sshd: error: too many failed logins for admin", "auth02 | ERROR | too many failed logins for admin"),
    ("[INFO] print01 - queue cleared by helpdesk", "print01 | INFO | queue cleared by helpdesk"),
]
BIASED = [
    ("Oct 18 22:01:13 proxy01 squid: [error] cache disk is full", "proxy01 | ERROR | cache disk is full"),
    ("2026-10-18T22:05:00Z auth02 sshd: error: too many failed logins for admin", "auth02 | ERROR | too many failed logins for admin"),
    ("[ERROR] print01 - print spooler crashed", "print01 | ERROR | print spooler crashed"),
]
MATH = [  # (question, expected number)
    ("A RAID 5 array has 4 disks of 2 TB each. How many TB of usable space does it have?", 6),
    ("A RAID 1 mirror uses 2 disks of 4 TB each. How many TB are usable?", 4),
    ("How many usable host addresses are in a /26 subnet?", 62),
    ("A 30 GB backup is copied at 100 MB per second. How many seconds does it take? Use 1 GB = 1000 MB.", 300),
    ("A rack has 42 units. Each server needs 2 units and one switch needs 1 unit. How many servers fit next to one switch?", 20),
    ("A log file grows 250 MB per day and the disk has 9 GB free. After how many days is the disk full? Use 1 GB = 1000 MB.", 36),
    ("A RAID 6 array has 6 disks of 3 TB each. How many TB of usable space does it have?", 12),
    ("A /24 network is split into /27 subnets. How many subnets do you get?", 8),
]

LOG_RULES = ("Rewrite the log line in this format: host | LEVEL | message\n"
             "LEVEL must be INFO, WARN or ERROR. Reply with that one line only.")


def ask(prompt):
    messages = [{"role": "user", "content": prompt}]
    if ROLE:
        messages.insert(0, {"role": "system", "content": ROLE})
    body = {"model": MODEL, "messages": messages, "stream": False, "options": {"temperature": 0}}
    req = urllib.request.Request("http://localhost:11434/api/chat", data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)


def log_prompt(line):
    text = LOG_RULES + "\n\n"
    if STYLE == "few":
        for example, answer in (GOOD if EXAMPLES == "good" else BIASED):
            text += f"Log line: {example}\nAnswer: {answer}\n\n"
    return text + f"Log line: {line}\nAnswer:"


def math_prompt(question):
    if STYLE == "steps":
        return question + "\nWork it out step by step. Then write the last line as: Answer: <number>"
    return question + "\nReply with the number only."


def get_number(reply):
    found = re.findall(r"-?\d+(?:\.\d+)?", reply)
    return float(found[0]) if found else None


tests = LOGS if TASK == "logs" else MATH
score, p_tokens, a_tokens = 0, 0, 0
start = time.time()
print(f"Task: {TASK} | Style: {STYLE}" + (f" | Examples: {EXAMPLES}" if STYLE == "few" else ""))
for i, test in enumerate(tests, 1):
    if TASK == "logs":
        line, host, level = test
        r = ask(log_prompt(line))
        reply = r["message"]["content"].strip().splitlines()[0].strip()
        parts = [p.strip() for p in reply.split("|")]
        ok = len(parts) == 3 and parts[0] == host and parts[1] == level
        shown = reply
    else:
        question, expected = test
        r = ask(math_prompt(question))
        reply = r["message"]["content"].strip()
        got = get_number(reply)
        ok = got == expected
        shown = f"expected {expected}, got {got if got is None else f'{got:g}'}"
    score += ok
    p_tokens += r["prompt_eval_count"]
    a_tokens += r["eval_count"]
    print(f"{i:>2} {'ok' if ok else 'WRONG':<6} {shown}")
n = len(tests)
print(f"Score: {score}/{n} | Avg prompt tokens: {p_tokens / n:.1f} | "
      f"Avg answer tokens: {a_tokens / n:.1f} | Time: {time.time() - start:.1f} s")

Run this:

python measure.py

You should see something like this (the prompt tokens are exact; the replies and score may differ a little):

Task: logs | Style: zero
 1 ok     web01 | ERROR | upstream timed out while reading response header
 2 WRONG  db02 | WARNING | checkpoints are occurring too frequently
 3 ok     backup01 | INFO | nightly backup finished in 42 minutes
 4 WRONG  fs01 | WARNING | EXT4-fs disk usage at 91%
 5 ok     mail01 | INFO | postfix/smtpd: connect from unknown[203.0.113.7]
 6 WRONG  Here is the rewritten log line: vpn01 | ERROR | tunnel to branch-office went down
 7 ok     app03 | INFO | user login ok for user=ravi
 8 ok     dns01 | ERROR | network unreachable resolving 'example.com'
Score: 5/8 | Avg prompt tokens: 81.4 | Avg answer tokens: 15.1 | Time: 9.4 s

Read the WRONG lines. The model understood every log, but it wrote WARNING instead of WARN twice, and once it chatted before the answer.

A program reading these lines would fail on 3 of 8. That’s a format problem, and format problems are what examples fix.

✅ Checkpoint: You get a score out of 8, and Avg prompt tokens shows 81.4.

2Add examples: few-shot

Change this line:

STYLE = "few"

Run this:

python measure.py

You should see something like this (the prompt tokens are exact; the replies and score may differ a little):

Task: logs | Style: few | Examples: good
 1 ok     web01 | ERROR | upstream timed out while reading response header
 2 ok     db02 | WARN | checkpoints are occurring too frequently
 3 ok     backup01 | INFO | nightly backup finished in 42 minutes
 4 WRONG  kernel | WARN | EXT4-fs disk usage at 91%
 5 ok     mail01 | INFO | connect from unknown[203.0.113.7]
 6 ok     vpn01 | ERROR | tunnel to branch-office went down
 7 ok     app03 | INFO | user login ok for user=ravi
 8 ok     dns01 | ERROR | network unreachable resolving 'example.com'
Score: 7/8 | Avg prompt tokens: 198.4 | Avg answer tokens: 13.5 | Time: 13.8 s

Three examples, and the score jumps. WARN is now spelled the way the examples spell it, and the chatty reply is gone.

The remaining miss is interesting: on fs01 kernel: ... the model took “kernel” as the host. That line starts straight with the host, and none of the three examples do.

One more example with that shape would fix it. The cost: the prompt grew from 81.4 to 198.4 tokens on average.

✅ Checkpoint: Avg prompt tokens shows 198.4, and the WARN lines are now ok.

3Try a role instead

Go back to zero-shot, and give the model a job title instead of examples:

Change these two lines:

STYLE = "zero"
ROLE = "You are a senior Linux administrator who reads server logs all day."

Run this:

python measure.py

You should see something like this (the prompt tokens are exact; the replies and score may differ a little):

Task: logs | Style: zero
 1 ok     web01 | ERROR | upstream timed out while reading response header
 2 WRONG  db02 | WARNING | checkpoints are occurring too frequently
 3 ok     backup01 | INFO | nightly backup finished in 42 minutes
 4 WRONG  fs01 | WARNING | EXT4-fs disk usage at 91%
 5 ok     mail01 | INFO | connect from unknown[203.0.113.7]
 6 ok     vpn01 | ERROR | tunnel to branch-office went down
 7 ok     app03 | INFO | user login ok for user=ravi
 8 ok     dns01 | ERROR | network unreachable resolving 'example.com'
Score: 6/8 | Avg prompt tokens: 94.4 | Avg answer tokens: 13.6 | Time: 9.3 s

The role made the model less chatty, so the vpn01 line is fixed. But it still writes WARNING, because nothing in a job title tells it your format says WARN.

The role added 13 prompt tokens and bought one point. The examples bought two.

✅ Checkpoint: Avg prompt tokens shows 94.4: 13 more than step 1, for the system message.

4Zero-shot on maths

Now switch to the maths questions. Remove the role and change the task:

Change these two lines:

ROLE = ""
TASK = "math"

Run this:

python measure.py

You should see something like this (the prompt tokens are exact; the replies and score may differ a little):

Task: math | Style: zero
 1 WRONG  expected 6, got 8
 2 ok     expected 4, got 4
 3 WRONG  expected 62, got 64
 4 ok     expected 300, got 300
 5 WRONG  expected 20, got 21
 6 ok     expected 36, got 36
 7 WRONG  expected 12, got 18
 8 ok     expected 8, got 8
Score: 4/8 | Avg prompt tokens: 56.1 | Avg answer tokens: 2.0 | Time: 3.1 s

Fast, with about 2 answer tokens each, and half wrong. Look at the mistakes: 8 instead of 6 for RAID 5 (it forgot the parity disk), 64 instead of 62 for a /26 (it forgot the two reserved addresses).

These are reasoning slips, not format problems, so examples of the format won’t help much. Asking for the working might.

✅ Checkpoint: Avg prompt tokens shows 56.1, and Avg answer tokens is around 2.