Apache Kafka on Linux with KRaft: a three-node cluster

Step 1 runs on the laptop and in the VirtualBox console, the rest on kafka01 as student over SSH (from ubu01 or the laptop on port 2231). kafka01 needs 4 GB of RAM, so on an 8 GB laptop keep only kafka01 and one other VM running.

1Build kafka01 with a data disk

On the laptop (PowerShell or a terminal; adjust the paths on macOS and Linux), create the VM the same way as rocky01 in Lesson 2, plus a 10 GB second disk:

VBoxManage createvm --name=kafka01 --ostype=RedHat_64 --basefolder="C:\Lab\Linux" --register
VBoxManage modifyvm kafka01 --memory=4096 --cpus=2 --firmware=efi --graphicscontroller=vmsvga --vram=16 --audio-enabled=off --nic1=natnetwork --nat-network1=LinuxLab --nic-type1=82540EM
VBoxManage createmedium disk --filename="C:\Lab\Linux\kafka01\kafka01.vdi" --size=20480 --format=VDI
VBoxManage createmedium disk --filename="C:\Lab\Linux\kafka01\kafka01-data.vdi" --size=10240 --format=VDI
VBoxManage storagectl kafka01 --name=SATA --add=sata --controller=IntelAhci --portcount=4
VBoxManage storageattach kafka01 --storagectl=SATA --port=0 --device=0 --type=hdd --medium="C:\Lab\Linux\kafka01\kafka01.vdi"
VBoxManage storageattach kafka01 --storagectl=SATA --port=1 --device=0 --type=hdd --medium="C:\Lab\Linux\kafka01\kafka01-data.vdi"
VBoxManage storagectl kafka01 --name=IDE --add=ide
VBoxManage storageattach kafka01 --storagectl=IDE --port=1 --device=0 --type=dvddrive --medium="C:\Lab\Linux\ISO\Rocky-10.2-x86_64-minimal.iso"
VBoxManage startvm kafka01 --type=gui

In the installer, repeat Lesson 2 with three changes: hostname kafka01.saatlab.test, IPv4 10.10.10.31/24 (gateway and DNS 10.10.10.1), and under Installation Destination tick only the 20 GB disk, so the 10 GB disk stays empty for Kafka. Create student (administrator) and set the root password as before.

After the first boot, from ubu01, copy your key in while password logins still work, then apply the Lesson 33 hardening on kafka01:

ssh-copy-id student@10.10.10.31
ssh student@10.10.10.31
printf 'PasswordAuthentication no\nPermitRootLogin no\n' | sudo tee /etc/ssh/sshd_config.d/00-saat-hardening.conf
sudo sshd -t && sudo systemctl reload sshd
printf '10.10.10.11  rocky01.saatlab.test  rocky01\n10.10.10.21  ubu01.saatlab.test  ubu01\n10.10.10.40  ad01.saatlab.test  ad01\n' | sudo tee -a /etc/hosts
sudo dnf -y update

Add kafka01 to /etc/hosts on rocky01 and ubu01 too if Lesson 2 didn’t already list it. Take a snapshot now (VBoxManage snapshot kafka01 take base), so you can always start Kafka again from a clean OS.

✅ Checkpoint: From ubu01, ssh student@10.10.10.31 hostname prints kafka01 without asking for a password.

2Prepare the data disk

On kafka01, find the empty disk and give Kafka an XFS file system on it:

lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS /dev/sdb
sudo mkfs.xfs -L kafkadata /dev/sdb
sudo mkdir -p /var/lib/kafka
UUID=$(sudo blkid -s UUID -o value /dev/sdb)
echo "UUID=$UUID  /var/lib/kafka  xfs  defaults,noatime  0 0" | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount -a
df -h /var/lib/kafka

You should see something like this (mkfs.xfs output cut):

NAME  SIZE TYPE MOUNTPOINTS
sdb    10G disk
UUID=8e3b6f21-4c7a-4d9e-b1f5-0a2c7d6e9b43  /var/lib/kafka  xfs  defaults,noatime  0 0
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         10G  228M  9.8G   3% /var/lib/kafka

Kafka uses the whole disk, so no partition table or LVM is needed here; at work, data disks for brokers are often whole LUNs like this one. noatime stops the kernel writing an access time on every read, which matters for a service that reads its files all day. Check that sdb is the 10 GB disk before you run mkfs.

✅ Checkpoint: df shows a 10G XFS file system mounted on /var/lib/kafka.

3Install Java 21 and create the kafka user

Run this:

sudo dnf install -y java-21-openjdk-headless tar
java -version
sudo useradd --system --home-dir /var/lib/kafka --no-create-home --shell /usr/sbin/nologin kafka
sudo mkdir -p /var/lib/kafka/node{1,2,3} /var/log/kafka/node{1,2,3}
sudo chown -R kafka:kafka /var/lib/kafka /var/log/kafka
sudo chmod 750 /var/lib/kafka /var/log/kafka
id kafka

You should see something like this:

openjdk version "21.0.12" 2026-07-21 LTS
OpenJDK Runtime Environment (Red_Hat-21.0.12.0.7-1) (build 21.0.12+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-21.0.12.0.7-1) (build 21.0.12+7-LTS, mixed mode, sharing)
uid=991(kafka) gid=991(kafka) groups=991(kafka)

The headless package has the Java runtime without desktop libraries, which is all a server needs. kafka is a system account (UID below 1000) that can’t log in, owns its data and log folders and nothing else. Your Java patch and build numbers may differ from the ones shown.

✅ Checkpoint: java -version prints version 21 and id kafka shows a system UID.

4Download Kafka 4.4.0, check it and unpack it

Run this:

cd /tmp
curl -fLO https://downloads.apache.org/kafka/4.4.0/kafka_2.13-4.4.0.tgz
curl -fLO https://downloads.apache.org/kafka/4.4.0/kafka_2.13-4.4.0.tgz.sha512
EXPECTED=$(sed 's/.*://' kafka_2.13-4.4.0.tgz.sha512 | tr -d ' \n' | tr 'A-F' 'a-f' | grep -oE '^[0-9a-f]{128}')
ACTUAL=$(sha512sum kafka_2.13-4.4.0.tgz | cut -d' ' -f1)
[ "$EXPECTED" = "$ACTUAL" ] && echo "SHA512 OK" || echo "SHA512 MISMATCH, do not use this file"
sudo tar -xzf kafka_2.13-4.4.0.tgz -C /opt
sudo chown -R root:root /opt/kafka_2.13-4.4.0
sudo ln -sfn /opt/kafka_2.13-4.4.0 /opt/kafka
/opt/kafka/bin/kafka-topics.sh --version

You should see something like this (curl progress cut):

SHA512 OK
4.4.0

The .sha512 file from Apache lists the hash in upper-case groups after the file name, so the sed, tr and grep line turns it into the plain form sha512sum prints, and works for either layout. Never install a tarball whose hash doesn’t match. When 4.4.0 moves off downloads.apache.org after newer releases, the same path works under archive.apache.org/dist/kafka/.

The /opt/kafka link lets a later upgrade unpack the new version next to this one and switch the link, with the configs safe in /etc/kafka.

✅ Checkpoint: The hash check printed SHA512 OK and kafka-topics.sh reports 4.4.0.

5Write the three node configs

Paste this whole block:

sudo mkdir -p /etc/kafka
for n in 1 2 3; do
  port=$((9092 + (n - 1) * 100))
  cport=$((port + 1))
  sudo tee /etc/kafka/node$n.properties >/dev/null <<EOT
# /etc/kafka/node$n.properties  (SaatVedha Linux course, Lesson 58)
process.roles=broker,controller
node.id=$n
controller.quorum.bootstrap.servers=10.10.10.31:9093,10.10.10.31:9193,10.10.10.31:9293
listeners=PLAINTEXT://10.10.10.31:$port,CONTROLLER://10.10.10.31:$cport
advertised.listeners=PLAINTEXT://10.10.10.31:$port,CONTROLLER://10.10.10.31:$cport
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
log.dirs=/var/lib/kafka/node$n
num.partitions=3
default.replication.factor=3
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
share.coordinator.state.topic.replication.factor=3
share.coordinator.state.topic.min.isr=2
log.retention.hours=168
EOT
done
grep -H -E '^(node.id|listeners|log.dirs)=' /etc/kafka/node*.properties

You should see this:

/etc/kafka/node1.properties:node.id=1
/etc/kafka/node1.properties:listeners=PLAINTEXT://10.10.10.31:9092,CONTROLLER://10.10.10.31:9093
/etc/kafka/node1.properties:log.dirs=/var/lib/kafka/node1
/etc/kafka/node2.properties:node.id=2
/etc/kafka/node2.properties:listeners=PLAINTEXT://10.10.10.31:9192,CONTROLLER://10.10.10.31:9193
/etc/kafka/node2.properties:log.dirs=/var/lib/kafka/node2
/etc/kafka/node3.properties:node.id=3
/etc/kafka/node3.properties:listeners=PLAINTEXT://10.10.10.31:9292,CONTROLLER://10.10.10.31:9293
/etc/kafka/node3.properties:log.dirs=/var/lib/kafka/node3

The three files differ only in node.id, ports and data folder. controller.quorum.bootstrap.servers tells every node where to find the controllers. The replication settings make every new topic, and Kafka’s internal topics for consumer offsets, transactions and share groups, keep three copies and demand two in sync. With a single data folder, log.dirs also holds the metadata log.

✅ Checkpoint: grep shows three different node.id values, ports 9092, 9192 and 9292, and three data folders.

6Format the storage with one cluster ID

Make the IDs, save them, and format each node as the kafka user:

CID=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
D1=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
D2=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
D3=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
echo "CID=$CID D1=$D1 D2=$D2 D3=$D3" | sudo tee /etc/kafka/cluster-ids.txt
IC="1@10.10.10.31:9093:$D1,2@10.10.10.31:9193:$D2,3@10.10.10.31:9293:$D3"
for n in 1 2 3; do
  sudo -u kafka /opt/kafka/bin/kafka-storage.sh format --cluster-id "$CID" \
    --initial-controllers "$IC" --config /etc/kafka/node$n.properties | grep Formatting
done
sudo cat /var/lib/kafka/node1/meta.properties

You should see something like this:

CID=tWqVaTLwEZAWWJeKqq3YEA D1=Mz7Gk7FKW-cAtnd_1_7EHQ D2=V5cGMZqTstCz2wdi4SMFZw D3=3Kh8DnGlNkHX9xXMHwJD0w
Formatting dynamic metadata voter directory /var/lib/kafka/node1 with metadata.version 4.4-IV2.
Formatting dynamic metadata voter directory /var/lib/kafka/node2 with metadata.version 4.4-IV2.
Formatting dynamic metadata voter directory /var/lib/kafka/node3 with metadata.version 4.4-IV2.
#
#Tue Oct 27 13:22:41 IST 2026
cluster.id=tWqVaTLwEZAWWJeKqq3YEA
directory.id=Mz7Gk7FKW-cAtnd_1_7EHQ
node.id=1
version=1

Your IDs will differ. Each format also prints a long “Bootstrap metadata” line, which the grep hides. The --initial-controllers value must be identical on all three: node ID @ host : controller port : directory ID.

“dynamic metadata voter directory” means each node starts as a voter in a dynamic quorum, which you can later grow or shrink with kafka-metadata-quorum.sh add-controller and remove-controller. Keep /etc/kafka/cluster-ids.txt: Lessons 59 and 60 and any rebuild of a node need the same cluster ID.

✅ Checkpoint: Three “Formatting dynamic metadata voter directory” lines appeared and node1’s meta.properties shows your cluster ID and node.id=1.

7Create the systemd template unit and the LimitNOFILE drop-in

Paste this whole block:

sudo tee /etc/systemd/system/kafka@.service >/dev/null <<'EOF'
# /etc/systemd/system/kafka@.service  (SaatVedha Linux course, Lesson 58)
[Unit]
Description=Apache Kafka KRaft node %i
Documentation=https://kafka.apache.org/documentation/
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/var/lib/kafka

[Service]
Type=simple
User=kafka
Group=kafka
Environment="KAFKA_HEAP_OPTS=-Xms512m -Xmx512m"
Environment="LOG_DIR=/var/log/kafka/node%i"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /etc/kafka/node%i.properties
SuccessExitStatus=143
TimeoutStopSec=120
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF
sudo mkdir -p /etc/systemd/system/kafka@.service.d
printf '[Service]\nLimitNOFILE=100000\n' | sudo tee /etc/systemd/system/kafka@.service.d/10-limits.conf
sudo systemctl daemon-reload
systemd-analyze verify kafka@1.service && echo "unit OK"
systemctl show kafka@1 -p LimitNOFILE

You should see something like this:

[Service]
LimitNOFILE=100000
unit OK
LimitNOFILE=100000

%i became 1 for kafka@1: its log folder is node1 and its config node1.properties. KAFKA_HEAP_OPTS replaces Kafka’s default 1 GB heap with 512 MB per node, which three nodes need to fit in 4 GB with room for the page cache; production brokers usually run 4 to 8 GB of heap and leave the rest of the RAM to the cache. LOG_DIR gives each node its own server.log, because three nodes writing one file would be unreadable.

SuccessExitStatus=143 tells systemd that a Java process ending on SIGTERM (128 + 15) stopped normally. systemctl edit kafka@.service would make the same kind of drop-in, named override.conf.

✅ Checkpoint: systemd-analyze verify prints nothing before “unit OK” and systemctl show reports LimitNOFILE=100000.

8Start the three nodes and check the quorum

Run this:

sudo systemctl enable --now kafka@1 kafka@2 kafka@3
systemctl is-active kafka@1 kafka@2 kafka@3
grep 'open files' /proc/$(systemctl show -p MainPID --value kafka@1)/limits
/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 10.10.10.31:9092 describe --status

You should see something like this:

Created symlink '/etc/systemd/system/multi-user.target.wants/kafka@1.service' → '/etc/systemd/system/kafka@.service'.
Created symlink '/etc/systemd/system/multi-user.target.wants/kafka@2.service' → '/etc/systemd/system/kafka@.service'.
Created symlink '/etc/systemd/system/multi-user.target.wants/kafka@3.service' → '/etc/systemd/system/kafka@.service'.
active
active
active
Max open files            100000               100000               files
ClusterId:              tWqVaTLwEZAWWJeKqq3YEA
LeaderId:               1
LeaderEpoch:            1
HighWatermark:          214
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   0
CurrentVoters:          [{"id": 1, "directoryId": "Mz7Gk7FKW-cAtnd_1_7EHQ", "endpoints": ["CONTROLLER://10.10.10.31:9093"]}, {"id": 2, "directoryId": "V5cGMZqTstCz2wdi4SMFZw", "endpoints": ["CONTROLLER://10.10.10.31:9193"]}, {"id": 3, "directoryId": "3Kh8DnGlNkHX9xXMHwJD0w", "endpoints": ["CONTROLLER://10.10.10.31:9293"]}]
CurrentObservers:       []

Node 1 is the leader in epoch 1 and the high watermark, the last metadata record committed by a majority, is 214. Three voters are listed with their directory IDs and endpoints, and there are no observers because every broker here is also a voter. The /proc line confirms the running Java process got the limit from the drop-in.

Now look at each voter’s progress:

/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 10.10.10.31:9092 describe --replication --human-readable

You should see something like this:

NodeId  DirectoryId             LogEndOffset    Lag     LastFetchTimestamp      LastCaughtUpTimestamp   Status
1       Mz7Gk7FKW-cAtnd_1_7EHQ  214             0       2 ms ago                2 ms ago                Leader
2       V5cGMZqTstCz2wdi4SMFZw  214             0       187 ms ago              187 ms ago              Follower
3       3Kh8DnGlNkHX9xXMHwJD0w  214             0       186 ms ago              186 ms ago              Follower

Lag 0 for every node means the followers have the whole metadata log. A controller whose lag keeps growing, or whose last fetch is many seconds old, is down or cut off, and this is the first command to run when a cluster acts strangely. Watch sudo journalctl -u kafka@2 -f or /var/log/kafka/node2/server.log for one node’s own messages.

✅ Checkpoint: All three units are active, open files is 100000, and describe --status shows a LeaderId and three CurrentVoters.

9Open the broker ports, create a topic, send and read

Open only the three broker ports:

sudo firewall-cmd --permanent --add-port={9092,9192,9292}/tcp
sudo firewall-cmd --reload
sudo firewall-cmd --list-ports

success
success
9092/tcp 9192/tcp 9292/tcp

Create a topic for SaatShop order events and look at it:

B=10.10.10.31:9092
/opt/kafka/bin/kafka-topics.sh --bootstrap-server $B --create --topic saatshop-orders \
  --partitions 3 --replication-factor 3 --config min.insync.replicas=2
/opt/kafka/bin/kafka-topics.sh --bootstrap-server $B --describe --topic saatshop-orders

You should see something like this:

Created topic saatshop-orders.
Topic: saatshop-orders	TopicId: 8Qm2c0yFRcS6n1d3kq2Vbw	PartitionCount: 3	ReplicationFactor: 3	Configs: min.insync.replicas=2
	Topic: saatshop-orders	Partition: 0	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1	Elr: 	LastKnownElr:
	Topic: saatshop-orders	Partition: 1	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2	Elr: 	LastKnownElr:
	Topic: saatshop-orders	Partition: 2	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3	Elr: 	LastKnownElr: 

Each partition has its leader on a different node (2, 3 and 1), so the write load is spread, and all three replicas are in sync. Elr and LastKnownElr (eligible leader replicas) stay empty while the ISR is healthy. Now send three events and read them back:

printf 'order-1001 placed\norder-1002 placed\norder-1003 paid\n' | \
  /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server $B --topic saatshop-orders
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server $B --topic saatshop-orders \
  --from-beginning --max-messages 3

You should see something like this:

order-1001 placed
order-1002 placed
order-1003 paid
Processed a total of 3 messages

Kafka keeps order only inside a partition. These three went into one partition together, so they came back in order; with keys or many producers, events for one order stay in order only if they share a key. On ubu01, timeout 3 bash -c '</dev/tcp/10.10.10.31/9092' && echo open should print “open”, proving the firewall lets clients in.

✅ Checkpoint: The consumer printed your three events and “Processed a total of 3 messages”, and port 9092 is open from ubu01.

10Lose a node and watch the cluster carry on

Stop node 3, then check the quorum and the topic:

sudo systemctl stop kafka@3
/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server $B describe --status | grep -E 'LeaderId|MaxFollowerLag:'
/opt/kafka/bin/kafka-topics.sh --bootstrap-server $B --describe --topic saatshop-orders | grep Partition:
echo 'order-1004 placed' | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server $B \
  --topic saatshop-orders --command-property acks=all && echo "write accepted"

You should see something like this (Elr columns cut):

LeaderId:               1
MaxFollowerLag:         41
	Topic: saatshop-orders	Partition: 0	Leader: 2	Replicas: 2,3,1	Isr: 2,1
	Topic: saatshop-orders	Partition: 1	Leader: 1	Replicas: 3,1,2	Isr: 1,2
	Topic: saatshop-orders	Partition: 2	Leader: 1	Replicas: 1,2,3	Isr: 1,2
write accepted

Two of three controllers are still a majority, so the leader stayed and node 3 falls behind (MaxFollowerLag grows). Partition 1 lost its leader and node 1 took over from the replica list; every ISR now has two members, which still meets min.insync.replicas=2, so the acks=all write was accepted. Stop a second node and two things would change: the controllers would lose their majority, and producers using acks=all would get NOT_ENOUGH_REPLICAS errors.

Don’t do that now; start node 3 again and check it catches up:

sudo systemctl start kafka@3
sleep 20
/opt/kafka/bin/kafka-topics.sh --bootstrap-server $B --describe --topic saatshop-orders | grep -c 'Isr: [123],[123],[123]'

3

✅ Checkpoint: With node 3 stopped the write was accepted, and after the restart all three partitions are back to three in-sync replicas.