ZooKeeper quorums for older Kafka clusters

Everything runs on kafka01 as student over SSH. Kafka 3.9.2 goes into its own folder, /opt/kafka_2.13-3.9.2, and the /opt/kafka link from Lesson 58 is never touched, so the two versions can’t mix. Take a snapshot first (VBoxManage snapshot kafka01 take before-l59), because it’s the easiest way back if anything goes wrong.

1Make room on kafka01

Three ZooKeeper members and a broker need about 1.5 GB, so stop the Lesson 58 cluster for now:

sudo systemctl stop kafka@1 kafka@2 kafka@3
systemctl is-active kafka@1 kafka@2 kafka@3
free -h

You should see something like this:

inactive
inactive
inactive
               total        used        free      shared  buff/cache   available
Mem:           3.6Gi       418Mi       2.6Gi       8.0Mi       806Mi       3.2Gi
Swap:          3.9Gi          0B       3.9Gi

“inactive” (not “failed”) means each node stopped cleanly on SIGTERM. The units stay enabled, so if kafka01 reboots in the middle of this lesson the Lesson 58 cluster comes back on its own; stop it again if that happens. The last part of Break It, Then Fix It starts it again for Lesson 60.

✅ Checkpoint: All three kafka@ units are inactive and “available” memory is above 3 GB.

2Install Kafka 3.9.2 next to Kafka 4.4

3.9.2 is an archived release, so it comes from archive.apache.org (slower than the main mirror):

cd /tmp
curl -fLO https://archive.apache.org/dist/kafka/3.9.2/kafka_2.13-3.9.2.tgz
curl -fLO https://archive.apache.org/dist/kafka/3.9.2/kafka_2.13-3.9.2.tgz.sha512
EXPECTED=$(sed 's/.*://' kafka_2.13-3.9.2.tgz.sha512 | tr -d ' \n' | tr 'A-F' 'a-f' | grep -oE '^[0-9a-f]{128}')
ACTUAL=$(sha512sum kafka_2.13-3.9.2.tgz | cut -d' ' -f1)
[ "$EXPECTED" = "$ACTUAL" ] && echo "SHA512 OK" || echo "SHA512 MISMATCH, do not use this file"
sudo tar -xzf kafka_2.13-3.9.2.tgz -C /opt
sudo chown -R root:root /opt/kafka_2.13-3.9.2
ls /opt/kafka_2.13-3.9.2/libs | grep -i zookeeper
readlink /opt/kafka

You should see something like this (curl progress cut):

SHA512 OK
zookeeper-3.8.4.jar
zookeeper-jute-3.8.4.jar
/opt/kafka_2.13-4.4.0

Kafka 3.x ships ZooKeeper inside its tarball: the jars in libs and the scripts zookeeper-server-start.sh and zookeeper-shell.sh in bin. Many companies instead run a separate Apache ZooKeeper install, where you’d use zkServer.sh status to read the mode; it runs the same srvr command you’ll use below. The link still points at 4.4.0, so every Lesson 58 command keeps working.

✅ Checkpoint: The hash printed SHA512 OK, the libs folder has zookeeper-3.8.4.jar, and /opt/kafka still points at kafka_2.13-4.4.0.

3Create the zookeeper user, myid files and three configs

Paste this whole block:

sudo dnf install -y nmap-ncat
sudo useradd --system --home-dir /var/lib/zookeeper --no-create-home --shell /usr/sbin/nologin zookeeper
sudo mkdir -p /etc/zookeeper
for n in 1 2 3; do
  sudo mkdir -p /var/lib/zookeeper/zk$n /var/log/zookeeper/zk$n
  echo "$n" | sudo tee /var/lib/zookeeper/zk$n/myid >/dev/null
  sudo tee /etc/zookeeper/zoo$n.cfg >/dev/null <<EOT
# /etc/zookeeper/zoo$n.cfg  (SaatVedha Linux course, Lesson 59)
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper/zk$n
clientPort=$((2180 + n))
clientPortAddress=10.10.10.31
maxClientCnxns=60
admin.enableServer=true
admin.serverAddress=127.0.0.1
admin.serverPort=$((8080 + n))
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
server.1=10.10.10.31:2888:3888
server.2=10.10.10.31:2889:3889
server.3=10.10.10.31:2890:3890
EOT
done
sudo chown -R zookeeper:zookeeper /var/lib/zookeeper /var/log/zookeeper
grep -H -E '^(clientPort|admin.serverPort)=' /etc/zookeeper/zoo*.cfg
sudo cat /var/lib/zookeeper/zk1/myid /var/lib/zookeeper/zk2/myid /var/lib/zookeeper/zk3/myid

You should see this:

/etc/zookeeper/zoo1.cfg:clientPort=2181
/etc/zookeeper/zoo1.cfg:admin.serverPort=8081
/etc/zookeeper/zoo2.cfg:clientPort=2182
/etc/zookeeper/zoo2.cfg:admin.serverPort=8082
/etc/zookeeper/zoo3.cfg:clientPort=2183
/etc/zookeeper/zoo3.cfg:admin.serverPort=8083
1
2
3

The three files differ only in dataDir and the two ports, and all of them carry the same three server lines. On separate servers every member would use 2181, 2888 and 3888; here each member needs its own ports because they share one IP address. clientPortAddress keeps clients on the lab network address, and the AdminServer listens only on 127.0.0.1. The autopurge lines clean old snapshots and transaction logs once a day, which a forgotten ensemble otherwise piles up until the disk fills.

✅ Checkpoint: grep shows ports 2181 to 2183 and 8081 to 8083, and the three myid files hold 1, 2 and 3.

4Run the ensemble under systemd

Create a template unit, start all three members and ask each one for its mode:

sudo tee /etc/systemd/system/zookeeper@.service >/dev/null <<'EOF'
# /etc/systemd/system/zookeeper@.service  (SaatVedha Linux course, Lesson 59)
[Unit]
Description=ZooKeeper member %i for the Kafka 3.9 lab
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=zookeeper
Group=zookeeper
Environment="KAFKA_HEAP_OPTS=-Xms256m -Xmx256m"
Environment="LOG_DIR=/var/log/zookeeper/zk%i"
ExecStart=/opt/kafka_2.13-3.9.2/bin/zookeeper-server-start.sh /etc/zookeeper/zoo%i.cfg
SuccessExitStatus=143
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now zookeeper@1 zookeeper@2 zookeeper@3
sleep 10
for n in 1 2 3; do echo "zk$n: $(echo srvr | nc 10.10.10.31 $((2180 + n)) | grep Mode)"; done
echo srvr | nc 10.10.10.31 2183

You should see something like this:

Created symlink '/etc/systemd/system/multi-user.target.wants/zookeeper@1.service' → '/etc/systemd/system/zookeeper@.service'.
Created symlink '/etc/systemd/system/multi-user.target.wants/zookeeper@2.service' → '/etc/systemd/system/zookeeper@.service'.
Created symlink '/etc/systemd/system/multi-user.target.wants/zookeeper@3.service' → '/etc/systemd/system/zookeeper@.service'.
zk1: Mode: follower
zk2: Mode: follower
zk3: Mode: leader
Zookeeper version: 3.8.4-9316c2a7a97e1666d8f4593f34dd6fc36ecc436c, built on 2024-02-12 22:16 UTC
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: leader
Node count: 5
Proposal sizes last/min/max: -1/-1/-1

zk3 won the first election because all three started with no data, so the zxids were equal and the highest myid won. The zxid 0x100000000 means epoch 1 (the high 32 bits), transaction 0: every new leader starts a new epoch, so you can tell from the zxid how many leader changes the ensemble has had. Node count 5 is the empty tree ZooKeeper creates for itself.

Each member logs to /var/log/zookeeper/zkN/server.log.

✅ Checkpoint: Two members say follower and one says leader.

5Four-letter words and the AdminServer

Ask for something other than srvr:

echo ruok | nc 10.10.10.31 2181; echo
echo stat | nc 10.10.10.31 2181; echo

You’ll get this:

ruok is not executed because it is not in the whitelist.
stat is not executed because it is not in the whitelist.

Only srvr is enabled by default (zkServer.sh needs it). Allow the four that monitoring uses, then restart one member at a time, followers first and the leader last, checking the mode after each:

for n in 1 2 3; do echo '4lw.commands.whitelist=srvr,stat,ruok,mntr' | sudo tee -a /etc/zookeeper/zoo$n.cfg >/dev/null; done
for n in 1 2 3; do
  sudo systemctl restart zookeeper@$n; sleep 10
  echo "zk$n: $(echo srvr | nc 10.10.10.31 $((2180 + n)) | grep Mode)"
done
echo ruok | nc 10.10.10.31 2181; echo
echo mntr | nc 10.10.10.31 2182 | grep -E 'server_state|znode_count|synced_followers|file_descriptor'
curl -s http://127.0.0.1:8082/commands/ruok

You should see something like this:

zk1: Mode: follower
zk2: Mode: follower
zk3: Mode: follower
imok
zk_server_state	leader
zk_znode_count	5
zk_synced_followers	2
zk_max_file_descriptor_count	524288
zk_open_file_descriptor_count	71
{
  "command" : "ruok",
  "error" : null
}

When zk3, the old leader, restarted, zk1 and zk2 elected zk2 (the higher myid of the two), and zk3 came back as a follower, so the ensemble never lost its majority. On the leader, zk_synced_followers 2 is the number worth alerting on: below 1 in a three-member ensemble means one more failure stops all writes. The AdminServer answered on 127.0.0.1:8082; curl -s http://127.0.0.1:8082/commands lists every command it knows.

✅ Checkpoint: ruok answers imok, mntr on zk2 shows leader with 2 synced followers, and the AdminServer returns JSON with “error” : null.

6Attach a ZooKeeper-mode Kafka 3.9.2 broker

Create the broker’s folders, config and unit, start it and look at what it wrote into ZooKeeper:

sudo -u kafka mkdir -p /var/lib/kafka/legacy-zk
sudo mkdir -p /var/log/kafka/legacy /etc/kafka39
sudo chown kafka:kafka /var/log/kafka/legacy
sudo tee /etc/kafka39/server.properties >/dev/null <<'EOF'
# /etc/kafka39/server.properties  (SaatVedha Linux course, Lesson 59: ZooKeeper-mode lab broker)
broker.id=0
listeners=PLAINTEXT://10.10.10.31:9392
advertised.listeners=PLAINTEXT://10.10.10.31:9392
log.dirs=/var/lib/kafka/legacy-zk
zookeeper.connect=10.10.10.31:2181,10.10.10.31:2182,10.10.10.31:2183
num.partitions=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=24
EOF
sudo tee /etc/systemd/system/kafka39.service >/dev/null <<'EOF'
# /etc/systemd/system/kafka39.service  (SaatVedha Linux course, Lesson 59)
[Unit]
Description=Kafka 3.9.2 broker in ZooKeeper mode (lab)
After=zookeeper@1.service zookeeper@2.service zookeeper@3.service

[Service]
Type=simple
User=kafka
Group=kafka
Environment="KAFKA_HEAP_OPTS=-Xms512m -Xmx512m"
Environment="LOG_DIR=/var/log/kafka/legacy"
ExecStart=/opt/kafka_2.13-3.9.2/bin/kafka-server-start.sh /etc/kafka39/server.properties
SuccessExitStatus=143
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl start kafka39
sleep 20
sudo grep -m1 'started (kafka.server.KafkaServer)' /var/log/kafka/legacy/server.log
ZS=/opt/kafka_2.13-3.9.2/bin/zookeeper-shell.sh
$ZS 10.10.10.31:2181 ls /brokers/ids 2>/dev/null | tail -n 1
$ZS 10.10.10.31:2181 get /controller 2>/dev/null | tail -n 1
$ZS 10.10.10.31:2181 get /cluster/id 2>/dev/null | tail -n 1

You should see something like this:

[2026-10-27 15:12:40,881] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[0]
{"version":2,"brokerid":0,"timestamp":"1793094158412","kraftControllerEpoch":-1}
{"version":"1","id":"Fz3mQe8URq2Yv0bK1xWJtA"}

Broker 0 registered itself as /brokers/ids/0, won the controller election (/controller says brokerid 0), and generated a cluster ID, because the cluster was new. kraftControllerEpoch -1 means no KRaft controller has ever taken over; during a migration that number changes, which is how ZooKeeper brokers know a KRaft controller is in charge. Now create a topic and see it appear as a znode:

/opt/kafka_2.13-3.9.2/bin/kafka-topics.sh --bootstrap-server 10.10.10.31:9392 --create --topic legacy-orders --partitions 1 --replication-factor 1
$ZS 10.10.10.31:2183 ls /brokers/topics 2>/dev/null | tail -n 1

Created topic legacy-orders.
[legacy-orders]

The topic was created through the broker, written by the controller into ZooKeeper through the leader, and read back from zk3, a different member. In Kafka 3.x every tool talks to the brokers with --bootstrap-server; only zookeeper-shell.sh and admin scripts for ZooKeeper itself still connect to port 2181.

✅ Checkpoint: server.log shows “[KafkaServer id=0] started”, /brokers/ids is [0], and legacy-orders is listed under /brokers/topics.

7Lose the leader and watch a new election

Find the current leader, stop it, and check that Kafka can still change metadata:

for n in 1 2 3; do echo srvr | nc 10.10.10.31 $((2180 + n)) | grep -q leader && L=$n; done; echo "leader is zk$L"
sudo systemctl stop zookeeper@$L
sleep 5
for n in 1 2 3; do echo "zk$n: $(echo srvr | nc -w 2 10.10.10.31 $((2180 + n)) 2>/dev/null | grep Mode)"; done
/opt/kafka_2.13-3.9.2/bin/kafka-topics.sh --bootstrap-server 10.10.10.31:9392 --create --topic legacy-returns --partitions 1 --replication-factor 1
sudo systemctl start zookeeper@$L
sleep 10
echo "zk$L: $(echo srvr | nc 10.10.10.31 $((2180 + L)) | grep Mode)"

You should see something like this:

leader is zk2
zk1: Mode: follower
zk2:
zk3: Mode: leader
Created topic legacy-returns.
zk2: Mode: follower

With zk2 gone, zk1 and zk3 were still two of three, so they elected zk3 within a second or two and writes carried on: the new topic was created while one member was down. zk2 came back as a follower and caught up from the leader. This is the failure a three-member ensemble is built for, and it’s why you patch and restart ZooKeeper members one at a time.

✅ Checkpoint: A new leader was elected, legacy-returns was created with one member down, and the stopped member rejoined as a follower.