Kafka security: SSL keystores, truststores and SASL

Steps 1 to 6 and 8 to 10 run on kafka01 as student; step 7 runs on ubu01. The cluster keeps serving the whole time, so leave a consumer running in a second window if you want to watch it. Take a snapshot first: VBoxManage snapshot kafka01 take before-l60.

1Check the cluster and give kafka01 its own name

The certificates will carry kafka01.saatlab.test, so kafka01 must resolve its own name:

systemctl is-active kafka@1 kafka@2 kafka@3
grep -q 'kafka01.saatlab.test' /etc/hosts || echo '10.10.10.31  kafka01.saatlab.test  kafka01' | sudo tee -a /etc/hosts
getent hosts kafka01.saatlab.test
mkdir -p ~/kafka-ssl ~/kafka-client
chmod 700 ~/kafka-ssl ~/kafka-client

You should see something like this:

active
active
active
10.10.10.31  kafka01.saatlab.test  kafka01
10.10.10.31     kafka01.saatlab.test kafka01

Lesson 58 gave kafka01 entries for the other VMs but not for itself, so the line is added once (the grep stops it being added twice). ubu01 and rocky01 already know kafka01.saatlab.test from Lesson 2. ~/kafka-ssl will hold the CA, which is the most sensitive file in this lesson, so it gets mode 700.

✅ Checkpoint: All three nodes are active and getent hosts kafka01.saatlab.test prints 10.10.10.31.

2Create a lab certificate authority

Run this:

cd ~/kafka-ssl
P=Lab-Only-Pass-60
keytool -genkeypair -alias saatlab-ca -keyalg RSA -validity 3650 \
  -dname "CN=SaatLab Kafka CA,O=SaatVedha Lab,C=IN" -ext bc:c \
  -keystore ca.p12 -storetype PKCS12 -storepass "$P"
keytool -exportcert -rfc -alias saatlab-ca -keystore ca.p12 -storepass "$P" -file ca.crt
openssl x509 -in ca.crt -noout -subject -enddate

You should see something like this:

Generating 3,072 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 3,650 days
	for: CN=SaatLab Kafka CA, O=SaatVedha Lab, C=IN
Certificate stored in file <ca.crt>
subject=C = IN, O = SaatVedha Lab, CN = SaatLab Kafka CA
notAfter=Oct 24 05:34:25 2036 GMT

-ext bc:c marks the certificate as a CA (basic constraints, critical), which is what lets it sign others. ca.p12 holds the CA’s private key, and anyone with it can mint certificates your clients will trust, so at work it never sits on a broker. ca.crt is the public half that goes into every truststore. Lab-Only-Pass-60 is a lab password; real store passwords come from your secrets manager.

✅ Checkpoint: ca.crt exists and openssl shows the subject “SaatLab Kafka CA” with an end date ten years away.

3Give each node a keystore with the right names

Create a key pair per node, sign it with the CA, and install the signed chain. Paste this whole block:

cd ~/kafka-ssl
P=Lab-Only-Pass-60
SAN="SAN=DNS:kafka01.saatlab.test,DNS:kafka01,IP:10.10.10.31"
for n in 1 2 3; do
  keytool -genkeypair -alias node$n -keyalg RSA -validity 730 \
    -dname "CN=kafka01.saatlab.test,OU=node$n,O=SaatVedha Lab,C=IN" -ext "$SAN" \
    -keystore node$n.keystore.p12 -storetype PKCS12 -storepass "$P"
  keytool -certreq -alias node$n -keystore node$n.keystore.p12 -storepass "$P" -file node$n.csr
  keytool -gencert -alias saatlab-ca -keystore ca.p12 -storepass "$P" -infile node$n.csr \
    -outfile node$n.crt -rfc -validity 730 -ext "$SAN" \
    -ext KU=digitalSignature,keyEncipherment -ext EKU=serverAuth,clientAuth
  keytool -importcert -alias saatlab-ca -file ca.crt -keystore node$n.keystore.p12 -storepass "$P" -noprompt
  keytool -importcert -alias node$n -file node$n.crt -keystore node$n.keystore.p12 -storepass "$P"
done
keytool -list -v -alias node1 -keystore node1.keystore.p12 -storepass "$P" | grep -E 'chain length|Owner|Issuer|until|DNSName|IPAddress'

You should see something like this (the first block repeats for node2 and node3):

Generating 3,072 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 730 days
	for: CN=kafka01.saatlab.test, OU=node1, O=SaatVedha Lab, C=IN
Certificate was added to keystore
Certificate reply was installed in keystore
Certificate chain length: 2
Owner: CN=kafka01.saatlab.test, OU=node1, O=SaatVedha Lab, C=IN
Issuer: CN=SaatLab Kafka CA, O=SaatVedha Lab, C=IN
Valid from: Tue Oct 27 11:05:14 IST 2026 until: Thu Oct 26 11:05:14 IST 2028
  DNSName: kafka01.saatlab.test
  DNSName: kafka01
  IPAddress: 10.10.10.31
Owner: CN=SaatLab Kafka CA, O=SaatVedha Lab, C=IN
Issuer: CN=SaatLab Kafka CA, O=SaatVedha Lab, C=IN
Valid from: Tue Oct 27 11:04:25 IST 2026 until: Fri Oct 24 11:04:25 IST 2036

genkeypair made a key and a temporary self-signed certificate, certreq turned it into a signing request, and gencert signed it with the CA. The SAN is passed again to gencert because keytool doesn’t copy extensions from the request on its own. Importing the CA first lets keytool build the chain, so “Certificate reply was installed” replaces the self-signed certificate with a chain of length 2: node, then CA.

All three nodes share one VM, so they carry the same names; on separate brokers each SAN would list that broker’s own name and IP.

✅ Checkpoint: The node1 listing shows chain length 2, issuer SaatLab Kafka CA, and the two DNS names plus the IP address.

4Build the truststore and install the files

Run this:

keytool -importcert -alias saatlab-ca -file ca.crt -keystore kafka.truststore.p12 \
  -storetype PKCS12 -storepass "$P" -noprompt
sudo install -d -o root -g kafka -m 750 /etc/kafka/ssl
sudo install -o root -g kafka -m 640 node1.keystore.p12 node2.keystore.p12 node3.keystore.p12 \
  kafka.truststore.p12 /etc/kafka/ssl/
install -m 600 kafka.truststore.p12 ca.crt ~/kafka-client/
sudo ls -l /etc/kafka/ssl

You should see something like this:

Certificate was added to keystore
total 24
-rw-r-----. 1 root kafka 1462 Oct 27 11:07 kafka.truststore.p12
-rw-r-----. 1 root kafka 6038 Oct 27 11:07 node1.keystore.p12
-rw-r-----. 1 root kafka 6038 Oct 27 11:07 node2.keystore.p12
-rw-r-----. 1 root kafka 6038 Oct 27 11:07 node3.keystore.p12

The kafka user can read the stores through its group but can’t change them, and other users can’t read them at all. install sets owner, group and mode in one go, which is less error-prone than cp followed by chown and chmod. The CA key stays behind in ~/kafka-ssl.

✅ Checkpoint: sudo ls -l /etc/kafka/ssl shows four files, all owned by root, group kafka, with mode -rw-r-----.

5Create the SCRAM users while PLAINTEXT still works

The brokers will log in to each other as admin, so that user must exist before any broker needs it:

B=10.10.10.31:9092,10.10.10.31:9192,10.10.10.31:9292
/opt/kafka/bin/kafka-configs.sh --bootstrap-server $B --alter --entity-type users --entity-name admin \
  --add-config 'SCRAM-SHA-512=[password=Lab-Only-Pass-60-admin]'
/opt/kafka/bin/kafka-configs.sh --bootstrap-server $B --alter --entity-type users --entity-name saatshop-app \
  --add-config 'SCRAM-SHA-512=[password=Lab-Only-Pass-60-app]'
/opt/kafka/bin/kafka-configs.sh --bootstrap-server $B --describe --entity-type users

You should see this:

Completed updating config for user admin.
Completed updating config for user saatshop-app.
SCRAM credential configs for user-principal 'admin' are SCRAM-SHA-512=iterations=4096
SCRAM credential configs for user-principal 'saatshop-app' are SCRAM-SHA-512=iterations=4096

Only the salt, the iteration count (4096 is the default and the minimum) and the derived keys are stored in the metadata log; describe never shows a password because Kafka doesn’t have it. Bootstrap lists all three nodes, so the tools keep working while any one of them restarts.

✅ Checkpoint: describe lists SCRAM-SHA-512 credentials for admin and saatshop-app.

6Rolling restart 1: open SASL_SSL next to PLAINTEXT

Rewrite the three configs with a SASL_SSL listener on 9094, 9194 and 9294 (inter-broker traffic stays PLAINTEXT for now). Paste this whole block:

for n in 1 2 3; do
  port=$((9092 + (n - 1) * 100))
  cport=$((port + 1))
  sport=$((port + 2))
  sudo tee /etc/kafka/node$n.properties >/dev/null <<EOT
# /etc/kafka/node$n.properties  (SaatVedha Linux course, Lessons 58 and 60)
process.roles=broker,controller
node.id=$n
controller.quorum.bootstrap.servers=10.10.10.31:9093,10.10.10.31:9193,10.10.10.31:9293
listeners=PLAINTEXT://10.10.10.31:$port,SASL_SSL://10.10.10.31:$sport,CONTROLLER://10.10.10.31:$cport
advertised.listeners=PLAINTEXT://10.10.10.31:$port,SASL_SSL://kafka01.saatlab.test:$sport,CONTROLLER://10.10.10.31:$cport
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SASL_SSL:SASL_SSL
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
listener.name.sasl_ssl.sasl.enabled.mechanisms=SCRAM-SHA-512
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="Lab-Only-Pass-60-admin";
ssl.keystore.type=PKCS12
ssl.keystore.location=/etc/kafka/ssl/node$n.keystore.p12
ssl.keystore.password=Lab-Only-Pass-60
ssl.key.password=Lab-Only-Pass-60
ssl.truststore.type=PKCS12
ssl.truststore.location=/etc/kafka/ssl/kafka.truststore.p12
ssl.truststore.password=Lab-Only-Pass-60
log.dirs=/var/lib/kafka/node$n
num.partitions=3
default.replication.factor=3
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
share.coordinator.state.topic.replication.factor=3
share.coordinator.state.topic.min.isr=2
log.retention.hours=168
EOT
done
sudo chown root:kafka /etc/kafka/node*.properties
sudo chmod 640 /etc/kafka/node*.properties
sudo grep -h '^listeners=' /etc/kafka/node*.properties

You should see this:

listeners=PLAINTEXT://10.10.10.31:9092,SASL_SSL://10.10.10.31:9094,CONTROLLER://10.10.10.31:9093
listeners=PLAINTEXT://10.10.10.31:9192,SASL_SSL://10.10.10.31:9194,CONTROLLER://10.10.10.31:9193
listeners=PLAINTEXT://10.10.10.31:9292,SASL_SSL://10.10.10.31:9294,CONTROLLER://10.10.10.31:9293

Open the new ports, define a small function that restarts one node at a time and waits for replication to catch up, and use it:

sudo firewall-cmd --permanent --add-port={9094,9194,9294}/tcp
sudo firewall-cmd --reload
roll() {
  # usage: roll BOOTSTRAP [extra kafka-topics options]
  local bs=$1; shift
  for n in 1 2 3; do
    sudo systemctl restart "kafka@$n"
    sleep 15
    until systemctl is-active --quiet "kafka@$n" && [ -z "$(/opt/kafka/bin/kafka-topics.sh \
        --bootstrap-server "$bs" "$@" --describe --under-replicated-partitions 2>&1)" ]; do
      sleep 5
    done
    echo "kafka@$n is back and nothing is under-replicated"
  done
}
roll "$B"
ss -ltn | grep -c -E ':(9094|9194|9294) '

You should see something like this (it takes a few minutes):

success
success
kafka@1 is back and nothing is under-replicated
kafka@2 is back and nothing is under-replicated
kafka@3 is back and nothing is under-replicated
3

The function restarts a node, waits until the unit is active and --under-replicated-partitions prints nothing, and only then moves on, which is the rule for any rolling restart. sasl.mechanism.inter.broker.protocol is already set but does nothing until the inter-broker listener changes in step 9. If you open a new shell later, paste the roll function again and set B (step 5) or S (step 9) before you use it. If a node doesn’t come back, sudo journalctl -u kafka@1 -n 50 usually names the keystore or password at fault.

✅ Checkpoint: roll printed three “is back” lines and three SASL_SSL ports are listening.

7Test the TLS listener from ubu01

On ubu01, fetch the CA certificate and connect the way any client would:

scp student@10.10.10.31:kafka-ssl/ca.crt ~/kafka-ca.crt
openssl s_client -connect kafka01.saatlab.test:9094 -CAfile ~/kafka-ca.crt -verify_hostname kafka01.saatlab.test -brief </dev/null

You should see something like this:

CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: C = IN, O = SaatVedha Lab, OU = node1, CN = kafka01.saatlab.test
Hash used: SHA256
Signature type: RSA-PSS
Verification: OK
Verified peername: kafka01.saatlab.test
Server Temp Key: X25519, 253 bits
DONE

“Verification: OK” and “Verified peername” are the two checks a Kafka client makes: the chain ends at your CA and the name is in the SAN. The broker and Java 21 picked TLS 1.3, Kafka’s default. openssl can’t do the SASL part, so this proves TLS only, and it’s the quickest test when a client team says “SSL doesn’t work”. Newer OpenSSL releases word a few of these lines differently; look for the Verification line.

✅ Checkpoint: openssl prints Verification: OK and Verified peername: kafka01.saatlab.test.

8Write client configs and send events over SASL_SSL

Back on kafka01, create one config file per user and use the application user:

cat > ~/kafka-client/admin.properties <<'EOF'
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="Lab-Only-Pass-60-admin";
ssl.truststore.type=PKCS12
ssl.truststore.location=/home/student/kafka-client/kafka.truststore.p12
ssl.truststore.password=Lab-Only-Pass-60
EOF
sed -e 's/username="admin"/username="saatshop-app"/' -e 's/Lab-Only-Pass-60-admin/Lab-Only-Pass-60-app/' \
  ~/kafka-client/admin.properties > ~/kafka-client/saatshop-app.properties
chmod 600 ~/kafka-client/*.properties
S=kafka01.saatlab.test:9094,kafka01.saatlab.test:9194,kafka01.saatlab.test:9294
APP=~/kafka-client/saatshop-app.properties
printf 'order-2001 placed\norder-2002 paid\n' | /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server $S --topic saatshop-orders --command-config $APP
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server $S --topic saatshop-orders \
  --from-beginning --max-messages 6 --command-config $APP

You should see something like this (your order of lines may differ):

order-1001 placed
order-1002 placed
order-1003 paid
order-2001 placed
order-2002 paid
order-1004 placed
Processed a total of 6 messages

The four events from Lesson 58 and the two new ones came back over an encrypted, authenticated connection. Kafka keeps order only inside a partition, so lines from different partitions can interleave. The truststore here is PKCS12; for non-Java clients, or to skip the store password, ssl.truststore.type=PEM with ssl.truststore.location pointing at ca.crt works too.

Without an authorizer, saatshop-app could read any topic; ACLs are the next step at work.

✅ Checkpoint: The consumer printed six events and “Processed a total of 6 messages” using saatshop-app.properties.

9Rolling restarts 2 and 3: brokers on SASL_SSL, PLAINTEXT closed

Move replication to SASL_SSL, roll, then remove PLAINTEXT, roll again and close the old ports:

S=kafka01.saatlab.test:9094,kafka01.saatlab.test:9194,kafka01.saatlab.test:9294
A=~/kafka-client/admin.properties
sudo sed -i 's/^inter.broker.listener.name=PLAINTEXT$/inter.broker.listener.name=SASL_SSL/' /etc/kafka/node*.properties
roll "$S" --command-config "$A"
sudo sed -i -E -e 's#PLAINTEXT://10\.10\.10\.31:9[0-9]92,##' -e 's#,PLAINTEXT:PLAINTEXT##' /etc/kafka/node*.properties
roll "$S" --command-config "$A"
sudo firewall-cmd --permanent --remove-port={9092,9192,9292}/tcp
sudo firewall-cmd --reload
sudo firewall-cmd --list-ports
sudo grep -E '^(listeners|advertised.listeners|inter.broker.listener.name|listener.security.protocol.map)=' /etc/kafka/node1.properties
/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server $S --command-config $A describe --status | grep -E 'LeaderId|MaxFollowerLag:'

You should see something like this:

kafka@1 is back and nothing is under-replicated
kafka@2 is back and nothing is under-replicated
kafka@3 is back and nothing is under-replicated
kafka@1 is back and nothing is under-replicated
kafka@2 is back and nothing is under-replicated
kafka@3 is back and nothing is under-replicated
success
success
9094/tcp 9194/tcp 9294/tcp
listeners=SASL_SSL://10.10.10.31:9094,CONTROLLER://10.10.10.31:9093
advertised.listeners=SASL_SSL://kafka01.saatlab.test:9094,CONTROLLER://10.10.10.31:9093
inter.broker.listener.name=SASL_SSL
listener.security.protocol.map=CONTROLLER:PLAINTEXT,SASL_SSL:SASL_SSL
LeaderId:               3
MaxFollowerLag:         0

During restart 2 every node still had both listeners, so a node replicating over SASL_SSL could reach nodes that hadn’t switched yet. Restart 3 could only begin once nothing used PLAINTEXT. The sed in the middle removes the PLAINTEXT entry from listeners, advertised.listeners and the protocol map in one pass.

From now on every Kafka command on this cluster needs --bootstrap-server with the SASL_SSL addresses and --command-config with a client file, including the Lesson 58 health checks.

✅ Checkpoint: Only 9094, 9194 and 9294 are open, node1 lists SASL_SSL and CONTROLLER only, and the quorum shows a leader with MaxFollowerLag 0.

10Check when the certificates expire

Read the end date from each keystore, and from the live listener the way a monitoring check would:

for f in /etc/kafka/ssl/node*.keystore.p12; do
  printf '%s  ' "$f"
  sudo keytool -list -v -keystore "$f" -storepass Lab-Only-Pass-60 | grep -m1 'until:' | sed 's/.*until: //'
done
echo | openssl s_client -connect kafka01.saatlab.test:9094 2>/dev/null | openssl x509 -noout -enddate -checkend $((30*24*3600))
echo "exit code $?"

You should see something like this:

/etc/kafka/ssl/node1.keystore.p12  Thu Oct 26 11:05:14 IST 2028
/etc/kafka/ssl/node2.keystore.p12  Thu Oct 26 11:05:16 IST 2028
/etc/kafka/ssl/node3.keystore.p12  Thu Oct 26 11:05:18 IST 2028
notAfter=Oct 26 05:35:14 2028 GMT
Certificate will not expire
exit code 0

-checkend 2592000 asks “will this expire in the next 30 days?” and answers with exit code 0 (no) or 1 (yes, renew now), which makes it easy to wrap in a monitoring item. keytool reads the file, so it also catches a renewed keystore that was never loaded; openssl reads what the broker is serving right now. Check both after every renewal.

✅ Checkpoint: All three keystores end in October 2028, and the live check prints “Certificate will not expire” with exit code 0.