Linux logging: journald, rsyslog, remote syslog and logrotate

Most of this lab runs on rocky01, with ubu01 as the log server in steps 5 and 6. Keep an SSH session open to each. Every file you create has a number in its name on purpose, because rsyslog reads its folder in name order.

1See where rocky01 keeps logs today

Run this on rocky01:

sudo ls -l /var/log/messages /var/log/secure
ls -ld /var/log/journal
journalctl --disk-usage
journalctl --list-boots

You should see something like this:

-rw-------. 1 root root 1843271 Oct 27 11:18 /var/log/messages
-rw-------. 1 root root   48212 Oct 27 11:18 /var/log/secure
ls: cannot access '/var/log/journal': No such file or directory
Archived and active journals take up 8.0M in the file system.
IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
  0 a41c7e0b92d84f3b8e6d1c5f2a9b7e30 Tue 2026-10-27 09:01:12 IST Tue 2026-10-27 11:18:40 IST

rsyslog’s text files are there, with the dot after the permissions telling you they carry an SELinux label. In this example /var/log/journal doesn’t exist, so the 8 MB of journal sits in /run/log/journal, in RAM, and the boot list shows only the current boot even though rocky01 has been restarted many times since Lesson 2.

If your VM already has /var/log/journal and --list-boots shows several boots with negative numbers, your journal is persistent already. Do step 2 anyway, so the setting and the size limit are written down in a file instead of depending on whether a folder exists.

✅ Checkpoint: You can say whether rocky01’s journal is in RAM or on disk right now, and how much space it uses.

2Make the journal persistent, with a size limit

Paste this whole block on rocky01:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/50-saatlab.conf > /dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=200M
SystemKeepFree=1G
EOF
sudo systemctl restart systemd-journald
sudo journalctl --flush
ls -ld /var/log/journal /var/log/journal/*
systemd-analyze cat-config systemd/journald.conf | grep -E '^(Storage|SystemMaxUse|SystemKeepFree)='
journalctl --disk-usage

You should see something like this:

drwxr-sr-x+ 3 root systemd-journal 46 Oct 27 11:20 /var/log/journal
drwxr-sr-x+ 2 root systemd-journal 53 Oct 27 11:20 /var/log/journal/6f2b9c1d4e8a4b0f9c3d2e1a7b5c8d90
Storage=persistent
SystemMaxUse=200M
SystemKeepFree=1G
Archived and active journals take up 16.0M in the file system.

A drop-in file in journald.conf.d is the recommended way to change journald, because package updates never touch it and anyone can see your changes in one small file. Storage=persistent creates /var/log/journal itself, and the folder inside it is named after the machine ID. --flush moves what was in RAM onto the disk, which is why the usage went up.

SystemMaxUse=200M caps the journal, and SystemKeepFree=1G makes journald stop growing when the file system has less than 1 GB free, whichever comes first. On a real server size these to the disk; on a 20 GB lab VM, 200 MB holds weeks of logs. When you need room right now, sudo journalctl --vacuum-size=100M or --vacuum-time=2weeks deletes the oldest archived journal files.

✅ Checkpoint: systemd-analyze cat-config shows your three settings and /var/log/journal/<machine-id> exists.

3Send test messages with logger and read them back

Run this on rocky01:

logger -p local3.warning -t saatapp "order 1044: payment gateway timeout"
logger -p local3.info -t saatapp "order 1045 placed"
journalctl -t saatapp -p warning --since "5 min ago" --no-pager
sudo tail -n 2 /var/log/messages

You should see something like this:

Oct 27 11:24:10 rocky01 saatapp[2391]: order 1044: payment gateway timeout
Oct 27 11:24:10 rocky01 saatapp[2391]: order 1044: payment gateway timeout
Oct 27 11:24:10 rocky01 saatapp[2392]: order 1045 placed

-p local3.warning sets facility and priority, and -t saatapp sets the tag that shows up as the program name. journalctl’s -p warning shows warning and anything more serious, so the info line is filtered out there, while /var/log/messages has both because its rule takes *.info and above. This is how you test any logging change: send a message you can recognise and look for it at every stop.

✅ Checkpoint: The warning appears in journalctl and both lines appear at the end of /var/log/messages.

4Give your app its own log file

Create the rule, check it, then restart rsyslog:

sudo tee /etc/rsyslog.d/30-saatapp.conf > /dev/null <<'EOF'
local3.*    /var/log/saatapp.log
& stop
EOF
sudo rsyslogd -N1
sudo systemctl restart rsyslog

You should see something like this:

rsyslogd: version 8.2412.0-1.el10, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

rsyslogd -N1 reads the whole configuration and checks it without touching the running daemon, so make it a habit before every restart. Now send a message and see where it goes:

logger -p local3.err -t saatapp "order 1046: card declined"
sudo tail -n 1 /var/log/saatapp.log
sudo ls -l /var/log/saatapp.log
sudo ls -Z /var/log/saatapp.log
sudo grep -c "order 1046" /var/log/messages

You should see something like this:

Oct 27 11:31:45 rocky01 saatapp[2448]: order 1046: card declined
-rw-------. 1 root root 59 Oct 27 11:31 /var/log/saatapp.log
system_u:object_r:var_log_t:s0 /var/log/saatapp.log
0

rsyslog created the file with root-only permissions and SELinux gave it the normal var_log_t label, because new files under /var/log inherit it. The count of 0 in /var/log/messages proves & stop worked: local3 messages now go only to the app’s own file.

✅ Checkpoint: order 1046 is in /var/log/saatapp.log and not in /var/log/messages.

5Turn ubu01 into a log server

Paste this whole block on ubu01:

sudo tee /etc/rsyslog.d/10-remote-in.conf > /dev/null <<'EOF'
# Receive syslog from other servers on TCP 514 and keep each host's logs apart
module(load="imtcp")
template(name="PerHostFile" type="string" string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
ruleset(name="fromRemote") {
    action(type="omfile" dynaFile="PerHostFile")
}
input(type="imtcp" port="514" ruleset="fromRemote")
EOF
sudo rsyslogd -N1
sudo systemctl restart rsyslog
sudo ss -tlnp | grep ':514'
sudo ufw status

You should see something like this:

rsyslogd: version 8.2512.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
LISTEN 0      25           0.0.0.0:514       0.0.0.0:*    users:(("rsyslogd",pid=1907,fd=6))
LISTEN 0      25              [::]:514          [::]:*    users:(("rsyslogd",pid=1907,fd=7))
Status: inactive

The ruleset keeps remote messages away from ubu01’s own /var/log/syslog, and the %HOSTNAME% and %PROGRAMNAME% properties in the template give each sender and each program its own file. rsyslog on Ubuntu opens port 514 as root and then drops to the syslog user, and its AppArmor profile allows writes anywhere under /var/log, so /var/log/remote needs no extra setup.

ufw is inactive on ubu01 in this lab. On a server where it is active, allow only your own network: sudo ufw allow from 10.10.10.0/24 to any port 514 proto tcp. If the receiver were a Rocky server, you’d open it with sudo firewall-cmd --permanent --add-port=514/tcp and sudo firewall-cmd --reload. Lesson 34 covers both firewalls.

✅ Checkpoint: ss shows rsyslogd listening on port 514 on ubu01.

6Forward rocky01’s logs to ubu01

Paste this whole block on rocky01:

sudo tee /etc/rsyslog.d/05-forward-ubu01.conf > /dev/null <<'EOF'
# Send a copy of every message to ubu01 over TCP; queue on disk while ubu01 is down
*.* action(type="omfwd" target="10.10.10.21" port="514" protocol="tcp"
           queue.type="LinkedList" queue.filename="fwd_ubu01"
           queue.maxDiskSpace="200m" queue.saveOnShutdown="on"
           action.resumeRetryCount="-1")
EOF
sudo rsyslogd -N1 && sudo systemctl restart rsyslog
logger -p local3.notice -t saatapp "order 1047: refund queued, test from rocky01"
sudo true

Then run this on ubu01:

sudo ls /var/log/remote/rocky01/
sudo tail -n 1 /var/log/remote/rocky01/saatapp.log /var/log/remote/rocky01/sudo.log | grep -v '^==' | grep -v '^$'

You should see something like this:

rsyslogd.log  saatapp.log  sudo.log  systemd.log
2026-10-27T11:41:07+05:30 rocky01 saatapp[2502]: order 1047: refund queued, test from rocky01
2026-10-27T11:41:07+05:30 rocky01 sudo[2503]:  student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/usr/bin/true

The file is named 05 on purpose. Your 30-saatapp.conf ends with & stop, so a forwarding rule in a file named 90 would never see local3 messages; at 05 it runs first and every message, the app’s included, gets a copy sent. The sudo line shows that authpriv messages travel too, which is the main reason security teams want a log server.

On ubu01 the lines carry an RFC 3339 timestamp with whole seconds, because rocky01 sends the classic syslog format and ubu01 writes it in its own file format. On the SELinux side nothing was needed: TCP port 514 is labelled rsh_port_t and the policy lets rsyslog connect to it. If you ever pick another port, label it first with sudo semanage port -a -t syslogd_port_t -p tcp 30514.

✅ Checkpoint: /var/log/remote/rocky01/saatapp.log on ubu01 contains order 1047.

7Rotate the app log, dry run first

Create the logrotate file on rocky01 and test it:

sudo tee /etc/logrotate.d/saatapp > /dev/null <<'EOF'
/var/log/saatapp.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0600 root root
    postrotate
        /usr/bin/systemctl kill -s HUP rsyslog.service >/dev/null 2>&1 || true
    endscript
}
EOF
sudo logrotate -d /etc/logrotate.d/saatapp

You should see something like this:

warning: logrotate in debug mode does nothing except printing debug messages!  Consider using verbose mode (-v) instead if this is not what you want.

reading config file /etc/logrotate.d/saatapp
Reading state from file: /var/lib/logrotate/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
...
Handling 1 logs

rotating pattern: /var/log/saatapp.log  after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/saatapp.log
Creating new state
  Now: 2026-10-27 11:46
  Last rotated at 2026-10-27 11:00
  log does not need rotating (log has already been rotated)

-d reads the file, reports any error and shows what it would do, without changing anything. It says “already been rotated” because logrotate records a brand new log as rotated now, so the first real rotation happens tomorrow. The lines mean: rotate daily, keep 7 old files, gzip them, but leave the newest old file uncompressed for a day (delaycompress), skip the file if it is missing or empty, create a new empty file, then send rsyslog a HUP so it reopens its files.

Force one rotation now to prove the postrotate step works:

sudo logrotate -f /etc/logrotate.d/saatapp
sudo ls -l /var/log/saatapp.log*
logger -p local3.info -t saatapp "order 1048: written after rotation"
sudo ls -l /var/log/saatapp.log; sudo tail -n 1 /var/log/saatapp.log
systemctl list-timers logrotate.timer

You should see something like this:

-rw-------. 1 root root  0 Oct 27 11:47 /var/log/saatapp.log
-rw-------. 1 root root 59 Oct 27 11:31 /var/log/saatapp.log.1
-rw-------. 1 root root 59 Oct 27 11:47 /var/log/saatapp.log
Oct 27 11:47:30 rocky01 saatapp[2561]: order 1048: written after rotation
NEXT                        LEFT     LAST                        PASSED  UNIT            ACTIVATES
Wed 2026-10-28 00:00:00 IST 12h left Tue 2026-10-27 00:00:04 IST 11h ago logrotate.timer logrotate.service

1 timers listed.

The old content moved to saatapp.log.1 and the new message landed in the fresh saatapp.log, so rsyslog reopened its file after the HUP. When the daily timer runs it reads /etc/logrotate.conf first, and on Rocky that file sets dateext, so the rotated files will be named like saatapp.log-20261028 instead of .1.

You ran this one file on its own. On Ubuntu, testing a single file that way can fail with “parent directory has insecure permissions”, because /var/log there is group-writable by syslog and the su root adm line that handles it lives in /etc/logrotate.conf. Test with sudo logrotate -d /etc/logrotate.conf there, or add an su line to your own file.

The log server needs the same care, or /var/log/remote grows until ubu01’s disk is full. Put this in /etc/logrotate.d/remote-hosts on ubu01 and check it with sudo logrotate -d /etc/logrotate.conf; the postrotate script is the one Ubuntu’s own rsyslog rotation uses.

/var/log/remote/*/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

✅ Checkpoint: saatapp.log.1 holds the old lines, order 1048 is in the new saatapp.log, and logrotate.timer shows a next run.