Linux log analysis: searching logs with grep, awk and journalctl

Real logs change every minute, so most of this lab runs on four small sample files that copy the exact format of each distro: messages and secure as Rocky writes them, syslog and auth.log as Ubuntu 26.04 writes them. Every output from those files below is exact. Steps 1, 8 and 9 use the real logs and the journal on the VMs.

1Find the log files on each distro

On rocky01, run this:

sudo ls -l /var/log/messages /var/log/secure
systemctl is-active rsyslog

You should see something like this:

-rw-------. 1 root root 412886 Oct 27 10:31 /var/log/messages
-rw-------. 1 root root  18204 Oct 27 10:24 /var/log/secure
active

On ubu01, run this:

ls -l /var/log/syslog /var/log/auth.log
systemctl is-active rsyslog

You should see something like this:

-rw-r----- 1 syslog adm  20531 Oct 27 10:24 /var/log/auth.log
-rw-r----- 1 syslog adm 388120 Oct 27 10:31 /var/log/syslog
active

Same job, different names and permissions. Only root can read the Rocky files. On Ubuntu the group is adm, and the installer puts the first user (student) in adm, so you can grep syslog without sudo there.

If rsyslog isn’t active on a server you inherit (some minimal and container images skip it), the files won’t exist and the journal is your only log.

✅ Checkpoint: You can name the general log and the login log on each VM and say who can read them.

2Create the sample logs

Paste this whole block on either VM:

mkdir -p ~/lab24 && cd ~/lab24
cat > messages <<'EOF'
Oct 27 09:58:12 rocky01 systemd[1]: Starting dnf-makecache.service - dnf makecache...
Oct 27 10:02:41 rocky01 saatapp[1440]: ERROR payment timeout after 30s order=10422
Oct 27 10:02:43 rocky01 saatapp[1440]: WARN retry queue length 12
Oct 27 10:07:55 rocky01 saatapp[1440]: ERROR payment timeout after 30s order=10431
Oct 27 10:12:30 rocky01 chronyd[801]: Selected source 10.10.10.1
Oct 27 10:15:02 rocky01 saatapp[1440]: error: database connection lost (errno 2013)
Oct 27 10:15:04 rocky01 saatapp[1440]: INFO reconnected to database
Oct 27 10:31:17 rocky01 systemd[1]: Finished dnf-makecache.service - dnf makecache.
EOF
cat > syslog <<'EOF'
2026-10-27T09:58:12.402117+05:30 ubu01 systemd[1]: Starting apt-daily.service - Daily apt download activities...
2026-10-27T10:02:41.118344+05:30 ubu01 saatapp[1520]: ERROR payment timeout after 30s order=10422
2026-10-27T10:02:43.020551+05:30 ubu01 saatapp[1520]: WARN retry queue length 12
2026-10-27T10:07:55.771230+05:30 ubu01 saatapp[1520]: ERROR payment timeout after 30s order=10431
2026-10-27T10:12:30.004512+05:30 ubu01 chronyd[790]: Selected source 10.10.10.1
2026-10-27T10:15:02.902113+05:30 ubu01 saatapp[1520]: error: database connection lost (errno 2013)
2026-10-27T10:15:04.330981+05:30 ubu01 saatapp[1520]: INFO reconnected to database
2026-10-27T10:31:17.640022+05:30 ubu01 systemd[1]: Finished apt-daily.service - Daily apt download activities.
EOF
cat > secure <<'EOF'
Oct 27 10:20:01 rocky01 sshd-session[2290]: Failed password for priya from 10.10.10.21 port 50122 ssh2
Oct 27 10:20:09 rocky01 sshd-session[2290]: Accepted password for priya from 10.10.10.21 port 50122 ssh2
Oct 27 10:21:40 rocky01 sshd-session[2301]: Invalid user admin from 10.10.10.35 port 41870
Oct 27 10:21:42 rocky01 sshd-session[2301]: Failed password for invalid user admin from 10.10.10.35 port 41870 ssh2
Oct 27 10:21:47 rocky01 sshd-session[2305]: Invalid user oracle from 10.10.10.35 port 41884
Oct 27 10:21:49 rocky01 sshd-session[2305]: Failed password for invalid user oracle from 10.10.10.35 port 41884 ssh2
Oct 27 10:21:55 rocky01 sshd-session[2309]: Failed password for root from 10.10.10.35 port 41890 ssh2
Oct 27 10:24:13 rocky01 sshd-session[2318]: Failed password for arjun from 10.10.10.48 port 52210 ssh2
Oct 27 10:24:20 rocky01 sshd-session[2318]: Accepted password for arjun from 10.10.10.48 port 52210 ssh2
EOF
cat > auth.log <<'EOF'
2026-10-27T10:20:01.746945+05:30 ubu01 sshd-session[2290]: Failed password for priya from 10.10.10.21 port 50122 ssh2
2026-10-27T10:20:09.401458+05:30 ubu01 sshd-session[2290]: Accepted password for priya from 10.10.10.21 port 50122 ssh2
2026-10-27T10:21:40.880593+05:30 ubu01 sshd-session[2301]: Invalid user admin from 10.10.10.35 port 41870
2026-10-27T10:21:42.611087+05:30 ubu01 sshd-session[2301]: Failed password for invalid user admin from 10.10.10.35 port 41870 ssh2
2026-10-27T10:21:47.191461+05:30 ubu01 sshd-session[2305]: Invalid user oracle from 10.10.10.35 port 41884
2026-10-27T10:21:49.228870+05:30 ubu01 sshd-session[2305]: Failed password for invalid user oracle from 10.10.10.35 port 41884 ssh2
2026-10-27T10:21:55.175457+05:30 ubu01 sshd-session[2309]: Failed password for root from 10.10.10.35 port 41890 ssh2
2026-10-27T10:24:13.203523+05:30 ubu01 sshd-session[2318]: Failed password for arjun from 10.10.10.48 port 52210 ssh2
2026-10-27T10:24:20.177601+05:30 ubu01 sshd-session[2318]: Accepted password for arjun from 10.10.10.48 port 52210 ssh2
EOF
wc -l messages syslog secure auth.log

You should see this:

   8 messages
   8 syslog
   9 secure
   9 auth.log
  34 total

The messages and syslog files hold the same eight events from an app called saatapp: two payment timeouts, a warning, a lost database connection and a reconnect. The secure and auth.log files hold the same SSH logins. Only the timestamp style and host name differ, which is the whole point of the comparison.

✅ Checkpoint: wc shows 8, 8, 9 and 9 lines.

3Search with grep

Run this:

grep -i error messages
grep -c -i error syslog
grep -n -E 'ERROR|WARN' syslog
grep -v saatapp messages | wc -l

You should see this:

Oct 27 10:02:41 rocky01 saatapp[1440]: ERROR payment timeout after 30s order=10422
Oct 27 10:07:55 rocky01 saatapp[1440]: ERROR payment timeout after 30s order=10431
Oct 27 10:15:02 rocky01 saatapp[1440]: error: database connection lost (errno 2013)
3
2:2026-10-27T10:02:41.118344+05:30 ubu01 saatapp[1520]: ERROR payment timeout after 30s order=10422
3:2026-10-27T10:02:43.020551+05:30 ubu01 saatapp[1520]: WARN retry queue length 12
4:2026-10-27T10:07:55.771230+05:30 ubu01 saatapp[1520]: ERROR payment timeout after 30s order=10431
3

-i catches ERROR and error, which matters because programs aren’t consistent. -c counts matching lines, -n adds line numbers, -E lets you write 'ERROR|WARN' for “either”, and -v throws matches away: three lines in messages are not from saatapp. Now look around one hit:

grep -B 1 -A 1 'connection lost' messages

You should see this:

Oct 27 10:12:30 rocky01 chronyd[801]: Selected source 10.10.10.1
Oct 27 10:15:02 rocky01 saatapp[1440]: error: database connection lost (errno 2013)
Oct 27 10:15:04 rocky01 saatapp[1440]: INFO reconnected to database

-B 1 adds one line before and -A 1 one line after (use -C 5 for five on each side). The line after shows the app reconnected two seconds later, which changes the story you tell in the ticket.

✅ Checkpoint: You get 3 error lines, a count of 3, two ERROR and one WARN line, and the reconnect line after the lost connection.

4Find the field numbers before you awk

Print the first fields of one line from each format:

head -n 1 messages | awk '{print NF": "$1" | "$2" | "$3" | "$4" | "$5}'
head -n 1 syslog | awk '{print NF": "$1" | "$2" | "$3" | "$4" | "$5}'

You should see this:

10: Oct | 27 | 09:58:12 | rocky01 | systemd[1]:
10: 2026-10-27T09:58:12.402117+05:30 | ubu01 | systemd[1]: | Starting | apt-daily.service

On Rocky the timestamp takes three fields and the program is $5. On Ubuntu the whole timestamp is $1, the host is $2 and the program is $3. Here is the older runbook builder, written for the Rocky style, run on the Ubuntu file:

grep -i "error" syslog | awk '{print $1, $2, $3, $4, $5}'

You should see this:

2026-10-27T10:02:41.118344+05:30 ubu01 saatapp[1520]: ERROR payment
2026-10-27T10:07:55.771230+05:30 ubu01 saatapp[1520]: ERROR payment
2026-10-27T10:15:02.902113+05:30 ubu01 saatapp[1520]: error: database

No error message, only odd columns: a host name and two words from the message where you expected the program and the date. Use the right fields for each format, or keep the whole message with cut:

grep -i error syslog | cut -d' ' -f1,3-
grep -i error messages | cut -d' ' -f1-3,5-

You should see this:

2026-10-27T10:02:41.118344+05:30 saatapp[1520]: ERROR payment timeout after 30s order=10422
2026-10-27T10:07:55.771230+05:30 saatapp[1520]: ERROR payment timeout after 30s order=10431
2026-10-27T10:15:02.902113+05:30 saatapp[1520]: error: database connection lost (errno 2013)
Oct 27 10:02:41 saatapp[1440]: ERROR payment timeout after 30s order=10422
Oct 27 10:07:55 saatapp[1440]: ERROR payment timeout after 30s order=10431
Oct 27 10:15:02 saatapp[1440]: error: database connection lost (errno 2013)

Careful with cut on Rocky files in the first nine days of a month: the traditional format pads the day with a space (Oct 7), so the line has two spaces in a row and cut counts an empty field. awk treats any run of spaces as one separator and doesn’t care.

Count lines per program, dropping the PID in brackets:

awk '{sub(/\[.*/, "", $3); print $3}' syslog | sort | uniq -c | sort -rn
awk '{sub(/\[.*/, "", $5); print $5}' messages | sort | uniq -c | sort -rn

You should see this:

      5 saatapp
      2 systemd
      1 chronyd
      5 saatapp
      2 systemd
      1 chronyd

Try it yourself: build the distro-aware error search in the Command Builder.

✅ Checkpoint: Both formats give 5 saatapp, 2 systemd and 1 chronyd once you use $3 on Ubuntu and $5 on Rocky.

5Count failed logins by IP

Run this:

grep -c "Failed password" secure
grep "Failed password" secure | grep -oE 'from [0-9.]+' | sort | uniq -c | sort -rn
awk '/Failed password/ {print $(NF-3)}' auth.log | sort | uniq -c | sort -rn
grep -oE 'Invalid user [^ ]+' auth.log | sort | uniq -c

You should see this:

5
      3 from 10.10.10.35
      1 from 10.10.10.48
      1 from 10.10.10.21
      3 10.10.10.35
      1 10.10.10.48
      1 10.10.10.21
      1 Invalid user admin
      1 Invalid user oracle

grep -o prints only the part that matched, so you get “from IP” instead of the whole line. The awk version counts from the end of the line: every “Failed password” line ends with “from IP port N ssh2”, so the IP is always $(NF-3), four fields from the end, whatever the timestamp style and whether “invalid user” is in the middle. 10.10.10.35 tried three times with admin, oracle and root, which looks like someone guessing, while priya and arjun mistyped once and then got in.

Try it yourself: build the failed-login report in the Command Builder.

✅ Checkpoint: Both methods put 10.10.10.35 on top with 3 failures.

6Cut out a time window

Run this:

awk '$1 >= "2026-10-27T10:05" && $1 < "2026-10-27T10:20"' syslog
awk '$3 >= "10:05" && $3 < "10:20"' messages

You should see this:

2026-10-27T10:07:55.771230+05:30 ubu01 saatapp[1520]: ERROR payment timeout after 30s order=10431
2026-10-27T10:12:30.004512+05:30 ubu01 chronyd[790]: Selected source 10.10.10.1
2026-10-27T10:15:02.902113+05:30 ubu01 saatapp[1520]: error: database connection lost (errno 2013)
2026-10-27T10:15:04.330981+05:30 ubu01 saatapp[1520]: INFO reconnected to database
Oct 27 10:07:55 rocky01 saatapp[1440]: ERROR payment timeout after 30s order=10431
Oct 27 10:12:30 rocky01 chronyd[801]: Selected source 10.10.10.1
Oct 27 10:15:02 rocky01 saatapp[1440]: error: database connection lost (errno 2013)
Oct 27 10:15:04 rocky01 saatapp[1440]: INFO reconnected to database

RFC 3339 timestamps sort as text in time order (year, month, day, hour, minute), so a plain string comparison in awk gives you an exact window, across days too. On the Rocky format the same trick only works within one day, because “Oct 27” has no year and month names don’t sort in calendar order. For anything wider on Rocky, use journalctl with --since and --until (step 8).

✅ Checkpoint: Both commands print the same four events, from 10:07:55 to 10:15:04.

7Search rotated logs with zgrep

Make a rotated set like Ubuntu keeps, then search all of it:

cp syslog syslog.1
gzip -c syslog > syslog.2.gz
zgrep -c ERROR syslog syslog.1 syslog.2.gz
zgrep -h 'order=10431' syslog.2.gz
zcat syslog.2.gz | awk '{print $3}' | sort | uniq -c | sort -rn | head -n 1

You should see this:

syslog:2
syslog.1:2
syslog.2.gz:2
2026-10-27T10:07:55.771230+05:30 ubu01 saatapp[1520]: ERROR payment timeout after 30s order=10431
      5 saatapp[1520]:

zgrep unpacks .gz files on the fly and reads plain files as they are, so one command covers the whole set (/var/log/syslog* on a real server). -h hides the file name. For awk or anything else, zcat feeds the unpacked text into the pipe. On rocky01 the rotated files are plain text named messages-20261025 and so on, and zgrep works on them too, so a habit of zgrep never hurts.

✅ Checkpoint: zgrep finds 2 ERROR lines in each of the three files, including the compressed one.

8Ask the journal on rocky01

Write your own error line, then find it three ways:

logger -i -t saatapp -p user.err "payment timeout after 30s order=10422"
sudo journalctl -t saatapp --since "10 min ago" --no-pager
sudo journalctl -t saatapp --since "10 min ago" -o short-iso --no-pager
sudo grep saatapp /var/log/messages | tail -n 1

You should see something like this:

Oct 27 10:41:05 rocky01 saatapp[2702]: payment timeout after 30s order=10422
2026-10-27T10:41:05+05:30 rocky01 saatapp[2702]: payment timeout after 30s order=10422
Oct 27 10:41:05 rocky01 saatapp[2702]: payment timeout after 30s order=10422

logger sends a message the way any program would: -t sets the program name, -p user.err the facility and priority, and -i adds its PID. The journal has it, -o short-iso prints the date with year and zone, and rsyslog wrote the same line to /var/log/messages. Now list everything at priority err or worse since the boot:

sudo journalctl -p err -b --no-pager

You should see something like this:

Oct 27 09:01:14 rocky01 kernel: piix4_smbus 0000:00:07.0: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
Oct 27 10:41:05 rocky01 saatapp[2702]: payment timeout after 30s order=10422

On a VirtualBox guest you’ll often see a harmless kernel line like the SMBus one at boot; on real servers the first few err lines after a boot are worth reading every time. Narrow by clock time with full dates, such as --since "2026-10-27 10:00" --until "2026-10-27 10:45". -b -1 asks for the previous boot, which only works when the journal is kept on disk; check with journalctl --list-boots (Lesson 29 makes it persistent).

✅ Checkpoint: Your logger line shows up in journalctl, in short-iso format, and in /var/log/messages.

9One service’s logs: sshd

From ubu01, try to log in to rocky01 as a user that doesn’t exist, type any password once, then press Ctrl+C:

ssh -o PubkeyAuthentication=no nobody1@10.10.10.11

On rocky01, look at the sshd unit:

sudo journalctl -u sshd --since "10 min ago" --no-pager | grep -iE "failed|invalid"
sudo journalctl -k -b --no-pager | head -n 2

You should see something like this:

Oct 27 10:46:12 rocky01 sshd-session[2731]: Invalid user nobody1 from 10.10.10.21 port 46882
Oct 27 10:46:15 rocky01 sshd-session[2731]: Failed password for invalid user nobody1 from 10.10.10.21 port 46882 ssh2
Oct 27 10:46:15 rocky01 sshd-session[2731]: Invalid user nobody1 from 10.10.10.21 port 46882
Oct 27 09:01:10 rocky01 kernel: Linux version 6.12.0-211.16.1.el10_2.0.1.x86_64 (mockbuild@...) ...
Oct 27 09:01:11 rocky01 kernel: e1000 0000:00:03.0 enp0s3: renamed from eth0

-u sshd gives only the SSH service’s lines, including those from the per-connection sshd-session processes, because they run inside the same systemd unit. On ubu01 the unit is called ssh, so the same command is sudo journalctl -u ssh. -k shows only kernel messages from this boot, with real wall-clock times instead of dmesg’s seconds since boot. To watch a log live, use sudo journalctl -fu sshd or sudo tail -f /var/log/secure and stop with Ctrl+C.

Try it yourself: build “one service’s logs since a time” in the Command Builder.

✅ Checkpoint: The journal shows “Invalid user nobody1 from 10.10.10.21”, the attempt you made from ubu01.