DNS and name resolution problems on Linux

Take snapshots of both VMs first (VBoxManage snapshot rocky01 take before-l76 and the same for ubu01). rocky01 becomes a small DNS server for one lab zone, apps.saatlab.test, using dnsmasq, and ubu01 sends only that zone to it. Everything is removed again at the end of the Break-It section, so both VMs go back to how they were.

1Build a lab DNS server on rocky01

On rocky01, install dnsmasq, give it the lab zone and open port 53:

sudo dnf install -y dnsmasq
sudo tee /etc/dnsmasq.d/saatlab-apps.conf >/dev/null <<'EOF'
# Lab DNS server for apps.saatlab.test (SaatVedha Linux course, Lesson 76)
listen-address=10.10.10.11
no-resolv
no-hosts
server=10.10.10.1
local=/apps.saatlab.test/
host-record=shop.apps.saatlab.test,10.10.10.11
host-record=api.apps.saatlab.test,10.10.10.21
local-ttl=300
log-queries
EOF
sudo dnsmasq --test
sudo systemctl enable --now dnsmasq
sudo firewall-cmd --permanent --add-service=dns
sudo firewall-cmd --reload
dig @10.10.10.11 +short shop.apps.saatlab.test

You should see something like this:

dnsmasq: syntax check OK.
Created symlink '/etc/systemd/system/multi-user.target.wants/dnsmasq.service' → '/usr/lib/systemd/system/dnsmasq.service'.
success
success
10.10.10.11

local=/apps.saatlab.test/ makes dnsmasq the final word for that zone: names it has get an answer, names it doesn’t have get NXDOMAIN, and nothing in the zone is forwarded. Everything else goes to server=10.10.10.1, VirtualBox’s DNS. no-hosts stops it from also serving rocky01’s /etc/hosts, local-ttl=300 gives the records a five-minute TTL so caches matter, and log-queries writes every query to the journal. The firewalld dns service opens 53 over both UDP and TCP; DNS needs TCP for large answers.

✅ Checkpoint: dig @10.10.10.11 +short shop.apps.saatlab.test prints 10.10.10.11 on rocky01.

2Ask the server directly from ubu01

On ubu01, ask rocky01 three questions and keep only the useful lines:

for n in shop.apps.saatlab.test nope.apps.saatlab.test one.one.one.one; do
  dig @10.10.10.11 $n | grep -E '^;; (->>HEADER|flags|SERVER)|^[^;].*IN[[:space:]]+A'
done

You should see something like this:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2231
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
shop.apps.saatlab.test.	300	IN	A	10.10.10.11
;; SERVER: 10.10.10.11#53(10.10.10.11) (UDP)
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 18164
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; SERVER: 10.10.10.11#53(10.10.10.11) (UDP)
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14646
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
one.one.one.one.	243	IN	A	1.0.0.1
one.one.one.one.	243	IN	A	1.1.1.1
;; SERVER: 10.10.10.11#53(10.10.10.11) (UDP)

Read the status and the flags. shop is NOERROR with the aa flag, because dnsmasq owns that name. nope is NXDOMAIN with no answer: the server is certain the name doesn’t exist. one.one.one.one is NOERROR without aa, forwarded to 10.10.10.1 and passed back; its TTL is whatever the real record had left. ra (recursion available) says this server will chase names for you.

✅ Checkpoint: You get NOERROR with aa for shop, NXDOMAIN for nope, and NOERROR without aa for one.one.one.one.

3Send only the lab zone to rocky01 (split DNS)

First prove that ubu01 can’t resolve the zone on its own:

getent hosts shop.apps.saatlab.test; echo "exit $?"
dig shop.apps.saatlab.test | grep HEADER

You should see something like this:

exit 2
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 40211

Now add a DNS delegate for the zone and restart systemd-resolved:

sudo mkdir -p /etc/systemd/dns-delegate.d
sudo tee /etc/systemd/dns-delegate.d/apps-saatlab.dns-delegate >/dev/null <<'EOF'
[Delegate]
DNS=10.10.10.11
Domains=~apps.saatlab.test
EOF
sudo systemctl restart systemd-resolved
resolvectl status

You should see something like this:

Global
           Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: stub

Link 2 (enp0s3)
      Current Scopes: DNS
           Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  Current DNS Server: 10.10.10.1
         DNS Servers: 10.10.10.1
          DNS Domain: saatlab.test

Delegate apps-saatlab
         DNS Servers: 10.10.10.11
          DNS Domain: ~apps.saatlab.test
       Default Route: no

A delegate is a third kind of scope next to Global and the links, added in systemd 258, which Ubuntu 26.04 has. The ~ in front of the domain makes it a routing domain only, not a search domain, and Default Route: no means the delegate gets nothing but names under apps.saatlab.test. On older Ubuntu releases you’d get the same effect with DNS= and Domains=~apps.saatlab.test on a link in netplan, or a drop-in in /etc/systemd/resolved.conf.d.

✅ Checkpoint: resolvectl status shows a “Delegate apps-saatlab” section with DNS Servers 10.10.10.11 and DNS Domain ~apps.saatlab.test.

4Ask the same question four ways

On ubu01, run this:

getent hosts shop.apps.saatlab.test
resolvectl query shop.apps.saatlab.test
resolvectl query -4 one.one.one.one
nslookup shop.apps.saatlab.test

You should see something like this:

10.10.10.11     shop.apps.saatlab.test
shop.apps.saatlab.test: 10.10.10.11

-- Information acquired via protocol DNS in 2.4ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network
one.one.one.one: 1.1.1.1                                    -- link: enp0s3
                 1.0.0.1                                    -- link: enp0s3

-- Information acquired via protocol DNS in 21.7ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	shop.apps.saatlab.test
Address: 10.10.10.11

getent, the application’s view, now gets 10.10.10.11. resolvectl query adds what the others hide: shop came from the network with no link shown, because the delegate answered, while one.one.one.one came through the enp0s3 link and its server 10.10.10.1. nslookup asked the stub at 127.0.0.53 and calls the answer non-authoritative: the stub sets the aa flag only on answers it makes up itself, never on answers it fetched from a server.

Try it yourself: build the name-resolution check for Rocky or Ubuntu in the Command Builder.

✅ Checkpoint: All four tools return 10.10.10.11 for shop.apps.saatlab.test, and resolvectl shows one.one.one.one going through enp0s3.

5See search domains and ndots in the server’s log

On ubu01, look up the same full name with ndots:5 and then with the default, using two glibc environment variables that override resolv.conf for one command. The flushes make sure every lookup reaches the server instead of resolved’s cache:

sudo resolvectl flush-caches
LOCALDOMAIN='apps.saatlab.test saatlab.test' RES_OPTIONS='ndots:5' getent ahostsv4 shop.apps.saatlab.test | head -n 1
sudo resolvectl flush-caches
LOCALDOMAIN='apps.saatlab.test saatlab.test' getent ahostsv4 shop.apps.saatlab.test | head -n 1
sudo resolvectl flush-caches
LOCALDOMAIN='apps.saatlab.test saatlab.test' getent ahostsv4 shop | head -n 1

You should see something like this:

10.10.10.11     STREAM shop.apps.saatlab.test
10.10.10.11     STREAM shop.apps.saatlab.test
10.10.10.11     STREAM shop.apps.saatlab.test

Then on rocky01, read what dnsmasq received in the last minute:

sudo journalctl -u dnsmasq --since "1 min ago" -o cat | grep -E 'query\[|config'

You should see something like this:

query[A] shop.apps.saatlab.test.apps.saatlab.test from 10.10.10.21
config shop.apps.saatlab.test.apps.saatlab.test is NXDOMAIN
query[A] shop.apps.saatlab.test from 10.10.10.21
config shop.apps.saatlab.test is 10.10.10.11
query[A] shop.apps.saatlab.test from 10.10.10.21
config shop.apps.saatlab.test is 10.10.10.11
query[A] shop.apps.saatlab.test from 10.10.10.21
config shop.apps.saatlab.test is 10.10.10.11

Same answer three times, different work behind it. shop.apps.saatlab.test has three dots; with ndots:5 that counts as “short”, so glibc tried shop.apps.saatlab.test.apps.saatlab.test first (NXDOMAIN, logged here), then shop.apps.saatlab.test.saatlab.test (sent to 10.10.10.1, so not in this log), and the real name last. With the default ndots:1 it went straight to the real name. The short name shop was completed with the first search domain and found in one query.

Kubernetes pods use ndots:5 with three or four search domains, so every lookup of an outside name like an API host costs several wasted queries, each one for A and AAAA. Write a trailing dot (api.example.com.) or lower ndots for that workload when it matters.

✅ Checkpoint: The dnsmasq log shows the extra .apps.saatlab.test query only for the ndots:5 lookup.

6Watch the cache, then flush it

On ubu01, ask twice and look at the source and the TTL:

resolvectl query shop.apps.saatlab.test
dig +noall +answer shop.apps.saatlab.test

You should see something like this:

shop.apps.saatlab.test: 10.10.10.11

-- Information acquired via protocol DNS in 214us.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: cache
shop.apps.saatlab.test.	251	IN	A	10.10.10.11

On rocky01, move shop to a new address:

sudo sed -i 's/^host-record=shop.apps.saatlab.test,10.10.10.11$/host-record=shop.apps.saatlab.test,10.10.10.21/' /etc/dnsmasq.d/saatlab-apps.conf
sudo systemctl restart dnsmasq

Back on ubu01, compare the server with what programs get:

dig @10.10.10.11 +short shop.apps.saatlab.test
resolvectl query shop.apps.saatlab.test
getent hosts shop.apps.saatlab.test

You should see something like this:

10.10.10.21
shop.apps.saatlab.test: 10.10.10.11

-- Information acquired via protocol DNS in 198us.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: cache
10.10.10.11     shop.apps.saatlab.test

The server already says 10.10.10.21, yet every program on ubu01 still gets 10.10.10.11 from resolved’s cache, and will until the 300-second TTL runs out. That’s normal DNS behaviour, and it’s why a migration plan lowers the TTL a day before an address changes. When you can’t wait, flush:

sudo resolvectl flush-caches
resolvectl query shop.apps.saatlab.test
getent hosts shop.apps.saatlab.test

You should see something like this:

shop.apps.saatlab.test: 10.10.10.21

-- Information acquired via protocol DNS in 2.1ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network
10.10.10.21     shop.apps.saatlab.test

rocky01 has no local cache, so a lookup there sees a change at once. Java applications often keep their own cache inside the JVM, which no flush on the server touches; restarting the app or setting networkaddress.cache.ttl is the fix there.

✅ Checkpoint: After flush-caches, resolvectl says “Data from: network” and getent returns 10.10.10.21.

7Tell NXDOMAIN, SERVFAIL and a timeout apart

On ubu01, run one of each:

getent hosts nope.apps.saatlab.test; echo "exit $?"
python3 -c 'import socket; socket.getaddrinfo("nope.apps.saatlab.test", 80)' 2>&1 | tail -n 1
curl -sS http://nope.apps.saatlab.test/
dig @10.10.10.99 shop.apps.saatlab.test
dig @1.1.1.1 dnssec-failed.org | grep HEADER
dig @1.1.1.1 +cd dnssec-failed.org | grep HEADER

You should see something like this (the dig to 10.10.10.99 waits several seconds):

exit 2
socket.gaierror: [Errno -2] Name or service not known
curl: (6) Could not resolve host: nope.apps.saatlab.test
;; communications error to 10.10.10.99#53: timed out
;; communications error to 10.10.10.99#53: timed out
;; communications error to 10.10.10.99#53: timed out

; <<>> DiG 9.20.24-1ubuntu0.4-Ubuntu <<>> @10.10.10.99 shop.apps.saatlab.test
; (1 server found)
;; global options: +cmd
;; no servers could be reached
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 51520
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7714

NXDOMAIN reaches programs as “Name or service not known” (errno -2): fix the name or add the record. “no servers could be reached” after “timed out” means nobody answered on port 53: wrong IP, dead server or a firewall. A program sees that, and SERVFAIL, as “Temporary failure in name resolution” (errno -3).

dnssec-failed.org is a test domain kept broken on purpose. 1.1.1.1 validates DNSSEC, the signatures don’t check out, so it answers SERVFAIL; the same query with +cd (checking disabled) gets NOERROR, which proves the data exists and the failure is validation. If your network blocks outside DNS servers, read along; at work, SERVFAIL from your own resolver usually means its upstream or the domain’s name servers are broken, and dig +trace is the next step.

✅ Checkpoint: You can name the status and the program message for a missing name, an unreachable server and a failed validation.

8Follow a name from the root with dig +trace

On ubu01, run this:

dig +trace +nodnssec wikipedia.org

You should see something like this (shortened; addresses depend on where you are):

; <<>> DiG 9.20.24-1ubuntu0.4-Ubuntu <<>> +trace +nodnssec wikipedia.org
;; global options: +cmd
.			86400	IN	NS	a.root-servers.net.
.			86400	IN	NS	b.root-servers.net.
[... 11 more root servers ...]
;; Received 239 bytes from 127.0.0.53#53(127.0.0.53) in 12 ms

org.			172800	IN	NS	a0.org.afilias-nst.info.
org.			172800	IN	NS	a2.org.afilias-nst.info.
[... 4 more .org servers ...]
;; Received 449 bytes from 199.7.83.42#53(l.root-servers.net) in 36 ms

wikipedia.org.		86400	IN	NS	ns0.wikimedia.org.
wikipedia.org.		86400	IN	NS	ns1.wikimedia.org.
wikipedia.org.		86400	IN	NS	ns2.wikimedia.org.
;; Received 177 bytes from 199.19.56.1#53(a0.org.afilias-nst.info) in 48 ms

wikipedia.org.		600	IN	A	103.102.166.224
;; Received 58 bytes from 208.80.154.238#53(ns0.wikimedia.org) in 214 ms

+trace ignores caches and does what a resolver does: it gets the root servers from your stub, asks a root server who handles .org, asks a .org server who handles wikipedia.org, and asks that domain’s own server for the address. Each “Received” line shows which server answered and how fast. When a domain fails for everyone, the step where the trace stops, or where the NS names point at servers that don’t answer, is the domain owner’s problem and not yours.

Through VirtualBox NAT this needs outbound UDP 53 from your laptop; some office networks block it.

✅ Checkpoint: Your trace reaches an A record for wikipedia.org from one of the ns0, ns1 or ns2.wikimedia.org servers.