Can’t SSH into the server: refused, timeout, permission denied

You’ll play client from ubu01 (and its ~/.ssh/config from Lesson 33, where rocky01 means student at 10.10.10.11 with your key) and server work on rocky01’s console. Snapshot both first (VBoxManage snapshot rocky01 take before-l72 and the same for ubu01). Every change here is undone by the end, so both VMs finish exactly as Lesson 33 left them: key-only SSH on port 22, firewalld on rocky01, ufw on ubu01.

1Tell the three failures apart

On ubu01, run these three. The first works; the other two are the errors you must recognise:

ssh rocky01 hostname
nc -zv 10.10.10.11 22
ssh -p 2222 rocky01 hostname
nc -zv 10.10.10.11 2222

You should see something like this:

rocky01
Connection to 10.10.10.11 22 port [tcp/ssh] succeeded!
ssh: connect to host 10.10.10.11 port 2222: Connection refused
nc: connect to 10.10.10.11 port 2222 (tcp) failed: Connection refused

Port 22 answers, so SSH is healthy. Port 2222 has nothing listening, so both ssh and nc say “Connection refused” straight away. That instant refusal means the host is reachable and the port is simply closed, which is a very different problem from a hang. nc -zv is the quick “is the port open” test before you even start ssh.

Try it yourself: build an nc or /dev/tcp port test in the Command Builder.

✅ Checkpoint: You can see “Connection refused” for the closed port and a success for port 22.

2Prove the server is listening

On rocky01’s console (or over SSH while it still works), run this:

sudo ss -tlnp '( sport = :22 )'
systemctl is-active sshd

You should see something like this:

State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port  Process
LISTEN  0       128            0.0.0.0:22          0.0.0.0:*      users:(("sshd",pid=905,fd=3))
LISTEN  0       128               [::]:22             [::]:*      users:(("sshd",pid=905,fd=4))
active

sshd is listening on 0.0.0.0:22 (all IPv4) and [::]:22 (all IPv6), and the service is active. If ss showed nothing on 22, a client would get “connection refused”, and you’d look at systemctl status sshd next. On Ubuntu the equivalent check is systemctl status ssh.socket, because the socket holds the port until a connection arrives.

✅ Checkpoint: ss shows sshd on port 22 and systemctl says active.

3Check the firewall on rocky01

Run this:

sudo firewall-cmd --list-all

You should see something like this:

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3
  sources:
  services: dhcpv6-client http ntp ssh
  ports: 8080/tcp 10050/tcp 10051/tcp
  forward: yes
  masquerade: no

The ssh service is in the active zone, which is why port 22 is allowed. If someone removed it, a client would see a timeout (if firewalld drops) or “No route to host” (if it rejects), not a refusal. On Ubuntu this is sudo ufw status, which should list OpenSSH as ALLOW.

✅ Checkpoint: firewalld’s public zone lists the ssh service.

4Move SSH to port 2222 the right way (Rocky)

Keep this rocky01 session open and the console window ready: a mistake here can close SSH. You’ll put it back in the next step.

On rocky01, teach SELinux the port, open the firewall, add a drop-in, test and restart:

sudo semanage port -a -t ssh_port_t -p tcp 2222
sudo firewall-cmd --permanent --add-port=2222/tcp
sudo firewall-cmd --reload
echo 'Port 22' | sudo tee /etc/ssh/sshd_config.d/50-port.conf
echo 'Port 2222' | sudo tee -a /etc/ssh/sshd_config.d/50-port.conf
sudo sshd -t && sudo systemctl restart sshd
sudo ss -tlnp '( sport = :2222 )' | tail -n 1

You should see something like this (last line):

LISTEN  0  128  0.0.0.0:2222  0.0.0.0:*  users:(("sshd",pid=3140,fd=3))

Keeping both Port 22 and Port 2222 means your current session survives while you test the new one. From ubu01, check the new port works:

ssh -p 2222 rocky01 hostname

You should see:

rocky01

The order matters: semanage port first, or sshd fails to bind with “Permission denied” even though the firewall and config are right. Confirm SELinux learned it with sudo semanage port -l | grep ssh_port_t, which now shows 2222, 22.

✅ Checkpoint: ssh -p 2222 to rocky01 prints rocky01, and semanage lists 2222 under ssh_port_t.

5Put rocky01 back to port 22 only

Undo the change so later lessons work:

sudo rm /etc/ssh/sshd_config.d/50-port.conf
sudo sshd -t && sudo systemctl restart sshd
sudo firewall-cmd --permanent --remove-port=2222/tcp
sudo firewall-cmd --reload
sudo semanage port -d -p tcp 2222
sudo ss -tlnp '( sport = :22 )' | tail -n 1

You should see sshd back on 22 only:

LISTEN  0  128  [::]:22  [::]:*  users:(("sshd",pid=3190,fd=4))

Removing the drop-in and restarting takes sshd back to the default 22 from Rocky’s own 50-redhat.conf. You also removed the firewall port and the SELinux port, so nothing is left behind. ssh -p 2222 rocky01 should now say “Connection refused” again.

✅ Checkpoint: rocky01 listens only on 22 again, and the 2222 firewall and SELinux entries are gone.

6Change the port on Ubuntu, where the socket owns it

On ubu01, look at the socket, then override its port:

systemctl cat ssh.socket | head -n 12
sudo systemctl edit ssh.socket

In the editor, add these lines in the blank area at the top, then save:

[Socket]
ListenStream=
ListenStream=2222

The empty ListenStream= clears the inherited port 22, and the next line sets 2222. Apply it and check:

sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -tlnp '( sport = :2222 )'
nc -zv localhost 2222

You should see something like this:

State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0      4096         0.0.0.0:2222      0.0.0.0:*     users:(("systemd",pid=1,fd=3))
LISTEN 0      4096            [::]:2222         [::]:*     users:(("systemd",pid=1,fd=4))
Connection to localhost 2222 port [tcp/*] succeeded!

Notice the process is systemd, not sshd: with socket activation systemd holds the port and starts sshd only when a connection arrives. That is why editing sshd_config alone and restarting ssh.service would not change the listening port. Because ufw allows only OpenSSH (port 22), reaching 2222 from another machine would also need sudo ufw allow 2222/tcp; the loopback check above skips ufw, so it just proves the socket binds.

Editing Port 2222 in a sshd_config.d drop-in and running daemon-reload works too, because a systemd generator reads sshd_config and rebuilds the socket. Editing the socket directly is the clearer way to see what owns the port.

Put ubu01 back:

sudo rm /etc/systemd/system/ssh.socket.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -tlnp '( sport = :22 )' | tail -n 1

LISTEN 0  4096  [::]:22  [::]:*  users:(("systemd",pid=1,fd=4))

✅ Checkpoint: ubu01 listened on 2222 through ssh.socket, then returned to 22 after you removed the override.

7Read ssh -vvv like a log

From ubu01, watch a healthy login in full detail (the output is long; these are the lines that matter):

ssh -vvv rocky01 true 2>&1 | grep -E 'Connecting|Connection established|host key|Offering|Server accepts|Authenticated|Authentications that can'

You should see something like this:

debug1: Connecting to 10.10.10.11 [10.10.10.11] port 22.
debug1: Connection established.
debug1: Authenticating to 10.10.10.11:22 as 'student'
debug1: Server host key: ssh-ed25519 SHA256:Q8m2kTnV0yJcR5aP1sXw4LhE9uDfG7bZ3oNqY6iKtUc
debug1: Host '10.10.10.11' is known and matches the ED25519 host key.
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering public key: /home/student/.ssh/id_ed25519 ED25519 SHA256:wRxHH/xEhT5QGNr7W2TW2Ljua2Qgpg31kVxviDxCfsE
debug1: Server accepts key: /home/student/.ssh/id_ed25519 ED25519 SHA256:wRxHH/xEhT5QGNr7W2TW2Ljua2Qgpg31kVxviDxCfsE
debug1: Authenticated to 10.10.10.11 ([10.10.10.11]:22) using "publickey".

Read it top down: the connection is made (so not refused or timed out), the host key matches known_hosts (so no “changed” warning), the key is offered and the server accepts it. When SSH fails, the last line before it stops tells you the layer: no “Connection established” means the path, no “Server accepts key” means authentication. Pair it with sudo journalctl -u sshd -n 20 on the server, which names the real reason.

✅ Checkpoint: You can point at the line where authentication succeeds in the -vvv output.

8Block yourself with AllowUsers, then fix it

Keep the console window ready; this step refuses SSH logins on purpose.

On rocky01, allow only a user that isn’t you, and restart:

echo 'AllowUsers arjun' | sudo tee /etc/ssh/sshd_config.d/60-allow.conf
sudo sshd -t && sudo systemctl restart sshd

From ubu01, a login now fails even though your key is correct:

ssh rocky01 hostname

student@10.10.10.11: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

The client just says “Permission denied (publickey)”, the same message a bad key gives, which is why this one wastes so much time. The server log is clear:

sudo journalctl -u sshd -n 2 --no-pager

Oct 27 12:41:03 rocky01 sshd-session[3301]: User student from 10.10.10.21 not allowed because not listed in AllowUsers
Oct 27 12:41:03 rocky01 sshd-session[3301]: Connection closed by invalid user student 10.10.10.21 port 51822 [preauth]

Fix: remove the drop-in and restart. Note that /etc/hosts.deny plays no part, OpenSSH ignores it.

sudo rm /etc/ssh/sshd_config.d/60-allow.conf
sudo sshd -t && sudo systemctl restart sshd
ssh rocky01 hostname

rocky01

While you’re in sshd settings, UseDNS is worth knowing: it defaults to off (commented out as #UseDNS no), but if someone sets it yes and the server’s DNS is slow, every login pauses for several seconds while sshd looks up the client’s name. A sudden “SSH is slow to log in” with no other change is often UseDNS yes plus a broken resolver.

✅ Checkpoint: Key login works again, and you can explain why “Permission denied” needed the server log to diagnose.