Hardening a VPS is a sequencing problem

I spent an evening hardening a fresh Hetzner box, and at the end of it I had changed about six lines of configuration.

Six lines. PermitRootLogin no. PasswordAuthentication no. A handful of ufw rules. Every one of them is a single line I could have found in a tutorial in thirty seconds, and none of them is interesting. If the value of this post were the lines, you should close it and read the Ubuntu server guide instead.

The evening didn’t go into finding those lines. It went into deciding what order to apply them in, and making sure that at every single point along the way, I could still get back into the machine if I was wrong.

That’s the actual job. And it’s the part no tutorial teaches, because a tutorial is a list and this is a sequence.

You’re standing on the branch you’re sawing

SSH hardening has an asymmetry that almost nothing else in server config has: the thing you are securing is the thing you are using to secure it.

Get a web server config wrong and you fix the config. Get a database permission wrong and you fix the permission. The feedback loop is cheap, so the order you do things in is a matter of taste.

Get SSH wrong and you fix it by… connecting over SSH. Which you can’t do. That’s what makes ordering load-bearing here instead of stylistic. A hardened box and a bricked box are running identical configs. The only difference is the order you applied them in, and whether you were still holding a way back when you got one wrong.

”Proven” is doing all the work

The rule I followed the whole evening sounds too obvious to state:

Never disable your current way in until you’ve proven the new way works.

Everyone nods at this. Then they lock themselves out anyway, because proven is quietly carrying the entire sentence.

Proven does not mean the config looks right. It doesn’t mean sshd -t passed. It doesn’t mean you reloaded sshd and nothing errored. All of those can be true of a box you can no longer reach.

Proven means: a brand-new connection, from a fresh terminal, that lands you at a shell prompt. Nothing less counts.

And here’s the part that actually bites people. Your existing SSH session is already authenticated. It will happily survive almost anything you do to sshd — you can disable password auth, ban root, reload the daemon, and break authentication completely, and that open session will not notice or care. It was authenticated before you broke anything and it stays that way until you close it.

So your open session is not evidence that anything works. It’s your escape hatch. Those are two different jobs, and quietly confusing them is how careful people end up in the web console at midnight.

Keep the old session open because it proves nothing. It’s there to save you, not to reassure you.

The sequence

Here’s what I actually did, in order. The right-hand column is the whole point of the post — every step is followed by the specific thing that proves it before the next step is allowed to happen.

Read the right column on its own and you have the method. Every arrow between those rows is a test, and no arrow gets crossed on faith.

The last two rows are the same move that matters most: add before delete. I added the Tailscale-scoped SSH rule and proved it before removing the public one, so there was never an instant where port 22 was allowed through zero doors. Reverse those two and you learn what the Hetzner console is for. (The interface-scoping trick itself — and the hour I later lost to it looking like it had broken — is its own story.)

Three places the order actually bites

Everything above is abstract until it isn’t. Three specific spots where getting the sequence wrong locks you out:

ufw enable is a cliff. The default deny incoming and the allow 22/tcp must both already exist before you enable. ufw doesn’t warn you. It starts enforcing the instant you enable it, and if the SSH rule isn’t in place yet, it drops your live connection mid-command.

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

Those two highlighted lines are in that order for a reason, and it is not aesthetic.

Deleting ufw rules renumbers them. You delete by number, and the numbers shift under you the moment you delete one. So it’s delete, re-check, delete — never read the list once and fire twice. There are also usually two rules to remove, not one: IPv4 and IPv6 are listed separately, and deleting only the v4 rule leaves the door open in a way ufw status will happily show you if you look.

sudo ufw status numbered
sudo ufw delete 5
sudo ufw status numbered # numbers have shifted — look again
sudo ufw delete 7

SSH keys are per-user, not per-machine. “I added my key to the server” isn’t a thing that happened. What happened is your public key went into /root/.ssh/authorized_keys. A new user gets their own home directory and their own authorized_keys, and it starts empty — so key login for that user won’t just work, and SSH will fall back to a password prompt. Same key, different file. This is exactly why the sudo user gets tested before root login gets disabled: those two steps are one step apart, and in the wrong order they’re terminal.

What I didn’t do

Two pieces of standard advice I looked at and declined, which I think is more informative than the ones I took.

I didn’t change the SSH port. Moving 22 → 2222 is security through obscurity. It cuts log noise, because unsophisticated bots only scan 22 — but it stops zero real attackers, since any actual scan finds sshd in seconds regardless of port. It’s a speed bump, not a wall, and the wall is key-only auth plus a firewall. In exchange you pay a -p flag or a config entry forever. And once SSH only listens on the tailnet, the question evaporates: the public internet can’t reach it on any port.

I didn’t block outbound. Plenty of tutorials tell you to lock down inbound and outbound, and it sounds thorough. It’s the one piece of hardening advice here with real teeth, and they point at you. Tailscale works by dialing out and holding a connection open — it never waits for an inbound connection on a fixed port. That’s exactly why it needed no firewall rule at all. A Cloudflare tunnel, if I add one later, works the same way. Block outbound and you cut the phone line both of them depend on, and they die together, and now you’re in the console.

What’s still missing

The box is hardened. It is not finished, and I’d rather say what’s on the list than imply otherwise.

unattended-upgrades isn’t set up. This is the real gap. I rebooted for a kernel upgrade once, by hand, because I happened to be looking at the box that evening. Nothing currently keeps it patched. Every hardening step in this post is about reachability — who can knock on which door — and none of it does anything about a vulnerability in software that’s already running with the door legitimately open. Automatic security updates are probably worth more than everything above combined, and they’re the next thing I do.

Tailnet ACLs aren’t scoped. The ufw rule trusts the tailscale0 interface, which means it trusts every device on my tailnet. Today that’s a laptop, so the blast radius is one machine. But that boundary is silent and it moves on its own: add a phone, add a CI runner with an ephemeral key, and each one quietly becomes a thing that can reach SSH. Tailscale ACLs are how you scope that back down to specific devices, and I haven’t written any.

fail2ban isn’t installed. In fairness to it, right now it would have nothing to do — its SSH jail watches for brute-force against a daemon the public internet can’t reach, and its web jails want an app with auth failures I haven’t deployed. It becomes worth something once there’s a public-facing app on this box with a login on it. I want it there for that, not for SSH.

The thing underneath all of it

None of this sequencing discipline is what actually keeps you safe from lockout. It just keeps you from needing the thing that does.

Right now, if Tailscale went down, I could not SSH into this machine at all. Port 22 only answers on tailscale0, and public 22 is closed at both ufw and Hetzner’s firewall. The SSH daemon is running fine, waiting patiently, completely unreachable.

That’s not a flaw. It’s a known state with a plan, and the plan rests on the Hetzner web console — the virtual keyboard and monitor that plugs into the machine underneath SSH, underneath the firewall, underneath all networking. It bypasses every layer in this post. It always works.

Which reframes what the keys were ever doing. Your SSH keys don’t guarantee you’re never locked out. They guarantee that once you can reach the door, you get in. The console guarantees you can always reach a door. Those are different promises, and only one of them is unconditional.

So the recovery plan is two branches and it fits in a sentence: get in via the console, then either revive Tailscale, or temporarily re-open public 22, sort it out, and close it again.

Everything else is just deciding which door to reopen.