Master Linux Security Hardening: What Every DevOps Engineer Should Actually Do

Introduction

Let me be honest with you. When I spun up my first production Linux server years ago, I did exactly what most engineers do — I installed the application, opened the ports, and called it a day. Firewall? I'll get to it. SSH hardening? The default settings are probably fine. Fail2ban? Sounds complicated, maybe later.

Three weeks later, my server was part of a botnet sending spam emails. The attacker had brute-forced a weak SSH password on port 22. That was the moment I stopped treating security as an afterthought.

Here is the uncomfortable truth: a fresh Ubuntu or Debian server is not secure out of the box. It is a starting point. The default configuration trusts everything, allows root SSH logins, runs services you do not need, and has no rate limiting on login attempts. If you ship your application on a fresh install and move on, you are one automated scan away from a very bad day.

This guide covers the actual hardening steps I apply to every Linux server I manage — not a theoretical checklist, but real commands with real explanations. We are going to talk about SSH hardening, user management, firewall rules, automatic updates, intrusion detection, and kernel-level protections. By the end, your server will be meaningfully harder to attack.

Step 1: Disable Root Login and Lock Down SSH

The root account is the first thing attackers go after. Every automated scanner on the internet is trying root with common passwords on port 22 right now — not hypothetically, literally right now. The first thing you do on any new server is take root SSH access off the table entirely.

Before you change anything, make sure you have a non-root sudo user created:

adduser devopsuser
usermod -aG sudo devopsuser

Now open the SSH daemon configuration:

sudo nano /etc/ssh/sshd_config

Change or add these lines:

PermitRootLogin no
MaxAuthTries 3
LoginGraceTime 20
AllowUsers devopsuser
Protocol 2
Port 2222

That last line — Port 2222 — changes SSH off the default port. This alone eliminates a massive amount of automated noise from your auth logs. It is not a substitute for real security, but it cuts your attack surface from millions of automated attempts down to a much smaller number of targeted ones.

Restart SSH to apply changes. Do this carefully — if you lock yourself out, you will need console access to recover:

sudo systemctl restart sshd

Step 2: Set Up Key-Based Authentication and Remove Password Login

Passwords are guessable. SSH keys are not. This is the single most impactful change you can make to your server's SSH security, and it takes about five minutes to set up. Once you have key-based auth working, you turn off password authentication entirely — no exceptions.

On your local machine, generate an ED25519 key pair (better than RSA for modern servers):

ssh-keygen -t ed25519 -C "devops-server-key"

Copy your public key to the server:

ssh-copy-id -p 2222 devopsuser@your_server_ip

Test the key-based login before you do anything else:

ssh -p 2222 devopsuser@your_server_ip

Once confirmed working, go back into /etc/ssh/sshd_config and disable password authentication completely:

PasswordAuthentication no
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
ChallengeResponseAuthentication no

sudo systemctl restart sshd

Step 3: Configure UFW Firewall Properly

UFW is Ubuntu's simplified firewall wrapper around iptables, and it is genuinely good for server management. The default approach most people take is wrong though — they enable UFW and then open ports one by one as they need them. The correct approach is: deny everything by default, then explicitly allow only what needs to be allowed.

Set the default policies first:

sudo ufw default deny incoming
sudo ufw default allow outgoing

Allow your new SSH port — do this before enabling UFW or you will lock yourself out:

sudo ufw allow 2222/tcp

Allow only what your application actually needs. For a typical web server:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

If you have a database server on a separate machine, allow that specific private IP only:

sudo ufw allow from 10.0.0.5 to any port 5432

Enable and verify:

sudo ufw enable
sudo ufw status verbose

One thing I see regularly in team environments is people opening ports with sudo ufw allow 3306 during debugging and forgetting to close them afterward. Make it a habit to audit your firewall rules every time you do a deployment review. What is open should match exactly what your architecture diagram says should be open.

Step 4: Install and Configure Fail2ban

Even with key-based auth and a non-standard SSH port, you will still get people poking at your server. Fail2ban watches your log files and automatically bans IP addresses that show signs of brute-force behavior. It is lightweight, runs quietly in the background, and has saved me more times than I can count.

sudo apt install fail2ban -y

Create a local configuration file rather than editing the default — this survives package updates:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo nano /etc/fail2ban/jail.local

Set these values in the [DEFAULT] section:

[DEFAULT]
bantime  = 3600
findtime = 600
maxretry = 3
ignoreip = 127.0.0.1/8 your_home_ip

Add the SSH jail configuration — make sure the port matches your custom SSH port:

[sshd]
enabled  = true
port     = 2222
logpath  = /var/log/auth.log
maxretry = 3
bantime  = 86400

sudo systemctl enable fail2ban
sudo systemctl restart fail2ban

Check who is currently banned and what jails are active:

sudo fail2ban-client status
sudo fail2ban-client status sshd

Step 5: Enable Automatic Security Updates

Unpatched packages are one of the most common entry points for real-world server compromises. The conversation I always have with teams is this: yes, automatic updates can occasionally cause issues. But the risk of running known-vulnerable packages for weeks because nobody remembered to run apt upgrade is far worse. Enable unattended upgrades for security patches and keep your system current.

sudo apt install unattended-upgrades -y
sudo dpkg-reconfigure --priority=low unattended-upgrades

Edit the configuration to be explicit about what gets auto-updated:

sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

Make sure these lines are active:

Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "false";

I keep Automatic-Reboot set to false for production servers — kernel updates require a reboot, and I want that scheduled during a maintenance window, not triggered automatically at 3am. You will get a notification that a reboot is needed, and you can schedule it properly.

Step 6: Harden Kernel Parameters with sysctl

The Linux kernel exposes a set of runtime parameters through /proc/sys/ that control how the system handles network traffic, memory, and inter-process communication. A lot of these defaults are set for compatibility, not security. Tightening them takes five minutes and meaningfully reduces your exposure to network-level attacks.

sudo nano /etc/sysctl.d/99-security.conf

Add the following:

# Disable IP forwarding (unless this server is a router)
net.ipv4.ip_forward = 0
 
# Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1
 
# Enable SYN flood protection
net.ipv4.tcp_syncookies = 1
 
# Disable source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
 
# Ignore redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
 
# Log suspicious packets
net.ipv4.conf.all.log_martians = 1
 
# Protect against time-wait assassination
net.ipv4.tcp_rfc1337 = 1

Apply the changes immediately without a reboot:

sudo sysctl --system

Step 7: Audit Running Services and Remove What You Don't Need

Every service running on your server is an attack surface. A fresh Ubuntu install often has services running that you will never use — Avahi daemon, cups (print server), and others that make zero sense on a headless cloud server. List everything that is running and kill what does not belong there.

sudo systemctl list-units --type=service --state=running

For each service you do not recognize or need, stop it and disable it from starting at boot:

sudo systemctl stop avahi-daemon
sudo systemctl disable avahi-daemon
 
sudo systemctl stop cups
sudo systemctl disable cups

Also check what ports are actually listening and cross-reference with what you expect:

sudo ss -tlnp

If you see a port listening that you did not put there, find out what process owns it before you do anything else. Sometimes it is benign — sometimes it is something that was installed as a dependency and quietly opened a port. Either way, you want to know.

Step 8: Set Up Auditd for Login and File Access Logging

When something goes wrong on a server — and eventually something will — you want logs that actually tell you what happened and when. Auditd is the Linux audit framework that gives you detailed logging of logins, privilege escalations, and sensitive file access. Without it, you are often left guessing during incident response.

sudo apt install auditd -y
sudo systemctl enable auditd
sudo systemctl start auditd

Add rules to watch critical files and actions:

sudo nano /etc/audit/rules.d/audit.rules

# Log all authentication events
-w /var/log/auth.log -p wa -k auth_log
 
# Watch sudoers file changes
-w /etc/sudoers -p wa -k sudoers_change
 
# Log passwd file changes
-w /etc/passwd -p wa -k passwd_change
 
# Track use of the sudo command
-w /usr/bin/sudo -p x -k sudo_use
 
# Capture failed login attempts
-a always,exit -F arch=b64 -S open -F exit=-EACCES -k access_denied

sudo systemctl restart auditd

Search audit logs for specific events:

sudo ausearch -k sudo_use
sudo ausearch -k auth_log --start today

Common Security Mistakes in Production

Frequently Asked Questions

Do I need to do all of this for a personal or hobby project?

Honestly? Most of it, yes. The SSH hardening and firewall rules take under thirty minutes and protect you from the automated noise that hits every server with a public IP. Whether it is a personal project or a Fortune 500 deployment, the threat actors scanning for weak passwords do not care. At minimum — disable root login, use SSH keys, and run a firewall.

Will changing the SSH port actually improve security?

It is not real security through obscurity in the traditional sense — a targeted attacker will port scan and find your SSH port quickly. What it does is eliminate the enormous volume of automated brute-force bots that only try port 22. Your auth logs will go from thousands of failed attempts per day to near silence. Less noise means real anomalies are easier to spot.

How often should I review these security settings?

Every time your server's purpose or architecture changes, review the firewall rules. Every time a new team member gets access, audit the authorized SSH keys. Run a full security review — check services, ports, audit rules, and update status — at least every quarter. Set a recurring calendar event. It is one of those things that never feels urgent until suddenly it is the only thing that matters.

Should I also set up something like CrowdSec or OSSEC?

For a single server, Fail2ban plus Auditd is a solid baseline. Once you are managing more than three or four servers, a centralized intrusion detection system like CrowdSec or OSSEC starts making real sense. The main value is correlated log analysis across servers — an attacker probing multiple machines shows up as a pattern in aggregate even if each individual server only sees a small number of attempts.

Conclusion

None of this is exotic. These are not advanced security concepts that require a dedicated security team. SSH key authentication, a properly configured firewall, Fail2ban, and automatic security updates — these are table stakes for any Linux server that faces the internet. The reason servers get compromised is almost never a sophisticated zero-day attack. It is usually a default password, a forgotten open port, or a package that stopped receiving patches six months ago.

The mindset shift that actually made me a better infrastructure engineer was treating security as part of the initial setup, not something to layer on later. A hardened server takes maybe two hours to configure from a fresh install. Recovering from a compromised server — rebuilding, rotating credentials, auditing what data was accessed, notifying users — takes days, and the reputational cost can last much longer.

Apply these steps in order, test each one before moving to the next, and document what you have done so the next engineer on your team knows the baseline they are working with. Security is a team discipline, not a solo task — and it starts with a server that was locked down properly from day one.