Locking the Gate: Practical Infrastructure Hardening for Production Linux

The moment a fresh Linux server is provisioned on a public cloud subnet or bare-metal edge facility, the clock starts ticking. Telemetry from project Honeypot, Censys, and GreyNoise reveals that an unfirewalled IPv4 address is probed by automated port scanners, credential-stuffing engines, and botnets within 60 to 90 seconds of coming online.

Stock operating system images are intentionally configured for low friction and developer convenience, not resilience under siege. Default packages ship with permissive permissions, open loopback interfaces, shared memory mounted with executable rights, and kernel debugging hooks left wide open.

“Locking the gate” is the operational discipline of systematically stripping away these attack vectors, enforcing strict least-privilege boundaries, and converting a generic Linux box into a hardened production bastion.

This guide provides an actionable, 30-step hardening blueprint specifically calibrated for Ubuntu Server LTS (and applicable to Debian and enterprise Linux derivatives). Each tip breaks down what it is, the threat model it neutralizes, the exact configuration commands, and how to verify your changes.

0. The Server Landscape: Choosing Your Production Foundation

Before executing a single command, your security baseline depends on selecting the right distribution for your workloads. Linux distributions share the upstream kernel, but their release lifecycles, default Mandatory Access Control (MAC) frameworks, and package supply-chain policies differ substantially:

While Red Hat families rely on SELinux’s type enforcement, Ubuntu Server LTS excels in modern infrastructure due to its seamless cloud-init automation, robust AppArmor confinement, Canonical Livepatch kernel updates, and long-term security backports.

Below are 30 battle-tested hardening techniques organized across six strategic defense layers:

+-----------------------------------------------------------------------------------------+
|                        DEFENSE-IN-DEPTH HARDENING HIERARCHY                             |
|                                                                                         |
|  [LAYER 1: IDENTITY & ACCESS]    --> SSH Keys, Root Prohibition, MFA, Sudo Lockdown    |
|  [LAYER 2: NETWORK PERIMETER]    --> UFW Default-Deny, IPS (CrowdSec), Protocol Banish  |
|  [LAYER 3: FILESYSTEM & STORAGE] --> Noexec Mounts, SUID Scans, Unused FS Blacklists    |
|  [LAYER 4: KERNEL & PROCESSES]   --> Sysctl Fortification, ASLR, Systemd Sandboxing   |
|  [LAYER 5: SUPPLY CHAIN & APPS]  --> Unattended Upgrades, Daemon Purging, Cron Locks    |
|  [LAYER 6: FORENSICS & AUDIT]    --> Auditd Syscalls, FIM (AIDE), Off-Host TLS Logs     |
+-----------------------------------------------------------------------------------------+

Layer 1: Identity, Authentication & SSH Fortification

Your SSH daemon is the primary external entrance. If this gate is breached or poorly defended, all underlying kernel defenses become secondary.

Tip 01: Enforce Modern Ed25519 Public Key Authentication & Disable Passwords

• The Threat Vector: Automated dictionary bots and rainbow-table tools try tens of thousands of common password combinations per hour. Password authentication also exposes systems to brute-force sprays, credential stuffing, and credential re-use attacks.

• Why It Matters: Is Ed25519 the undisputed best? In classical computing and daily sysadmin operations, yes—with one caveat: Ed25519-SK (hardware-bound FIDO2 security keys) is even more resilient against endpoint compromise.

  To understand why Ed25519 is preferred over other algorithms, let’s examine the alternatives:

  Algorithm	Key Size	Classical Security Level	Side-Channel Immunity	Hardware Token Binding	Verdict
  Ed25519	256-bit (68 chars)	~128-bit (~3000-bit RSA equivalent)	Native (Constant-Time)	No (Software file)	Gold Standard for general SSH
  Ed25519-SK	256-bit + Token	~128-bit + Hardware Bound	Native (Constant-Time)	Yes (FIDO2 / U2F YubiKey)	Absolute Best for high-value root/prod access
  RSA (4096-bit)	4096-bit (700+ chars)	~140-bit	Vulnerable if unblinded	No	Legacy fallback; bulky, slow, prone to timing leaks
  RSA (2048-bit)	2048-bit	~112-bit (Deprecated)	Vulnerable	No	Avoid: Disallowed by modern compliance standards
  ECDSA (P-256/384/521)	256–521 bit	128–256 bit	Poor	Yes (ecdsa-sk)	Dangerous: Total key recovery if RNG leaks or biases
  DSA (1024-bit)	1024-bit	Insecure (<80-bit)	Severely flawed	No	Dead: Disabled by default since OpenSSH 7.0

Why Ed25519 Outclasses the Alternatives:

1. Immunity to Timing Attacks: Designed by Daniel J. Bernstein (djb) et al., Ed25519 operations are strictly constant-time. They do not use secret-dependent branch conditions or memory lookups, inherently neutralizing cache-timing and microarchitectural side-channel snooping.
2. Deterministic Signatures (No Flawed Random Nonces): Unlike ECDSA—where a weak, biased, or broken Pseudo-Random Number Generator (PRNG) exposes the private key after just two signatures (the exact flaw that famously broke the PlayStation 3 code signing and multiple Bitcoin wallets)—Ed25519 computes the signature nonces deterministically from the private key and the message. It cannot leak the private key through a bad random seed.
3. No Suspicious Constants: Standard ECDSA relies on NIST-curated curves (secp256r1) whose seed constants originated from unspecified government agency sources. Curve25519 constants are fully transparent and mathematically verifiable.
4. Performance & Footprint: Key generation, signing, and verification are orders of magnitude faster than 4096-bit RSA, and public keys are short enough to safely paste without terminal truncation errors.

• Implementation: Generate an Ed25519 key with maximum Key Derivation Function (KDF) rounds for brute-force resistance on your local machine:

  ssh-keygen -t ed25519 -a 100 -C "secops@production-infra"
  

  (Or, if you use a FIDO2/U2F hardware token like a YubiKey, generate a hardware-backed key):

  ssh-keygen -t ed25519-sk -O resident -O verify-required -C "yubikey-fido2-secops"
  

  Append the public key to ~/.ssh/authorized_keys on the server with strict permissions:

  chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys
  

  Create an override file in /etc/ssh/sshd_config.d/50-hardening.conf:

  PasswordAuthentication no
  KbdInteractiveAuthentication no
  PubkeyAuthentication yes
  AuthenticationMethods publickey
  

• How to Verify: Test the configuration syntax and reload:

  sudo sshd -t && sudo systemctl restart ssh
  

  Attempt connecting without an SSH key: ssh -o PubkeyAuthentication=no user@your-server-ip. The connection must be rejected with Permission denied (publickey).

Tip 02: Banish Direct Root Logins & Enforce Scoped Privileged Sudo Access

• The Threat Vector: Attackers targeting Linux assume the existence of an account named root. If root login is permitted, an attacker only has to guess half the credential equation.
• Why It Matters: Disabling root login forces every operator to authenticate as a discrete human user, creating a mandatory audit trail in /var/log/auth.log whenever sudo is invoked.
• Implementation: In /etc/ssh/sshd_config.d/50-hardening.conf:

  PermitRootLogin no
  

  Ensure your primary administration user belongs to the sudo group:

  sudo usermod -aG sudo sysadmin_ops
  

• How to Verify: Attempt logging in directly as root: ssh root@your-server-ip. The server must refuse authentication immediately, regardless of what keys root might hold.

Tip 03: Obscure, Relocate, or VPN-Isolate the SSH Port

• The Threat Vector: Internet-wide scanners (such as Shodan and Censys) and commodity botnets continuously target standard port 22. The resulting connection churn floods auth logs and wastes CPU cycles.

• Why It Matters: Shifting SSH to a non-standard port cuts down 98% of automated noise. Even better: binding SSH strictly to a WireGuard overlay network (Tailscale, Headscale, Netmaker) removes SSH entirely from the public internet.

• Implementation: In /etc/ssh/sshd_config.d/50-hardening.conf:

  # Change port to a non-standard high port (e.g., 2222 or 54322)
  Port 2222
  # Or bind strictly to a private management interface / VPN IP:
  # ListenAddress 10.8.0.5
  

  Important Note for Ubuntu 22.10 and 24.04 LTS: Modern Ubuntu uses systemd socket activation (ssh.socket) by default. In this mode, editing Port in sshd_config alone will not change the listening port. You must either update the socket override:

  sudo systemctl edit ssh.socket
  

  Add the following lines:

  [Socket]
  ListenStream=
  ListenStream=2222
  

  Then run sudo systemctl daemon-reload && sudo systemctl restart ssh.socket. Alternatively, revert to the classic standalone service:

  sudo systemctl disable --now ssh.socket
  sudo systemctl enable --now ssh.service
  

• How to Verify: Inspect listening sockets:

  sudo ss -tulpn | grep -E 'ssh|2222'
  

Tip 04: Restrict SSH Cryptographic Ciphers, KEX, and MACs

• The Threat Vector: By default, sshd negotiates backward-compatible cryptographic algorithms, some of which suffer from theoretical vulnerabilities, weak random number generators, or small block sizes (e.g., 3DES, CBC modes, SHA-1 MACs).
• Why It Matters: Restricting SSH to modern Authenticated Encryption with Associated Data (AEAD) ciphers and curve-based key exchanges ensures quantum-resistant key generation and cryptographic integrity.
• Implementation: Add modern crypto suites to /etc/ssh/sshd_config.d/50-hardening.conf:

  # Key Exchange Algorithms
  KexAlgorithms curve25519-sha256,[email protected],diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
  
  # Ciphers (AEAD only)
  Ciphers [email protected],[email protected],[email protected]
  
  # Message Authentication Codes
  MACs [email protected],[email protected]
  

• How to Verify: Run ssh -Q cipher and ssh -Q kex to list available algorithms, then verify active sshd negotiation using ssh-audit localhost -p 2222.

Tip 05: Enforce Idle SSH Session Timeouts

• The Threat Vector: Developers and system administrators often leave active shell sessions running inside terminals on laptops. If an operator walks away in a shared space or coffee shop, an unauthorized person has unhindered access.
• Why It Matters: Automated inactivity termination tears down unmonitored TCP sockets and invalidates the session after a predetermined idle period.
• Implementation: In /etc/ssh/sshd_config.d/50-hardening.conf:

  # Send an alive probe every 300 seconds (5 minutes)
  ClientAliveInterval 300
  # Terminate connection if client fails to respond twice
  ClientAliveCountMax 2
  

• How to Verify: Open an SSH connection and leave the terminal idle for 10 minutes. The daemon will close the pipe with Timeout, client not responding.

Tip 06: Implement Multi-Factor Authentication (TOTP / FIDO2)

• The Threat Vector: If an administrator’s private key is leaked through a compromised local workstation, malware, or an insecure backup, the attacker can authenticate instantly.
• Why It Matters: Enforcing Multi-Factor Authentication (MFA) requires both “something you have” (the private key or TOTP seed) and “something you generate” (a 6-digit dynamic code or hardware touch token like YubiKey/Nitrokey FIDO2).

Vendor-Neutral & Open-Source Options (Working Globally, Including China)

Many guides suggest libpam-google-authenticator, but proprietary app ecosystems or services tied to Google Play can be problematic in restricted network environments (such as behind China’s Great Firewall) or on de-Googled devices.

Fortunately, TOTP (RFC 6238) is an open international mathematical standard. You can implement completely vendor-neutral, offline MFA:

1. Server Engine: libpam-oath (OATH Toolkit) — Part of the GNU project. It is 100% open source (LGPL/GPL), completely decoupled from any commercial vendor, and requires zero external network connections.
2. Client Apps: Open-source, offline authenticators that require no Google Play Services, no internet access, and work anywhere in the world:
   • Aegis Authenticator (Android): Free, open-source (GPLv3), available via F-Droid and GitHub. Encrypted offline vault with zero network permissions.
   • Ente Auth (Cross-platform / iOS / Android / Desktop): 100% open-source, end-to-end encrypted, standalone.
   • FreeOTP+ (Android / iOS): Open-source implementation maintained by Red Hat community developers.
3. Hardware Keys (FIDO2 / U2F): Open-source hardware tokens like Nitrokey (100% open-source hardware and firmware, made in Germany) or CanoKey (open-source hardware with dual NFC/USB, popular worldwide). Hardware cryptographic handshakes take place entirely across the USB bus and never touch the internet.

• Implementation Option A: Vendor-Neutral TOTP via libpam-oath: Install the OATH toolkit:

  sudo apt install -y libpam-oath oathtool
  

  Generate a random 32-character hexadecimal secret key:

  SECRET=$(head -c 16 /dev/urandom | xxd -p)
  echo "HOTP/T30/6 $USER - $SECRET" | sudo tee -a /etc/users.oath
  sudo chmod 600 /etc/users.oath
  

  Convert the hex secret to a standard base32 string and QR code for Aegis/FreeOTP:

  # Display the secret for manual entry into Aegis or Ente Auth:
  python3 -c "import base64; print(base64.b32encode(bytes.fromhex('$SECRET')).decode())"
  

  In /etc/pam.d/sshd, add:

  auth required pam_oath.so usersfile=/etc/users.oath window=30 digits=6
  

• Implementation Option B: Classic libpam-google-authenticator:

  sudo apt install -y libpam-google-authenticator
  google-authenticator -t -d -f -r 3 -R 30 -W
  

  In /etc/pam.d/sshd:

  auth required pam_google_authenticator.so nullok
  

• Enable in OpenSSH: In /etc/ssh/sshd_config.d/50-hardening.conf, require public key + interactive code:

  KbdInteractiveAuthentication yes
  AuthenticationMethods publickey,keyboard-interactive
  

• How to Verify: Initiate a new SSH session. After validating your SSH key, sshd must prompt for your one-time verification code: One-time password (OATH) for user: or Verification code: .

Tip 07: Harden Sudoers with PTY Allocation, Logfiles, and Strict Timouts

• The Threat Vector: By default, sudo tickets remain valid in memory for 15 minutes. Background scripts running in user space can hijack the cached credentials to execute unmonitored root actions.
• Why It Matters: Enforcing dedicated pseudo-terminals (use_pty) prevents attacks where a malicious background process injects input into a parent shell, while timestamp timeouts limit privilege persistence.
• Implementation: Create /etc/sudoers.d/01-security-defaults:

  Defaults use_pty
  Defaults logfile="/var/log/sudo.log"
  Defaults timestamp_timeout=5
  Defaults passwd_tries=3
  Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  

• How to Verify: Run sudo visudo -c to validate syntax. Run a sudo command, verify /var/log/sudo.log records the command, and observe that credentials expire after 5 minutes of inactivity.

Layer 2: Network Perimeter & Active Intrusion Defense

Every open network port is an invitation. Your network layer must act as a filter that only recognizes authorized conversations.

Tip 08: Implement a Strict Default-Deny Firewall with UFW or Native Nftables

• The Threat Vector: Developers frequently spin up temporary databases, diagnostic HTTP servers (e.g., Python http.server), or debug APIs on 0.0.0.0, inadvertently broadcasting sensitive data to the world.
• Why It Matters: A strict default-drop inbound policy guarantees that no service can communicate across public interfaces unless explicitly granted permission by network rules.
• Implementation: Using UFW (Uncomplicated Firewall):

  sudo ufw default deny incoming
  sudo ufw default allow outgoing
  sudo ufw default deny routed
  
  # Allow explicit services
  sudo ufw allow 2222/tcp comment 'Hardened SSH'
  sudo ufw allow 80/tcp comment 'Web HTTP'
  sudo ufw allow 443/tcp comment 'Web HTTPS'
  
  sudo ufw enable
  

• How to Verify: Check verbose status:

  sudo ufw status verbose
  

  Port scan your server from an external system: nmap -sS -Pn your-server-ip. Only specified ports should show as open; all others should report filtered (dropped packets).

Tip 09: Dynamic Intrusion Prevention with Fail2ban or CrowdSec

• The Threat Vector: Continuous authentication attempts consume system bandwidth, pollute audit logs, and risk credential stuffing breakthroughs.

• Why It Matters: Intrusion Prevention Systems (IPS) parse live application and system logs, dynamically inserting temporary kernel-level packet drops for abusive IPs.

• Implementation: Install and configure Fail2ban:

  sudo apt install -y fail2ban
  

  Create /etc/fail2ban/jail.d/custom-sshd.local:

  [sshd]
  enabled   = true
  port      = 2222
  mode      = aggressive
  filter    = sshd
  backend   = systemd
  maxretry  = 3
  findtime  = 15m
  bantime   = 48h
  banaction = ufw
  

  (Note: On Ubuntu 24.04 LTS, rsyslog is no longer installed by default. Using backend = systemd reads authentication logs directly from journald without needing a legacy /var/log/auth.log file).

  Restart service: sudo systemctl restart fail2ban.

• How to Verify: Inspect the ban status:

  sudo fail2ban-client status sshd
  

Tip 10: Block IP Spoofing and Enable SYN Cookies via Sysctl

• The Threat Vector: In a SYN Flood attack, an attacker sends an avalanche of TCP SYN packets with forged source addresses, exhausting the server’s TCP connection backlog queue and freezing network responsiveness.
• Why It Matters: TCP SYN Cookies allow the kernel to reply with a cryptographic cookie in the sequence number without allocating memory state until the client returns a valid ACK packet.
• Implementation: Add network tuning to /etc/sysctl.d/99-networking-hardening.conf:

  # Enable SYN Flood resistance
  net.ipv4.tcp_syncookies = 1
  net.ipv4.tcp_max_syn_backlog = 4096
  net.ipv4.tcp_synack_retries = 2
  
  # Enable Source Address Verification (Anti-Spoofing / Reverse Path Filtering)
  net.ipv4.conf.all.rp_filter = 1
  net.ipv4.conf.default.rp_filter = 1
  
  # Disable IPv4 and IPv6 packet routing (unless building a gateway router)
  net.ipv4.ip_forward = 0
  net.ipv6.conf.all.forwarding = 0
  

  Apply immediately: sudo sysctl --system.
• How to Verify: Query running kernel parameters:

  sysctl net.ipv4.tcp_syncookies net.ipv4.conf.all.rp_filter
  

Tip 11: Disable Insecure ICMP Redirects and Bogus Error Responses

• The Threat Vector: ICMP Redirect messages can be weaponized by a malicious actor on a local network segment to alter routing tables, forcing the server to route traffic through an attacker-controlled Man-in-the-Middle (MITM) proxy.
• Why It Matters: Production servers have static routes or dedicated DHCP gateways; they have zero legitimate need to let arbitrary network packets redesign their routing topology.
• Implementation: Add to /etc/sysctl.d/99-networking-hardening.conf:

  # Do not accept ICMP redirects
  net.ipv4.conf.all.accept_redirects = 0
  net.ipv4.conf.default.accept_redirects = 0
  net.ipv6.conf.all.accept_redirects = 0
  
  # Do not send ICMP redirects
  net.ipv4.conf.all.send_redirects = 0
  net.ipv4.conf.default.send_redirects = 0
  
  # Ignore broadcast ping requests (Smurf attack defense)
  net.ipv4.icmp_echo_ignore_broadcasts = 1
  
  # Ignore bogus ICMP error responses
  net.ipv4.icmp_ignore_bogus_error_responses = 1
  

  Reload: sudo sysctl --system.
• How to Verify: Verify with:

  sysctl net.ipv4.conf.all.accept_redirects net.ipv4.icmp_echo_ignore_broadcasts
  

Tip 12: Blacklist Unused and Vulnerable Network Protocols

• The Threat Vector: The Linux kernel includes drivers for niche and legacy network protocols (such as DCCP, SCTP, RDS, and TIPC). Historical CVEs show that obscure network protocols are a goldmine for local privilege escalation and remote code execution exploits.
• Why It Matters: If your production application communicates over TCP/UDP and IPv4/IPv6, compiling or loading legacy protocol modules provides unnecessary attack surface for zero-day exploits.
• Implementation: Create /etc/modprobe.d/blacklist-network-protocols.conf:

  install dccp /bin/true
  install sctp /bin/true
  install rds /bin/true
  install tipc /bin/true
  

• How to Verify: Attempt to load one of the modules:

  sudo modprobe sctp
  lsmod | grep sctp
  

  The module should return empty, proving the kernel redirected the loader to /bin/true.

Layer 3: Filesystem, Storage & Memory Protections

Filesystems are the physical or virtual bedrock where unauthorized payloads are dropped. Restricting file execution boundaries paralyzes web shells and script droppers.

Tip 13: Neutralize Malicious Payloads in /dev/shm, /tmp, and /var/tmp

• The Threat Vector: Web shells and remote code execution vulnerabilities in web servers (such as Nginx, PHP, Node.js) almost always download initial binary droppers or crypto-miners into universally writable directories: /tmp or /dev/shm (shared memory).
• Why It Matters: Mounting temporary locations with noexec, nosuid, and nodev ensures that even if an attacker manages to download an exploit binary to disk, the kernel refuses to execute it or grant SUID elevation.
• Implementation: Ensure /etc/fstab contains restrictive mount flags:

  # Shared memory hardening
  tmpfs  /dev/shm  tmpfs  defaults,noexec,nosuid,nodev  0  0
  
  # Temp filesystem isolation
  tmpfs  /tmp      tmpfs  defaults,noexec,nosuid,nodev,size=2G  0  0
  

  Remount the active mounts:

  sudo mount -o remount,noexec,nosuid,nodev /dev/shm
  sudo mount -o remount,noexec,nosuid,nodev /tmp
  

• How to Verify: Write a simple test script in /dev/shm:

  echo -e '#!/bin/bash\necho "exploit"' > /dev/shm/test.sh && chmod +x /dev/shm/test.sh
  /dev/shm/test.sh
  

  The shell must return: bash: /dev/shm/test.sh: Permission denied. Clean up: rm /dev/shm/test.sh.

Tip 14: Restrict SUID and SGID Executables

• The Threat Vector: SUID (Set User ID) binaries run with the permissions of the file owner (often root), regardless of who executes them. Vulnerabilities in poorly maintained SUID binaries (or unintended execution parameters like those cataloged in GTFOBins) permit instant root escalation.
• Why It Matters: Eliminating SUID bits on binaries that do not need them strips away privilege-escalation trampolines.
• Implementation: Locate all SUID binaries on your filesystem:

  sudo find / -perm -4000 -type f -exec ls -ld {} + 2>/dev/null
  

  Remove SUID from non-essential utilities (e.g., legacy print utilities, mount helpers, or diagnostic network tools):

  sudo chmod u-s /usr/bin/chfn /usr/bin/chsh /usr/bin/newgrp
  

• How to Verify: Run ls -l /usr/bin/chfn. The file permissions must display -rwxr-xr-x instead of -rwsr-xr-x.

Tip 15: Find and Eliminate World-Writable Files and Directories

• The Threat Vector: If a configuration file, library, or cron script is world-writable (mode 0777 or -rw-rw-rw-), any unprivileged service account can inject arbitrary commands that subsequently execute with elevated privileges.
• Why It Matters: Maintaining strict ownership discipline guarantees that only administrators and designated service users can modify executable code and configuration paths.
• Implementation: Search for world-writable files excluding proc and sysfs:

  sudo find / -xdev -type f -perm -0002 -exec ls -ld {} + 2>/dev/null
  

  Remedy any discovered files by removing write privileges for others:

  sudo chmod o-w /path/to/vulnerable-file
  

• How to Verify: Re-run the find command. It should yield zero matches across production paths.

Tip 16: Blacklist Rare and Legacy Filesystem Kernel Modules

• The Threat Vector: Attackers can mount corrupted or maliciously crafted disk image files formatted as legacy filesystems (e.g., cramfs, hfs, jffs2, freevxfs) to trigger memory corruption bugs in kernel filesystem parsing drivers.

• Why It Matters: Standard production servers exclusively use ext4, xfs, or btrfs. Disabling legacy drivers eliminates attack surface from untrusted storage devices or rogue USB attachments.

• Implementation: Create /etc/modprobe.d/blacklist-filesystems.conf:

  install cramfs /bin/true
  install freevxfs /bin/true
  install jffs2 /bin/true
  install hfs /bin/true
  install hfsplus /bin/true
  install udf /bin/true
  # Only blacklist squashfs if you DO NOT use Snap packages (snapd):
  # install squashfs /bin/true
  

  Warning regarding Ubuntu Snap packages: Ubuntu relies on squashfs to mount Snap packages (such as lxd, core, or canonical utilities). If you use Snaps, do not blacklist squashfs unless you have completely uninstalled and purged snapd.

• How to Verify: Attempt to load a blacklisted filesystem: sudo modprobe jffs2. Verify that lsmod | grep jffs2 produces no output.

Tip 17: Enforce Full Disk Encryption (LUKS) at Rest

• The Threat Vector: Decommissioned cloud volumes, stolen physical servers, or un-scrubbed backup images allow attackers to bypass all OS-level authentication by mounting the raw block storage offline.
• Why It Matters: LUKS (Linux Unified Key Setup) full-disk encryption guarantees that sensitive data, credentials, and configuration files are unreadable without the cryptographic decryption key.
• Implementation: For secondary storage volumes and data drives:

  # Format volume with LUKS2
  sudo cryptsetup luksFormat --type luks2 /dev/sdb1
  
  # Map encrypted partition to virtual block device
  sudo cryptsetup open /dev/sdb1 secure_storage
  
  # Format virtual block device
  sudo mkfs.ext4 /dev/mapper/secure_storage
  

• How to Verify: Check block device status: sudo cryptsetup status secure_storage. Verify cipher algorithm (aes-xts-plain64) and key size (512-bit).

Layer 4: Kernel Defenses & Process Sandboxing

The Linux kernel manages memory, system calls, and hardware. Hardening kernel parameters turns memory safety features on and locks down debugging backdoors.

Tip 18: Restrict Kernel Pointer Leaks and Protect dmesg

• The Threat Vector: Unprivileged processes can inspect /proc/kallsyms and dmesg (the kernel ring buffer) to discover real memory addresses of kernel structures. Attackers use these addresses to defeat modern memory protections and craft Return-Oriented Programming (ROP) exploits.
• Why It Matters: Restricting pointer exposure obscures kernel layout and limits information leakage to unprivileged users.
• Implementation: Add to /etc/sysctl.d/99-kernel-security.conf:

  # Hide kernel pointers from unprivileged users
  kernel.kptr_restrict = 2
  
  # Restrict access to dmesg to users with CAP_SYSLOG
  kernel.dmesg_restrict = 1
  
  # Restrict BPF (Berkeley Packet Filter) JIT compiler to root
  net.core.bpf_jit_harden = 2
  kernel.unprivileged_bpf_disabled = 1
  

  Reload: sudo sysctl --system.
• How to Verify: Switch to an unprivileged user (su - nobody -s /bin/bash) and run dmesg. The command must return: dmesg: read kernel buffer failed: Operation not permitted.

Tip 19: Harden Memory Space Layout with ASLR and Restrict ptrace Scope

• The Threat Vector: Local processes can attach to other processes via the ptrace syscall (used by debuggers like GDB), allowing an attacker to read passwords, private keys, or session tokens out of adjacent processes in memory.
• Why It Matters: ASLR (Address Space Layout Randomization) randomizes program memory structures, making buffer overflow predictions nearly impossible. The Yama LSM ptrace_scope ensures that a process cannot inspect another process unless it is a direct child process.
• Implementation: Add to /etc/sysctl.d/99-kernel-security.conf:

  # Maximum randomization of memory address spaces (ASLR)
  kernel.randomize_va_space = 2
  
  # Restrict ptrace to ancestor processes only
  kernel.yama.ptrace_scope = 1
  
  # Prevent links from being exploited in world-writable sticky directories
  fs.protected_hardlinks = 1
  fs.protected_symlinks = 1
  fs.protected_fifos = 2
  fs.protected_regular = 2
  

  Apply: sudo sysctl --system.
• How to Verify: Verify active settings:

  sysctl kernel.randomize_va_space kernel.yama.ptrace_scope fs.protected_symlinks
  

Tip 20: Disable Core Dumps to Prevent Memory Harvesting

• The Threat Vector: When an application crashes, the Linux kernel can dump the entire contents of its active memory to disk in a core file. If a database or web server crashes, this core dump frequently contains plaintext passwords, session tokens, and TLS private keys.
• Why It Matters: Disabling core dumps ensures that transient secrets held in RAM are wiped cleanly upon application termination rather than serialized to disk.
• Implementation: In /etc/security/limits.d/10-disable-coredumps.conf:

  * hard core 0
  * soft core 0
  

  In /etc/sysctl.d/99-kernel-security.conf:

  # Disable SUID program core dumps
  fs.suid_dumpable = 0
  

  Reload: sudo sysctl --system.
• How to Verify: Query limits in your current shell: ulimit -c. Output must be 0.

Tip 21: Enforce Mandatory Access Control with Active AppArmor Profiles

• The Threat Vector: Traditional Discretionary Access Control (DAC) means that if your nginx process is compromised, the attacker inherits all permissions of the www-data user, including reading readable system files and running local scripts.
• Why It Matters: AppArmor enforces Mandatory Access Control (MAC). Even if an attacker achieves arbitrary code execution inside Nginx, AppArmor restricts the process strictly to the files, sockets, and capabilities declared in its profile.
• Implementation: Check AppArmor status:

  sudo aa-status
  

  Install profile utilities:

  sudo apt install -y apparmor-utils apparmor-profiles
  

  Enforce profiles on exposed daemons (e.g., Nginx):

  sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx
  

• How to Verify: Run sudo aa-status. Verify that your public daemons appear in the list under profiles are in enforce mode.

Tip 22: Sandbox Custom Daemons Using Modern Systemd Security Directives

• The Threat Vector: Background services written in Go, Node.js, Python, or Rust often run with far more system privileges than they actually require, exposing /home, /etc, and device interfaces to potential web application bugs.
• Why It Matters: Modern systemd comes with built-in, lightweight cgroup-based and namespace-based sandboxing directives that can isolate a daemon with zero overhead.
• Implementation: Edit your custom service unit file (e.g., /etc/systemd/system/myapp.service):

  [Service]
  ExecStart=/usr/local/bin/myapp
  User=appuser
  Group=appuser
  
  # Sandboxing Directives
  NoNewPrivileges=true
  ProtectSystem=strict
  ProtectHome=true
  ProtectKernelTunables=true
  ProtectKernelModules=true
  ProtectControlGroups=true
  PrivateTmp=true
  PrivateDevices=true
  CapabilityBoundingSet=CAP_NET_BIND_SERVICE
  ReadOnlyPaths=/
  ReadWritePaths=/var/log/myapp /var/run/myapp
  

  Reload and restart:

  sudo systemctl daemon-reload && sudo systemctl restart myapp
  

• How to Verify: Audit the security posture of your services with systemd’s built-in analyzer:

  systemd-analyze security myapp.service
  

  Aim for an exposure score below 2.5 / 10 (OK / Protected).

Layer 5: Supply Chain, Package Auditing & Exposure Minimization

Unpatched vulnerabilities in common libraries and unneeded background processes are the low-hanging fruit exploited by scanning engines.

Tip 23: Automate Security Patching with Unattended Upgrades & Kernel Livepatch

• The Threat Vector: Zero-day vulnerabilities in common packages (like OpenSSL, glibc, curl) and Linux kernel vulnerabilities (like Dirty COW, Dirty Pipe) are weaponized within hours of public CVE disclosure.
• Why It Matters: Automated unattended upgrades ensure that critical and high security patches apply overnight without waiting for manual operational maintenance windows.
• Implementation: Install unattended-upgrades:

  sudo apt install -y unattended-upgrades update-notifier-common
  sudo dpkg-reconfigure --priority=low unattended-upgrades
  

  Configure /etc/apt/apt.conf.d/50unattended-upgrades:

  Unattended-Upgrade::Allowed-Origins {
      "${distro_id}:${distro_codename}-security";
  };
  Unattended-Upgrade::Package-Blacklist {
  };
  Unattended-Upgrade::AutoFixInterruptedDpkg "true";
  Unattended-Upgrade::MinimalSteps "true";
  Unattended-Upgrade::InstallOnShutdown "false";
  Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
  Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
  Unattended-Upgrade::Automatic-Reboot "false";
  

• How to Verify: Simulate an unattended upgrade run:

  sudo unattended-upgrades --dry-run --debug
  

Tip 24: Ruthlessly Purge Unneeded Daemons and Compilers

• The Threat Vector: Default Linux distributions ship with legacy print spoolers (cups), RPC mappers (rpcbind), and local mail transport agents that listen on localhost or local subnets. Furthermore, having compilers (gcc, clang, make) readily available on production hosts allows attackers to compile privilege-escalation exploits on the fly.
• Why It Matters: If a daemon is not running, its bugs cannot be exploited. If compilers are removed, an attacker cannot easily build exploit code from C sources.
• Implementation: Audit all active listening ports:

  sudo ss -tulpn
  

  Stop, disable, and purge common legacy bloatware:

  sudo systemctl stop cups rpcbind avahi-daemon 2>/dev/null
  sudo systemctl disable cups rpcbind avahi-daemon 2>/dev/null
  sudo apt purge -y cups rpcbind avahi-daemon
  

  Restrict compiler execution to root:

  sudo chmod 700 /usr/bin/gcc /usr/bin/make 2>/dev/null || true
  

• How to Verify: Re-run sudo ss -tulpn. The output should only show the absolute minimum services required for your production workload.

Tip 25: Restrict Cron and At Scheduling to Authorized Users Only

• The Threat Vector: Persistent backdoors are frequently installed by dropped scripts into /var/spool/cron/crontabs or /etc/cron.* to survive reboots.
• Why It Matters: By default, any non-system user can create recurring cron tasks. Restricting cron access via allowlists ensures only explicitly authorized service accounts can schedule automated tasks.
• Implementation: Remove open permissions and establish explicit allowlists:

  sudo rm -f /etc/cron.deny /etc/at.deny
  echo "root" | sudo tee /etc/cron.allow
  echo "sysadmin_ops" | sudo tee -a /etc/cron.allow
  echo "root" | sudo tee /etc/at.allow
  

  Lock down cron directory permissions:

  sudo chmod 700 /etc/cron.daily /etc/cron.hourly /etc/cron.monthly /etc/cron.weekly /var/spool/cron
  

• How to Verify: Log into an unprivileged test account and attempt to edit crontab: crontab -e. The system will refuse: You (user) are not allowed to use this program (crontab).

Layer 6: Forensics, Auditing & Continuous Compliance

Defenses are incomplete without active telemetry. When an intrusion attempt occurs, high-fidelity logging allows you to reconstruct the incident and prove compliance.

Tip 26: Continuous Kernel-Level Auditing with auditd Rules

• The Threat Vector: Attackers who gain initial shell access will modify /etc/passwd, alter /etc/sudoers, or execute reconnaissance commands like whoami and id. Standard syslog captures none of this system call activity.
• Why It Matters: The Linux Audit Framework (auditd) hooks directly into kernel system calls, logging immutable audit events whenever critical files are accessed or modified.
• Implementation: Install auditd:

  sudo apt install -y auditd audispd-plugins
  

  Create /etc/audit/rules.d/99-production-hardening.rules:

  # Reset rules
  -D
  -b 8192
  -f 1
  
  # Monitor changes to user and identity databases
  -w /etc/group -p wa -k identity
  -w /etc/passwd -p wa -k identity
  -w /etc/shadow -p wa -k identity
  -w /etc/security/opasswd -p wa -k identity
  
  # Monitor privilege escalation configurations
  -w /etc/sudoers -p wa -k sudoers_tamper
  -w /etc/sudoers.d/ -p wa -k sudoers_tamper
  
  # Monitor SSH configuration changes
  -w /etc/ssh/sshd_config -p wa -k sshd_config
  -w /etc/ssh/sshd_config.d/ -p wa -k sshd_config
  
  # Monitor execution of privilege commands
  -a always,exit -F arch=b64 -S execve -F euid=0 -k root_exec
  

  Restart daemon or load rules into kernel:

  sudo augenrules --load
  sudo systemctl restart auditd 2>/dev/null || sudo service auditd restart
  

• How to Verify: Query audit logs for recent events:

  sudo ausearch -k identity --raw | aureport -f -i
  

Tip 27: Cryptographic File Integrity Monitoring (FIM) with AIDE

• The Threat Vector: Sophisticated attackers replace system binaries (like /bin/login, /usr/bin/ssh, or PAM modules) with trojaned versions that log passwords or install persistent rootkits.
• Why It Matters: AIDE (Advanced Intrusion Detection Environment) computes SHA-256/SHA-512 cryptographic hashes of all static binaries, libraries, and configurations, comparing them against a known-good database to alert on unauthorized modifications.
• Implementation: Install and initialize AIDE:

  sudo apt install -y aide
  sudo aideinit
  sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
  

  Create a daily automated cron job in /etc/cron.daily/aide-check:

  #!/bin/bash
  /usr/bin/aide --check | /usr/bin/mail -s "AIDE Integrity Alert: $(hostname)" [email protected]
  

  Make executable: sudo chmod +x /etc/cron.daily/aide-check.
• How to Verify: Run an ad-hoc integrity check:

  sudo aide --check
  

  If no files have changed, it will output: AIDE found NO differences between database and filesystem. Looks okay!!.

Tip 28: Forward Encrypted Logs to an Off-Host Aggregator

• The Threat Vector: One of the first actions an attacker takes after escalating privileges is to delete /var/log/auth.log, /var/log/syslog, and clear bash history with rm -rf /var/log/* to erase forensic breadcrumbs.
• Why It Matters: Streaming logs off-host over an encrypted TLS transport in real time ensures that even if a server is completely compromised, forensic records are preserved in an external SIEM or log cluster.
• Implementation: Using rsyslog or vector to forward logs over TLS. For rsyslog, create /etc/rsyslog.d/60-remote-tls.conf:

  # Setup TLS certificate validation
  $DefaultNetstreamDriver gtls
  $DefaultNetstreamDriverCAFile /etc/ssl/certs/internal-ca.pem
  $ActionSendStreamDriverMode 1
  $ActionSendStreamDriverAuthMode x509/name
  
  # Stream all events in RFC5424 format to remote collector
  *.* @@logs.secops.internal.net:6514;RSYSLOG_SyslogProtocol23Format
  

  Restart logging daemon: sudo systemctl restart rsyslog.
• How to Verify: Generate a synthetic test log: logger -t SECURITY_TEST "Hardening verification token 984321". Check your central log dashboard (Grafana Loki, Elastic, or Datadog) to verify immediate arrival.

Tip 29: Automated Security Auditing and Benchmark Verification with Lynis & OpenSCAP

• The Threat Vector: Security posture degrades over time as developers install dependencies, alter permissions, and tweak test configs. Without automated auditing, security regressions go unnoticed.
• Why It Matters: Lynis and OpenSCAP evaluate your server against established industry standards (such as the CIS Ubuntu Benchmarks and DISA STIG), providing actionable remediation guidance and a quantitative hardening score.
• Implementation: Install and run Lynis:

  sudo apt install -y lynis
  sudo lynis audit system --quick
  

  For formal compliance reporting with OpenSCAP:

  sudo apt install -y ssg-deb-reg ssg-non-standard libopenscap8
  oscap xccdf eval \
    --profile xccdf_org.ssgproject.content_profile_cis_level2_server \
    --report /var/log/cis-benchmark-report.html \
    /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml
  

• How to Verify: Review the Lynis Hardening Index in the terminal output. A well-hardened production server should score 80 or higher with zero critical warnings.

Tip 30: Harden Container Runtimes (Docker / Podman)

• The Threat Vector: The default Docker daemon runs as root. If an attacker executes a container breakout exploit or has mounted the Docker socket (/var/run/docker.sock) into a container, they immediately inherit root privileges over the host operating system.
• Why It Matters: Enforcing rootless containers, disabling inter-container communication, and dropping kernel capabilities limits container escapes to an unprivileged sandbox.
• Implementation: In /etc/docker/daemon.json:

  {
    "icc": false,
    "no-new-privileges": true,
    "userns-remap": "default",
    "live-restore": true,
    "userland-proxy": false,
    "log-driver": "json-file",
    "log-opts": {
      "max-size": "10m",
      "max-file": "3"
    }
  }
  

  Restart docker: sudo systemctl restart docker. When launching containers, always drop unnecessary kernel capabilities:

  docker run --read-only --cap-drop=ALL --cap-add=NET_BIND_SERVICE -d nginx
  

• How to Verify: Inspect a running container to verify capabilities and non-root execution:

  docker inspect --format '{{ .HostConfig.CapDrop }}' <container-id>
  

  The output should confirm [ALL].

7. The 30-Point Production Verification Checklist

Print this reference card or incorporate it into your automated provisioning pipelines (Ansible, Terraform, or Packer) to verify every node before it receives production traffic:

• 1. Ed25519 Keys Enforced: Passwords disabled in sshd_config.
• 2. Root Login Prohibited: PermitRootLogin no active.
• 3. Non-Standard / Private SSH Port: SSH moved off port 22 or bound to VPN.
• 4. Modern SSH Crypto Suite: Deprecated ciphers, MACs, and KEX removed.
• 5. SSH Idle Timeouts: ClientAliveInterval terminates stale sessions.
• 6. Multi-Factor Authentication: TOTP or FIDO2 hardware token enforced.
• 7. Sudoers Hardened: use_pty, command logfiles, and 5-minute timeouts active.
• 8. Default-Deny Firewall: ufw or nftables drops all unsolicited inbound packets.
• 9. Dynamic IPS Active: Fail2ban / CrowdSec scanning auth logs.
• 10. Anti-Spoofing & SYN Cookies: net.ipv4.tcp_syncookies = 1 active.
• 11. ICMP Redirects Disabled: Routing table poisoning vectors blocked.
• 12. Obscure Protocols Blacklisted: DCCP, SCTP, RDS disabled in modprobe.d.
• 13. Restrictive Mounts: /dev/shm and /tmp mounted noexec,nosuid,nodev.
• 14. SUID Binaries Scanned: Unneeded elevated binaries stripped of SUID bits.
• 15. World-Writable Files Cleared: Zero unauthorized world-writable paths.
• 16. Legacy Filesystems Blocked: cramfs, jffs2, hfs drivers disabled.
• 17. Encrypted Disks (LUKS): Full-disk or partition encryption at rest.
• 18. Kernel Pointers Concealed: kptr_restrict = 2, dmesg_restrict = 1.
• 19. Memory Randomization (ASLR): randomize_va_space = 2, ptrace_scope = 1.
• 20. Core Dumps Disabled: ulimit -c 0 and fs.suid_dumpable = 0.
• 21. AppArmor Enforced: Profiles active for internet-facing daemons.
• 22. Systemd Sandboxing: Custom units configured with ProtectSystem=strict.
• 23. Automated Security Patches: unattended-upgrades applied automatically.
• 24. Unused Daemons Purged: Extraneous network services uninstalled.
• 25. Cron Whitelisting: /etc/cron.allow restricts task scheduling.
• 26. Kernel Auditing (auditd): System call rules active for identity and sudo files.
• 27. File Integrity Monitoring: AIDE initialized with baseline hashes.
• 28. Centralized TLS Logging: Syslog stream forwarded to an off-host SIEM.
• 29. Benchmark Audited: Lynis Hardening Index >= 80.
• 30. Container Runtime Hardened: Rootless Docker / dropped capabilities.

Conclusion: Security as an Operational State

Infrastructure hardening is not a single checkbox on a deployment ticket—it is a continuous operational discipline. Attack surfaces evolve as new CVEs are uncovered and modern threat landscapes shift.

By applying these 30 hardening controls, you build a multi-layered defense-in-depth architecture. Even if an attacker finds an application-level flaw, the underlying system is locked down: they cannot run scripts from /tmp, cannot escalate privileges via SUID binaries, cannot snoop adjacent memory via ptrace, and their actions leave an indelible trail across remote audit logs.

Lock the gate early, test continuously, and let automation keep it barred.