I manage a few servers for myself, friends and family as well as for the Libravatar project. Here is how I customize recent releases of Debian on those servers.

Hardware tests

apt install memtest86+ smartmontools e2fsprogs rasdaemon

Prior to spending any time configuring a new physical server, I like to ensure that the hardware is fine.

To check memory, I boot into memtest86+ from the grub menu and let it run overnight.

Then I check the hard drives using:

smartctl -t long /dev/sdX badblocks -swo badblocks.out /dev/sdX

Configuration

apt install etckeeper git sudo vim

To keep track of the configuration changes I make in /etc/ , I use etckeeper to keep that directory in a git repository and make the following changes to the default /etc/etckeeper/etckeeper.conf :

turn off daily auto-commits

turn off auto-commits before package installs

and then put these config files in /etc/.gitignore :

/aliases.db /group- /gshadow- /passwd- /shadow- /subgid- /subuid-

and this in /etc/.git/config :

[commit] gpgsign = false

Note that in order to fully turn off auto-commits, it's also necessary to run the following:

systemctl stop etckeeper.timer systemctl disable etckeeper.timer

To get more control over the various packages I install, I change the default debconf level to medium:

dpkg-reconfigure debconf

Since I use vim for all of my configuration file editing, I make it the default editor:

update-alternatives --config editor

and I turn on syntax highlighting and visual beeping globally by adding the following to /etc/vim/vimrc.local :

syntax on set background=dark set visualbell set nomodeline

ssh

apt install openssh-server mosh fail2ban

Since most of my servers are set to UTC time, I like to use my local timezone when sshing into them. Looking at file timestamps is much less confusing that way.

I also ensure that the locale I use is available on the server by adding it the list of generated locales:

dpkg-reconfigure locales

Make sure the default locale is using the UTF-8 encoding since that will ensure that things like Postgres default to the One True Encoding when you install/bootstrap them.

Other than that, I harden the ssh configuration and end up with the following settings in /etc/ssh/sshd_config (jessie):

HostKey /etc/ssh/ssh_host_ed25519_key HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_ecdsa_key KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256 Ciphers chacha20-poly1305@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com AuthenticationMethods publickey PasswordAuthentication no PermitRootLogin no AcceptEnv LANG LC_* TZ AllowGroups sshuser

On those servers where I need duplicity/paramiko to work, I also add the following:

KexAlgorithms ...,diffie-hellman-group-exchange-sha1 MACs ...,hmac-sha1

Since fail2ban is used to rate-limit attempts to brute-force ssh connections, you may want to whitelist your own IP addresses by adding them to /etc/fail2ban/jail.d/local.conf :

[DEFAULT] ignoreip = 127.0.0.1/8 1.2.3.4

Then I remove the "Accepted" filter in /etc/logcheck/ignore.d.server/ssh (first line) to get a notification whenever anybody successfully logs into my server.

I also create a new group and add the users that need ssh access to it:

addgroup sshuser adduser francois sshuser

and add a timeout for root sessions by putting this in /root/.bash_profile :

TMOUT=600

Security checks

apt install logcheck logcheck-database fcheck tiger debsums systemd-coredump rkhunter apt purge john john-data rpcbind tripwire unhide unhide.rb

Logcheck is the main tool I use to keep an eye on log files, which is why I add a few additional log files to the default list in /etc/logcheck/logcheck.logfiles :

/var/log/apache2/error.log /var/log/mail.err /var/log/mail.warn /var/log/mail.info /var/log/fail2ban.log

while ensuring that the apache logfiles are readable by logcheck:

chmod a+rx /var/log/apache2 chmod a+r /var/log/apache2/*

and fixing the log rotation configuration by adding the following to /etc/logrotate.d/apache2 :

create 644 root adm

I usually create these empty files to silence logcheck errors:

touch /var/log/mail.{err,warn} chown root:adm /var/log/mail.{err,warn}

I also modify the main logcheck configuration file ( /etc/logcheck/logcheck.conf ):

INTRO=0 FQDN=0

Other than that, I enable daily checks in /etc/default/debsums and customize a few tiger settings in /etc/tiger/tigerrc :

Tiger_Check_RUNPROC=Y Tiger_Check_DELETED=Y Tiger_Check_APACHE=Y Tiger_FSScan_WDIR=Y Tiger_SSH_Protocol='2' Tiger_Passwd_Hashes='sha512' Tiger_Running_Procs='rsyslogd cron /usr/sbin/apache2 postgres' Tiger_Listening_ValidProcs='sshd|mosh-server|ntpd'

I also add these to /etc/fcheck/fcheck.local.cfg :

Directory = /var/www Exclusion = /etc/.git/ Exclusion = /etc/.etckeeper Exclusion = /etc/.gitignore Exclusion = /etc/mtab

and these to /etc/rkhunter.conf.local :

ALLOWHIDDENDIR=/etc/.git ALLOWHIDDENFILE=/etc/.etckeeper ALLOWHIDDENFILE=/etc/.gitignore ALLOWDEVFILE=/dev/shm/logcheck.* ALLOWDEVFILE=/dev/shm/logcheck.*/ignore/* ALLOWDEVFILE=/dev/shm/logcheck.*/violations/* ALLOWDEVFILE=/dev/shm/logcheck.*/violations-ignore/* ALLOWDEVFILE=/dev/shm/logcheck.*/cracking/* ALLOWDEVFILE=/dev/shm/logcheck.*/logoutput/* ALLOWDEVFILE=/dev/shm/logcheck.*/logoutput-sorted ALLOWDEVFILE=/dev/shm/logcheck.*/checked ALLOWDEVFILE=/dev/shm/PostgreSQL.*

General hardening

apt install apparmor apparmor-profiles apparmor-profiles-extra libpam-tmpdir

Certain kernel security features can be enabled by putting the following in /etc/sysctl.d/local.conf to hide kernel pointers and messages from unprivileged processes:

kernel.dmesg_restrict = 1 kernel.kptr_restrict = 1

and the following to harden the TCP stack:

net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.all.rp_filter=1 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.send_redirects = 0 net.ipv4.tcp_syncookies=1 net.ipv6.conf.all.accept_redirects = 0 net.ipv6.conf.all.accept_source_route = 0 net.ipv6.conf.default.accept_redirects = 0 net.ipv6.conf.default.accept_source_route = 0

before reloading these settings using sysctl -p .

Sandboxing in apt can be enabled by putting the following in /etc/apt/apt.conf.d/30-seccomp :

APT::Sandbox::Seccomp "1";

I also restrict the use of cron to the root user by putting the following in /etc/cron.allow :

root

Entropy and timekeeping

apt install rng-tools chrony

To keep the system clock accurate and increase the amount of entropy available to the server, I install the above packages and add the tpm_rng module to /etc/modules .

I also set the server timezone to UTC using dpkg-reconfigure tzdata .

Preventing mistakes

apt install molly-guard safe-rm sl

The above packages are all about catching mistakes (such as accidental deletions).

apt install apticron unattended-upgrades deborphan debfoster apt-listchanges reboot-notifier popularity-contest needrestart debian-security-support distro-info

These tools help me keep packages up to date and remove unnecessary or obsolete packages from servers. On Rackspace servers, a small configuration change is needed to automatically update the monitoring tools.

On jessie or later, I install reboot-notifier to send me a notification whenever a kernel update requires a reboot to take effect.

In addition to knowing when you need to reboot your machine, the needrestart package will let you know (and offer to do it for you) when you need to restart a daemon using an obsolete library.

Handy utilities

apt install renameutils atool iotop sysstat lsof mtr-tiny mc netcat-openbsd command-not-found nocache apt-file

Most of these tools are configuration-free, except for sysstat, which requires enabling data collection in /etc/default/sysstat to be useful.

Also, command-not-found won't work until you update the apt cache:

apt update update-command-not-found

Apache configuration

apt install apache2 a2enmod mpm_event a2dismod status

While configuring apache is often specific to each server and the services that will be running on it, there are a few common changes I make.

I enable these in /etc/apache2/conf-enabled/security.conf :

<Directory /> AllowOverride None Require all denied </Directory> ServerTokens Prod ServerSignature Off

I also create a new /etc/apache2/conf-available/servername.conf which contains:

ServerName machine_hostname.example.com

and then run:

a2enconf servername

Mail

apt install postfix libsasl2-modules apt purge exim4-base exim4-daemon-light exim4-config

Configuring mail properly is tricky but the following has worked for me.

In /etc/hostname , put the bare hostname (no domain), but in /etc/mailname put the fully qualified hostname.

In /etc/hosts , make sure that the fully qualified hostname is the first alias for 127.0.0.1 , followed by the bare hostname and then anything else. For example:

127.0.0.1 hostname.example.com hostname localhost

Change the following in /etc/postfix/main.cf :

inet_interfaces = loopback-only myhostname = (fully qualified hostname) smtpd_banner = $myhostname ESMTP smtp_tls_security_level = may smtp_tls_protocols = !SSLv2, !SSLv3

Set the following aliases in /etc/aliases :

set francois as the destination of root emails

as the destination of emails set an external email address for francois

set root as the destination for mon and www-data emails

before running newaliases to update the aliases database.

Create a new cronjob ( /etc/cron.hourly/checkmail ):

#!/bin/sh ls /var/mail

to ensure that email doesn't accumulate unmonitored on this box. Don't forget to make the script executable:

chmod +x /etc/cron.hourly/checkmail

Finally, set reverse DNS for the server's IPv4 and IPv6 addresses and then test the whole setup using mail root . You should also use this online tool to make sure everything looks good.

To monitor that mail never stops flowing, add this machine to a free healthchecks.io account and create a /etc/cron.d/healthchecks-io cronjob:

0 1 * * * root echo "ping" | mail xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@hc-ping.com

Monitoring

apt install --no-install-recommends mon libfilesys-diskspace-perl libfilesys-df-perl

In order to ensure that the root partition never has less than 1G of free space, I put the following in /etc/mon/mon.cf :

serverbind = 127.0.0.1 trapbind = 127.0.0.1 clientallow = 127.0.0.1 watch localhost service freespace interval 10m monitor freespace.monitor /:1048576 ;; period numalerts 10 alert mail.alert root upalert mail.alert root alertevery 60m

and then systemctl restart mon.service .

Network tuning

To reduce the server's contribution to bufferbloat I change the default kernel queueing discipline (jessie or later) by putting the following in /etc/sysctl.d/local.conf :

net.core.default_qdisc=fq

and the following to improve congestion control and HTTP/2 prioritization: