Tags: howtos, linux, software

« Previous post: In Defence of the Boring Web — Next post: Watching the Watchmen: Pitfalls in Graph … »

One of the unexpected¹ outcomes of the Russian invasion in Ukraine is a large increase in nefarious access attempts to servers under my control, not necessarily appearing to arise from Russian IP addresses. There are the usual attempts at brute-forcing SSH access, for instance, but also a flurry of malevolent HTTP(S) requests, looking for insecure WordPress installations etc. With my logs starting to get clogged, I decided to improve security a little bit.

Caveat lector: The tips in this article will help keep out some obnoxious folks. They are not enough to fight professionals. If you are running critical infrastructure, please use my advice only as a starting point.

The Case for Pattern Matching

As a machine learning researcher, I know that I have different tools at my disposal: I could monitor connections over a prolonged period of time, categorise them as hostile or not, and train a transformer to detect hostile connections in real time. All this would take is a long time for data collection and labelling, followed by a lot of GPU hours. Alternatively, I could just do some simple pattern matching. Feeling the disapproving gaze of the lords and ladies of deep learning upon me, I nevertheless opted for pattern matching—at least for now.²

Strategies

The best strategy to make a server secure is to minimise its attack surface. That means cutting down the number of services that are available or exposed to the internet. A classical configuration error is to run a service listening on a public interface instead of localhost. If you cannot disable the service, at least make sure that it’s not publicly accessible.

Containers. There’s also something to be said about using containers or virtualisation—my needs for my server are too modest to make use of this. If it works for you, make sure that the container is secure. It is of no use to you if attackers can easily break into the container itself because that already provides them with a foothold.

In my case, I only run two services that are publicly available, viz. a webserver and SSH:

~ % nmap taliesin.annwfn.net   
Starting Nmap 7.92 ( https://nmap.org ) at 2022-03-12 08:00 CET
Nmap scan report for taliesin.annwfn.net (178.62.197.121)
Host is up (0.023s latency).
Not shown: 997 closed tcp ports (conn-refused)
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.41 seconds

Webserver

The webserver—nginx—serves multiple virtual hosts, both for my own needs and that of my friends. Websites are rather lightweight and do not make use of additional servers, so there’s nothing to do here. I am also running an OwnCloud instance for my family. This requires using php-fpm and creating a separate pool that can only ever access one base directory. This is probably too specific to my situation and I won’t go into additional details here; see this guide for more details.

SSH

SSH, on the other hand, is a different beast: with journalctl --follow, I can observe failed login attempts in real time. There are numerous users that are just attempting to brute-force their way into the server. As a starting point, I decided to be a little bit more severe in my sshd configuration:

LoginGraceTime 30
PermitRootLogin no
MaxAuthTries 3

X11Forwarding no
PrintMotd no

This is not super strict yet, but it prevents root login altogether, and reduces the wait time for a login to 30 seconds. Authentication failures will be logged after one incorrect attempt;³ this will turn out to be relevant later on.

The other settings in sshd_config are sane defaults, in my opinion. If you have a lot of users, you can also set PermitEmptyPasswords no. I put all of these values in a local.conf file that is deployed to /etc/ssh/sshd_config.d/local.conf.

There’s additional options that might be appropriate for your use case: disabling password authentication altogether, preventing tunnelling, and much more. However, I do not want to decrease the utility of my server for my own users, so I tend to keep all of these extra settings.

fail2ban

Logging and preventing some services is all nice and good, but we need one additional way to deter attackers. fail2ban is a great tool that can scan all types of logs, look for authentication failures or other nefarious incidents, and react accordingly. This is a really powerful tool, and I’ve only started scratching its surface.

Our installation of fail2ban consists of multiple parts:

1. A fail2ban server.
2. A global configuration file fail2ban.conf, which controls general settings such as log verbosity.
3. A set of filters to detect authentication failures.
4. A set of actions to ‘ban’ and ‘unban’ IP addresses.
5. A set of jails that combine filters and actions.

After installing fail2ban, I checked the global configuration /etc/fail2ban/fail2ban.conf, but its defaults appear very good to me: everything is logged into /var/log/fail2ban.log at an INFO verbosity level. The default distribution of fail2ban comes with numerous filters, to be found in /etc/fail2ban/filter.d. There is of course an sshd.conf filter available—no need to parse log files manually.

Moreover, there is even a default jail for sshd available, and we just have to enable it:

$ cat /etc/fail2ban/jail.d/defaults-debian.conf 
[sshd]
enabled = true

After making sure that the service is enabled, we are good to go:

$ systemctl status fail2ban
● fail2ban.service - Fail2Ban Service
     Loaded: loaded (/lib/systemd/system/fail2ban.service; enabled; vendor preset: enabled)
     Active: active (running) since [...] 

That’s it! fail2ban will now happily run along, screen your SSH logs, and start banning and unbanning IP addresses automatically. Of course, this is not an entirely-foolproof way of securing your server, but it at least prevents brute-force break-in attempts, thus reducing the attack surface.

Visualising Potential Attackers

I also wanted to know where the potential attackers are coming from. Luckily, the fail2ban log output is relatively easy to parse, making it possible to create a .tsv file: From this file, it is

echo "count\tip"

awk '($(NF-1) = /Ban/){print $NF}' /tmp/fail2ban.log \
  | sort                                             \
  | jdresolve -                                      \
  | uniq -c                                          \
  | sort -n -r                                       \
  | sed -e 's/^[ \t]*//'                             \
  | sed -e 's/ /\t/'                                 \
  | head -n 100

This file can be read and processed, resulting in the following map of attackers (updated automatically every hour; larger circles indicate more attacks from a specific city or region):

What’s Next?

I am quite happy with the setup at the moment. In the future, I am considering additional measures to take: for instance, it would be interesting to automatically start an nmap scan for all IP addresses that are banned for longer than a few minutes. Going even further, I am wondering whether it would be legal to try to automatically check for known exploits, in order to ‘p0wn’ the wannabe-attacker and disable their system instead.⁴ This is all idle speculation, though, since both the legality and the practicality of such actions are questionable.

Epilogue

To answer the question of this post, I parsed the list I created above and queried another IP address database. Here’s a partial answer about the country the nefarious connection requests are originating from:

China: 30
Hong Kong: 15
United States: 8
India: 6
Germany: 6
Singapore: 5
Viet Nam: 4
Netherlands: 3
Russian Federation: 2
Colombia: 2
United Kingdom: 2
Korea, Republic of: 2
Spain: 1
Brazil: 1
Argentina: 1
Japan: 1
Indonesia: 1
Ukraine: 1
Senegal: 1
Poland: 1

Interestingly, the connections from China are all created by a single host, who is rather persistent in its brute-force attack, despite getting banned for prolonged periods of time. I don’t know what to make of this.

Stay safe, until next time!

(PS: If you are one of the persons trying to break into my server, please stop. There’s nothing of interest here, and I have a day job that keeps me sufficiently occupied.)