UPDATE: I upgraded the whole ssh setup on client and server to be much more secure, thanks to PengouinBSD's comment!
as I wrote in my last post I want to take care a bit about ssh connection and its security. So today's task will be:
- Create an ssh-key on my workstation and add the public key to the server's authorized keys
- Configure the ssh daemon (sshd) on the server (no root permit, no password permit, only public key)
- Do the same for each virtual machine
- Enable a
fail2banlike functionality via
pf, banning specific IP addresses that tried to connect too often or too fast
A short outline for jumping around:
Establishing an ssh connection to the server is great if you want to remotely sanitize the system and do some maintenance work on the server. But of course by opening a new way to access the server for yourself you are also opening a new way for potential attacks on your server.
But as usual there are ways to make this connection more secure than default.
I had a server setup operating openssh on OpenBSD 6.1 that was disallowing root login totally and only allowing users to login ssh via their passwords. SSH service is running on port 22. This configuration is the default if you enabled ssh and chose not to permit root login for ssh during the installation process of OpenBSD. As I did for example do in this installation guide.
I want to change that into only allowing the user to login with public keys and the Google Authenticator App as a second factor.
SSH - Public key authentication for the server and each VM
In this section I will write how I set up the ssh public key connection for my workstations and the server with the VMs running on it. This will include some paranoid precaution, the creation of the keys, the transferral of the public keys to the servers (main server and VMs) and setting up the ssh config on the workstation for establishing easy connections.
This section will be completely updated as for the comment of PengouinBSD. As I read by his resources, the public key setup that I chose was not really secure (at least it can be much more secure). At first I want to share those resources that PengouinBSD shared at the top of this new section:
The resource by Martin Kleppmann is showing how you can implement a key derivation for ssh private keys. In the header of his post he is pointing out that his article is not relevant for newer versions of OpenSSH anymore as those are supporting the
-o Option mentioned by PengouinBSD which integrates a new stronger format with key derivation.
The second resource by Ted Unangst is about a new OpenSSH key format. The keyformat is using stronger key derivation functions which are enabled by default for keys using ed25519 signatures. Also he mentions how to upgrade the old keyformat to the new one.
In the last resource stribika provides a profound explanation on how to setup your server/client key authentication. I will mainly follow his setup and make my access security much better.
For setting up all the ssh configurations I disconnected my server and my workstation from the internet (not from the LAN) so there could not be any interceptions to the internal traffic when I setup the public key authentication.
Create an ssh-key
On the workstation I wanted to be able to reach my server from, I created an ssh-key:
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -o -a 64
Copy the key to the server
Copying the key to the server for public key authentication of ssh sessions is easy.
After generating the key I needed to transfer it to my server.
There are several ways to do this. The first thing to try out is using
ssh-copy-id like so (the server's IP is at 192.168.1.250 with the user
ssh-copy-id -i ~/.ssh/id_ed25519 -p 22 firstname.lastname@example.org
There were cases when I couldn't use
ssh-copy-id, what I usually did then was:
- Opening an ssh session in a terminal to the ssh public key receiver via
ssh -p 22 email@example.com
- Open another terminal and copy the content of the public key file to the clipboard (Do not use the file
~/.ssh/id_ed25519that is the private key file which should never leave your workstation but the file
~/.ssh/id_ed25519.pubthe public key file belonging to that key)
cat ~/.ssh/id_ed25519.pub | xclip -selection clipboard
This last command needs
xclip on your workstation (on a Debian like system install it via
sudo apt install xclip).
Append the clipboard content to the authorized-keys file like so (pasting in the terminal
Ctrl-Shift-V or with an editor of your choice like vim):
echo "ssh-ed25519 AAF23409SLFKJE02394FFTtlksdfowie3422DKdcweoDweoDFJwode99973fdOdjf9WT jan@jans-workstation" >> ~/.ssh/authorized_keys
Changing ssh configuration on client and server
There are a lot of things to consider when changing the clients' and servers' configuration for OpenSSH.
For most of the configurations (nearly all), I followed the documentation by stribika. Thank you for this great setup!
Additionally I also set up a two factor authentication with the Google Authenticator App on my android device.
Setting up (client and server)
Stribika proposed to change or generate the
/etc/ssh/moduli file. Because mine already existed, I modified my
/etc/ssh/moduli file like so (explained here):
1 2 3 4
- Line 1: Get all lines from
/etc/ssh/modulithat have a fifth column of a value greater than 2000 and copy them into a file in
- Line 2: Make sure that there are still lines left in the generated file.
- Line 3: Make a backup of
moduliin case something goes wrong in the future.
- Line 4: Copy the generated file over the existing one.
SSH configuration (server)
To alter the configuration of my ssh daemon on the server, I had to manipulate the file
/etc/ssh/sshd_config as shown below.
The code snippet below shows the configuration that I definitely set for my sshd service (comments point out the different functionalities):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
New user groups and keyfiles
After changing the configuration file I created a group called
ssh-user on the server:
doas groupadd ssh-user
and added my local user
jan-server to that group:
doas user mod -G ssh-user jan-server
Then I deleted all host key files on the server and generated new ones:
cd /etc/ssh # delete the host key files existing: rm ssh_host_*key* # generate new ones that we use (only those two mentioned in the sshd_config) ssh-keygen -t ed25519 -f ssh_host_ed25519_key -N "" < /dev/null ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key -N "" < /dev/null
Google authenticator integration
As an additional instance of security I wanted to integrate Google Authenticator as a second factor. I chose that because I already had it installed on my Android device. It is also open source.
I installed the following packages:
doas pkg_add login_oath doas pkg_add node
login_oathis used for using the time based one time password compatible to the google authenticator.
nodeis used for installing a convenient wrapper for setting up the key-file for the one-time-password service.
Afterwards I installed a tool for creating the keyfile for the Google Authenticator, I read the sourcecode beforehand to be sure its safe.:
npm install -g https://github.com/WIZARDISHUNGRY/totp-util
Set up the key file and give it to my Android device
On the server I then set up the keyfile by running
totp-util as the user I would like to be able to log in with (in my case that user was
I scanned the QR-code appearing on the console with my google authenticator app on my android phone.
Configure login.conf for the OTP integration
I added a new login class to
/etc/login.conf at the end of the file (appending) like so:
1 2 3
- Line 1: the name of the new login class.
- Line 2: specifying the allowed login methods for ssh authentication to -totp (timebased one-time-password) and skey.
- Line 3: use the defaults for everything else.
I then recompiled the
doas cap_mkd /etc/login.conf
And changed my user's login class to the newly generated
doas usermod -L totppw jan-server
At the end of the server configuration
I needed to reload the configuration file for the service:
doas rcctl reload sshd
SSH configuration (client)
The configuration for the client is done in
/etc/ssh/ssh_config and corresponds with the configurations that I chose for the server:
Host * PasswordAuthentication yes ChallengeResponseAuthentication yes PubkeyAuthentication yes # this will choose the following algorithms for hostkey, ciphers and hmacs in the order from left to right (see Stribika's documentation for more info): HostKeyAlgorithms firstname.lastname@example.org,email@example.com,ssh-ed25519,ssh-rsa Ciphers firstname.lastname@example.org,email@example.com,firstname.lastname@example.org,aes256-ctr,aes192-ctr,aes128-ctr MACs email@example.com,firstname.lastname@example.org,email@example.com,firstname.lastname@example.org,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,email@example.com
That's what I can do now
Now when I do
ssh -p 13423 -i ~/.ssh/id_ed25519 firstname.lastname@example.org, I am able to connect to my server via public key authentication and google authenticator like so (password are of course not shown in terminal input but I included them for better presentation):
ssh -p 13423 -i ~/.ssh/id_ed25519 email@example.com Enter passphrase for key '.ssh/id_ed25519': mypubkeypasswd Authenticated with partial success. firstname.lastname@example.org's password: 123456
For my convenience I also added these lines into my workstation's
# ~/.ssh/config Host server-local Port 13423 HostName 192.168.1.250 User jan-server IdentityFile ~/.ssh/id_ed25519 Host server-remote Port 13423 HostName hermes-technology.de User jan-server IdentityFile ~/.ssh/id_ed25519
Now I can just do
ssh server-local when I want to connect to my server from the local network or
ssh server-remote when I want to connect from a remote place.
Configure the ssh daemon (sshd) on the virtual machines
As you may have noticed from my last posts: I have a server infrastructure, where different subdomain requests are forwarded to virtual machines on the main server.
So for being able to do the same convenient ssh maintenance on my virtual machines, without ssh'ing to my main server and then connecting to the VMs via
vmctl as explained here, I have to configure the ssh daemons on the virtual machines as well and enable the public key authentication.
I will explain how I did it on the
Because OpenBSD on the
host-vm was installed as explained in the installation guide in my last post, the ssh daemon is enabled by default and root login is prohibited.
On my workstation I logged in via ssh into my main server:
Then I logged into the
vmctl console host-vm
Thereafter I first configured the
/etc/ssh/moduli file as explained here.
Then I followed the sshd configrations on the server and did some modifications in the
line 14I changed the ssh Port to 22 (Nobody can access the ssh service on my virtual machine without reaching my main server, so if the attacker is already on the main server nothing matters anymore :-D).
- I also deleted
line 62about allowing tcp forwarding (
AllowTcpForwarding yes), because it is unlikely that I will have nested VMs in the near future so I don't need to forward ssh requests to somebody else.
Of course I also changed the user
host as this is my username on the
I followed the creation of the user group
ssh-user for allowing ssh login and integrated the Google Authenticator setup thereafter.
At the end I had to reload the
doas rcctl reload sshd.
Now also the
host-vm accepts the same ssh-key that was generated for the main server.
Access virtual machines via ssh on my workstation
For an easy connection to my virtual machines from my workstation I added some lines to my
~/.ssh/config file (for each vm):
1 2 3 4 5 6 7 8 9 10 11 12 13
- Line 3: The ssh port on the virtual machine.
- Line 4: The local IP of the virtual machine as reachable from the main server.
- Line 5: Enables the forwarding of the ssh request to the virtual machine.
- Line 6: The user on the virtual machine.
Logging in to virtual machines from my workstation now can easily be done by executing e.g.
Make PF ban IPs that do malicious ssh attempts
I wanted to block certain IPs from reconnecting via ssh if there were too many attempts with wrong credentials or if those attempts have a frequency that exceeded a certain limitation.
So in short I wanted to react to brute force attackers like
In addition I want those addresses to expire after a certain amount of time.
Enable blocking of bruteforcers
For this task I had to modify the
/etc/pf.conf and reload the firewall configuration.
I added a table that is persistent and can be updated with new members:
table <bruters> persist
pf that IPs listed in this list should be blocked by the server (this rule should be inserted early in your ruleset):
block quick from <bruters>
Then I created a rule to update the brutes table for general connection attempts (not only ssh):
pass inet proto tcp from any to $localnet port $services \ flags S/SA keep state \ (max-src-conn 80, max-src-conn-rate 15/5, \ overload <bruters> flush global)
max-src-conn 80limits the maximum number of simultaneous TCP connections on my tcp services to 80.
max-src-conn-rate 15/5limits the rate of new connections over a time interval to 15 connections in 5 seconds.
This will allow 80 connections from the same source and a connection rate of 15 connections in 5 seconds. Connections exceeding this limit will be added to the
bruters table and are blocked from any connection to my server until they are removed from the
Then I added another rule early in my ruleset that specifically blocks ssh bruteforcers. This rule is a little bit stricter on its limits:
pass quick proto tcp from any to any port 13423 \ flags S/SA keep state \ (max-src-conn 5, max-src-conn-rate 5/3, \ overload <bruters> flush global)
Enable expiration of bruterforcers over time
For expiring IP addresses in the
bruters table I added a cron job into the crontab file:
# /etc/crontab @daily root pfctl -t brutes -T expire 86400
This cronjob will be executed daily and remove entries in the
bruters table that exceed the duration of presence for 86400 seconds (24 hours).
Now I'm able to login to ssh sessions from my workstation, only using my ed25519 key and the Google Authenticator prompt as second factor.
SSH sessions can be opened as easy as:
# for server from local network ssh server-local # for server from remote network ssh server-remote # for vms from local network (exchange <vm> with e.g. host-vm as showed above) ssh <vm>-local # for vms from remote network ssh <vm>-remote
"Insecure" plain password connections are not supported anymore.
Failing ssh connections will remembered and according IPs will be banned for the specified amounts of time as presented in the last section of this post.
I'm not sure yet what I will be writing about in my next post. Maybe I will find out how to use
relayd as a substitute of
nginx for redirecting subdomain requests to my local VM IPs, then I will probably be writing about that (there is also an open question for this on stack exchange). Maybe I will also write about a backup utility that I set up as a cronjob for doing frequent backups of my main-server and the VMs.
But until then I'm happy about comments and any suggestions that you have for me, "see" you soon.