2019-01-20T18:52:55Z

SSH Security For Teams

Posted by Miguel Grinberg under Cloud.

Authentication Tokens

There are plenty of tutorials that cover the basic measures you need to take to secure a Linux server, including my own. What usually falls outside the scope of these tutorials is what other steps are recommended for a server that is going to be accessed by multiple people, such as a group of developers all working together as a team. Group access to a server introduces some challenges, as you will need to implement procedures to grant and revoke access as team members come and go, and do so without any compromises on security.

Basic SSH Security

A fact that a lot of people overlook is that all servers on the Internet are under constant attacks from hackers. If you think the chances your own little server is hacked are next to none, think again. Just by being on the Internet and having a public IP address assigned, your server is going to be found. Your server will be frequently prodded by these malicious scanners that are looking for machines they can penetrate and control. You can't hide from them, because they patiently scan all IP addresses!

While there is nothing you can do to prevent these attempts, there are measures you can take to ensure that they don't have any chance of success. In terms of the security of your SSH service, the important things you can do to make it much harder for an intruder to gain access are:

  • Always log in to your server using a public key (never with a password)
  • Disable logins with username and password for all users (only public key logins should be allowed)
  • Disable logins of any kind with the root account (only regular users can log in, and then elevate to superuser when needed)

If you don't know how to configure your server to work in this way, I suggest you read a more basic tutorial than this one first. One of my Flask Mega-Tutorial chapters is dedicated to the topic of server deployments and covers SSH security among other equally important topics: https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux.

Extending SSH Security to a Team

The vast majority of software developers understand the importance of securing the SSH service in their servers and have implemented some or all of the measures I have summarized in the previous section. But when it comes to team access to a server, most simply duplicate the same set up for all members of the team, which means that:

  • All members of the team have a copy of the private key that allows access to the server in their systems, and use it to log in as needed.
  • When a new member joins the team, they are given a copy of that private key
  • When a member leaves the team, they are asked to delete the key from their system

There are a few issues with this approach:

  • All the members of the team are logging into the server using the same key and same user account, making it really hard or sometimes impossible to determine who logs in to the server and when. With this system there is basically no accountability.
  • There is no easy way to ensure that when a member leaves the team they have their server access revoked, it all hinges on the developer deleting the key on good faith.
  • If the private key used by the team is compromised (for example due to a team member's laptop being stolen), it is difficult to roll out a new key, since all members of the team have to be given the new private key before the old key can be fully replaced.

In the following sections I'm going to discuss a slightly more elaborate approach that addresses all these issues.

One Key Per Developer!

As I'm sure you know, public key authentication in SSH is implemented by adding a public key to the ~/.ssh/authorized_keys file in the server, and then logging in using the matching private key. What a lot of people don't realize is that you can write a list of public keys in ~/.ssh/authorized_keys with one key per line, and all those keys are granted login access to the account.

Given that key pairs are very easy to generate, then why not require each developer in the team to use their own key? As an added benefit, this is more convenient for developers that already have a personal key that they use elsewhere, for example on open source work they do on GitHub. The important concept is to move away from the idea that a key belongs to a server, and instead associate keys with people.

Using a key per developer is much more secure. Let's review the list of issues from the previous section to see how things improve after this change is implemented:

  • It is now easier to keep track of who is logging into the server, because the SSH logs are going to show which key was used in each login, even though the same server account is still used for all members of the team.
  • When a new member joins the team, you ask them to provide their public key and add it to the ~/.ssh/authorized_keys file to grant them access.
  • When a member leaves the team, all you need to do is remove their public key from the ~/.ssh/authorized_keys file, and their access is immediately revoked.
  • If one of the keys is compromised, you just remove it from the list and it won't be accepted anymore. The affected developer will need to generate a new key and register it with the server, but the remaining team members do not need to be bothered, their access remains as secure as before.

An example ~/.ssh/authorized_keys file for a team of three people could look like this:

# Miguel
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdUT8T+hv82ZI4gidMi+Fx4jGkANvE+PvG3HTuBuCfwHu2tGGk6o69BCr0c765olCKOsyLqNEtagLap/Zz3xLrwHJkmhiy8pvFhLeat+V7bZqBXLIvz5JZ6F0l1OoosptEtZxOIlUf8Bu5ytdDwCEcMFPzV7Wr+0rSuAQXo2nD4z9LBGp7jBmCpHQEdu7XkWomqTv0KTleqXdPAo+QWUtC7fJJ90PdXMeEOmsmtBUibDWDqbZiIHo7y0XuCt/CGKP4shCIciRPeBaBo9dx8Pud4WaghfvZCmFgICFqdd4gIuKQ+2vERAz2lBn6/9gfx0gPxN68jL6s2P1Nk1A4ypNd miguelgrinberg@Miguels-MBP

# Jen
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDY3PP9srR59wI42Gt35buM/WRVcZpS+MN0U7VrbJT6AgJ9TaSlASd8DM1Bm1e6zLFaNjwXJ9GLY7xG0UOiSd66T4rmjgYjIqdwSy5sgZfNOWMrso5GCFMwzkxIjtUeezwwbS6/oQnPTvMPFpl6+NTs/5qT59jGaiQj37aBo13HkS5EG5c44OMC2mhU4mXF7PUbYLp02uTy53UGTfD5uweXIq0od/pORCDXbc6bjL4VQk7Zqyn8n0sljM4LZoSbiPkUS5GMNdnvIuUJbwib2NFVFRk/RAI8oemvQpCCJx5Ep/flY8NbMrdYLdiimdzZ0eXwI4gAnOOjm/zVBP7ZJNjD jd@new-host-2.ftrdhcpuser.net
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCn+ltAn4BDJt2kQIc0cqznx0rV+0XHIhKJC1/AhFVM3P+J+dzG8c7PWPsdjROErJRANVI5jpkG703FKF7McRL+EypioOOThNYMl/R1gcVb1+/EJeroWk/q97UsQJAkWKo3TEP+aHzzQVAto63kMiPgAP6qYUbjMPMU9a5NGa+UgV+U96mz+JE9gE+Rz7uSTk1jJpBS2wEoiMBQ0g1IevG9UDuaDjkx6bHqijNMHCZpR+LvMMdnsL5l3BVMEpvmcX+rOPBmalZxeiCQGLiFxkoWHcUUivUxpx/6Eivy3+33tl7Z5ghlp3oxrkhn+kU1MLt1bqcjPe1DLWi+l8X1s4CH jen@microsoft.com

# AG
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6fDKZm8jo/kYeBxf9tqy0iqf+Rh5xEXhl86uL/2U1jr1+A45ULUhBcU3oHrYN/ToZAKsrIrJlehGvGbrhZtWVTKaUfCA+00K+FUqBY+gT4VcwgbW+WRA/drG/e8SAlCOP/AaK6ZfweLTdhxHNu8ORDNEWuqlWycMYLAoMROwkOtaYCIxNTEH/oIOTVXEmMlswbnVLtUuqbXKaxI9IGYinfx8l+F5taoaDE0lDCg+Ao+XLo6lP76FzGvsrQQeR1xL0tNAc0+EjGTY7IoqJy1kac6wDNch7O7aUZG0PR0hmELqB11PV7w1PAsfvv3o4W8lFrRqJF+FYDcRZRg1JP5DH alicia@mac.com

As you can see, you can insert comments to help you identify the owner of each key. Also note how the second person in the team has two keys, which could be for two different machines, as many people prefer to use a different key on each device they have.

This idea can be extended to "virtual" team members as well. For example, if you have a continuous integration build that runs whenever a developer makes a commit to your source code repository, you can assign a key to the build process so that it can login to the server to deploy the changes.

The Authorized Keys Repository

One problem with the solution I presented in the previous section is that there is some management burden in terms of adding and removing keys from the ~/.ssh/authorized_keys file each time the team changes. This isn't a big deal when you have one or two servers to manage, but it could become tedious if you have a large number of them, or if you dynamically create and destroy cloud servers as part of your development workflow.

To make it easier for your team to manage access to your servers, I suggest that you put a complete copy of your authorized_keys file in a public git repository, on GitHub or similar service. Since this file only contains public keys, there is no security risk in hosting it in a public repository.

You can then manage updates to your access list with the git flow:

  • When the team receives a new member, you ask them to create a pull request on this repository adding their public keys. One of the repository admins can then merge the PR.
  • When the team has a departure, any of the remaining members can create a PR that removes the public keys for the person that is leaving. Alternatively you can bypass the PR route in this case and have an admin of the repository just make a commit with those keys removed.
  • A continuous integration build can be added to this repository. Each time a new commit is made to master, the build can use scp to refresh the authorized_keys file in all the servers used by the team. Of course this CI build process will use its own key, which will also be listed in the authorized_keys file!

The Master Key

You may want to consider having a master key for your server that also has the public key included in the authorized_keys file of your server. The idea is that in the event of being locked out of your server, you can use this key to regain access.

Of course using this key as a last resort to access your server means that you have to protect it extremely well. So this key should not be installed on anyone's computer, should not be used for more than one server, and should be kept safely stored away in a place from where it can be retrieved when an emergency arises. For example, you could keep this key in a USB stick in a safe. Or if you are paranoid, you can print the key on paper and store that instead, leaving no digital version of the file that can be stolen!

Protecting Your Team's Private Keys

When you use keys to log in to your server, a big part of keeping your server secure is to protect the private keys from being stolen. These keys have to be installed as regular files in the systems used by your team, so a stolen laptop could give an attacker a way to obtain one of the private keys.

Luckily you can take a few measures to make extracting private keys (or any other files) from a stolen laptop virtually impossible:

  • Use an encrypted file system.
  • Configure your machine to always prompt for a password when turning it on and when waking it from sleep
  • Never leave your unlocked machine unattended. If you are going to take a break, put the machine to sleep or lock it before leaving. With a laptop you can do this easily by getting used to closing the lid when you leave your desk. Of course you need to make sure the machine is configured to go to sleep when the lid is closed!

Conclusion

While I hope I'm wrong, my impression is that the techniques I presented in this article are fairly unusual, with most teams just sharing a single key to access their servers. I hope more and more teams start adopting more secure practices!

If you have any questions or have used a different configuration with your own team, let me know below in the comments.

6 comments

  • #1 Alastair said 2019-01-20T22:48:20Z

    Hi Miguel, great info! With the security approach (no root logins etc) you end up needing to use ‘sudo’ to gain privileges. In a build process tool like Terraform or any bash script you end up having to write “echo ‘password’ | sudo -S command” or similar. I don’t like this because it leaks the password in logs. Do you know of any other strategy? I haven’t found one searching. Thanks!

  • #2 Miguel Grinberg said 2019-01-20T23:17:01Z

    @Alastair: when I do my deployments I arrange everything so that I do not need to use root for day to day work. The only time I use the root account is in the initial deployment, which is the process that runs immediately after the server is created. This process installs dependencies and also copies the authorized_keys file. But if you must use root in a build script, you can configure the build user to not require a password when using sudo. Look for the NOPASSWD directive in the sudoers man page at https://linux.die.net/man/5/sudoers.

  • #3 Aurélien Gâteau said 2019-01-23T16:20:25Z

    Great article! Another protection for your private keys is to define a passphrase for them. This prevents the attacker from using it if they can get a copy of the private key file. You can use ssh-agent (coupled with good desktop locking habits!) to avoid typing it every time it is needed.

  • #4 Doug Rohm said 2019-01-23T20:54:11Z

    Great stuff here. I really like the CI build for the authorized_keys file idea for managing multiple servers!

  • #5 Ssuching Yu said 2019-01-26T15:05:01Z

    The infrastructure department in my company plans to manage server SSH keys for every developer. This article is timely and helpful for me.

  • #6 Bernardo Gomes de Abreu said 2019-12-05T10:36:59Z

    Good tips !! Keep sharing !!

Leave a Comment