SSH Audit Made Simple

If you work with systems that run any variety of Linux or BSD then the probability is high that you have dealt with SSH. Invented in 1995 and established as an internet standard by the IETF in 2006, Secure Shell has become the default mechanism for remote access to servers by individuals and teams everywhere.

SSH Authentication

Authenticating yourself to Unix or Linux servers can take a variety of forms, but the most common among them are simple pairings of username and password or a public and private cryptographic key. Key-based authentication is widely regarded as the preferred method due to the greater security that it provides.

Passwords

Using a password as a means of establishing your identity is subject to a variety of weaknesses. If you monitor the system logs of any server that has an SSH server accessible to the internet, you will see a nearly constant barrage of login attempts from attackers attempting brute force techniques to gain unauthorized access to your SSH client, often from hostnames indicating they are compromised servers. It is also essentially impossible to ensure that the user who enters a given password is the individual who that credential was intended for. Social engineering, poor password hygiene (using the same GitHub and Gmail password, for example), and shared logins all increase the probability that your servers will be compromised.

Key Pairs

While it isn't the only alternative to passwords, key-based authentication is by far the most commonly used and widely supported. The primary advantages that it provides over passwords are the ability to add a password as a second factor and the need for the private key to be present on the device that is logging in.

The first step to using this mechanism is to generate a pair of keys, one that is "public" and can be shared freely without risk of compromise, and one that is "private" which needs to be kept secret and secure. The public key or host-key is then placed on the target system(s) in an "authorized_keys" file which is owned by the user account that you would like to log in to. The private key will be placed in a default directory which varies depending on the platform that you are using. Alternatively, you can use the "ssh-agent" process to load and manage your private keys on your behalf.

When generating your key pair, there are a few choices to be made which can affect the overall security of your system. The first decision is the type and strength of the key, followed by whether to protect it with a password. Unless the key is going to be used by an automated process, it is a good practice to use a password to protect it from unauthorized use in the event that it is leaked to an attacker.

The default key formats are usually RSA or ECDSA, with RSA being the older and more established algorithm. Within those formats are the option to create keys of varying sizes. In general, a larger key is more difficult to compromise, with the tradeoff being that encrypting and decrypting messages with it will take longer, though on modern hardware, that time difference is negligible.

As an example, the following command will generate an RSA key with a size of 2048 bits that is password-protected and save it to the default location for your operating system, including Linux (Ubuntu / Unix) and OSX computers.

ssh-keygen -t rsa -b 2048 -N a_secure_password -C 'A Comment Goes Here' -f ~/.ssh/id_rsa

Server Side

On the server side, you can enforce which authentication options are supported and for which users. Assuming that you have decided to rely on key-based login, you should disable the option to use passwords on your servers. It is also advisable to disable logging in as the root user. By preventing root logins, you are forced to either grant sudo permissions to your users or configure access to specific commands and directories, both of which ensure that you are considering who should be granted which powers.

Adding the following settings to your `/etc/ssh/sshd_config` file will configure your SSH server process with the desired behaviors.

The challenge that comes with key-based authentication is that as soon as you move beyond a single user and a single server, you have to consider how to track, control, and rotate those keys. An audit tool can make this easier for teams, and StrongDM allows you to bypass message authentication code algorithms since SSH passwords or keys are never stored on the local client.

🕵 Learn how Coveo gained complete visibility across their entire stack with centralized and granular audit logs and simplified compliance audits.

Considerations For Your Key Management Strategy

There are a variety of factors that you should be thinking of as you determine how best to design and implement your SSH key management strategy. When using public-key authentication as your default (or preferably only) login mechanism, each key that is provisioned on your infrastructure grants access to someone. Access, however, is a rather nuanced concept in this situation.

Authentication

Authenticating a login session on a server requires the user who initiates that connection to have access to the combination of private key and username that are provisioned in advance. At face value this would appear to allow for easy access control and attribution of user actions, but only if each member of your team uses their own set of credentials. An additional concern is how to effectively modify or revoke access when an employee leaves the organization.

Authorization

Beyond the comparatively simple matter of access and authentication is the process of managing the authorization of a user once they have gained access to a server. What permissions should each user be given? Should you grant superuser (sudo) access to everyone, or do you need more fine-grained control over what commands are allowed?

Beyond the question of what permissions to grant and to whom is the matter of managing that set of rules in a consistent manner. The more users and systems that are under your purview, the greater the complexity and the need for a scalable solution to the problem.

Auditability

Once you have determined how to grant access to your servers and manage permissions, you will still need a way to keep a record of who did what and when. SSH server auditing trails are an essential requirement of most compliance regulations, in addition to being an important part of proper systems administration and a security best practice.

SSH audit logs allow you to determine, either retroactively or in real time, when an unauthorized or destructive action was taken, who performed that action, and the broader context of the user session at the time. By maintaining historical information with these records, it is easier, in the event of a malicious actor, to identify weaknesses or vulnerabilities in your servers to be addressed. If the damage is a result of inadequate education or guard rails for your employees, then you can review what mistakes were made and update your tooling and documentation to avoid the situation in the future.

Now you have a better idea of the challenges and concerns that are involved in the seemingly simple practice of managing SSH access to your servers. While it can be straightforward at small scales of users and machines, as you scale along one or both of those axes, the challenge and complexity of building and maintaining a secure and easy-to-use workflow grow exponentially.

Solutions

Each User Has Their Own Key

When you first start building your systems, it is common to either share credentials with anyone who needs access or to provision accounts and keys for each user on each server. Both of these solutions are manageable when you are dealing with quantities in the single or low double digits of either employees or servers.

Some of the challenges associated with the practice of letting each user manage their own keys are:

• Inconsistent key strength between users
• Keys proliferate with no oversight
• It becomes unwieldy to share keys or to provision every server with every user’s keys

We will take each of these points in turn.

Key Strength

The overall security of a system is only as strong as its weakest link. As we discussed in the last post, there are a number of options that can be specified when generating an SSH key, and some of the possible configurations can result in a key that is easily compromised. When every user is responsible for generating and maintaining their own keys, there is the potential for them to select one of these configurations, thereby putting all of your infrastructure at risk.

To avoid this situation, you can provide educational material to your employees to make them aware of what the current best practices are. In addition to, or instead of, education, you can provide tooling that only allows for known good settings when creating key pairs such as a simple bash script that embeds your preferred options and eliminates the need to look up parameters. A third option is to generate keys on their behalf, but this requires that you have an established channel for securely transferring the sensitive private key.

Key Proliferation

Another problem with allowing each of your users to create and maintain their own keys is that you have no oversight as to who has which keys and on what machines. This leads to a situation where you can never be sure whether a given SSH session is being conducted by an authorized individual or if their private key was compromised. Having no visibility into the location and controls over each individual’s keys also means that it is difficult, if not impossible, to enforce key rotation policies.

Server Provisioning

The last item to discuss relating to individual management of SSH keys is that, in order for any of those key pairs to be useful, they need to be provisioned on the servers that each user requires access to. In a small team that manages a small number of instances, this task can be performed manually or via your configuration management strategy of choice without too much difficulty.

Once you reach a measure of scale for either axis of people or servers, it becomes impractical to provision, track, and expire access across your infrastructure. At small scales, it is likely that every user has access to every server, but with growth comes an inevitable level of specialization, requiring greater care when granting permissions. In an environment where any measure of compliance is mandatory, this is even more important to get right.

Bastion Host

Once you reach the tipping point where individual key management is no longer practical or possible, the next common strategy is to use some form of bastion host. A bastion host is a server that is publicly accessible and can be provisioned with each user’s key who should be granted access to your network. From that host, the user can then access the private network and log in to the destination servers.

An initial benefit of the bastion host is that it reduces the many-to-many relationship of users to servers to a many-to-one situation. Now you only have one location to populate with your users’ keys, meaning that there is only one place to look when revoking access as well. By consolidating the potential entry points to your network, it is also possible to more closely monitor access patterns, though not the contents of the session.

This approach works particularly well when your network topology is relatively flat, and the bastion host is able to access the destination servers directly. As you add more network segments, you can either add more bastions or update the routing tables to allow your existing bastion access to the new environments. By scaling to additional bastions, you begin approaching the original problem of managing multiple keys on multiple hosts.

Another consideration when working with bastion hosts is how to control access for specific users to specific hosts. You can provision the bastion with multiple keys, each of which grants access to a specific set of servers; otherwise, you begin to enter the realm of integrating additional components such as identity management to the login path of your server. There are projects that can help with this, but they introduce another moving piece that diverts more of your time and attention away from the primary goal of solving your business problems.

In essence, a bastion host is just a proxy for connecting from your laptop to your servers, but one that requires a dedicated server and all of the maintenance and management that goes along with it.

Infrastructure Access Platform

The proxy approach to managing SSH access to your systems is a viable one in principle; what makes the difference is how it is executed in practice. The StrongDM architecture is also proxy-based, with the critical difference being how it manages authentication and authorization.

Rather than needing to integrate your identity management system with OpenSSH, or the PAM system, on each server, you only need to do it once when you set up StrongDM. Alternatively, you can use the built-in user and role management within the StrongDM admin interface. This will give you a single location to manage role-based access control for all of your servers, as well as assign your users to the appropriate role based on their needs and responsibilities.

To set up your environment to work with StrongDM, you will still need to provision keys on the destination hosts (server and host keys for SSH1 , and just host keys for SSH2), but only the one needed by the StrongDM gateway. Once that is in place, your users will request a session from the gateway and be granted a temporary credential that is only valid for single use. Upon connection, all of their activity will be logged within StrongDM for your later review and analysis. Because all of the traffic is routed through the StrongDM gateway, this audit log also applies when using a bastion or jump host to access additional servers.

By delegating authentication to StrongDM and removing it as a concern of your SSH servers, you also gain easy access to integrations with multi-factor authentication (MFA) providers. For further scaling, StrongDM natively supports hierarchical deployment patterns so that your users don’t have to keep track of which proxy to use for which environment. StrongDM also provides native SSH2 protocol server support and provides ssh version compatibility with IPv6, command line or GUI access, and drag and drop role-based access controls.

If you want to learn more about how StrongDM can simplify your SSH strategy, gain visibility as an auditing tool, and make it scale with your business, book a time today to talk to one of our experts.

To learn more about how StrongDM helps companies with auditing, make sure to check out our Auditing Use Case.