A guide to a "safer" SSH!
August 11, 2019
Now after working on remote VPS for a good decent time, I have understood that it's not difficult to connect to remote nodes via SSH. Rather the concern is to connect to these remote nodes in a more safe and secure manner. Therefore, when this thought is taken into consideration, we need to work on certain things to ensure the security aspect.
So, this post will walk you through a demonstrated list of almost all the major steps that can be taken in this regard.
And now that when I am trying to learn writing "infrastructure as code" using ansible. I will try implementing this task of efficient configuration management, SSH-hardening & SSH-security via writing ansible roles again.
I recommend you to go and checkout some of the previous posts on ansible and iptables. That way, it would be quite easy for you to understand the norms used later in this post.
Taking this basic foundation further and not wasting a second more now, let's quickly start writing our ansible code for the today's use-case.
PREREQUISITES:
- A newly created server/VM. (I will be writing it for Ubuntu or Debian based machines, but you can use the same code for other distros as well, with little alterations)
- An ansible controller machine (i.e. a machine with ansible installed on it). It could be your local machine or a remote server again which can connect to other nodes via SSH.
- And lastly, an ansible "hosts" file made ready with the IP addresses of your required remote servers.
TODO's:
- Create a non-root user with sudo access.
- Upgrade all the installed packages to ensure that they are in their latest state.
- Install few more basic required packages to make the initial configuration management easier.
- Copy the local SSH key to the remote node to enable passwordless logins.
- Perform SSH-hardening via altering sshd_config file in accordance with some basic security measures.
- Create some basic required iptables rules to improve and ensure security.
- Setup fail2ban to prevent SSH brute-force connection attacks.
IMPLEMENTATION:
- The very first step is to create an ansible directory in our ansible-controller machine. This will contain the main provisioning ansible playbook, "playbook.yml" file and, all the other required ansible roles. (you can make this directory anywhere in your machine. I am creating one for me in "\etc\ansible\" directory path). Run the following commands for the same:
$ sudo mkdir playbooks
$ cd playbooks
$ sudo touch playbook.yml
$ sudo ansible-galaxy init users
$ sudo ansible-galaxy init packages
$ sudo ansible-galaxy init ssh
$ sudo ansible-galaxy init iptables
$ sudo ansible-galaxy init fail2banAnd this will give you a directory tree structure like one shown below ( and all the nested roles will be similar to ones like "users").
.
├── playbook.yml
├── users
│ ├── defaults
│ │ └── main.yml
│ ├── files
│ ├── handlers
│ │ └── main.yml
│ ├── meta
│ │ └── main.yml
│ ├── README.md
│ ├── tasks
│ │ └── main.yml
│ ├── templates
│ ├── tests
│ │ ├── inventory
│ │ └── test.yml
│ └── vars
│ └── main.yml
├── packages
├── ssh
├── iptables
└──fail2ban
- The next step is to write our main provisioning ansible playbook, "playbook.yml". This will define the flow of commands to be run later on, in order to achieve the purpose. Meanwhile, we will create some variables as well here, to facilitate the latter tasks of multiple ansible roles like username, password etc.
---
- name: Provisioning a new SSH-hardened and more secure server.
hosts: sshnodes
become_user: root
become: true
become_method: sudo
vars:
username: testuser
# For password, we are required to pass a password hash/digest.
# So, I will be using python's hashlib module to create secure hash(or message digest) for our password.
# passwordhash = python -c 'import hashlib; print(hashlib.pbkdf2_hmac('sha256', b'examplepassword', b'examplesalt', 100000))'
password: passwordhash
publickey: ~/.ssh/id_rsa.pub
roles:
- users
- packages
- ssh
- iptables
- fail2ban
...Here, we have our driving file ready. So, let's move further writing the actual hardening elements now.
Role: users
- Change directory (cd) to "/users/tasks/main.yml" and write the basic tasks to check for the pre-requisites of creating a non-root sudo user.
- A new thing, we will see here is "lineinfile" ansible module which is used to alter some specific lines in a file (file given through "dest") using regexp (regular expressions).
- The tasks written in this role will perform the following jobs sequentially:
- Check/Ensure whether "wheel" group is present or not. If not, it will create one.
- Check whether the "wheel" group has sudo privileges or not. If not, check for the given regexp "^%wheel" (line starting with wheel) in the "/etc/sudoers" file and replace it with "%wheel ALL=(ALL:ALL) ALL".
- Next comes installing the "sudo" package if it is not there.
- And finally, create the non-root sudo user account as per the variables specified in "playbook.yml".
---
# tasks file for users
- name: Ensure wheel group is present
group:
name: wheel
state: present
- name: Ensure wheel group has sudo privileges
lineinfile:
dest: /etc/sudoers
state: present
regexp: "^%wheel"
line: "%wheel ALL=(ALL:ALL) ALL"
validate: "/usr/sbin/visudo -cf %s"
- name: Install the `sudo` package
package:
name: sudo
state: latest
- name: Create the non-root user account
user:
name: ""
password: ""
shell: /bin/bash
update_password: on_create
groups: wheel
append: yes
...Role: packages
- This role is a pretty simple one and aims to keep the required packages intact and to ensure automatic upgrades of these packages.
- Change directory to "/packages/tasks/main.yml" and create tasks to do the following:
- Upgrade all the already installed/available packages in the remote node.
- Install some extra packages which, in our case, are vim, htop and net-utils.
- Finally, install the "Unattended-upgrades" package.
- And copy the corresponding "20-auto-upgrdaes.j2" configuration file from the current role's template path to remote nodes "/etc/apt/apt.conf.d/20auto-upgrades" file. As we are trying to copy the data into a root privileged directory, we need to change the owner, group and mode as well.
---
# tasks file for packages
- name: Upgrading all packages (Ubuntu/Debian)
apt:
upgrade: dist
- name: Install a few more packages
package:
name: ""
state: present
with_items:
- vim
- htop
- net-tools
- name: Install the `unattended-upgrades` package
package:
name: unattended-upgrades
state: present
- name: Copy the `20auto-upgrades` configuration file
copy:
src: /etc/ansible/playbooks/packages/templates/20-auto-upgrdaes.j2
dest: /etc/apt/apt.conf.d/20auto-upgrades
owner: root
group: root
mode: 0644
...- Copy the following configuration in "packages/templates/20auto-upgrdades.j2" file to enable automatic security upgrades meanwhile ensuring that the server won't automatically reboot when these updates requires for it.
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";Role: ssh
- This role aims to achieve passwordless remote logins via passing local SSH-key to remote node's newly created non-root sudo user account.
- Change directory to "/ssh/tasks/main.yml" and create tasks to do the following:
- Check for the local machines SSH public key at the location provided in the "vars" section of "playbook.yml" and copy it to remote node at ".ssh/authorized_keys" file.
- Perform SSH-hardening again using ansible's "lineinfile" module to alter the default ansible configuration to a more secure and restricted one.
- finally, in the last task, restart the sshd daemon service to reflect the configuration changes.
---
# tasks file for ssh
- name: Add local public key for key-based SSH authentication
authorized_key:
user: ""
state: present
key: ""
with_fileglob: public_keys/*.pub
- name: Harden sshd configuration
lineinfile:
dest: /etc/ssh/sshd_config
regexp: ""
line: ""
state: present
with_items:
- regexp: "^#?PermitRootLogin"
line: "PermitRootLogin no"
- regexp: "^^#?PasswordAuthentication"
line: "PasswordAuthentication no"
- regexp: "^#?AllowAgentForwarding"
line: "AllowAgentForwarding no"
- regexp: "^#?AllowTcpForwarding"
line: "AllowTcpForwarding no"
- regexp: "^#?MaxAuthTries"
line: "MaxAuthTries 2"
- regexp: "^#?MaxSessions"
line: "MaxSessions 2"
- regexp: "^#?TCPKeepAlive"
line: "TCPKeepAlive no"
- regexp: "^#?UseDNS"
line: "UseDNS no"
- regexp: "^#?AllowAgentForwarding"
line: "AllowAgentForwarding no"
- name: Restart sshd
systemd:
state: restarted
daemon_reload: yes
name: sshd
...Role: iptables
- If you are done with reading this previous article on iptables, you all how know what iptables are meant for. So, I am skipping directly to the tasks for this ansible role here.
- Change directory to "/ssh/tasks/main.yml" and create tasks to do the following:
- Check for "iptables" package in the remote node. If it's not present, install it.
- Flush all the already written iptables firewall rules to start from scratch.
- Create a firewall rule to allow all loopback traffic that might exist between various applications and services on the remote node.
- Create another firewall rule to allow established connections like already establish SSH connections for both outgoing and incoming packet transfer.
- The following next 4 firewall rules are meant to allow or open the frequently required ports i.e "ping, 22/ssh, 80/http & 443/https".
- The next task appends another rule to "drop" any other traffic (packets) which don't match the above defined firewall rules.
- And finally in order to retain these ephemeral rules, install "netfilter-persistent" and "iptables-persistent" packages.
---
# tasks file for iptables
- name: Install the `iptables` package
package:
name: iptables
state: latest
- name: Flush existing firewall rules
iptables:
flush: true
- name: Firewall rule - allow all loopback traffic
iptables:
action: append
chain: INPUT
in_interface: lo
jump: ACCEPT
- name: Firewall rule - allow established connections
iptables:
chain: INPUT
ctstate: ESTABLISHED,RELATED
jump: ACCEPT
- name: Firewall rule - allow port ping traffic
iptables:
chain: INPUT
jump: ACCEPT
protocol: icmp
- name: Firewall rule - allow port 22/SSH traffic
iptables:
chain: INPUT
destination_port: 22
jump: ACCEPT
protocol: tcp
- name: Firewall rule - allow port80/HTTP traffic
iptables:
chain: INPUT
destination_port: 80
jump: ACCEPT
protocol: tcp
- name: Firewall rule - allow port 443/HTTPS traffic
iptables:
chain: INPUT
destination_port: 443
jump: ACCEPT
protocol: tcp
- name: Firewall rule - drop any traffic without rule
iptables:
chain: INPUT
jump: DROP
- name: Firewall rule - drop any traffic without rule
iptables:
chain: INPUT
jump: DROP
- name: Install `netfilter-persistent` && `iptables-persistent` packages
package:
name: ""
state: present
with_items:
- iptables-persistent
- netfilter-persistent
...Role: fail2ban
- This role again is a simple one to help prevent brute force attacks on the remote node.
- Copy the following custom fail2ban configurations in "/fail2ban/templates/jail.local.j2" file. This configuration instructs to limit the maximum tries to establish a SSH connection to 3. Also, it bans the trouble-making node's IP address for an hour.
[DEFAULT]
# Ban hosts for one hour:
bantime = 3600
# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport
[sshd]
enabled = true
maxretry = 3- Change directory to "/fail2ban/tasks/main.yml" and create tasks to do the following:
- Install "fail2ban" package.
- Overide the basic fail2ban configurations by the custom "jail.local.j2" file present in the templates section of the current role directory. Again, we need to change the owner, group and mode as well because of the path being a privileged one.
---
# tasks file for fail2ban
- name: Install the `fail2ban` package
package:
name: fail2ban
state: latest
- name: Override some basic fail2ban configurations
copy:
src: /etc/ansible/playbooks/fail2ban/templates/jail.local.j2
dest: /etc/fail2ban/jail.local
owner: root
group: root
mode: 0644
...So, finally we are done writing our all 5 ansible roles required to establish and ensure a secure SSH connection, the last thing is to try testing if all the plays work fine way or not.
$ ansible-playbook playbook.ymland if you get an output like one shown below. You are good to go.
____________
< PLAY RECAP >
------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
23.251.151.76 : ok=24 changed=9 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 Cheers, We have reached our goal for this article.
But I have something more before I end.
The above ansible playbook which we have written for securing the SSH remote connection is undoubtedly serving our purpose to a large extend. But nothing can be made perfect in just one go.
It is always recommended to keep an eye on the system's processes and services to get a deeper insight of what things are working properly and what other things requires more attention from our end. And for this purpose, an auditing tool sounds handy.
So, our last step would be to quickly setup an auditing tool for us, here, the Lynis.
Lynis is a battle-tested security tool for systems running Linux, macOS, or Unix-based operating system. It performs an extensive health scan of your systems to support system hardening and compliance testing. The project is open source software with the GPL license and available since 2007.
- SSH into our newly deployed VM and run the following commands to install lynis from the official git repository.
$ sudo apt-get install git
$ git clone https://github.com/CISOfy/lynis
$ sudo chown -R 0:0 lynis
$ cd lynis- And finally run an audit with the following commands.
$ su -
# ./lynis audit systemAnd after a long dedicated scan, it will give you a detailed report which includes measures like Hardening index, Tests performed etc.
As we see, the Lynis scan on our newly deployed VM (hardened using the above playbook) has a Hardening index of 74 which is highly appreciable for our work.
Along with this report, it will also output warnings and suggestions that can be used to improve the results to higher extents. You can check the entire Lynis scan log at "/var/log/lynis.log".
And bam! 🙌 This is the end of this article. :D
[UPDATE: I have improved some part of the above playbook for optimized execution and simpler look. You can check the updated post here. ]