Remote Deployment Management through Reverse SSH Tunnels
In my research I’ve overseen a number of deployments of novel prototypes – usually running on an embedded system such as the Raspberry Pi – in public settings. No lab study could ever replace the rich and deeply contextual data and insight you gain from such public deployments. However, such deployments can be tricky to manage. Especially when embedded systems are connected using unreliable public WiFi or 4G networks, they can be hard to reach as they are de-facto hidden behind Network Area Translation (NAT) rules, firewalls, or proxies. One strategy that I’ve found particularly helpful, especially when working with a Raspberry Pi, is to leverage a reverse SSH tunnel. Here the Rasperry Pi connects to a server via SSH, but then also creates a tunnel between an unused port on the remote server (e.g. 2200) and an active local port on the Raspberry Pi (e.g. 22 – the port commonly used by SSH servers) that is otherwise inaccessible because of firewall or NAT rules. So long as the tunnel is active and the server is accessible you can then reach or SSH into the Raspberry Pi via the remote server. In this post, I’ll cover how to setup such a tunnel, and just as importantly, how to ensure that the tunnel remains active in contexts where electricity supplies are intermittent and 4G/WiFi networks are unreliable. I do this with the help of:
- SSH keys that enable passwordless authentication
- SSH tunneling also referred to as SSH port forwarding
- autossh a program that automatically restarts SSH sessions and tunnels
- systemd service manager
Creating and securing user accounts
We begin by creating a dedicated user –
autotunnel – in charge of the tunnel. Since we only plan on using the
autotunnel user account to create a reverse ssh tunnel, we set the shell to
/sbin/nologin. That way if the
autotunnel user logs in, they’ll get a polite message saying: “This account is currently not available.” This is similar to many “user” accounts on Linux/Unix systems that do not have a valid shell, as they are only used to execute specific programs.
# Create autotunnel user account with /sbin/nologin shell # pi@raspberrypi:~ $ sudo useradd -m -s /sbin/nologin autotunnel
You’ll also notice that the
adduser command does not prompt for a password. This is further assurance that no user will be able to login with the
autotunnel account, since accounts without a password are disabled. The only exception to this rule is the
root (or superuser) account which can access and execute programs as any user – even disabled ones such as our
autotunnel user account.
We leverage this fact in the next step, where we create an SSH Key for the
autotunnel user on the Raspberry Pi through the
sudo command, which provides superuser (or root) access to the
autotunnel account. SSH Keys serve similar functions to user names and passwords, but are primarily used for automated processes. We’ll use the SSH Key as a passwordless form of authentication when connecting to the remote server, so you do not want to enter a passphrase.
# create SSH Key for autotunnel user. Do not use a passphrase here. # pi@raspberrypi:~ $ sudo -u autotunnel -- ssh-keygen -t ed25519
This produces the following output:
Generating public/private ed25519 key pair. Enter file in which to save the key (/home/autotunnel/.ssh/id_ed25519): Created directory '/home/autotunnel/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/autotunnel/.ssh/id_ed25519. Your public key has been saved in /home/autotunnel/.ssh/id_ed25519.pub. The key fingerprint is: SHA256:swjDpidQkbe7DamljdKFxBy5nGnDaPph/XQ6eRI288I autotunnel@raspberrypi The key's randomart image is: +--[ED25519 256]--+ | .o | | +.. | | *.* . | |.o%.. | |+o +=o S | |..+oBo*..o | | +oO.O.O. | |. *oo E o | | . = | +----[SHA256]-----+
Next we’ll configure a corresponding
autotunnel user on the remote server.
Configure the remote server
On the server side, we want the
autotunnel user to be able to login to the server. So this time we give the
autotunnel user access to the
# user@server:~ $ sudo useradd -m -s /bin/bash autotunnel
We also set a new, random 12 character password (including symbols & numbers) for the user using a secure password generator (
pwgen) and the
# install password generator: pwgen # user@server:~ $ sudo apt install pwgen
# generate, set, and print password for autotunnel user # user@server:~ $ sudo autotunnelpass=`pwgen -s -y 12 -N 1` \ sh -c 'printf "autotunnel:$autotunnelpass" | chpasswd && printf " Username: autotunnel\n Password: $autotunnelpass"'
Make a note of the password that was generated for the
autotunnel user account, as we’ll need it in the next step to login to the server from the Raspberry Pi.
Username: autotunnel Password: _i\+.I^618^E
Copy SSH Keys
To enable passwordless and thus automated logins between the Raspberry Pi and the Remote Server, you’ll need to copy the newly created SSH keys to the server.
# Copy SSH Key from Raspberry Pi to Server # pi@raspberrypi:~ $ sudo -u autotunnel -- ssh-copy-id server.url.com
When prompted enter the password of the
autotunnel user account on the server.
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/autotunnel/.ssh/id_ed25519.pub" The authenticity of host 'server.url.com (188.8.131.52)' can't be established. ECDSA key fingerprint is SHA256:QMuoMB7QNaUFy9ksp7bMPzvZVrvzQeJAfWqLmWAjh38. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys firstname.lastname@example.org's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'server.url.com'" and check to make sure that only the key(s) you wanted were added.
After completing the steps above you should now be able log in to the remote server without being prompted for a password, as the connecting is authenticated using the SSH key we just copied.
# Login to the server using SSH Keys # pi@raspberrypi:~ $ sudo -u autotunnel -- ssh server.url.com
You should now see the shell prompt of the
autotunnel user on the server. If you type
exit, you’ll return back to the Raspberry Pi.
autotunnel@server:~ $ exit
Make the SSH tunnel accessible externally
If you want to connect to the pi directly through your server’s external IP address, you’ll need to ensure that the SSH tunnel is bound to your server’s external address.
To do this, edit
/etc/ssh/sshd_config and go to
GatewayPorts and enable it by setting it to
yes. You’ll need to restart the ssh service on the server for this change to take effect. Be careful when you do this, as any error in the config file could result in the ssh server not starting and you’ll be locked out of the server.
# user@server:~ $ sudo systemctl restart ssh
Configure the reverse tunnel
If you haven’t already, make sure that SSH is enabled on the Pi using the
raspi-config command. When enabling SSH on a Pi that may be connected to the internet, you should change its default password to ensure that it remains secure.
# enable ssh # pi@raspberrypi:~ $ sudo raspi-config
- Navigate to and select
We can then setup the reverse tunnel between the Raspberry Pi and the remote server. We’ll forward the remote TCP port
2200 on the server to the local TCP port
22 on the Raspberry Pi. This is achieved using the
ssh -R 2200:localhost:22 command switch. The standard TCP port for SSH is 22, so you’ll be able to SSH into the Raspberry Pi via port
2200 on the remote server.
# setup reverse ssh tunnel between raspberry pi & server # pi@raspberrypi:~ $ sudo -u autotunnel -- \ ssh -R 2200:localhost:22 server.url.com
Next check if the SSH Tunnel is properly setup.
# Check that the SSH Tunnel is bound correctly # autotunnel@server:~ $ sudo netstat -lntp
Make sure that the tunnel is bound to the Local Address 0.0.0.0:2200, if you want to utilise the tunnel from the server’s external address. If it shows 127.0.0.1:2200 instead, you’ll only be able to utilise the SSH tunnel while logged into the server.
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:2200 0.0.0.0:* LISTEN 626/sshd: autotunne tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 670/sshd tcp6 0 0 :::2200 :::* LISTEN 626/sshd: autotunne tcp6 0 0 :::22 :::* LISTEN 670/sshd
You should now be able to
ssh back to the Pi on the server’s local port
# Connect back to Raspberry Pi # autotunnel@server:~ $ ssh -p 2200 pi@localhost
You’ll likely need to accept the SSH Key fingerprint and enter the password of the
pi user of the Raspberry Pi, in order to connect. You should see the familiar shell prompt of the Raspberry Pi.
The authenticity of host '[localhost]:2200 ([::1]:2200)' can't be established. ECDSA key fingerprint is SHA256:QMuoMB7QNaUFy9ksp7bMPzvZVrvzQeJAfWqLmWAjh38. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '[localhost]:2200' (ECDSA) to the list of known hosts. pi@localhost's password: Linux raspberrypi 5.4.83-v7l+ #1379 SMP Mon Dec 14 13:11:54 GMT 2020 armv7l The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Wed Mar 3 13:47:16 2021 from ::1 pi@raspberrypi:~ $
If this doesn’t work, you’ll likely need to tweak the SSH or Firewall settings on the Pi.
You can also try connecting to the pi from a different computer using the server’s external network interface.
# Test if pi can be accessed through server's external network interface # user@anycomputer:~ $ ssh -p 2200 email@example.com
If this doesn’t work, you’ll likely need to tweak the remote server’s firewall settings and enable incoming TCP connections to port
2200 to allow connections to the Pi from outside of the server.
Now that we tested the connection, you can close all ssh connection with the
exit command. We’ll next turn our attention to automating and persisting the reverse tunnel.
Maintaining the SSH Tunnel through AutoSSH
To proactively maintain the SSH tunnel, we begin by installing the
autossh package, which as the name suggests, is used to automate establishing, monitoring, and if necessary restarting an SSH connection.
# install autossh # pi@raspberrypi:~ $ sudo apt install autossh
Next we test the
autossh connection using the parameters explained below.
||Do not execute remote command (only establish a tunnel)|
||Disable legacy monitoring. Use below options for monitoring.|
||Close current connection if 3 consecutive alive messages (see above) did not receive a reply|
||Terminate current ssh connection if unable to setup port forwarding tunnel|
||Forward remote port
# test autossh connection # pi@raspberrypi:~ $ sudo -u autotunnel -- \ autossh -N -M 0 -o "ServerAliveInterval 10" -o "ServerAliveCountMax 3" -o "ExitOnForwardFailure=yes" -R 2200:localhost:22 -vvv firstname.lastname@example.org
We’ll forward the remote port
2200 on the server to the local port
22 on the Raspberry Pi. The
-vvv command switch increases the verbosity level. So initially you’ll get a range of messages ending with the following confirmation.
... debug1: remote forward success for: listen 2200, connect localhost:22 debug1: All remote forwarding requests processed
And then every 30 seconds a null packet gets sent to keep the connection alive.
debug3: send packet: type 80 debug3: receive packet: type 82
If this works, you can terminate the (auto)ssh connection.
Secure the server
Now that we have confirmed that the reverse SSH tunnel is working correctly, and that
autossh is able to keep the connection alive and restart it when it fails, we can take additional steps to secure the server. First we can restrict ssh access to the
autotunnel user account on the server to only allow port-forwarding. To do this edit
~/.ssh/authorized_keys file, and find the entry that corresponds to the
autotunnel user on the raspberry pi, it should start with
ssh-ed25519 and end with
autotunnel@raspberrypi. Next add the following configurations to the beginning of that line:
These parameters limit access to the
autotunnel user on the server to port-forwarding and listening on port
2200. After making the edits the line should look similar to this:
restrict,port-forwarding,permitlisten="localhost:2200" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKVgR4S8boThO5fj8hmPEUewyj2XeQvemO0Lic5tpSwi autotunnel@raspberrypi
Finally, as an added layer of protection, we can also remove the password of the account. This effectively disables username/password based authentication and only allows SSH key authentication we configured earlier.
# Remove user password # user@server:~ $ sudo passwd -d autotunnel
Automatically starting the tunnel
The last piece of the puzzle is to automatically start the tunnel using a
systemd service configured in the
/etc/systemd/system/autossh.service file on the Raspberry Pi:
# /etc/systemd/system/autossh.service [Unit] Description=Keeps an ssh tunnel to server.url.com open # Run after network connections are established and the ssh server is started After=network-online.target ssh.service [Service] # Disable monitoring & gatetime Environment="AUTOSSH_PORT=0" Environment="AUTOSSH_GATETIME=0" # Restart on failure RestartSec=3 Restart=always ExecStart=/usr/bin/autossh -NT -o "ExitOnForwardFailure=yes" -o "ServerAliveInterval=10" -o "ServerAliveCountMax=3" -i /home/autotunnel/.ssh/id_ed25519 -R 2200:127.0.0.1:22 email@example.com TimeoutStopSec=10 [Install] WantedBy=multi-user.target
Finally, we can start and enable the autossh service. Here enabling the service means it will start at boot while starting a service is a once off operation.
# Start and enable the # pi@raspberrypi:~ $ sudo systemctl start autossh.service sudo systemctl enable autossh.service