Remote Deployment Management through Reverse SSH Tunnels

In my research I’ve overseen a number of deployments of novel prototypes – usually running on an embedded system such as the Raspberry Pi – in public settings. No lab study could ever replace the rich and deeply contextual data and insight you gain from such public deployments. However, such deployments can be tricky to manage. Especially when embedded systems are connected using unreliable public WiFi or 4G networks, they can be hard to reach as they are de-facto hidden behind Network Area Translation (NAT) rules, firewalls, or proxies. One strategy that I’ve found particularly helpful, especially when working with a Raspberry Pi, is to leverage a reverse SSH tunnel. Here the Rasperry Pi connects to a server via SSH, but then also creates a tunnel between an unused port on the remote server (e.g. 2200) and an active local port on the Raspberry Pi (e.g. 22 – the port commonly used by SSH servers) that is otherwise inaccessible because of firewall or NAT rules. So long as the tunnel is active and the server is accessible you can then reach or SSH into the Raspberry Pi via the remote server. In this post, I’ll cover how to setup such a tunnel, and just as importantly, how to ensure that the tunnel remains active in contexts where electricity supplies are intermittent and 4G/WiFi networks are unreliable. I do this with the help of:

• SSH keys that enable passwordless authentication
• SSH tunneling also referred to as SSH port forwarding
• autossh a program that automatically restarts SSH sessions and tunnels
• systemd service manager

Creating and securing user accounts

We begin by creating a dedicated user – autotunnel – in charge of the tunnel. Since we only plan on using the autotunnel user account to create a reverse ssh tunnel, we set the shell to /sbin/nologin. That way if the autotunnel user logs in, they’ll get a polite message saying: “This account is currently not available.” This is similar to many “user” accounts on Linux/Unix systems that do not have a valid shell, as they are only used to execute specific programs.

# Create autotunnel user account with /sbin/nologin shell
# pi@raspberrypi:~ $
sudo useradd -m -s /sbin/nologin autotunnel

You’ll also notice that the adduser command does not prompt for a password. This is further assurance that no user will be able to login with the autotunnel account, since accounts without a password are disabled. The only exception to this rule is the root (or superuser) account which can access and execute programs as any user – even disabled ones such as our autotunnel user account.

We leverage this fact in the next step, where we create an SSH Key for the autotunnel user on the Raspberry Pi through the sudo command, which provides superuser (or root) access to the autotunnel account. SSH Keys serve similar functions to user names and passwords, but are primarily used for automated processes. We’ll use the SSH Key as a passwordless form of authentication when connecting to the remote server, so you do not want to enter a passphrase.

# create SSH Key for autotunnel user. Do not use a passphrase here.
# pi@raspberrypi:~ $
sudo -u autotunnel -- ssh-keygen -t ed25519

This produces the following output:

Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/autotunnel/.ssh/id_ed25519):
Created directory '/home/autotunnel/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/autotunnel/.ssh/id_ed25519.
Your public key has been saved in /home/autotunnel/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:swjDpidQkbe7DamljdKFxBy5nGnDaPph/XQ6eRI288I autotunnel@raspberrypi
The key's randomart image is:
+--[ED25519 256]--+
|  .o             |
|  +..            |
| *.* .           |
|.o%..            |
|+o +=o  S        |
|..+oBo*..o       |
| +oO.O.O.        |
|. *oo E o        |
| .     =         |
+----[SHA256]-----+

Next we’ll configure a corresponding autotunnel user on the remote server.

Configure the remote server

On the server side, we want the autotunnel user to be able to login to the server. So this time we give the autotunnel user access to the /bin/bash shell.

# user@server:~ $
sudo useradd -m -s /bin/bash autotunnel

We also set a new, random 12 character password (including symbols & numbers) for the user using a secure password generator (pwgen) and the chpasswd command.

# install password generator: pwgen
# user@server:~ $
sudo apt install pwgen

# generate, set, and print password for autotunnel user
# user@server:~ $
sudo autotunnelpass=`pwgen -s -y 12 -N 1` \
  sh -c 'printf "autotunnel:$autotunnelpass" | chpasswd && printf " Username: autotunnel\n Password: $autotunnelpass"'

Make a note of the password that was generated for the autotunnel user account, as we’ll need it in the next step to login to the server from the Raspberry Pi.

 Username: autotunnel
 Password: _i\+.I^618^E

Copy SSH Keys

To enable passwordless and thus automated logins between the Raspberry Pi and the Remote Server, you’ll need to copy the newly created SSH keys to the server.

# Copy SSH Key from Raspberry Pi to Server
# pi@raspberrypi:~ $
sudo -u autotunnel -- ssh-copy-id server.url.com

When prompted enter the password of the autotunnel user account on the server.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/autotunnel/.ssh/id_ed25519.pub"
The authenticity of host 'server.url.com (239.136.124.174)' can't be established.
ECDSA key fingerprint is SHA256:QMuoMB7QNaUFy9ksp7bMPzvZVrvzQeJAfWqLmWAjh38.

Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
autotunnel@server.url.com's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'server.url.com'"
and check to make sure that only the key(s) you wanted were added.

After completing the steps above you should now be able log in to the remote server without being prompted for a password, as the connecting is authenticated using the SSH key we just copied.

# Login to the server using SSH Keys
# pi@raspberrypi:~ $
sudo -u autotunnel -- ssh server.url.com

You should now see the shell prompt of the autotunnel user on the server. If you type exit, you’ll return back to the Raspberry Pi.

autotunnel@server:~ $ exit

Make the SSH tunnel accessible externally

If you want to connect to the pi directly through your server’s external IP address, you’ll need to ensure that the SSH tunnel is bound to your server’s external address.

To do this, edit /etc/ssh/sshd_config and go to GatewayPorts and enable it by setting it to yes. You’ll need to restart the ssh service on the server for this change to take effect. Be careful when you do this, as any error in the config file could result in the ssh server not starting and you’ll be locked out of the server.

# user@server:~ $
sudo systemctl restart ssh

Configure the reverse tunnel

If you haven’t already, make sure that SSH is enabled on the Pi using the raspi-config command. When enabling SSH on a Pi that may be connected to the internet, you should change its default password to ensure that it remains secure.

# enable ssh
# pi@raspberrypi:~ $
sudo raspi-config

• Select Interfacing Options
• Navigate to and select SSH
• Choose Yes
• Select Ok
• Choose Finish

We can then setup the reverse tunnel between the Raspberry Pi and the remote server. We’ll forward the remote TCP port 2200 on the server to the local TCP port 22 on the Raspberry Pi. This is achieved using the ssh -R 2200:localhost:22 command switch. The standard TCP port for SSH is 22, so you’ll be able to SSH into the Raspberry Pi via port 2200 on the remote server.

# setup reverse ssh tunnel between raspberry pi & server
# pi@raspberrypi:~ $
sudo -u autotunnel -- \
ssh -R 2200:localhost:22 server.url.com

Next check if the SSH Tunnel is properly setup.

# Check that the SSH Tunnel is bound correctly
# autotunnel@server:~ $
sudo netstat -lntp

Make sure that the tunnel is bound to the Local Address 0.0.0.0:2200, if you want to utilise the tunnel from the server’s external address. If it shows 127.0.0.1:2200 instead, you’ll only be able to utilise the SSH tunnel while logged into the server.

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:2200            0.0.0.0:*               LISTEN      626/sshd: autotunne
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      670/sshd
tcp6       0      0 :::2200                 :::*                    LISTEN      626/sshd: autotunne
tcp6       0      0 :::22                   :::*                    LISTEN      670/sshd

You should now be able to ssh back to the Pi on the server’s local port 2200.

# Connect back to Raspberry Pi
# autotunnel@server:~ $
ssh -p 2200 pi@localhost

You’ll likely need to accept the SSH Key fingerprint and enter the password of the pi user of the Raspberry Pi, in order to connect. You should see the familiar shell prompt of the Raspberry Pi.

The authenticity of host '[localhost]:2200 ([::1]:2200)' can't be established.
ECDSA key fingerprint is SHA256:QMuoMB7QNaUFy9ksp7bMPzvZVrvzQeJAfWqLmWAjh38.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:2200' (ECDSA) to the list of known hosts.
pi@localhost's password:
Linux raspberrypi 5.4.83-v7l+ #1379 SMP Mon Dec 14 13:11:54 GMT 2020 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Mar  3 13:47:16 2021 from ::1
pi@raspberrypi:~ $

If this doesn’t work, you’ll likely need to tweak the SSH or Firewall settings on the Pi.

You can also try connecting to the pi from a different computer using the server’s external network interface.

# Test if pi can be accessed through server's external network interface
# user@anycomputer:~ $
ssh -p 2200 pi@server.url.com

If this doesn’t work, you’ll likely need to tweak the remote server’s firewall settings and enable incoming TCP connections to port 2200 to allow connections to the Pi from outside of the server.

Now that we tested the connection, you can close all ssh connection with the exit command. We’ll next turn our attention to automating and persisting the reverse tunnel.

Maintaining the SSH Tunnel through AutoSSH

To proactively maintain the SSH tunnel, we begin by installing the autossh package, which as the name suggests, is used to automate establishing, monitoring, and if necessary restarting an SSH connection.

# install autossh
# pi@raspberrypi:~ $
sudo apt install autossh

Next we test the autossh connection using the parameters explained below.

# test autossh connection
# pi@raspberrypi:~ $
sudo -u autotunnel -- \
autossh -N -M 0 -o "ServerAliveInterval 10" -o "ServerAliveCountMax 3" -o "ExitOnForwardFailure=yes" -R 2200:localhost:22 -vvv autotunnel@server.url.com

We’ll forward the remote port 2200 on the server to the local port 22 on the Raspberry Pi. The -vvv command switch increases the verbosity level. So initially you’ll get a range of messages ending with the following confirmation.

...
debug1: remote forward success for: listen 2200, connect localhost:22
debug1: All remote forwarding requests processed

And then every 30 seconds a null packet gets sent to keep the connection alive.

debug3: send packet: type 80
debug3: receive packet: type 82

If this works, you can terminate the (auto)ssh connection.

Secure the server

Now that we have confirmed that the reverse SSH tunnel is working correctly, and that autossh is able to keep the connection alive and restart it when it fails, we can take additional steps to secure the server. First we can restrict ssh access to the autotunnel user account on the server to only allow port-forwarding. To do this edit ~/.ssh/authorized_keys file, and find the entry that corresponds to the autotunnel user on the raspberry pi, it should start with ssh-ed25519 and end with autotunnel@raspberrypi. Next add the following configurations to the beginning of that line:

restrict,port-forwarding,permitlisten="localhost:2200"

These parameters limit access to the autotunnel user on the server to port-forwarding and listening on port 2200. After making the edits the line should look similar to this:

restrict,port-forwarding,permitlisten="localhost:2200" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKVgR4S8boThO5fj8hmPEUewyj2XeQvemO0Lic5tpSwi autotunnel@raspberrypi

Finally, as an added layer of protection, we can also remove the password of the account. This effectively disables username/password based authentication and only allows SSH key authentication we configured earlier.

# Remove user password
# user@server:~ $
sudo passwd -d autotunnel

Automatically starting the tunnel

The last piece of the puzzle is to automatically start the tunnel using a systemd service configured in the /etc/systemd/system/autossh.service file on the Raspberry Pi:

# /etc/systemd/system/autossh.service
[Unit]
Description=Keeps an ssh tunnel to server.url.com open

# Run after network connections are established and the ssh server is started
After=network-online.target ssh.service

[Service]
# Disable monitoring & gatetime
Environment="AUTOSSH_PORT=0"
Environment="AUTOSSH_GATETIME=0"

# Restart on failure
RestartSec=3
Restart=always

ExecStart=/usr/bin/autossh -NT -o "ExitOnForwardFailure=yes" -o "ServerAliveInterval=10" -o "ServerAliveCountMax=3" -i /home/autotunnel/.ssh/id_ed25519 -R 2200:127.0.0.1:22 autotunnel@server.url.com

TimeoutStopSec=10

[Install]
WantedBy=multi-user.target

Finally, we can start and enable the autossh service. Here enabling the service means it will start at boot while starting a service is a once off operation.

# Start and enable the
# pi@raspberrypi:~ $
sudo systemctl start autossh.service
sudo systemctl enable autossh.service