Reverse ssh tunnel with port forwarding - Apache Linux

I am feeling happy today, To have done this. Because I have been thinking of it in my mind for years. And here it is..

First we need to check if we can connect to our remote server via ssh

ssh user@<REMOTE_IP_ADDRESS>

If this is successful we need to set Gateway ports to yes

echo "GatewayPorts yes" >> /etc/ssh/sshd_config

if this command complains about permission denied error,
use nano to append the line at the bottom of the file.

Now restart the ssh service

service ssh restart
exit

we will be prompoted to reauthenticate

Port forwarding Basic

Original source: https://www.everythingcli.org/ssh-tunnelling-for-fun-and-profit-local-vs-remote/

Local vs Remote SSH port forwarding

When it comes to the art of SSH tunnelling, there are basically two options where to relay a port to.

You can relay a port from a remote server to your local machine with ssh -L, hence called local port forwarding. A very basic use-case is if your remote server has a MySQL database daemon listening on port 3306 and you want to access this daemon from your local computer.

The second option is to make your local port available on a remote server (ssh -R). Remote port forwarding might come in handy if you for example want to make your local web-server available on a port of a public server, so that someone can quickly check what your local web-server provides without having to deploy it somewhere publicly.

It should now be pretty easy to remember: Local and remote port forwarding always refers to where to relay the port to. The SSH command syntax uses the same easy to remember abbreviations: -L (forward to my local machine) and -R (forward to my remote machine).

TL;DR

Remote MySQL server (remote port 3306) to local machine on local port 5000:

ssh -L 5000:localhost:3306 cytopia@everythingcli.org

Local web-server (local port 80) to remote server on remote port 5000:

ssh -R 5000:localhost:80 cytopia@everythingcli.org

Local port forwarding

(Make a remote port available locally).

In this example we are going to make a remote MySQL Server (Port 3306) available on our local computer on port 5000.

Let’s start with the general syntax of local port forwarding:

ssh -L <LocalPort>:<RemoteHost>:<RemotePort> sshUser@remoteServer

Now let’s simply forward our remote MySQL server to our local machine on port 5000.

ssh -L 5000:localhost:3306 cytopia@everythingcli.org

That’s all the magic! You can now simply reach the remote database from your local machine with mysql --host=127.0.0.1 --port=5000 or any other client.

But wait… which local address does it listen on?

Yes, you are right! The complete syntax is:

ssh -L [<LocalAddress>]:<LocalPort>:<RemoteHost>:<RemotePort> sshUser@remoteServer

The local address is an optional parameter. If you do not specify it, the remote port will be bound locally to all interfaces (0.0.0.0). So you can also only bind it locally to your 127.0.0.1 (on your local machine).

This is the full example:

ssh -L 127.0.0.1:5000:localhost:3306 cytopia@everythingcli.org

Remote port forwarding

(Make a local port available remotely).

In this example we are going to make our local web-server (Port 80) available on a remote server on Port 5000.

Let’s start with the general syntax of remote port forwarding:

ssh -R <RemotePort>:<LocalHost>:<LocalPort> sshUser@remoteServer

Now let’s simply forward our local web-server to our remote machine on port 5000.

ssh -R 5000:localhost:80 cytopia@everythingcli.org

That’s all the magic! You can now simply reach your local webserver via http://everythingcli.org:5000.

But wait… which remote address does it listen on?

Yes, you are right! The complete syntax is:

ssh -R [<RemoteAddress>]:<RemotePort>:<LocalHost>:<LocalPort> sshUser@remoteServer

The remote address is an optional parameter. If you do not specify it, the remote port will be bound remotely (on remoteServer) to all interfaces (0.0.0.0). So you can also only bind it remotely to a specific interface.

This is the full example:

Assuming the IP address of everythingcli.org is 109.239.48.64 and you only want to bind it to this IP.

ssh -R 109.239.48.64:5000:localhost:80 cytopia@everythingcli.org

But wait… it doesn’t work
By default, the listening socket on the server will be bound to the loopback interface only. This may be overridden by specifying RemoteAddress. Specifying a RemoteAddress will only succeed if the server’s GatewayPorts option is enabled (on the remote server):

$ vim /etc/ssh/sshd_config
GatewayPorts yes

Some more details

Ports below 1024

Every system user can allocate ports above and including 1024 (high ports). Ports below that require root privileges.
So If you want to relay any port to a port to for example 10, you must do that like so:

As you allocate a low port on your local machine, you must either do that as root (locally) or with sudo (locally):

sudo ssh -L 10:localhost:3306 cytopia@everythingcli.org

As you allocate a low port on the remote server, you will need to ssh into the machine as root:

ssh -R 10:localhost:80 root@everythingcli.org

Now we can run this on our host and EXPOSED!!.

ssh -R 8080:localhost:3000 root@111.111.111.111

Here we are forwarding our local port 3000 to remote 111.111.111.111:8080

For my case I tunneled my webcam stream running on port 8081 to my pi

ssh -R 8080:localhost:8081 ubuntu@192.168.1.250

For making it stable I am going to use autossh

sudo apt install autossh

Use it like:

autossh -M -f -R

Without autossh, we can use ssh and a monitoring script to make it persistant

#!/bin/bash
createTunnel() {
  /usr/bin/ssh -N -R 2222:localhost:22 serverUser@25.25.25.25
  if [[ $? -eq 0 ]]; then
    echo Tunnel to jumpbox created successfully
  else
    echo An error occurred creating a tunnel to jumpbox. RC was $?
  fi
}
/bin/pidof ssh
if [[ $? -ne 0 ]]; then
  echo Creating new tunnel connection
  createTunnel
fi

Next make it executable by doing the following:

chmod 700 ~/create_ssh_tunnel.sh

Now start the crontab.

crontab -e

Place this in as your cron job (every minute check if the ssh connection is up, if not, attempt to bring it up)

*/1 * * * * ~/create_ssh_tunnel.sh > tunnel.log 2>&1

To troubleshoot any problems in this you can view the tunnel.log file.

Setting up a proxy to the SSH tunnel

Now that you have reverse SSH port forwarding set up, next step is to forward traffic to the SSH tunnel. Let’s say your remote server accepts internet traffic at www.my-web-address.com. You want traffic at the URL _www.my-web-address.com/my-path_ to be forwarded to your local application server via the SSH tunnel. One way to do this is to set up a Location directive in your Apache server settings like so:

<Location "/my-path">
   ProxyPass "http://127.0.0.1:8000"
   ProxyPassReverse "http://127.0.0.1:8000"
</Location>

This assumes that port 8000 is the port you opened on the remote host side while setting up the SSH tunnel. The syntax for other servers such as nginx will vary, but the idea is similar.

Setting up a virtual host on the Apache server on the remote host

We need a similar set up on the local host as well. The Apache server listening on the local host side of the SSH tunnel needs to proxy traffic to the application server. You may also want to serve some static content as the front end for your application. This can be done by setting up a virtual host, as shown below.

 Listen 8000
 Listen 8082
 <VirtualHost *:8000>

	#ServerName <em>www.my-web-address.com/my-path</em> 
	
        # some-dir is a directory in the docker container that will be mapped to a directory on the your local file system
        # through docker volume mapping
	DocumentRoot /usr/local/apache2/some-dir

	<Directory "/usr/local/apache2/some-dir">
		Order allow,deny
		AllowOverride All
		Allow from all
		Require all granted
	</Directory>

    #Load the SSL module that is needed to terminate SSL on Apache
    LoadModule ssl_module modules/mod_ssl.so

    #This directive toggles the usage of the SSL/TLS Protocol Engine for proxy. Without this you cannot use HTTPS URL as your Origin Server
    SSLProxyEngine on
    
    # write logs to stderr so you can see them using docker logs
    ErrorLog /dev/stderr
    CustomLog /dev/stderr combined

    # The ProxyPass directive specifies the mapping of incoming requests to the backend server 

    ProxyPass /my-endpoint http://172.17.0.1:5000/my-endpoint
    ProxyPassReverse /my-endpoint http://172.17.0.1:5000/my-endpoint

</VirtualHost>

# for local testing: forward traffic on port 8082 to local Flask server. Don't use 8080! Jenkins may be running on it
<VirtualHost *:8082>

	# ServerName localhost

	DocumentRoot /usr/local/apache2/some-dir
        Alias /some-dir /usr/local/apache2/some-dir
	<Directory "/usr/local/apache2/some-dir">
		Order allow,deny
		AllowOverride All
		Allow from all
		Require all granted
	</Directory>

    ErrorLog /dev/stderr
    CustomLog /dev/stderr combined

    # local testing Flask server is running as is (not as a container) on port 5001
    ProxyPass /my-path/my-endpoint/ http://172.17.0.1:5001/my-endpoint/
    ProxyPassReverse /my-path/my-endpoint/ http://172.17.0.1:5001/my-endpoint/

</VirtualHost>

Future reference

### ssh tunnel
# first make sure the permission on PEM key file is correct (-rw------)
chmod 600 <path to your pem file>

# first close open ssh tunnels.
# this prints out the process ids for all ssh tunnels. The awk {'print$2'} corresponds to the process ids of the ssh tunnels
sudo lsof -i -n | egrep '\<ssh\>' | awk {'print$2'}
# Out of these ssh tunnels, we are only interested in those to our remote server. So output of the lsof is passed to
# grep <remote host IP>, which is then passed to kill -9
kill -9 $(sudo lsof -i -n | egrep '\<remote IP\>' | awk {'print$2'})

# Now set up the remote port forward
#   -f    Requests ssh to go to background just before command execution.
#   -N    Do not execute a remote command.
#   -T    Disable pseudo-tty allocation.
#   -R    Remote Port Forward.
# This allows anyone on the remote server to connect to TCP port 8000 (first 8000) on the remote server.
# The connection will then be tunneled back to the client host, and the client then makes a TCP connection to port
# 8000 (second 8000) on local host (127.0.0.1)

ssh -fNT -R 8000:127.0.0.1:8000 username@ip_of_remote_host -i <path to your pem file>
### NOTE:
# for this to work, open /etc/ssh/sshd_config and add the following line at the bottom:
# GatewayPorts yes
# then restart ssh:
# sudo service ssh restart

### Some useful commands: To see open SSH connections:
# sudo lsof -i -n | egrep '\<ssh\>'
# to manually kill open ssh tunnels:
# kill -9 <process_id>