Advanced Topics

This page covers topics for advanced users of POD who would like tighter integration with their existing workflows and on-premise resources. The topics on this page are not required for all users. Proceed with caution. If you would like assistance please contact the POD Support team by e-mailing pod@penguincomputing.com.

Persistent SSH Tunnels on Windows

License server administrators serving licenses using an on-premise Windows server may be required to respond to license requests from their users running jobs on POD. This will require establishing an SSH tunnel to POD such that requests to specific network ports are automatically forwarded from POD to the Windows license server.

For best results, we recommend setting up a Windows service to automatically start and restart the SSH tunnel connection to POD with minimal input from the license server administrator. This documentation describes the setup process using a Windows application called the Non-Sucking Service Manager (NSSM) and a PuTTY utility called Plink. The advantage of this configuration is that Windows administrators can use the regular Windows Services Management Console to stop and start this service as needed. Below are links to download the required software and instructions for setting up the Windows service using NSSM and Plink.

Please Note: When downloading the NSSM applciation, make sure to download the “Latest release” and NOT the “Featured pre-release”!

Configure New Service using NSSM GUI

Set the configuration parameters for each tab in the NSSM GUI as described below. No changes are required for the Dependencies, Shutdown, File Rotation, and Environment tabs. You may need to use the scroll buttons to see and select all the different tabs. Pleaes Note: Included with the GUI instructions below are corresponding NSSM commands that achieve the same configuration.

  1. Application: Specify how to run Plink.

  2. Details: Specify details about the new Windows service.

  3. Log on: Identify the user account to run the new Windows service.

  4. Process: Specify CPU affinity requirements.

  5. Exit Actions: Identify restart actions when Plink application crashes.

  6. I/O: Specify any input/output redirection. Useful for capturing logging information for debugging.

Application Tab

Plink is a simple PuTTY utility for creating SSH tunnels. It can be run unattended without any user input as long as the appropriate options are provided on the command line. Please Note: Your install paths may vary.

  • Path: C:\Program Files\PuTTY\plink.exe

  • Startup directory: C:\Program Files\PuTTY

  • Arguments: This is the heart of the configuration. Including an example and details below.

-batch -load keepalive -i C:\pod\keys\POD-license-key.ppk -N -R *:27000:localhost:27000 -R *:92709:localhost:92709 penguin@login-12-34.pod.penguincomputing.com

PS C:\pod> nssm set POD-Plink-Tunnel Application "C:\Program Files\PuTTY\plink.exe"
PS C:\pod> nssm set POD-Plink-Tunnel AppDirectory "C:\Program Files\PuTTY"
PS C:\pod> nssm set POD-Plink-Tunnel AppParameters "-batch -load keepalive -i C:\pod\keys\POD-license-key.ppk -N -R *:27000:localhost:27000 -R *:92709:localhost:92709 penguin@login-12-34.pod.penguincomputing.com"

Application Tab: Arguments

The Plink utility will need a number of arguments to correctly establish and connect the SSH tunnel such that licensing requests can be forwarded back to the license server. Some of these arguments will be provided by the license administrator and others by the users and administrators of the POD cluster. Please Note: Make sure to test the Plink command before setting up the service using NSSM.

  1. POD Username: The POD account user can identify their username on POD. For this example, we will assume the case-sensitive username is penguin.

  2. POD Login hostname or IP address: The POD account user can identify the hostname or IP address of their pod.free login node. For this example, we will assume the fully-qualified domain name is: login-12-34.pod.penguincomputing.com.

  3. Private SSH key file: The SSH tunnel connection will need to be authenticated using an SSH key pair. Instructions for creating a new SSH key pair can be found here. The private key needs to be installed in C:\pod\keys while the public key needs to be uploaded to the POD portal by the penguin user. For this example, we will assume the private key is: C:\pod\keys\POD-license-key.ppk.

  4. License Daemon port number(s): Licensed applications typically use known port(s) for making requests to the license server Daemon. The License Server Administrator needs to identify the persistent port(s) used. For this example, we will assume the License Daemon port is: 27000.

  5. Vendor Daemon port number(s): Licensed application also use ephemeral port(s) for making requests to the license server Vendor Daemon. The License Server Administrator needs to lock down and identify the port(s) used. For this example, we will assume the Vendor Daemon is locked down to port: 92709.

  6. Host Key fingerprint: The easiest way to collect the host key fingerprint for the POD Login node is to fail a connection attempt and look in the logs. More details on this here.

Putting these options and values together, without the host key fingerprint, the Arguments parameter should be look like:

-batch -load keepalive -i C:\pod\keys\POD-license-key.ppk -N -R *:27000:localhost:27000 -R *:92709:localhost:92709 penguin@login-12-34.pod.penguincomputing.com

Details Tab

For this section, specify configuration that will help the License Manager identify the purpose of this service. Be as specific as required but make sure that the Startup type is set to Automatic to start the service on system startup.

  • Display name: POD SSH Tunnel

  • Description: Penguin On Demand (POD) persistent SSH tunnel

  • Startup type: Automatic

PS C:\pod> nssm set POD-Plink-Tunnel DisplayName "POD SSH Tunnel"
PS C:\pod> nssm set POD-Plink-Tunnel Description "Penguin On Demand (POD) persistent SSH tunnel for licensing requests."
PS C:\pod> nssm set POD-Plink-Tunnel Start SERVICE_AUTO_START

Log On Tab

For this section, please keep in mind that the Windows service will need to run with elevated privileges. Select the Local System account radio button or create and specify a service account on your Active Directory.

  • Local System account: Selected

PS C:\pod> nssm set POD-Plink-Tunnel ObjectName LocalSystem
PS C:\pod> nssm set POD-Plink-Tunnel Type SERVICE_WIN32_OWN_PROCESS

Process Tab

For this section, specify any processor affinity configuration you require for the Plink process. We recommend allowing Plink to run on all CPU processors.

  • Priority: High

  • Console window: Selected

  • Affinity: All processors Selected

PS C:\pod> nssm set POD-Plink-Tunnel AppPriority HIGH_PRIORITY_CLASS
PS C:\pod> nssm set POD-Plink-Tunnel AppNoConsole 0
PS C:\pod> nssm set POD-Plink-Tunnel AppAffinity All

Exit Actions Tab

For this section, we want to make sure the Plink tunnel is restarted if it is killed or crashes. Make sure to set the drop-down menu to Restart application.

  • Action to take when application exits…: Restart application

PS C:\pod> nssm set POD-Plink-Tunnel AppThrottle 1500
PS C:\pod> nssm set POD-Plink-Tunnel AppExit Default Restart
PS C:\pod> nssm set POD-Plink-Tunnel AppRestartDelay 0

I/O Tab

For this section, using the C:\pod\logs folder we already created, specify a single log file to capture the STDOUT and STDERR streams from Plink. We will use this file for debugging purposes.

  • Output (stdout): C:\pod\logs\log.txt

  • Error (stderr): C:\pod\logs\log.txt

PS C:\pod> nssm set POD-Plink-Tunnel AppStdout "C:\pod\logs\log.txt"
PS C:\pod> nssm set POD-Plink-Tunnel AppStderr "C:\pod\logs\log.txt"

Install new Windows Service

With all the tabs configured, now click the Install service button to create a new Windows service named POD-Plink-Tunnel. This service can now be managed from the normal Windows Services Management Console.

Managing new Windows Service

Open the Windows Services Management Console by running services.msc and find the new POD-Plink-Tunnel service. Right click and select Start to start the service. Please Note: This first attempt should fail to start correctly. Here, you could also select Stop to stop the service if it was already running.

We need to modify our call to Plink to include the POD host key fingerprint from the log file: C:\pod\logs\log.txt. If the fingerprints do not match then the connection should fail. This guarantees the connection can only be made successfully to a known host on POD. The relevant host key fingerprint from the log snippet is: 12:b5:b8:34:eb:74:27:e5:2d:47:60:f6:b2:39:cc:1a.

The server's host key is not cached in the registry. You
have no guarantee that the server is the computer you
think it is.
The server's ssh-ed25519 key fingerprint is:
ssh-ed25519 255 12:b5:b8:34:eb:74:27:e5:2d:47:60:f6:b2:39:cc:1a Connection abandoned.

Updating the Host Key Fingerprint

After running the update command the NSSM Graphical User Interface (GUI) window will open allowing you to update the existing POD-Plink-Tunnel Windows service.

  1. Make sure to stop the POD-Plink-Tunnel service in the Windows Services Management Console.

  2. In an Elevated Command Prompt run the command: nssm edit POD-Plink-Tunnel.

  3. Under the Application tab update the Arguments parameter to include the -hostkey option and fingerprint from the log file: -hostkey 12:b5:b8:34:eb:74:27:e5:2d:47:60:f6:b2:39:cc:1a

  4. Click the Edit Service button to commit the changes.

  5. Restart the POD-Plink-Tunnel service in the Windows Services Management Console and verify a Running state.

You should now have a persistent SSH license tunnel connected from the Window License server to POD. Please Note: Any changes to your login node will require you to update the host key fingerprint and hostname options. Using the example from above, with the host key fingerprint, the Arguments parameter should now look like:

-batch -hostkey 12:b5:b8:34:eb:74:27:e5:2d:47:60:f6:b2:39:cc:1a -load keepalive -i C:\pod\keys\POD-license-key.ppk -N -R *:27000:localhost:27000 -R *:92709:localhost:92709 penguin@login-12-34.pod.penguincomputing.com