Singularity on POD

Back to documentation index

Introduction to Singularity

POD supports Singularity, a powerful Linux container platform designed for HPC clusters.

Singularity enables users to have full control of their environment, enabling a non-privileged user to "swap out" the operating system on the host for one they control.  As an example, Singularity can provide a user with the ability to create an Ubuntu image of their application, and run the containerized application on POD in its native Ubuntu environment.

Links to Singularity Information

Using Singularity on POD

Creating Singularity Containers

To use Singularity on POD one needs to either create a Singularity container or use one provided by someone else.  To build a container, one needs root access on the build system so one cannot build one on POD.   Some options for a build environment are:

  • If you have root access to a Linux system you can install Singularity and build containers there.
  • If you don't have a Linux system you could easily install one in a virtual machine using software like VirtualBox or  VMware.
  • You can create containers and download them from

Once you have built and tested your container locally, just use scp or rsync to upload it to your POD storage.

remote$ scp -C my_container.img [login node address]:~/


remote$ rsync -z my_container.img [login node address]:~/

Using Docker Images in Singularity

Docker images can be used to create Singularity or can even be run directly. More information can be found at Singularity's home page but we’ve used two different methods to incorporate Docker into Singularity on POD.

Method 1 - Importing a Docker image into a Singularity container

Singularity communicates with Docker Hub using the Docker Remote API. One can create a Singularity image and import from a docker::. Note that this is a one way operation. Changes made to the image will not be propagated to the Docker image.

remote$ sudo singularity build tensorflow.img docker://tensorflow/tensorflow:latest

Method 2 - Using a Singularity Recipe file to build from Docker Hub

In the recipe file, specify docker as the Bootstrap source and specify the image in Docker syntax

Bootstrap: docker
From: tensorflow/tensorflow:latest

Singularity Recipe files created by Penguin for POD

While running single node containers on POD is very easy, creating a container for MPI is a bit more complex. Penguin has created several skeleton recipe files as examples of creating images suitable for running on POD. You can find them in the PenguinComputing/pod public git repository.

There you will find example recipe files for Centos and Ubuntu with OpenMPI using various network transports supported on POD and a script to make creating and bootstrapping containers easier. Check back for additional spec files and send email to if you have special requests.


# copy pod repo to your local machine
remote$ git clone
remote$ cd pod/singularity

# if you don’t have git installed on your local machine you can also 
# browse to and download a spec file

# build the pod centos7 recipe (.def) into a sandbox folder
remote$ sudo singularity build --sandbox ./pod-ompi2-centos7 ./pod-ompi2-centos7.def

# convert the sandbox into a read-only, squashfs image
remote$ sudo singularity build ./pod-ompi2-centos7.img ./pod-ompi2-centos7

# copy the final image to pod; compressed transfers are recommended (rsync -z)
remote$ rsync -azvH ./pod-centos7-ompi2.img [pod username]@[login node address]:~/

# login into pod, and submit a job in the B30 queue; this example uses interactive qsub
remote$ ssh [pod username]@[login node address]
pod$ qsub -I -q B30 -l nodes=2:ppn=28,walltime=00:15:00

# in your job environment, load singularity and ompi 2.0.1
n550$ module load singularity/2.4
n550$ module load openmpi/2.0.1/gcc.4.8.5

# run the mpi_ring application inside the container using pod's mpirun
n550$ mpirun singularity exec pod-ompi2-centos7.img /usr/bin/mpi_ring

Tips for running MPI applications with Singularity on POD

  • Always compile MPI applications inside a container image with the same MPI implementation and version you plan to use on POD. Refer to the Singularity documentation for currently supported MPI implementations.

  • POD's B30 queue is enabled by Intel's Omni-Path fabric, which works optimally with the PSM2 transport for MPI communication. Not all Operating Systems support PSM2. CentOS 7 and Ubuntu's Beta Zesy release (17.04) can be built with PSM2 support. Ubuntu 16 does not support PSM2 and will require the use of IPoIB for MPI communication.