Singularity on POD

Back to documentation index

Introduction to Singularity

POD supports Singularity, a powerful Linux container platform designed for HPC clusters by Lawrence Berkeley National Laboratory.

Singularity enables users to have full control of their environment, enabling a non-privileged user to "swap out" the operating system on the host for one they control.  As an example, Singularity can provide a user with the ability to create an Ubuntu image of their application, and run the containerized application on POD in its native Ubuntu environment.

Links to Singularity Information

Using Singularity on POD

Creating Singularity Containers

To use Singularity on POD one needs to either create a Singularity container or use one provided by someone else.  To build a container, one needs root access on the build system so one cannot build one on POD.   Some options for a build environment are:

  • If you have root access to a Linux system you can install Singularity and build containers there.
  • If you don't have a Linux system you could easily install one in a virtual machine using software like VirtualBox or  VMware.
  • You can create containers and download them from

Once you have built and tested your container locally, just use scp or rsync to upload it to your POD storage.

$ scp -C my_container.img [login node address]:


$ rsync -z my_container.img [login node address]:

Using Docker Images in Singularity

Docker images can be used to create Singularity or can even be run directly. More information can be found at Singularity's home page but we’ve used three different methods to incorporate Docker into Singularity on POD.

Method 1 - Importing a DOCKER image into a Singularity container

Singularity communicates with Docker Hub using the Docker Remote API. One can create a Singularity image and import from a docker::. Note that this is a one way operation. Changes made to the image will not be propagated to the Docker image.

sudo singularity create --size 3000 tensorflow.img
sudo singularity import tensorflow.img docker://tensorflow/tensorflow:latest

Method 2 - Using a Singularity spec file to bootstrap from Docker Hub

In the spec file, specify docker as the Bootstrap source and specify the image in Docker syntax

Bootstrap: docker
From: tensorflow/tensorflow:latest

Method 3 - Directly execute a Docker image

If you have a Docker image that doesn’t require any modification you can also directly execute it on POD. Simply start an interactive job on a compute node:

$ qsub -I -qB30 -l nodes=1:ppn=28
qsub: waiting for job 27057 to start    

qsub: job 27057 ready

[user@n550 ~]$

And shell into your docker image:

[user@n550~]$ singularity shell docker://ubuntu:latestlibrary/ubuntu:latest    

Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4Downloading layer: sha256:6d9ef359eaaa311860550b478790123c4b22a2eaede8f8f46691b0b4433c08cf

Downloading layer: sha256:9654c40e9079e3d5b271ec71f6d83f8ce80cfa6f09d9737fc6bfd4d2456fed3fDownloading layer: sha256:e8db7bf7c39fab6fec91b1b61e3914f21e60233c9823dd57c60bc360191aaf0

Downloading layer: sha256:f8b845f45a87dc7c095b15f3d9661e640ebc86f42cd8e8ab36674846472027f7Downloading layer: sha256:d54efb8db41d4ac23d29469940ec92da94c9a6c2d9e26ec060bebad1d1b0e48d

Singularity: Invoking an interactive shell within container... Singularity.ubuntu:latest> cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"

Spec files created by Penguin for POD

While running single node containers on POD is very easy, creating a container for MPI is a bit more complex. Penguin has created several skeleton definition files as examples of creating images suitable for running on POD. You can find them in the PenguinComputing/pod public git repository.

There you will find example spec files for Centos and Ubuntu with OpenMPI using various network transports supported on POD and a script to make creating and bootstrapping containers easier. Check back for additional spec files and send email to if you have special requests.


# copy pod repo to your local machine
remote$ git clone
remote$ cd pod/singularity

# if you don’t have git installed on your local machine you can also 
# browse to and download a spec file

# create an empty container image
remote$ sudo singularity create --size 1512 ./pod-centos7-ompi2.img

# bootstrap the pod centos7 definition file into the previously created image
remote$ sudo singularity bootstrap ./pod-centos7-ompi2.img ./pod-centos7-ompi2.def

# make any changes or install any software desired
remote$ sudo singularity shell -w ./pod-centos7-ompi2.img

# copy the final image to pod; compressed transfers are recommended (rsync -z)
remote$ rsync -azvH ./pod-centos7-ompi2.img <pod username>@<loginnode ip>:~/

# login into pod, and submit a job in the B30 queue; this example uses interactive qsub
remote$ ssh <pod username>@<loginnode ip>
pod$ qsub -I -q B30 -l nodes=2:ppn=28,walltime=00:15:00

# in your job environment, load singularity and ompi 2.0.1n550$ module load singularity
n550$ module load openmpi/2.0.1/gcc.4.8.5

# run the mpi_ring application inside the container using pod's mpirun
n550$ mpirun singularity exec pod-centos7-ompi2.img /usr/bin/mpi_ring

Tips for running MPI applications with Singularity on POD

  • Always compile MPI applications inside a container image with the same MPI implementation and version you plan to use on POD. Refer to the Singularity documentation for currently supported MPI implementations.

  • POD's B30 queue is enabled by Intel's Omni-Path fabric, which works optimally with the PSM2 transport for MPI communication. Not all Operating Systems support PSM2. CentOS 7 and Ubuntu's Beta Zesy release (17.04) can be bootstrapped with PSM2 support. Ubuntu 16 does not support PSM2 and will require the use of IPoIB for MPI communication.