Using the S30 Skylake Queue

Back to documentation index

S30 Class Compute Server
Intel® Xeon® Scalable Processor (Skylake-SP)


• POD Location: MT2
• Intel® Xeon® Gold 6148 
• 40 Bare-metal Compute Cores Per Node
• 384 GB DDR4 RAM Per Node
• Intel Omni-Path 100 Gb/s non-blocking fabric
• Lustre parallel file system

S30 Usage

This section assumes that you hav worked with POD's job queues already. If you need a brief introduction on building job scripts for job submission to using POD's queues, please see: POD 101: Quick Start for POD.

POD's S30, Skylake-SP queue allows for single node jobs to have 1 through 40 cores.  Each core is allotted 9.6GB of RAM.  For example, a single node, 10 core, 64GB RAM job for a maximum of 1 hour is submitted as such:

qsub -q S30 -l nodes=1:ppn=10,walltime=01:00:00 jobscript.sub

Multi-node jobs must request all 40 cores for each server.

qsub -q S30 -l nodes=5:ppn=40,walltime=01:00:00 mpi-jobscript.sub

Interactive jobs for compiling applications can be requested with the -I flag. Please note that all interactive sessions are billed for until you exit the interactive shell to terminate the job.

[user@loginnode ~]$ qsub -q S30 -I -l nodes=1:ppn=40
qsub: waiting for job 14701947.pod to start
qsub: job 14701947.pod ready

[user@n700 ~]$
[user@n700 ~]$ logout
qsub: job 14701947.pod completed
[user@loginnode ~]$

Compiling for Skylake-SP

The S30 queue's processors allow for AVX2 and AVX512 extention sets.  The use of AVX2 or AVX512 extention sets requires at least GCC 4.9.0.  Since AVX512 is a new feature for Skylake-SP processors, AVX512 enabled binaries will only execute on S30 nodes.  For binaries that are optimized for portability vs the performance characteristics of the Skylake-SP processor, you may use AVX2 which is supported on T30, B30, and S30 processors.

GCC users can leverage a number of compile flags for AVX512 extentions, including:

AVX-512 foundation instructions: -mavx512f
AVX-512 prefetch instructions: -mavx512pf
AVX-512 exponential and reciprocal instructions: -mavx512er
AVX-512 conflict detection instructions: -mavx512cd

[user@loginnode ~]$ qsub -q S30 -I -l walltime=01:00:00 -l nodes=1:ppn=40

qsub: waiting for job 4287980 to start
qsub: job 4287980

[user@n700 ~]$ module load gcc/4.9.0
[user@n700 ~]$ gcc -mavx512f -mavx512pf -mavx512er -mavx512cd mycode.c 

Customers using Intel compilers on POD, can use -x CORE-AVX512 to generate AVX-512F, AVX-512CD, AVX-512BW, AVX-512DQ and AVX-512VL binaries.  For more information, read Intel's documentation on AVX512 optimizations using the Intel C, C++ and Fortran compilers. 

Tundra Open Compute Platform

POD's new Intel® S30 cores are delivered through Tundra, Penguin Computing's Open Compute HPC platform. Penguin Computing’s Tundra Extreme Scale Series provides density, serviceability, reliability and optimized total cost of ownership for highly-demanding computing requirements. As such, Penguin Computing is able to pass these savings to POD customers.

Read more about Penguin Computing's Tundra solutions which was awarded the Department of Energy’s National Nuclear Security Administration (NNSA) Advanced Simulation and Computing (ASC) CTS-1 contract

• NNSA CTS-1 Tundra Award Press Release
• Tundra Extreme Scale (ES) Open Compute Solutions Guide
• Penguin Computing Showcases OCP Platforms for HPC at SC15
• Tundra Extreme Scale (ES) Series Specifications