A Guide to using Scyld Cloud Management APIs on POD

Back to documentation index

The Scyld Cloud Management Platform is a distributed set of software components that provides HPC cloud account and resource management. These software components work tightly together to provide authentication, authorization, and cluster resource provisioning services through their APIs.

At Penguin Computing, we've deployed SCM for our POD clusters and supporting infrastructure. Users register at the POD Portal and use the portal to manage their account. In addition, each component within SCM exposes an API that users can utilize for programmatic management of their account and POD cluster resources.

The purpose of this guide is to demonstrate the functionality of the APIs by example. We'll show some code examples; we'll be using python but the basic interaction pattern should look similar in other languages.

We'll go through these steps:

  • Prerequisite: Creating a POD Account on the portal
  • Authentication
  • Get a list of available PODs (Scyld Cloud Controller instances and their API endpoints)
  • Create a system account on a POD cluster
  • Provision a storage volume on a POD cluster
  • Create a server instance (VM) on a POD cluster
  • Install ssh keys to access your VM

Prerequisite: Create an Account on the POD Portal

Registering for an account on the POD portal establishes your Scyld Cloud Auth user identity and provides you with an API key and secret. All Scyld Cloud Management APIs require you to provide an authentication token and to get that token, you'll use that API key and secret.

In the portal, clicking on your username in the upper right triggers a pop-out menu with a link to your "Account Settings". On that page you'll find your User ID, your API key and your API secret. If your API secret is empty, click the link to regenerate the secret and it will be regenerated and displayed on the page. If you choose to revoke your secret, you will not be able to use the APIs. And we didn't name it "secret" for nothing... be sure to keep your API secret protected as you would your passwords!

Authentication

Now that you have your API credentials, you can use them to get an authentication token from the Scyld Cloud Auth service. To get a token you submit a request to the service using OAuth. OAuth is an open web protocol for token-based authentication (see the bottom of this page for references). This is best done using a client library since OAuth requests involve manipulating HTTP headers, concatenating multiple strings including timestamps... not something easily done on the command line.

import oauth2 as oauth
import json

CLOUDAUTH_ID = '880da6d718d54acc745ab9fea7b2c090'
API_KEY = 'my_key'
API_SECRET = 'my_secret'
AUTH_API_ENDPOINT = 'https://auth.pod.penguincomputing.com'

auth_url = AUTH_API_ENDPOINT + '/v1/auth/request_token'

# setup the oauth client
consumer = oauth.Consumer(key=API_KEY, secret=API_SECRET)
client = oauth.Client(consumer)

# make the HTTP request using GET method 
resp, content = client.request(auth_url, "GET")

# we're looking for HTTP status code of 200 for success
if resp['status'] == '200':
    result = json.loads(content)
    auth_token = result['authentication_token']
    auth_token_expires_at = result['expires_at_utc']
else:
    # an error occured.
    pass

Once you've acquired your auth_token, you'll use it every subsequent SCM API call by including in the HTTP headers as the X-Auth-Token header. Notice also the expiration timestamp (in GMT) accompanying the token. Once its expired, it's no longer valid and any API calls will fail.

Get a List of Available PODs

Now that we have an authentication token, we'll make a request to retrieve the list of Scyld Cloud Controller instances available within the POD domain.

import requests

# construct the url for the request
url = AUTH_API_ENDPOINT + '/v1/cloud_controller/list'

# all SCM APIs require this header 
headers = {'X-Auth-Token': auth_token}

# make the HTTP GET request
r = requests.get(url, headers=headers)

# check for success
if r.status_code == requests.codes.ok:
    response = r.json()
    if response['success']:
        data = response['data']

The response is a list of Scyld Cloud Controllers in the data field:

'data': {
    'cloud_controllers': [{
        'id': 16,
        'name': 'POD MT1',
        'active': True,
        'api_endpoint': 'mt1-api.pod.penguincomputing.com:443',
        'api_ssl': True,
        'api_version': '1',
        'beoweb_endpoint': 'mt1-beoweb.pod.penguincomputing.com:443',
        'beoweb_ssl': True,
        'active': True,
        'created_at_utc': '2013-02-13 17:03:31',
        'timezone': 'US/Mountain'}],
    'total': 1
}

A cloud controller object contains API endpoint information so now we know where to aim our cluster resource requests.

Create a system account on a POD cluster

The first thing we'll need to do to get set up on the cluster is to create a user account on the POD cluster, which is a UNIX-style system account. We'll make the API request to the cloud controller endpoint from our previous call.

import requests
import json

# result from our previous query 
CLOUDCON_ENDPOINT = 'https://mt1-api.pod.penguincomputing.com:443' 
CLOUDCON_VERSION = 'v1'

# construct URL
url = CLOUDCON_ENDPOINT + '/' + CLOUDCON_VERSION + '/user' 

# construct headers: the cloud controller requires an additional header 
headers = {
    'X-Auth-Token': auth_token,
    'X-Auth-Cloudauth-Id': CLOUDAUTH_ID} 

# construct the required parameters
# note that the cloud controller requires JSON encoded parameters 
params = json.dumps({
    'cloudauth-id': CLOUDAUTH_ID, 
    'system-name': 'penguin'})

# make the HTTP POST request
r = requests.post(url, headers=headers, data=params)

# check for success
if r.status_code == requests.codes.ok:
    response = r.json()
    if response['success']:
        data = response['data']
        # the response is a list of user objects, in this case a list of 1
        cloudcon_user = data[0]

The successful response from this call to the cloud controller returns a Cloud Controller User.

Provision Storage on a POD cluster

Next, let's set up a storage volume on the cluster.

import requests
import json

# construct URL
url = CLOUDCON_ENDPOINT + '/' + CLOUDCON_VERSION + '/storage-volume' 

# construct headers
headers = {
    'X-Auth-Token': auth_token}
    'X-Auth-Cloudauth-Id': CLOUDAUTH_ID} 

# construct the required parameters: volume-size is GB.
params = json.dumps({
    'cloudauth-id': CLOUDAUTH_ID, 
    'volume-size': 50,
    'volume-type': {'visibility': 'cluster'}})

# make the HTTP POST request
r = requests.post(url, headers=headers, data=params)

# check for success
if r.status_code == requests.codes.ok:
    response = r.json()
    if response['success']
        data = response['data']
        # the response is a list of storage volumes
        volume = data[0]

The successful response from this call to the cloud controller returns a Storage Volume object.

Create a server instance (VM) on a POD cluster

We have now created a system account and storage volume on the POD cluster. These steps are required before you can create a server instance. Your server instance serves as your gateway or login node to the cluster. You'll log into your VM in order to run jobs and manage the jobs' input and output files.

When we create our server instance, we'll be required to specify both a server-image, which indicates the boot image and a server-flavor, which defines the VM specs (cores, memory, disk space). In order to determine what server images and flavors are available from the cloud controller, we'll make 2 requests to find out: one each type.

Here's the result from those 2 requests:

# GET /server-instance payload 
'data': [{
    'id': 'd36256c9-9fcb-4178-83b0-ead2243fc0e1',
    'active': True,
    'min-disk': 5,
    'min-ram': 256,
    'name': 'MT1 Login Node (CentOS 6)',
    'operating-system': {
        'major_version': '',
        'minor_version': '',
        'name': ''},
    'creation-date': '2013-04-17T18:50:19Z',
    'public': True}]

# GET /server-flavor payload 
'data': [{
    'id': '1',
    'disk': 5,
    'memory': 256,
    'name': 'pod.free',
    'vcpu': 1}]

Now we make the request to create a new server instance. Note the users parameter; this is your cloud controller userid.

import requests
import json

# construct URL
url = CLOUDCON_ENDPOINT + '/' + CLOUDCON_VERSION + '/server-instance

# construct headers
headers = {
    'X-Auth-Token': auth_token}
    'X-Auth-Cloudauth-Id': CLOUDAUTH_ID} 

# construct the required parameters: volume-size is GB.
params = json.dumps({
    'cloudauth-id': CLOUDAUTH_ID,
    'server-image': server_image_id, 
    'server-flavor': server_flavor_id, 
    'users': [cloudcon_user['id']]})

# make the HTTP POST request
r = requests.post(url, headers=headers, data=params)

# check for success
if r.status_code == requests.codes.ok:
    response = r.json()
    if response['success']
        data = response['data']
        # the response is a list of server instance objects
        vm = data[0]

A successful call will return a server instance object. You'll have the id of the instance so you can query it for status and the public IP. Here's the GET (i.e. query) output:

# GET /server-instance/vm['id']
'data': [{
    'cloudauth-id': '880da6d718d54acc745ab9fea7b2c090',
    'creation-date': '2013-04-12T00:01:21Z',
    'id': 'cbaadf63529df080885497cbaad1f635',
    'private-ip': '10.10.100.37',
    'public-ip': '192.41.74.108',
    'server-flavor': '1',
    'server-image': '66d63657-570b-4a49-9560-5427c861c5f6',
    'status': 'ACTIVE',
    'users': ['880da6d718d54acc745ab9fea7b2c090']}]

Once the status is ACTIVE, the instance is accessible via the public IP.

But wait! There is one more task to be done before you can log in to your server instance. POD does not permit password logins; users must install one or more ssh keys in order to authenticate.

Install SSH Keys to Access Your VM

This guide assumes you are familiar with SSH keys and their use. As stated above, in order to log in to your server instance, you'll need to upload a public ssh key. SSH keys are a property of a cloud controller User object.

import requests
import json

# construct URL
url = CLOUDCON_ENDPOINT + '/' + CLOUDCON_VERSION + '/user/' + cloudcon['id']

# the public key string, shortened here
mykey = 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA1u'
# construct headers
headers = {
    'X-Auth-Token': auth_token}
    'X-Auth-Cloudauth-Id': CLOUDAUTH_ID}

# construct the required parameters
params = json.dumps({
    'cloudauth-id': CLOUDAUTH_ID,
    'action': 'add', 
    'ssh-public-keys': [{'key': mykey}] 

# make the HTTP PUT request
r = requests.put(url, headers=headers, data=params)

# check for success
if r.status_code == requests.codes.ok:
    response = r.json()
    if response['success']
        # success
    else:
        # did not succeed. the msg may indicate the specific error.
        print response['msg']

Now you are ready to log into your server instance and prepare to run jobs on the cluster.

References, Links, Et cetera

POD Portal - https://pod.penguincomputing.com

POD API Endpoints (authentication required)

  • POD's Cloud Auth Service (Scyld Cloud Auth): https://auth.pod.penguincomputing.com
  • POD's Cloud Accountant Service (Scyld Cloud Accountant): https://accountant.pod.penguincomputing.com

See the documentation for the Scyld Cloud Management Platform for more details.

OAuth