2014-08-04T05:58:08Z

The Rackspace Cloud API

Most of you know by now that not too long ago I joined Rackspace. As you can imagine, I am now learning tons of new things as I familiarize myself with all the OpenStack projects, none of which I have used before.

In this article I'm going to show you a few ways to work more efficiently with your Rackspace cloud account (or any OpenStack cloud for that matter). I will begin with the introduction of a command line tool that you can use to manage your cloud servers, and then go even lower level and show you how you can do the same thing using a Python SDK. To end this article I'm going to show you a complete script that creates a cloud server, configures it as a Web server and deploys a Flask application to it, all completely unattended.

Why do it?

As a developer, being able to spin up a clean server instance to run a given process or application with just a few keystrokes is invaluable, as is to destroy it when I'm done. Having scripted access to my cloud account gives me the ability to include the server provisioning right in my scripts to help me create a fully automated experience.

Using the "nova" command line client

The nova command line client gives you access to the compute resources in your cloud account with simple commands. This tool is written in Python, so you need a Python interpreter installed on your system to use it. Assuming you have Python and the pip package manager installed, then you can download and install nova with the following command:

$ pip install rackspace-novaclient

Note that the above command will install the open source nova client with the Rackspace extensions to it. If you need help with installation issues I recommend that you review the installation instructions in the Getting Started Guide.

Before you can use nova you have to define a few environment variables that will be used to authenticate you with the cloud services. If you are using a bash shell or similar, then you can add the following lines to your ~/.bash_profile file:

export OS_AUTH_URL=https://identity.api.rackspacecloud.com/v2.0/
export OS_AUTH_SYSTEM=rackspace
export OS_USERNAME=<your username>
export OS_PASSWORD=<your API key>
export OS_TENANT_NAME=<your account number>
export OS_PROJECT_ID=<your account number>
export OS_REGION_NAME=<your desired region>
export NOVA_RAX_AUTH=1
export OS_NO_CACHE=1

Note that if you are not authenticating to a Rackspace cloud you will need to change some of the values above to the appropriate ones for your OpenStack cloud. Also for those of you that are on a Windows machine be aware that you will need to adapt the above assignments to the form set VAR=<value>.

It is important to note that Rackspace needs the OS_PASSWORD variable to be set to your API key, not your account password (it would be a terrible idea to have your account password exposed in a config file!). You can find your API key in the Web portal, in the Account Settings page.

Another piece of information you need is your Rackspace account number, which appears when you expand the dropdown at the top right corner of the navigation bar in the Web portal.

Finally, each session is attached to a specific Rackspace data center, so you need to choose what region you want to authenticate to, using the three letter code for it (i.e. DFW for Dallas, LON for London and so on).

Listing your servers

With all these variables set in your environment you are ready to use nova. To see the list of servers active in your account for the chosen region, run the following command:

$ nova list

Flavors

If you used the Web portal to create cloud servers you know that one of the decisions you need to make when you create a new server is what flavor to use for it. The flavor that you choose determines the specs of your server, and directly related to that, its cost. The same flavor options you have in the Web portal are available from nova. You can obtain the complete list of supported flavors with the following command:

$ nova flavor-list

Each flavor listed will have an ID associated with it, which you will need later when I show you how to create a new server.

Images

Another decision you need to make before you create a new server is what image to install in it. Rackspace offers images for most operating systems. The complete list can be obtained with this command:

$ nova image-list

As with the flavors, images have an ID associated with them.

Keypairs

This is entirely optional, but it is useful to have your SSH public key installed on new servers during creation, so that you can use a password-less login from the start. You can upload your public key(s) in the Web portal, but nova includes a full set of management commands for keypairs as well:

$ nova keypair-list
$ nova keypair-show <key-name>
$ nova keypair-add --pub-key <pub-key-file> <key-name>
$ nova keypair-delete <key-name>

Booting a new server

Once you have a flavor and an image selected, you can bring a new server to life with the following command:

$ nova boot my_new_server --image <image-id> --flavor <flavor-id> [--key-name <key-name>]

Yes, it's that simple! Note that nova boot is an asynchronous command, it normally takes a minute or two for the server to be ready.

The output of the nova boot command will show a few pieces of information that are very useful, so while you wait for your server to be created write down the adminPass value (the root password of your server) and the id your server was assigned, which you will use to query its status.

To find out if your server is ready to use you can issue the following command:

$ nova show <server-id>

When the status of your server appears as active you are ready to roll. The output of nova show will include the IP address that was assigned to the server on the public network. Using this IP address and either your SSH key or the root password you wrote down earlier you can SSH into your server and start using it.

It's important to note that once you create a server you start incurring in charges. If you create a server just to run some short tasks, do not forget to delete it when you are done, because if not it will continue to generate exposes, even if you don't use it. The command to delete a server is:

$ nova delete <server-id>

Using the Python SDK

The nova client is fun to use, but using it in scripts to automate your workflows requires that you add some logic to parse the command output and scrape the useful bits, and that makes it not fun anymore.

For a complete and very rewarding automation experience nothing beats connecting directly to the Rackspace API. The examples I'm going to show you are written in Python (we seem to like Python a lot at Rackspace). But note that you don't have to use Python, as there are SDKs for several other languages as well, and they all work in a similar way. Head over to the SDK documentation page for more information.

To install the Rackspace SDK for Python you can use pip:

$ pip install pyrax

Note that you may need to run the command above on an account with administration rights, or else you have the option to create a Python virtual environment and install locally to it.

Listing your servers

Assuming you kept your OS_* environment variables that you were using with nova, here is a short Python script that shows the id, name and status of all your cloud servers:

#!/usr/bin/env python
import os
import pyrax

pyrax.set_setting('identity_type', os.environ['OS_AUTH_SYSTEM'])
pyrax.set_default_region(os.environ['OS_REGION_NAME'])
pyrax.set_credentials(os.environ['OS_USERNAME'], os.environ['OS_PASSWORD'])
nova = pyrax.cloudservers

for server in nova.servers.list():       
    print(server.id, server.name, server.status)

If you store the above script in a file named list-servers.py then you can execute it as follows:

$ python list-servers.py

If you are on Unix or OS X you can also make this file executable with chmod +x list-servers.py and then run it directly as a command:

$ ./list-servers.py

There are a couple important notes to make about this example. The pyrax.cloudservers object is not Rackspace specific, it is a Client instance, from the novaclient open source project. Anything you do with this object can be translated to other OpenStack based clouds. You can even install DevStack on your computer and test your scripts against it!

The other note I wanted to make is that the Server objects returned by list() have more information available to you. See the documentation if you want to see what you can do with them.

Flavors and Images

We've seen above how to obtain the list of flavors using nova. Here is a list-flavors.py script that achieves the same goal:

#!/usr/bin/env python
import os
import pyrax

pyrax.set_setting('identity_type', os.environ['OS_AUTH_SYSTEM'])
pyrax.set_default_region(os.environ['OS_REGION_NAME'])
pyrax.set_credentials(os.environ['OS_USERNAME'], os.environ['OS_PASSWORD'])
nova = pyrax.cloudservers

for flavor in nova.flavors.list():
    print(flavor.id, flavor.name)

And following the same style we can write a list-images.py script to list images:

#!/usr/bin/env python
import os
import pyrax

pyrax.set_setting('identity_type', os.environ['OS_AUTH_SYSTEM'])
pyrax.set_default_region(os.environ['OS_REGION_NAME'])
pyrax.set_credentials(os.environ['OS_USERNAME'], os.environ['OS_PASSWORD'])
nova = pyrax.cloudservers

for image in nova.images.list():
    print(image.id, image.name)

Like with servers, the Flavor and Image objects have a number of additional properties and methods.

Creating a server

Creating a cloud server using the nova APIs requires a script that is a bit longer than the ones we've been using so far, but the process is still fairly simple. Here is a script that creates a low-end Ubuntu 14.04 server named "my-server":

#!/usr/bin/env python
import os
import sys
import time
import re
import pyrax

pyrax.set_setting('identity_type', os.environ['OS_AUTH_SYSTEM'])
pyrax.set_default_region(os.environ['OS_REGION_NAME'])
pyrax.set_credentials(os.environ['OS_USERNAME'], os.environ['OS_PASSWORD'])
nova = pyrax.cloudservers

flavor = nova.flavors.find(name='512MB Standard Instance')
image = nova.images.find(name='Ubuntu 14.04 LTS (Trusty Tahr)')
key_name = None  # set your public key name here!

# create the server
server = nova.servers.create(name='my-server', flavor=flavor.id,
                             image=image.id, key_name=key_name)
print 'Building, please wait...'

# wait for server create to be complete
while server.status == 'BUILD':
    time.sleep(5)
    server = nova.servers.get(server.id)  # refresh server

# check for errors
if server.status != 'ACTIVE':
    print 'ERROR!'
    sys.exit(1)

# the server was assigned IPv4 and IPv6 addresses, locate the IPv4 address
ip_address = None
for network in server.networks['public']:
    if re.match('\d+\.\d+\.\d+\.\d+', network):
        ip_address = network
        break
if ip_address is None:
    print 'No IP address assigned!'
    sys.exit(1)
print 'The server is waiting at IP address {0}.'.format(ip_address)

In this script you can see how it is possible to find flavors or images by name using the find() method. Also recall that server creation is an asynchronous task, so we need to wait for the server to reach the "ACTIVE" state before it can be used. Once the server is ready we need to do a little bit of regex parsing to locate the IPv4 address was assigned to it.

Deleting a server

The other important operation you may need to automate is server destruction. Below is a script that deletes the server created above, waiting until the operation is complete:

#!/usr/bin/env python
import os
import time
import pyrax
from novaclient.exceptions import NotFound

pyrax.set_setting('identity_type', os.environ['OS_AUTH_SYSTEM'])
pyrax.set_default_region(os.environ['OS_REGION_NAME'])
pyrax.set_credentials(os.environ['OS_USERNAME'], os.environ['OS_PASSWORD'])
nova = pyrax.cloudservers

# find the server by name
server = nova.servers.find(name='my-server')
server.delete()
print 'Deleting, please wait...'

# wait for server to be deleted
try:
    while True:
        server = nova.servers.get(server.id)
        time.sleep(5)
except NotFound:
    pass
print 'The server has been deleted.'

A complete example

This was all too easy, right? In this section I present a practical application of the techniques I showed you above. We are going to write a script that creates a cloud server, configures it as a web server, and finally deploys a Python web application to it, all unattended. After this script runs you will have a fully enabled website online that you can connect to with a web browser from anywhere in the world!

You probably know that I like Flask very much, so for this example I'm going to deploy Flasky, the social blogging application featured in my book, straight from its GitHub project. To run the installation steps in the server I will use the Python package paramiko to create a SSH connection to the server. To install paramiko you need to run the following command:

$ pip install paramiko

The following sections describe the different parts of this deployment script, which is actually two, though in my defense the first script calls the second automatically. You can download the two complete scripts from this gist.

The main function

I called the deployment script rackspace-flasky.py. When this script executes it invokes the main() function. You can see this function below:

def main():
    args = process_args()
    gmail_password = getpass('gmail password: ')

    # instantiate the nova client
    global nova
    pyrax.set_setting('identity_type', os.environ['OS_AUTH_SYSTEM'])
    pyrax.set_default_region(os.environ['OS_REGION_NAME'])
    pyrax.set_credentials(os.environ['OS_USERNAME'], os.environ['OS_PASSWORD'])
    nova = pyrax.cloudservers

    flavor = get_flavor(args.flavor)
    image = get_image(args.image)

    ip_address = create_web_server(flavor, image, args.key_name)
    install_flasky(ip_address, args.gmail_username, gmail_password)
    print 'Flasky is now running at http://{0}.'.format(ip_address)

So as you can see here the script begins by processing the command line arguments. Function process_args() uses argparse for this task and is not interesting enough to show here. Below you can see the help message it generates:

$ ./rackspace-flasky.py --help
usage: rackspace-flasky.py [-h] [--flavor FLAVOR] [--image IMAGE]
                           KEY_NAME GMAIL_USERNAME

Deploy Flasky to a Rackspace cloud server.

positional arguments:
  KEY_NAME              Keypair name to install on the server. Must be
                        uploaded to your cloud account in advance.
  GMAIL_USERNAME        The username of the gmail account that the web app
                        will use to send emails.

optional arguments:
  -h, --help            show this help message and exit
  --flavor FLAVOR, -f FLAVOR
                        Flavor to use for the instance.Default is 512MB
                        Standard Instance.
  --image IMAGE, -i IMAGE
                        Image to use for the server. Default is Ubuntu 14.04.

The only two required arguments are the public key name to install on the new server and a username for a Gmail account. Flasky, the application I'm going to install, sends emails to users at different times. The easiest way to allow it to send emails is to use a Gmail account for that task. In a real production server you would have your own email server instead, since Gmail has quotas that may prevent you from sending all the emails that you need to send.

The password to the Gmail account is not given in the command line for security reasons, and instead is requested using Python's package getpass.

After all the arguments have been collected the script instantiates a nova client object, relying on the OS_* variables in the environment as we did before. The client object is stored in a global variable, so that all functions in this script can access it.

Next we retrieve the flavor and image objects, using the names given as command line arguments. These are very simple functions, shown below:

def get_flavor(name):
    return nova.flavors.find(name=name)

def get_image(name):
    return nova.images.find(name=name)

To complete the process, create_web_server() deals with the creation of the cloud server, while function install_flasky() transforms the clean instance into a complete web server. These two functions are described in the following sections.

Server creation

The function that creates the cloud server is not going to be a surprise for you, as it is very similar to the one we've seen before:

def create_web_server(flavor=None, image=None, key_name=None):
    server = nova.servers.create(name='flasky', flavor=flavor.id,
                                 image=image.id, key_name=key_name)
    print 'Building, please wait...'

    # wait for server create to be complete
    while server.status == 'BUILD':
        time.sleep(5)
        server = nova.servers.get(server.id)  # refresh server

    # check for errors
    if server.status != 'ACTIVE':
        raise RuntimeError('Server did not boot, status=' + server.status)

    # the server was assigned IPv4 and IPv6 addresses, locate the IPv4 address
    ip_address = None
    for network in server.networks['public']:
        if re.match('\d+\.\d+\.\d+\.\d+', network):
            ip_address = network
            break
    if ip_address is None:
        raise RuntimeError('No IP address assigned!')
    print 'Server is running at IP address ' + ip_address
    return ip_address

Server configuration

This is the function that connects to the cloud server and runs the installer for Flasky:

def install_flasky(ip_address, gmail_username, gmail_password):
    print 'Installing Flasky...'

    # establish a SSH connection
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.load_system_host_keys()
    retries_left = 3
    while True:
        try:
            ssh.connect(ip_address, username='root')
            break
        except socket_error as e:
            if e.errno != errno.ECONNREFUSED or retries_left <= 1:
                raise e
        time.sleep(10)  # wait 10 seconds and retry
        retries_left -= 1

    # upload deployment script
    ftp = ssh.open_sftp()
    ftp.put('install-flasky.sh', 'install-flasky.sh')
    ftp.chmod('install-flasky.sh', 0544)

    # deploy
    stdin, stdout, stderr = ssh.exec_command(
        './install-flasky.sh "{0}" "{1}"'.format(
            gmail_username, gmail_password))
    status = stdout.channel.recv_exit_status()
    open('stdout.log', 'wt').write(stdout.read())
    open('stderr.log', 'wt').write(stderr.read())
    if status != 0:
        raise RuntimeError(
            'Deployment script returned status {0}.'.format(status))

As you can see, I'm using paramiko to establish a SSH connection to the server. A password is not required because the server creation function installed the public key. In this function paramiko reads the corresponding private key from the default location, which on a Unix based system is ~/.ssh/.

I have found that even when the Rackspace API reports the server as active it may take a few seconds for SSH to respond to requests. For that reason I allow up to three retries to the connect() function, separated by 10 seconds of wait.

Once the SSH link is established I use SFTP to upload a bash script called install-flasky.sh to the server, and then execute it. In case there are problems, the stdout and stderr output from the command running on the cloud server are saved to files stdout.log and stderr.log.

Remote installer script

I'm not going to show the installer script here because it is largely unrelated to the topic of this article, but you can view it, along with the main Python script, here.

The script takes two arguments, the Gmail username and password for Flasky to use when sending emails. The actions performed by this script are listed below, you can easily find the proper sections in the script if you are interested in learning more:

  • Install dependencies, such as git, nginx and supervisor
  • Create a flasky user
  • Clone the Flasky application from the public GitHub repository
  • Create a configuration file for Flasky with the Gmail credentials and other variables.
  • Create a Python virtualenv and install all the Flasky dependencies in it.
  • Install Gunicorn, the web server that will run Flasky.
  • Run the Flasky setup function to create and populate the database.
  • Install a config file for supervisor to monitor Gunicorn. The command that runs Flasky is given here.
  • Install a config file for nginx. This is the front-end web server. It serves static files directly and proxies dynamic requests to the Gunicorn server.

Conclusion

I hope you decide to give this little project a try. If you do, then remember to set your OS_* environment variables, as this script authenticates to the Rackspace cloud APIs with them.

As I mentioned above, the two required files are in a gist here. Below is an example run of the script:

(venv) $ ./rackspace-flasky.py <key-name> <gmail-username>
gmail password: <gmail-password>
Building, please wait...
Server is running at IP address 162.242.209.20
Installing Flasky...
Flasky is now running at http://162.242.209.20.

Once you verify that the server is running and everything is all right don't forget to delete the server, if not it will continue to generate charges to your account.

I hope this article piqued your interest enough that you decide to give the cloud a try. To find more information you can always head over to the Rackspace Developer page, where you will find documentation for all the APIs and SDKs, or else go to the ultimate source, at openstack.org.

In future posts I will cover other cloud related topics such as scaling, load balancing and whatever else I find worthy of mention during my travels through the land of OpenStack.

Miguel

3 comments

  • #1 Dylan said 2015-03-04T08:13:14Z

    Fantastic example, very clear and helpful! Also, congrats on becoming a Racker.

  • #2 Graham Land said 2016-11-22T08:55:36Z

    Great article and fantastic book - thank you. Now that you're working with OpenStack by any chance do you have examples of using flask with keystone v3 authentication? I've got some python scripts that I put together to help admins but I'd like to give the scripts a web front-end using flask. The scripts run against OpenStack Keystone v3 and I thought I could use this to authenticate the admins - project admins in OpenStack terms. Many thanks, Graham

  • #3 Miguel Grinberg said 2016-11-22T20:01:05Z

    @Graham: I don't have any examples, sorry. But I would think it should be fairly to request or validate a token using standard keystone API calls.

Leave a Comment