Integrating OpenStack with Ceph

In this document we describe how to intergrate OpenStack Cinder with Ceph to enable volume based back end storage as well as image management and storage.

We assume that you do not have administrative permissions to the Ceph service and 'volumes' and 'images' pools have been created for you. To access these pools, we also assume the users 'volumes' and 'images' have been created and that the pools can be accessed by the provided keyrings (ceph.client.volumes.keyring, ceph.client.images.keyring) and a ceph configuration file (ceph.conf).

Sources of Information

Currently there exists good documentation on integrating Ceph with OpenStack however dependent on the package versions installed, the integration may not be successful hence why our approach is outlined here. The documentation from other sources can be found at:

Installing Ceph and Required Packages

Firstly, it is unlikely that the correct QEMU version is installed on your compute nodes. This needs to be QEMU version 0.12.1 and needs to be installed from the Ceph repository. Other 0.12.1 versions exists however produce errors when used with Ceph. If you believe you have the correct packages installed, please jump to the 'Ceph Volume Configuration' section otherwise remove qemu and related packages:

Note: the following command will remove openstack cinder on the controller node and openstack-nova-compute on compute nodes hence these must be re-installed afterwards. Hence it is a good idea to copy the configuration files to a safe place so they can be returned after re-installation

Install the Ceph dependency packages:

and then the correct QEMU and Ceph packages from the respective repositories found at:

Cinder / Volume Configuration

On each of the OpenStack nodes (controller and compute) copy the ceph configuration file and keyrings you were given to /etc/ceph/. Make sure the configuration file has the correct permissions for the cinder service to access it:

Also check that the keyring files are readable by all and if not:

In order to access the 'volumes' pool, we need to instruct Ceph where and what keyrings to use. We can do this by adding the following to the Ceph configuration file:

In order to use the command line tools 'rbd' and 'rados' we need to specify the arguments to Ceph via the CEPH_ARGS environment variable:

This specifies that the user 'volumes' should access the Ceph service and with the combination of adding the above to ceph.conf, we should be able to sucessfully use Ceph. Test this by trying:

Now we need to re-install openstack-cinder and openstack-nova-compute on the appropriate nodes and ensure that the appropriate services are running again:

It would be wise to check that the service is functioning normally by executing:

If not, fix these problems first before proceeding with the integration. You may find a solution within OpenStackErrors.

Now we need to export the user that the Cinder and Nova services, on the controller and compute nodes respectively, must use to access Ceph. We do this by adding export CEPH_ARG="--id volumes" to /etc/init.d/openstack-cinder-volume and /etc/init.d/openstack-nova-compute and then restart these services:

For hosts running nova-compute (controller and compute nodes in our case), these services actually do not need a keyring as they instead store a secret key in libvirt. On the controller host, first create the secret.xml file containing:

and then create/output the secret key by executing:

Copy this secret key and add it so libvirt by executing:

where the_client_key is the key found within ceph.client.volumes.keyring.

On the OpenStack compute nodes, we ensure that each host has the same secret key by creating the following secret.xml file:

And then execute the same virsh commands as above

Now Ceph, RDB and Rados should successfully work from the command line. Finally, in order to give OpenStack the ability to create, attach and delete volumes via the Horizon interface, we must add the following to the Cinder configuration file (/etc/cinder/cinder.conf):

The pool name and user to access the pool may be different in your case. After restarting the Cinder volume service, it should be successfully integrated with Ceph.

Glance / Image Configuration

On the controller node (where Glance is installed), we must install the python libraries for RBD; this is available from Ceph:

yum -y install http://ceph.com/rpm-cuttlefish/el6/x86_64/python-ceph-0.61.7-0.el6.x86_64.rpm

We must also edit the /etc/glance/glance-api.conf file to include the following variables within the [DEFAULT] section.

In order to access the 'images' pool, we need to instruct Ceph where and what keyrings to use. We can do this by adding the following to the Ceph configuration file:

Restart the Glance API to pick up the configuration changes:

Important: Note that some of these variables may already exist and simply might need changing to reflect your pool name and username to access RBD; adding these variables before RBD variables that already exist will result in your changes being overwritten hence connection errors are likely to occur.

Also ensure that the ceph.client.images.keyring within the /etc/ceph directory and has the correct permissions (cinder:cinder).

Now test image creation via Glance (an example):

Finally, check that this image appears within the pool 'images':

Note: you may have to export CEPH_ARGS="--id images" to get the above command to work.

You should now be able to create images and boot instances from these images!

Troubleshooting

Error Deleting Volume

This error typically occurs when Ceph is incorrectly configured or in the case when configuration is correct, deleting a large volume can cause this error to appear both within the logs and on the Horizon interface. To remove the volume from Ceph, we need to obtain the ID of the volume which can be found by executing:

and copying the ID of the volume where the state is error_deleting.

We then remove the Ceph volume:

Now that the volume is deleted, the volume will still appear on the Horizon interface hence we need to manually remove the volume entry from the MySQL database:

It is likely that when an such an error occurs, the amount of volumes and total volume GB's reserved are not updated hence one may see the subsequent error in the future. It would be wise to check that the correct amount of volumes and volume GB's in use are correct after such an error, hence following the solution to the error below.

VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded

When deleting a Ceph volume, it may fail as described above hence the usage of volumes and total volume GB's reserved may not be updated. This will result in the "VolumeLimitExceeded" error when actually less than 10 (the default value) volumes are in use.

Check that the correct number of volumes and GB's are recorded in the cinder.quotas_usages table:

If the figures are incorrect, there are two ways to solve this problem:

1. Increase the quota of allowed volumes:

2. Update the database with the correct number of volumes and volume GB's currently in use. This is the prefered as method (1) does not provide a long term solution but instead delays this error from appearing again:

The correct volumes and volume GB's in use can actually be seen from the Horizon interface when one selects 'Create Volume'.

Caught signal (Segmentation fault)

When executing a Ceph, rbd or rados command, one may see the output of a core dump specifing a segmentation fault as the cause of the problem. This was noticed when Ceph was installed via the method outlined above but later changed to an older version, likely by automatic configuration changes. For example, Ceph 0.61.7 was initially installed however was downgraded to version 0.56.3. Ensure the correct version of Ceph is installed to solve this problem.