General OpenStack Errors and Solutions

After installing OpenStack via Packstack, one may see errors both on the Horizon interface and the service logs. Those outlined are ones which we have encountered.

Error: unable to retrieve...

In our experience, sometimes a service may not function correctly straight after installation when installing via Packstack.

After logging into Horizon after an installation, a number of errors may be seen such as:

"unable to retrieve quota information"
"unable to retrieve instances"

Also within the Horizon logs you will see the following error:

"The server has either erred or is incapable of performing the requested operation"

This is due to the service being incorrectly configured by Packstack.

The problem emerges from database password problems where the specified sql_connection password within the service.conf file (e.g /etc/nova/nova.conf or /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini in the case of Quantum) does not allow access to the datasbase. The password specified in the configuration line below may not allow access to the database:

sql_connection = mysql://quantum:dfj9234nef923r90@lxfssmXXXX.cern.ch/ovs_quantum

To solve the error in this case, we have to manually change the service's database password to the one specified in all configuration files for '%', '127.0.0.1' and the 'hostname' of the machine. This can be performed via the following MySQL commands within the mysql DB and table user:

SET PASSWORD FOR 'quantum'@'%' = PASSWORD('dfj9234nef923r90');
SET PASSWORD FOR 'quantum'@'127.0.0.1' = PASSWORD('dfj9234nef923r90');
SET PASSWORD FOR 'quantum'@'lxfssmXXXX.cern.ch' = PASSWORD('dfj9234nef923r903');

After making the appropriate change, restart OpenStack and the problem should be solved!

Error: unauthorized to retrieve ... via Horizon interface

This likely occurs when you add the 'admin' user to another project. Simply log out and back in again.

you must have a tty to run sudo

Various OpenStack services may not run due to not having the appropriate access they require. Within a service log, an error such as the following may appear:

"you must have a tty to run sudo"

To solve this problem, the service has to be given sudo access and hence a file must be inserted into /etc/sudoers.d/ to give the service access. For example, the following text would be placed in /etc/sudoers.d/nova :

Defaults:nova requiretty
nova ALL = (root) NOPASSWD: /usr/bin/nova-rootwrap

After making the change, restart OpenStack and the problem should be solved!

Address already in use/{service} dead but pid file exists (upon OpenStack restart)

After restarting OpenStack, a service (typically nova-api) may not start despite being correctly configured. Within the service log you will see and error such as:

"Address already in use"

Furthermore, querying the status of the service (e.g /etc/init.d/openstack-nova-api status) may show

"{service} dead but pid file exists"

Simply remove the .pid file within /var/run/{service}/{service.pid} and start the service to solve the problem.

An Instance is Stuck in an Intermediate State (e.g Error, Deleting etc)

In some cases, an instance may remain in an intermediate state and not delete. For example it remains in the Error or Deleteing state and hence will not actually delete after you select 'Terminate'.

You can reset the state of an instance or force the instance back into an Active state and then delete it via the command line to (sometimes!) solve this problem:

nova reset-state {instance_id}
nova delete {instance_id}

OR

nova reset-state --active {instance_id}
nova delete {instance_id}

KeyError: <VerNum(161)>

Over time, your controller node will add new fields to the database and update its OpenStack packages. As such, the nova versions of each host may differ causing the KeyError: <VerNum(161)> error and hence stop compute nodes running. You can see this error when issuing a nova command on a compute host.

To solve this problem please ensure that all software packages (specifically nova!) installed on both controller and compute nodes have equal versions. Afterwards issue the following commands to synchronize with the database and restart the compute service:

nova-manage db sync
/etc/init.d/openstack-nova-compute

Re-installing OpenStack: Possible Errors and Solutions

If you have previously installed OpenStack on your server or are performing a re-installation, many errors are likely to occur. Ensure you have cleared your entire system of previous OpenStack databases and configuration files before installing OpenStack. For example, those found in:

/etc/nova, /var/lib/nova
/etc/glance, /var/lib/glance
/etc/cinder, /var/lib/cinder
/etc/keystone, /var/lib/keystone

If you have forgot to remove a configuration file, drop a database, or remove an OpenStack user from the database, these errors may occur:

* Could not evaluate: undefined method `creds_keystone' for #<Puppet::Type::Keystone_user::ProviderKeystone: 0x7f2e0fa3c390>
- Solution: remove the glance, cinder, nova and keystone databases from MySQL

* packstack installation may stall/hang when testing if quantum has been applied or not
- Solution: remove the ovs_quantum database from MySQL

* change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-nova-conductor' returned 1: Error: Nothing to do
- Solution: ensure the package is actually installed and configured correctly!!!!

* Access denied for user 'nova'@'controller_hostname' (using password: YES) (happens after an OpenStack re-install)
- Solution: after applying typical MySQL solutions and the service still can't access the database, first try the solution from the 'Error: unable to retrieve' solution above else remove ALL OpenStack configuration files and re-install.
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2013-08-14 - GaryMcGilvary
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback