General OpenStack Errors and Solutions
After installing
OpenStack via Packstack, one may see errors both on the Horizon interface and the service logs. Those outlined are ones which we have encountered.
Error: unable to retrieve...
In our experience, sometimes a service may not function correctly straight after installation when installing via Packstack.
After logging into Horizon after an installation, a number of errors may be seen such as:
"unable to retrieve quota information"
"unable to retrieve instances"
Also within the Horizon logs you will see the following error:
"The server has either erred or is incapable of performing the requested operation"
This is due to the service being incorrectly configured by Packstack.
The problem emerges from database password problems where the specified
sql_connection password within the service.conf file (e.g
/etc/nova/nova.conf or
/etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini in the case of Quantum) does not allow access to the datasbase. The password specified in the configuration line below may not allow access to the database:
sql_connection = mysql://quantum:dfj9234nef923r90@lxfssmXXXX.cern.ch/ovs_quantum
To solve the error in this case, we have to manually change the service's database password to the one specified in all configuration files for '%', '127.0.0.1' and the 'hostname' of the machine. This can be performed via the following
MySQL commands within the
mysql DB and table
user:
SET PASSWORD FOR 'quantum'@'%' = PASSWORD('dfj9234nef923r90');
SET PASSWORD FOR 'quantum'@'127.0.0.1' = PASSWORD('dfj9234nef923r90');
SET PASSWORD FOR 'quantum'@'lxfssmXXXX.cern.ch' = PASSWORD('dfj9234nef923r903');
After making the appropriate change, restart
OpenStack and the problem should be solved!
Error: unauthorized to retrieve ... via Horizon interface
This likely occurs when you add the '
admin' user to another project. Simply log out and back in again.
you must have a tty to run sudo
Various
OpenStack services may not run due to not having the appropriate access they require. Within a service log, an error such as the following may appear:
"you must have a tty to run sudo"
To solve this problem, the service has to be given sudo access and hence a file must be inserted into
/etc/sudoers.d/ to give the service access. For example, the following text would be placed in
/etc/sudoers.d/nova :
Defaults:nova requiretty
nova ALL = (root) NOPASSWD: /usr/bin/nova-rootwrap
After making the change, restart
OpenStack and the problem should be solved!
Address already in use/{service} dead but pid file exists (upon OpenStack restart)
After restarting
OpenStack, a service (typically nova-api) may not start despite being correctly configured. Within the service log you will see and error such as:
"Address already in use"
Furthermore, querying the status of the service (e.g
/etc/init.d/openstack-nova-api status) may show
"{service} dead but pid file exists"
Simply remove the .pid file within
/var/run/{service}/{service.pid} and start the service to solve the problem.
An Instance is Stuck in an Intermediate State (e.g Error, Deleting etc)
In some cases, an instance may remain in an intermediate state and not delete. For example it remains in the Error or Deleteing state and hence will not actually delete after you select 'Terminate'.
You can reset the state of an instance or force the instance back into an Active state and then delete it via the command line to (sometimes!) solve this problem:
nova reset-state {instance_id}
nova delete {instance_id}
OR
nova reset-state --active {instance_id}
nova delete {instance_id}
Over time, your controller node will add new fields to the database and update its
OpenStack packages. As such, the nova versions of each host may differ causing the
KeyError: <VerNum(161)> error and hence stop compute nodes running. You can see this error when issuing a nova command on a compute host.
To solve this problem please ensure that all software packages (specifically nova!) installed on both controller and compute nodes have equal versions. Afterwards issue the following commands to synchronize with the database and restart the compute service:
nova-manage db sync
/etc/init.d/openstack-nova-compute
Re-installing OpenStack: Possible Errors and Solutions
If you have previously installed
OpenStack on your server or are performing a re-installation, many errors are likely to occur. Ensure you have cleared your entire system of previous
OpenStack databases and configuration files before installing
OpenStack. For example, those found in:
/etc/nova, /var/lib/nova
/etc/glance, /var/lib/glance
/etc/cinder, /var/lib/cinder
/etc/keystone, /var/lib/keystone
If you have forgot to remove a configuration file, drop a database, or remove an
OpenStack user from the database, these errors may occur:
* Could not evaluate: undefined method `creds_keystone' for #<Puppet::Type::Keystone_user::ProviderKeystone: 0x7f2e0fa3c390>
- Solution: remove the glance, cinder, nova and keystone databases from
MySQL
* packstack installation may stall/hang when testing if quantum has been applied or not
- Solution: remove the ovs_quantum database from
MySQL
* change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-nova-conductor' returned 1: Error: Nothing to do
- Solution: ensure the package is actually installed and configured correctly!!!!
* Access denied for user 'nova'@'controller_hostname' (using password: YES) (happens after an
OpenStack re-install)
- Solution: after applying typical
MySQL solutions and the service still can't access the database, first try the solution from the 'Error: unable to retrieve' solution above else remove ALL
OpenStack configuration files and re-install.