BAAN LDAP login broken

Description

Users could not login on BAAN on Thu, July 15th from 9:55 to 10:25. Only the users defined in LDAP were affected (all system-related users were working OK).

Impact

(Input from GS-AIS) Users already connected were not affected. The problem occurred with users trying to open a new connection. Some people from the stores replenishment (msanchez, crechard) were unable to do their job till the problem was solved. Luckily the incident occurred late in the morning when most users were connected.

Time line of the incident

• 2010-06-29 14:32 - Email from IT-PES about "reconfiguration of LDAP for SLC5 / RHES5" done "in preprod only for SLC5 / RHES5 boxes"
• 2010-07-15 09:50 - All Quattor-managed nodes notified for a spma-ncm run by IT-PES
• 2010-07-15 09:55 - ncm-authconfig runs on dbsrvd236 , changing /etc/pam.d/system-auth and /etc/ldap.conf
• 2010-07-15 10:22 - Email from GS-AIS signaling the problem
• 2010-07-15 10:25 - Acknowledged the problem and started investigating by checking the usual suspects (/etc/ldap.conf and /etc/pam.d/system-auth)
• 2010-07-15 10:26 - Files were modified at 9:55, restored the old ones (ncm-authconfig renames the old ones with a .old suffix)
• 2010-07-15 10:28 - Notified GS-AIS of the workaround
• 2010-07-15 10:29 - Confirmation that users could successfully login

Analysis

• The change was affecting the LDAP authentication configuration and package version, and it turned out to affect also SLC4/RH4 that are using the so-called "Zuul" method for LDAP authentication.
• The culprit was the Kerberos method that was inserted in /etc/pam.d/system-auth by the new version of ncm-authconfig. This kind of setting makes system logins work, but BAAN logins fail since apparently pam_krb5 is not supported by blogind.
• The permanent fix is to explicitly disable Kerberos authentication on the ncm-authconfig template

Follow up

• The permanent fix is to explicitly disable Kerberos authentication on the ncm-authconfig template
• 2010-07-15 16:30 - Started testing a permanent fix on BAAN spare server
• 2010-07-16 12:45 - Permanent fix successfully deployed and tested on BAAN spare and BAANv6
• 2010-07-19 13:30 - Permanent fix successfully deployed on BAAN