Procedure in case of problems with LFC

Important :

Remember to call the operators (75011) before rebooting any machine, so that they (temporarily) ignore the alarms (and don't start a parallel intervention.)

Lemon monitoring
Note that the LFC daemon is monitored by Lemon : if the LFC daemon is down for any reason, it will be restarted automatically. After 3 unsuccessfull restarts, the operators will call the support phones.

You can check the list of alarms in: LemonAlarmList

Go through the LFCSmokeTest.
This includes the description of some kind of problem with the relative solution. It is not meant to be exhaustive and is under constant update. If you can solve the problem then:
  1. Log your action in the intervention log
  2. Email the Second Level Support mailing list : hep-service-sc-level2@cernNOSPAMPLEASE.ch

If you can not solve the problem then try to use the backup LFC host:
  1. Figure out which is the backup host : /afs/cern.ch/project/gd/SC3/machines_list.txt
  2. Run the LFCSmokeTest and make sure this host is working correctly
  3. If the backup is working:
    • set the alias of the problematic service to the IP address of such host (DNSSwitchProcedure)
    • modify /afs/cern.ch/project/gd/SC3/machines_list.txt
    • Log the intervention in the intervention log
    • Email the Second Level Support Mailing List
    • Email the appropriate Third Level Support mailing list

Note : If needed, several aliases (all from the Pilot or the Production) can point to the same host, as there is only one shared database account per service (Pilot and Production).

If the backup is not working then you are experiencing a more general problem.
  1. Try to think at general problems like network connections, firewalls, connection with database etc ... and look at the appropriate log files (both of the current service machine and the backup) for hints (/var/log/messages and /var/log/lfc/log are just an example)
  2. If you still are clueless then

-- SophieLemaitre - 07 Jul 2005

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r9 - 2006-02-07 - PeterJones
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback