What to log when there is a problem with the service

Whenever is a problem in the LFCs that has to be understood, you should log some data. The first thing to do is checking if there is a process taking completely one of the CPUs. For this you can use the command:
uptime
If it is, and nothing is being logged on /var/log/lfc/log then the service is stuck and you should collect and report the logs. You can start creating a directory with the date YYMMDD and put in it these files:
  • Log the last lines of the LFC log file with:
    tail -999 /var/log/lfc/log > log-YYMMDD.txt
  • Log all the processes running with:
    ps auxwwwfm > ps-YYMMDD.txt
  • Get the lfcdaemon process number with
    ps auxwww|grep lfc
    and log the list of open files with:
    lsof -p lfcdaemon_process_number > lsof-YYMMDD.txt
  • Log the top cpu processes:
    top -b -n 1 > top-YYMMDD.txt
  • The same for the netstat command:
    netstat -a > netstat-YYMMDD.txt
  • Obtain a core image with:
    gcore lfcdaemon_process_number
  • Log a list of the mapped memory regions via:
    cat /proc/lfcdaemon_process_number/maps > maps-YYMMDD.txt
  • Log gdb output:
    gdb -p lfcdaemon_process_number
    (gdb) info threads
    (gdb) set logging file gdb-YYMMDD.txt
    (gdb) set loggin on
    Call macro that does a stack dump of all threads:
    (gdb) allt 42 (42 means num of threads to log/check)
    (gdb) quit
    After running gdb, be sure that the lfcdaemon process is not blocked (does not have the T bit) when running:
    !ps
    If it has it, make the daemon continue with the command:
    kill -CONT lfcdaemon_process_number

-- Main.dcollado - 18 Dec 2006

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2006-12-18 - DavidCollados
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback