Crab vs snoopy.so on lxplus

references

Example on a normal lxplus node

SSH Secure Shell 3.2.9 (Build 283)
Copyright (c) 2000-2003 SSH Communications Security Corp - http://www.ssh.com/

This copy of SSH Secure Shell is a non-commercial version.
This version does not include PKI and PKCS #11 functionality.


Last login: Fri Jan  7 10:00:49 2011 from 151.65.43.167
*******************************************************************************
*      LXPLUS Public Login Service         Website:  http://cern.ch/plus      *
*    Scientific Linux CERN 5.5 x86_64      Support:   helpdesk@cern.ch        *
*                                                                             *
*    Reminder: You have agreed to comply with the CERN computing rules        *
*                                          http://cern.ch/ComputingRules      *
******************************************************************************* 
Wed  1st Dec 2010: Old /etc/motd messages moved to /etc/motd-archive/
Wed  1st Dec 2010: Reminder:
                   SLC4 LxBatch service will close on 13th December.
                   SLC4 LxPlus service will close on 14th December.
                   See http://cern.ch/ssb for more details.
Wed  1st Dec 2010: pexpect installed, a python version of expect.
Thu  2nd Dec 2010: /opt/atlas,cms,lhcb and sft are available via cvmfs for 
                   testing  on lxplus253,414 and 437.
Fri  3rd Dec 2010: colordiff installed - yet another comparison tool.
                   openssl-devel(i386) installed.
Mon  6th Dec 2010: Routine SLC5 upgrade. No updates.
Wed  8th Dec 2010: wxPython, wlxGTK and wxBase installed. GUI toolkit.
Mon 13th Dec 2010: Routine SLC5 upgrade including apr-util, phonebook,
                   gnome-screensaver, gtk2, krb5-libs, thundirbird and 
                   php-cli.
Mon 13th Dec 2010: Change in LxPlus account resolving, should be transparent.
Tue 14th Dec 2010: SLC4 LxPlus and LxBatch services are now closed.
Mon 10th Jan 2011: Routine SLC5 update including firefox, xulrunner, 
                   thunderbird, openssl, bind-utils, wxBase, wxGTK 
                   and Django-South.
                   A restart of firefox and thunderbird is advised.
Wed 12th Jan 2011: Eleven nodes added to lxplus5.
******************************************************************************

belforte@lxplus248/~> PROVA
Your identity: /C=IT/O=INFN/OU=Personal Certificate/L=Trieste/CN=Stefano Belforte
Creating temporary proxy ................................................... Done
Contacting  voms.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch] "cms" Done
Creating proxy .................................................................................... Done
Your proxy is valid until Thu Jan 13 22:03:53 2011
belforte@lxplus248/crabtest> cd ~/scratch0/casino
belforte@lxplus248/casino> crab -c DYToTauTau_M-20_bg -status
crab:  Version 2.7.6 running on Thu Jan 13 10:04:21 2011 CET (09:04:21 UTC)

crab. Working options:
        scheduler           glite
        job type            CMSSW
        server              OFF
        working directory   /afs/cern.ch/user/b/belforte/scratch0/casino/DYToTauTau_M-20_bg/

crab:  Checking the status of all jobs: please wait
*** glibc detected *** python: double free or corruption (!prev): 0x091add70 ***
======= Backtrace: =========
/lib/libc.so.6[0x6b75a5]
/lib/libc.so.6(cfree+0x59)[0x6b79e9]
/lib/snoopy.so[0xf7fb3830]
/lib/snoopy.so(execv+0x47)[0xf7fb38c7]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0[0xf7f4e736]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyCFunction_Call+0x19d)[0xf7eb0b6d]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x4d6f)[0xf7f12def]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x7c8)[0xf7f15538]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5952)[0xf7f139d2]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x6669)[0xf7f146e9]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x6669)[0xf7f146e9]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x7c8)[0xf7f15538]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0[0xf7e9aee9]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyObject_Call+0x5c)[0xf7e6c1dc]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0[0xf7e7cdf5]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyObject_Call+0x5c)[0xf7e6c1dc]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0[0xf7ed49f3]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0[0xf7ecacf5]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyObject_Call+0x5c)[0xf7e6c1dc]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x302d)[0xf7f110ad]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x7c8)[0xf7f15538]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5952)[0xf7f139d2]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x6669)[0xf7f146e9]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x7c8)[0xf7f15538]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5952)[0xf7f139d2]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x7c8)[0xf7f15538]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5952)[0xf7f139d2]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x7c8)[0xf7f15538]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5952)[0xf7f139d2]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x6669)[0xf7f146e9]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x6669)[0xf7f146e9]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x7c8)[0xf7f15538]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5952)[0xf7f139d2]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x6669)[0xf7f146e9]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x6669)[0xf7f146e9]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x7c8)[0xf7f15538]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyEval_EvalCode+0x63)[0xf7f15693]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyRun_FileExFlags+0xb4)[0xf7f365c4]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyRun_SimpleFileExFlags+0x1fc)[0xf7f3691c]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(PyRun_AnyFileExFlags+0x7a)[0xf7f36c0a]
/afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_7/external/slc5_ia32_gcc434/lib/libpython2.6.so.1.0(Py_Main+0xdda)[0xf7f4494a]
python(main+0x32)[0x80485c2]
/lib/libc.so.6(__libc_start_main+0xdc)[0x663e9c]
python[0x80484d1]
======= Memory map: ========
0064e000-007a1000 r-xp 00000000 08:06 1900620                            /lib/libc-2.5.so
007a1000-007a3000 r--p 00152000 08:06 1900620                            /lib/libc-2.5.so
007a3000-007a4000 rw-p 00154000 08:06 1900620                            /lib/libc-2.5.so
007a4000-007a7000 rw-p 007a4000 00:00 0 
007e2000-007e5000 r-xp 00000000 08:06 1900571                            /lib/libdl-2.5.so
007e5000-007e6000 r--p 00002000 08:06 1900571                            /lib/libdl-2.5.so
007e6000-007e7000 rw-p 00003000 08:06 1900571                            /lib/libdl-2.5.so
08048000-08049000 r-xp 00000000 00:17 880863260                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/bin/python
08049000-0804a000 rw-p 00000000 00:17 880863260                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/bin/python
08d01000-091d1000 rw-p 08d01000 00:00 0                                  [heap]
f7050000-f7051000 ---p f7050000 00:00 0 
f7051000-f7a51000 rwxp f7051000 00:00 0 
f7a51000-f7a61000 r-xp 00000000 08:06 1900593                            /lib/libresolv-2.5.so
f7a61000-f7a62000 r--p 0000f000 08:06 1900593                            /lib/libresolv-2.5.so
f7a62000-f7a63000 rw-p 00010000 08:06 1900593                            /lib/libresolv-2.5.so
f7a63000-f7a65000 rw-p f7a63000 00:00 0 
f7a65000-f7a69000 r-xp 00000000 08:06 1900581                            /lib/libnss_dns-2.5.so
f7a69000-f7a6a000 r--p 00003000 08:06 1900581                            /lib/libnss_dns-2.5.so
f7a6a000-f7a6b000 rw-p 00004000 08:06 1900581                            /lib/libnss_dns-2.5.so
f7a6b000-f7a75000 r-xp 00000000 08:06 1900583                            /lib/libnss_files-2.5.so
f7a75000-f7a76000 r--p 00009000 08:06 1900583                            /lib/libnss_files-2.5.so
f7a76000-f7a77000 rw-p 0000a000 08:06 1900583                            /lib/libnss_files-2.5.so
f7a79000-f7a85000 r-xp 00000000 00:17 880739764                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/gcc/4.3.4/lib/libgcc_s.so.1
f7a85000-f7a86000 rw-p 0000b000 00:17 880739764                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/gcc/4.3.4/lib/libgcc_s.so.1
f7a86000-f7a91000 r-xp 00000000 00:17 880858750                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/_sha512.so
f7a91000-f7a92000 rw-p 0000a000 00:17 880858750                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/_sha512.so
f7a92000-f7a96000 r-xp 00000000 00:17 880858748                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/_sha256.so
f7a96000-f7a97000 rw-p 00004000 00:17 880858748                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/_sha256.so
f7a97000-f7a99000 r-xp 00000000 00:17 880858730                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/_hashlib.so
f7a99000-f7a9a000 rw-p 00002000 00:17 880858730                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/_hashlib.so
f7a9a000-f7a9b000 rw-p f7a9a000 00:00 0 
f7a9b000-f7b04000 r-xp 00000000 00:17 880746782                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/sqlite/3.6.10/lib/libsqlite3.so.0.8.6
f7b04000-f7b06000 rw-p 00069000 00:17 880746782                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/sqlite/3.6.10/lib/libsqlite3.so.0.8.6
f7b06000-f7b12000 r-xp 00000000 00:17 1303125888                         /afs/cern.ch/user/b/belforte/scratch0/CRAB_PROVA/external/py2-pysqlite/2.4/site-packages/pysqlite2/_sqlite.so
f7b12000-f7b14000 rw-p 0000b000 00:17 1303125888                         /afs/cern.ch/user/b/belforte/scratch0/CRAB_PROVA/external/py2-pysqlite/2.4/site-packages/pysqlite2/_sqlite.so
f7b14000-f7b17000 r-xp 00000000 00:17 880858736                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/_json.so
f7b17000-f7b18000 rw-p 00002000 00:17 880858736                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/_json.so
f7b18000-f7b79000 rw-p f7b18000 00:00 0 
f7b79000-f7b7d000 r-xp 00000000 00:17 880858800                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/math.so
f7b7d000-f7b7f000 rw-p 00003000 00:17 880858800                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/math.so
f7b7f000-f7b86000 r-xp 00000000 00:17 880858764                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/array.so
f7b86000-f7b88000 rw-p 00006000 00:17 880858764                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/array.so
f7b88000-f7b8e000 r-xp 00000000 00:17 880858806                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/operator.so
f7b8e000-f7b90000 rw-p 00005000 00:17 880858806                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/operator.so
f7b90000-f7b95000 r-xp 00000000 00:17 880858712                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/_collections.so
f7b95000-f7b96000 rw-p 00005000 00:17 880858712                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/python/2.6.4-cms6/lib/python2.6/lib-dynload/_collections.so
f7b96000-f7c8d000 r-xp 00000000 00:17 880743672                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/openssl/0.9.7m/lib/libcrypto.so.0.9.7
f7c8d000-f7c9e000 rw-p 000f7000 00:17 880743672                          /afs/cern.ch/cms/slc5_ia32_gcc434/external/openssl/0.9.7m/lib/libcrypto.so.0.9.7
f7c9e000-f7ca2000 rw-p f7c9e000 00:00 0 
f7ca2000-f7cd1000 r-xp 00000000 00:17 88074
crab: Status Query failed with message : error executing GLiteStatusQuery


Log file is /afs/cern.ch/user/b/belforte/scratch0/casino/DYToTauTau_M-20_bg/log/crab.log
belforte@lxplus248/casino> 

Example on lxplus401 where Steve Trayled disabled snoopy

belforte@lxplus248/~> ssh lxplus401
The authenticity of host 'lxplus401 (137.138.210.200)' can't be established.
RSA key fingerprint is a4:9f:57:6b:d5:4e:4d:56:85:ba:99:db:8c:2a:8e:b7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxplus401,137.138.210.200' (RSA) to the list of known hosts.
*******************************************************************************
*      LXPLUS Public Login Service         Website:  http://cern.ch/plus      *
*    Scientific Linux CERN 5.5 x86_64      Support:   helpdesk@cern.ch        *
*                                                                             *
*    Reminder: You have agreed to comply with the CERN computing rules        *
*                                          http://cern.ch/ComputingRules      *
******************************************************************************* 
Wed  1st Dec 2010: Old /etc/motd messages moved to /etc/motd-archive/
Wed  1st Dec 2010: Reminder:
                   SLC4 LxBatch service will close on 13th December.
                   SLC4 LxPlus service will close on 14th December.
                   See http://cern.ch/ssb for more details.
Wed  1st Dec 2010: pexpect installed, a python version of expect.
Thu  2nd Dec 2010: /opt/atlas,cms,lhcb and sft are available via cvmfs for 
                   testing  on lxplus253,414 and 437.
Fri  3rd Dec 2010: colordiff installed - yet another comparison tool.
                   openssl-devel(i386) installed.
Mon  6th Dec 2010: Routine SLC5 upgrade. No updates.
Wed  8th Dec 2010: wxPython, wlxGTK and wxBase installed. GUI toolkit.
Mon 13th Dec 2010: Routine SLC5 upgrade including apr-util, phonebook,
                   gnome-screensaver, gtk2, krb5-libs, thundirbird and 
                   php-cli.
Mon 13th Dec 2010: Change in LxPlus account resolving, should be transparent.
Tue 14th Dec 2010: SLC4 LxPlus and LxBatch services are now closed.
Mon 10th Jan 2011: Routine SLC5 update including firefox, xulrunner, 
                   thunderbird, openssl, bind-utils, wxBase, wxGTK 
                   and Django-South.
                   A restart of firefox and thunderbird is advised.
Wed 12th Jan 2011: Eleven nodes added to lxplus5.
******************************************************************************

belforte@lxplus401/~> PROVA
Your identity: /C=IT/O=INFN/OU=Personal Certificate/L=Trieste/CN=Stefano Belforte
Creating temporary proxy ........................................ Done
Contacting  voms.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch] "cms" Done
Creating proxy .................................... Done
Your proxy is valid until Thu Jan 13 22:04:39 2011
belforte@lxplus401/crabtest> cd ~/scratch0/casino
belforte@lxplus401/casino> crab -c DYToTauTau_M-20_bg -status
crab:  Version 2.7.6 running on Thu Jan 13 10:04:47 2011 CET (09:04:47 UTC)

crab. Working options:
        scheduler           glite
        job type            CMSSW
        server              OFF
        working directory   /afs/cern.ch/user/b/belforte/scratch0/casino/DYToTauTau_M-20_bg/

crab:  Checking the status of all jobs: please wait
crab:  
ID    END STATUS            ACTION       ExeExitCode JobExitCode E_HOST
----- --- ----------------- ------------  ---------- ----------- ---------
1     Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
2     Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
3     Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
4     Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
5     Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
6     Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
7     Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
8     Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
9     Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
10    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
--------------------------------------------------------------------------------
11    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
12    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
13    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
14    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
15    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
16    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
17    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
18    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
19    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
20    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
--------------------------------------------------------------------------------
21    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
22    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
23    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
24    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
25    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
26    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
27    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
28    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
29    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
30    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
--------------------------------------------------------------------------------
31    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
32    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
33    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
34    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
35    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
36    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
37    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
38    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
39    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
40    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
--------------------------------------------------------------------------------
41    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
42    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
43    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
44    Y   Done              Terminated                           rossmann-osg.rcac.purdue.edu
45    Y   Done              Terminated                           oberon.hep.kbfi.ee
46    Y   Done              Terminated                           oberon.hep.kbfi.ee
47    Y   Done              Terminated                           oberon.hep.kbfi.ee
48    Y   Done              Terminated                           oberon.hep.kbfi.ee
49    Y   Done              Terminated                           oberon.hep.kbfi.ee
50    Y   Done              Terminated                           oberon.hep.kbfi.ee
--------------------------------------------------------------------------------
51    Y   Done              Terminated                           oberon.hep.kbfi.ee
52    Y   Done              Terminated                           oberon.hep.kbfi.ee
53    Y   Done              Terminated                           oberon.hep.kbfi.ee
54    Y   Done              Terminated                           oberon.hep.kbfi.ee
55    Y   Done              Terminated                           oberon.hep.kbfi.ee
56    Y   Done              Terminated                           oberon.hep.kbfi.ee
57    Y   Done              Terminated                           oberon.hep.kbfi.ee
58    Y   Done              Terminated                           oberon.hep.kbfi.ee
59    Y   Done              Terminated                           oberon.hep.kbfi.ee
60    Y   Done              Terminated                           oberon.hep.kbfi.ee
--------------------------------------------------------------------------------
61    Y   Done              Terminated                           oberon.hep.kbfi.ee
62    Y   Done              Terminated                           oberon.hep.kbfi.ee
63    Y   Done              Terminated                           oberon.hep.kbfi.ee
64    Y   Done              Terminated                           oberon.hep.kbfi.ee
65    Y   Done              Terminated                           oberon.hep.kbfi.ee
66    Y   Done              Terminated                           oberon.hep.kbfi.ee
67    Y   Done              Terminated                           oberon.hep.kbfi.ee
68    Y   Done              Terminated                           oberon.hep.kbfi.ee
69    Y   Done              Terminated                           oberon.hep.kbfi.ee
70    Y   Done              Terminated                           oberon.hep.kbfi.ee
--------------------------------------------------------------------------------
71    Y   Done              Terminated                           oberon.hep.kbfi.ee
72    Y   Done              Terminated                           oberon.hep.kbfi.ee
73    Y   Done              Terminated                           oberon.hep.kbfi.ee
74    Y   Done              Terminated                           oberon.hep.kbfi.ee
75    Y   Done              Terminated                           oberon.hep.kbfi.ee
76    Y   Done              Terminated                           oberon.hep.kbfi.ee
77    Y   Done              Terminated                           oberon.hep.kbfi.ee
78    Y   Done              Terminated                           oberon.hep.kbfi.ee
79    Y   Done              Terminated                           oberon.hep.kbfi.ee
80    Y   Done              Terminated                           oberon.hep.kbfi.ee
--------------------------------------------------------------------------------
81    Y   Done              Terminated                           oberon.hep.kbfi.ee
82    Y   Done              Terminated                           oberon.hep.kbfi.ee
83    Y   Done              Terminated                           oberon.hep.kbfi.ee
84    Y   Done              Terminated                           oberon.hep.kbfi.ee
85    Y   Done              Terminated                           oberon.hep.kbfi.ee
86    Y   Done              Terminated                           oberon.hep.kbfi.ee
87    Y   Done              Terminated                           oberon.hep.kbfi.ee
88    Y   Done              Terminated                           oberon.hep.kbfi.ee
89    Y   Done              Terminated                           oberon.hep.kbfi.ee
90    Y   Done              Terminated                           oberon.hep.kbfi.ee
--------------------------------------------------------------------------------
91    Y   Done              Terminated                           oberon.hep.kbfi.ee
92    Y   Done              Terminated                           oberon.hep.kbfi.ee
93    Y   Done              Terminated                           oberon.hep.kbfi.ee
94    Y   Done              Terminated                           oberon.hep.kbfi.ee
95    Y   Done              Terminated                           oberon.hep.kbfi.ee
96    Y   Done              Terminated                           oberon.hep.kbfi.ee
97    Y   Done              Terminated                           oberon.hep.kbfi.ee
98    Y   Done              Terminated                           oberon.hep.kbfi.ee
99    Y   Done              Terminated                           oberon.hep.kbfi.ee
100   Y   Done              Terminated                           oberon.hep.kbfi.ee
--------------------------------------------------------------------------------
101   Y   Done              Terminated                           oberon.hep.kbfi.ee
102   Y   Done              Terminated                           oberon.hep.kbfi.ee
103   Y   Done              Terminated                           oberon.hep.kbfi.ee
104   Y   Done              Terminated                           oberon.hep.kbfi.ee

crab:   104 Total Jobs 
 >>>>>>>>> 104 Jobs Done
        Jobs terminated: retrieve them with: crab -getoutput <List of jobs>
        List of jobs: 1-104 

crab:  You can also follow the status of this task on :
        CMS Dashboard: http://dashb-cms-job-task.cern.ch/taskmon.html#task=sani_DYToTauTau_M-20_bg_q2o5f9
        Your task name is: sani_DYToTauTau_M-20_bg_q2o5f9 

Log file is /afs/cern.ch/user/b/belforte/scratch0/casino/DYToTauTau_M-20_bg/log/crab.log
belforte@lxplus401/casino> 


A bit of details

what my commands/aliases are doing
belforte@lxplus401/casino> alias PROVA
alias PROVA='cd ~/w0/crabtest; source SETUP.PROVA; <here it goes some secret stuff to get me a valid proxy>''
belforte@lxplus401/casino> cat ~/w0/crabtest/SETUP.PROVA
#source /afs/cern.ch/cms/LCG/LCG-2/UI/cms_ui_env.sh
#source /afs/cern.ch/user/b/belforte/grid/GRID-ALIAS
source  /afs/cern.ch/user/b/belforte/grid/CERN-280
cd /afs/cern.ch/user/b/belforte/w0/CMSSW_3_8_7
eval `scramv1 runtime -sh`
cd /afs/cern.ch/user/b/belforte/w0/crabtest
#source /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_7_5/crab.sh
source /afs/cern.ch/user/b/belforte/scratch0/CRAB_PROVA/crab.sh
belforte@lxplus401/casino> 

How to reproduce

Assuming you are not a CMS member but in the hope it works nevertheless, please give it a try in a clean bash shell. This also removes some customizations in my tests above. Note that grid proxy is not needed (the crab command will not work,in the end, but the snoopy problem happens before that).

mkdir /tmp/mytest
cd /tmp/mytest
source /afs/cern.ch/cms/cmsset_default.sh
source /afs/cern.ch/project/gd/LCG-share/3.2.8-0/etc/profile.d/grid_env.sh
export PYTHONPATH=${PYTHONPATH}:${GLITE_LOCATION}/lib
cmsrel CMSSW_3_8_6
cd CMSSW_3_8_6
cmsenv
cd ..
tar xjvf ~belforte/scratch0/casino.tj
cd casino
source /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_7_6_patch1/crab.sh
crab -c DYToTauTau_M-20_bg -status




Where it all happens

Where in our code the problem happens and what details I found

in /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_7_6_patch1/external/ProdCommon/BossLite/Common/System.py there is

def executeCommand( command, timeout=None ):
    """
    _executeCommand_

    Util it execute the command provided in a popen object with a timeout
    """

    start = time.time()
    p = Popen( command, shell=True, \
               stdin=PIPE, stdout=PIPE, stderr=STDOUT, \
               close_fds=True, preexec_fn=setPgid )

the traceback is orginated when that is called with command set to the string in here ~belforte/scratch0/casino/prova/command

the code that calls that is likely not very instructive, but I can point to it

it is 5423 bytes, driven by the list of grid jobId's for which status is queried. Users reported that things are OK when the number of jobs is small.

My attempts to execute that Popen in a clean python session failed to reproduce the problem

Fixing the Situation

Update Jan 13th 2011 16:20

  • snoopy is now disabled on lxplus except for lxplus250.
  • Luis provided a patch 0001-Fix-a-problem-with-strings-not-necessarily-being-nul.patch however after testing the lxplus250 the situation is still present. Luis investigating more.

Update Jul 30 2014 ()

-- StefanoBelforte - 13-Jan-2011

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2014-07-30 - StefanoBelforte
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback