Test of CMSSW_4_1_2

Tests performed on 2011/03/04.

Configuration

  • Job type:MTR3 job .
  • GT: START42_V3::All.
  • CRAB: 2_7_7, stand alone (no server)

Checked both with ("dpm w/a") and without ("vanilla") A.Sartirana's dpm workaround, using the script:

#!/bin/bash

LOG="cmssw"

eval `scram ru -sh`
export LD_PRELOAD=${GLOBUS_LOCATION}/lib/libglobus_gssapi_gsi_gcc64dbgpthr.so
cmsRun -j ${LOG}.xml pset.py

Summary of tests

Site SE Technology Vanilla DPM w/a
T2_BE_IIHE dCache smile -
T2_CH_CSCS dCache smile smile
T2_ES_CIEMAT dCache smile  
T2_FR_GRIF_LLR dpm smile / frown (*) smile
T2_IT_Bari Lustre+Storm smile smile
T2_DE_DESY dCache(+xRootd?) smile  
Overall   smile smile

(*) On LLR, the workaround is already deployed on polgrid1. Where the workaound is not available (e.g. grid36.lal.in2p3.fr or grid10.lal.in2p3.fr), the jobs fail for sure with error: send2dpm: DP002 - send error : client_establish_context: Could not find or use a credential

Summary:

  • The workaround seems to properly work and to create not issues on not-DPM sites
  • Strange results for LLR. polgrid1.in2p3.fr and llrcream.in2p3.fr work fine, while grid36.lal.in2p3.fr and grid10.lal.in2p3.fr segfault when the workaround is in place. The segfault is different from the previously seen one, but still it is related to Globus authentication. The mylib directory is loaded:

LD_LIBRARY_PATH=/var/spool/pbs/tmpdir/2439384.grid33.lal.in2p3.fr/CREAM752907455/CMSSW_4_1_2/lib/slc5_amd64_gcc434:/var/spool/pbs/tmpdir/2439384.grid33.lal.in2p3.fr/CREAM752907455/CMSSW_4_1_2/external/slc5_amd64_gcc434/lib:/swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434:/swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/external/slc5_amd64_gcc434/lib:/swareas/cms/slc5_amd64_gcc434/external/gcc/4.3.4-cms/lib64:/swareas/cms/slc5_amd64_gcc434/external/gcc/4.3.4-cms/lib:/swareas/cms/mylib64:/opt/c-ares/lib:/opt/classads/lib64:/opt/glite/lib64:/opt/glite/lib:/opt/lcg/lib64:/opt/globus/lib:/opt/c-ares/lib:/opt/classads/lib64:/opt/glite/lib64:/opt/glite/lib:/opt/lcg/lib64:/opt/globus/lib:/opt/d-cache//dcap/lib:/opt/d-cache//dcap/lib

A. Sartirana has been notified.

Show segfault error... Hide

Thread 3 (Thread 0x40c4a940 (LWP 32498)):
#0  0x0000003f1f00e838 in do_sigwait () from /lib64/libpthread.so.0
#1  0x0000003f1f00e8dd in sigwait () from /lib64/libpthread.so.0
#2  0x00002ad8b31b5039 in globus_l_callback_thread_signal_poll (user_arg=0x0)
    at globus_callback_threads.c:2841
#3  0x00002ad8b31ccd5a in thread_starter (temparg=0x1d7d7ac0)
    at globus_thread_pthreads.c:508
#4  0x0000003f1f00673d in start_thread () from /lib64/libpthread.so.0
#5  0x0000003f1e4d3f6d in clone () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x42124940 (LWP 32499)):
#0  0x0000003f1f00aee9 in pthread_cond_wait

GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00002ad8b31cd518 in globus_cond_wait (cv=0x2ad8b33e33e8,
    mut=0x2ad8b33e33c0) at globus_thread_pthreads.c:939
#2  0x00002ad8b31b464a in globus_l_callback_thread_poll (
    user_arg=0x2ad8b33e3380) at globus_callback_threads.c:2433
#3  0x00002ad8b31cdb1f in globus_l_thread_pool_thread_start (
    user_arg=0x1d7db280) at globus_thread_pool.c:217
#4  0x00002ad8b31ccd5a in thread_starter (temparg=0x1d7d7ae0)
    at globus_thread_pthreads.c:508
#5  0x0000003f1f00673d in start_thread () from /lib64/libpthread.so.0
#6  0x0000003f1e4d3f6d in clone () from /lib64/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x2ad8b3b74bf0 (LWP 32497)):
#0  0x0000003f1e499fff in waitpid () from /lib64/libc.so.6
#1  0x0000003f1e43c331 in do_system () from /lib64/libc.so.6
#2  0x0000003f1e43c687 in system () from /lib64/libc.so.6
#3  0x00002ad8b0dbb2c2 in TUnixSystem::StackTrace() ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/external/slc5_amd64_gcc434/lib/libCore.so
#4  0x00002ad8b0dbbd7c in TUnixSystem::DispatchSignals(ESignals) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/external/slc5_amd64_gcc434/lib/libCore.so
#5  <signal handler called>
#6  X509_get_issuer_name (a=0x0) at x509_cmp.c:120
#7  0x00002ad8b36f3a96 in X509_check_issued (issuer=0x1, subject=0x0)
    at v3_purp.c:614
#8  0x00002ad8b246c327 in globus_gsi_callback_check_issued (
    context=0x7fff1c120440, cert=0x0, issuer=0x1) at globus_gsi_callback.c:583
#9  0x00002ad8b36e725c in X509_verify_cert (ctx=0x7fff1c120440)
    at x509_vfy.c:305
#10 0x00002ad8b246bff9 in globus_gsi_callback_X509_verify_cert (
    context=0x7fff1c120440, arg=0x0) at globus_gsi_callback.c:378
#11 0x00002ad8b36225e6 in ssl_verify_cert_chain (s=0x1d803590,
    sk=<value optimized out>) at ssl_cert.c:525
#12 0x00002ad8b360cc8d in ssl3_get_server_certificate (s=0x1d803590)
    at s3_clnt.c:903
#13 0x00002ad8b360e108 in ssl3_connect (s=0x1d803590) at s3_clnt.c:271
#14 0x00002ad8b3629f45 in ssl_ctrl (b=0x1d7fe9f0, cmd=<value optimized out>,
    num=0, ptr=0x0) at bio_ssl.c:417
#15 0x00002ad8af1f2ebc in globus_i_gsi_gss_handshake (
    minor_status=0x7fff1c1207fc, context_handle=0x1d801fa0)
    at globus_i_gsi_gss_utils.c:843
#16 0x00002ad8af1ed8ba in gss_init_sec_context (minor_status=0x7fff1c1210d8,
    initiator_cred_handle=0x1d7f2740, context_handle_P=0x7fff1c122a58,
    target_name=0x1d801780, mech_type=0x0, req_flags=6, time_req=0,
    input_chan_bindings=0x0, input_token=0x7fff1c121100,
    actual_mech_type=0x0, output_token=0x7fff1c121110,
    ret_flags=0x7fff1c1232bc, time_rec=0x0) at init_sec_context.c:185
#17 0x00002ad8bc739e6e in Csec_client_establish_context_GSI_pthr (
    csec_funcptr=0x7fff1c1211b0, ctx=0x7fff1c122a40, s=134)
    at Csec_plugin_GSS.c:831
#18 0x00002ad8bc71b6fb in Csec_client_establish_context_caller (
    ctx=0x7fff1c122a40, t2=134) at ../h/Csec_plugin.h:137
#19 0x00002ad8bc718cbd in Csec_client_establishContext (ctx=0x7fff1c122a40,
    s=134) at Csec_api.c:425
#20 0x00002ad8bc6b4572 in send2dpnsx (socketp=0x0,
    host=0x7fff1c1237d0 "polgrid4.in2p3.fr",
    reqp=0x7fff1c123810 "\003\016\023\001", reql=189,
    user_repbuf=0x7fff1c124340 "\001", user_repbuf_len=57, repbuf2=0x0,
    nbstruct=0x0) at send2nsd.c:248
#21 0x00002ad8bc6b6198 in send2dpns (socketp=0x0,
    host=0x7fff1c1237d0 "polgrid4.in2p3.fr",
    reqp=0x7fff1c123810 "\003\016\023\001", reql=189,
    user_repbuf=0x7fff1c124340 "\001", user_repbuf_len=57) at send2nsd.c:695
#22 0x00002ad8bc6812e2 in dpns_statx (
    path=0x1d7d1c70 "/dpm/in2p3.fr/home/cms/trivcat/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_38Y_V13_JobRobot-v1/0010/24DBDEBD-DAE8-DF11-A7E3-0030487CAF0E.root", file_uniqueid=0x7fff1c124420, statbuf=0x7fff1c1244b0)
    at Cns_stat.c:185
#23 0x00002ad8bc6817b7 in dpns_stat (
    path=0x1d7d1c70 "/dpm/in2p3.fr/home/cms/trivcat/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_38Y_V13_JobRobot-v1/0010/24DBDEBD-DAE8-DF11-A7E3-0030487CAF0E.root", statbuf=0x7fff1c1244b0) at Cns_stat.c:216
#24 0x00002ad8bc6aed41 in rfio_HsmIf_open (
    path=0x1d7d1c70 "/dpm/in2p3.fr/home/cms/trivcat/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_38Y_V13_JobRobot-v1/0010/24DBDEBD-DAE8-DF11-A7E3-0030487CAF0E.root", flags=0, mode=438, mode64=1) at rfio_HsmIf.c:428
#25 0x00002ad8bc6a1e25 in rfio_open64_ext (
    filepath=0x1d7d1bc8 "rfio:///dpm/in2p3.fr/home/cms/trivcat/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_38Y_V13_JobRobot-v1/0010/24DBDEBD-DAE8-DF11-A7E3-0030487CAF0E.root", flags=0, mode=438, uid=0, gid=0, passwd=0,
    reqhost=0x7fff1c1266cf "") at open64.c:209
#26 0x00002ad8bc6a1c43 in rfio_open64_v2 (
    filepath=0x1d7d1bc8 "rfio:///dpm/in2p3.fr/home/cms/trivcat/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_38Y_V13_JobRobot-v1/0010/24DBDEBD-DAE8-DF11-A7E3-0030487CAF0E.root", flags=0, mode=438) at open64.c:132
#27 0x00002ad8bc6a1bf8 in rfio_open64 (
    filepath=0x1d7d1bc8 "rfio:///dpm/in2p3.fr/home/cms/trivcat/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_38Y_V13_JobRobot-v1/0010/24DBDEBD-DAE8-DF11-A7E3-0030487CAF0E.root", flags=0, mode=438) at open64.c:121
#28 0x00002ad8bc6571a2 in RFIOFile::open(char const*, int, int) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/libUtilitiesRFIOAdaptor.so
#29 0x00002ad8bc6577de in RFIOFile::RFIOFile(std::string const&, int, int) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/libUtilitiesRFIOAdaptor.so
#30 0x00002ad8bc64e9f5 in RFIOStorageMaker::open(std::string const&, std::string const&, int) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/pluginUtilitiesRFIOAdaptorPlugin.so
#31 0x00002ad8b43b127e in StorageFactory::open(std::string const&, int) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/libUtilitiesStorageFactory.so
#32 0x00002ad8b4351a94 in TStorageFactoryFile::TStorageFactoryFile(char const*, char const*, char const*, int) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/pluginIOPoolTFileAdaptor.so
#33 0x00002ad8b43580f9 in G__TFileAdaptorLinkDef_215_0_13(G__value*, char const*, G__param*, int) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/pluginIOPoolTFileAdaptor.so
#34 0x00002ad8b1327bbc in Cint::G__CallFunc::Execute(void*) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/external/slc5_amd64_gcc434/lib/libCint.so
#35 0x00002ad8b0d78841 in TCint::CallFunc_ExecInt(void*, void*) const ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/external/slc5_amd64_gcc434/lib/libCore.so
#36 0x00002ad8b0da3294 in TMethodCall::Execute(void*, long&) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/external/slc5_amd64_gcc434/lib/libCore.so
#37 0x00002ad8b0d0e279 in TPluginHandler::ExecPlugin(int, ...) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/external/slc5_amd64_gcc434/lib/libCore.so
#38 0x00002ad8b0a2a1da in TFile::Open(char const*, char const*, char const*, int, int) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/external/slc5_amd64_gcc434/lib/libRIO.so
#39 0x00002ad8bc6004e1 in edm::RootInputFileSequence::initFile(bool) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#40 0x00002ad8bc604b7f in edm::RootInputFileSequence::RootInputFileSequence(edm::ParameterSet const&, edm::PoolSource const&, edm::InputFileCatalog const&, edm::PrincipalCache&, bool) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#41 0x00002ad8bc5d3212 in edm::PoolSource::PoolSource(edm::ParameterSet const&, edm::InputSourceDescription const&) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#42 0x00002ad8bc5d0bb4 in edmplugin::PluginFactory<edm::InputSource* ()(edm::ParameterSet const&, edm::InputSourceDescription const&)>::PMaker<edm::PoolSource>::create(edm::ParameterSet const&, edm::InputSourceDescription const&) const
    ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#43 0x00002ad8af543695 in edm::InputSourceFactory::makeInputSource(edm::ParameterSet const&, edm::InputSourceDescription const&) const ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#44 0x00002ad8af4e8652 in edm::makeInput(edm::ParameterSet&, edm::CommonParams const&, edm::ProductRegistry&, edm::PrincipalCache&, boost::shared_ptr<edm::ActivityRegistry>, boost::shared_ptr<edm::ProcessConfiguration>) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#45 0x00002ad8af4ea6da in edm::EventProcessor::init(boost::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#46 0x00002ad8af4f10af in edm::EventProcessor::EventProcessor(boost::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) ()
   from /swareas/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_2/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#47 0x000000000040e24b in main ()

Update on LAL issue: after a chat with A.Sartirana, mylib64 was removed from LD_LIBRARY_PATH (those are old libs non needed anymore), and everything seems fine.

export LD_LIBRARY_PATH=`echo $LD_LIBRARY_PATH  | sed "s=/swareas/cms/mylib64:==g" `

-- LeonardoSala - 07-Mar-2011

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r3 - 2011-03-07 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback