c.f. explanations by Danilo:

-------- Original Message --------
Date:    Sun, 2 Dec 2012 18:08:07 +0100
Subject:    Re: IgProf usage with LHCb Software
From:    Danilo Piparo <danilo.piparo@cern.ch>
To:    Ben Couturier <Ben.Couturier@cern.ch>

Hi Ben,

it seems that the problem is related to the slowness of AFS these days. IgProf is indeed sensible to the latency with which the libs are opened by dlopen. There is not much to do in this case: people are for example running the application for a couple of events to put the libs in the cache and then start igprof.
This explaination would perfectly fit with the fact that I did not encounter any problem running the instructions at 8:30 am and you did later in the morning or I did at around 18:00.

Sorry for not coming up with a real solution, but this is the state of the art.

Cheers and enjoy your evening,

2012/12/1 Danilo Piparo <danilo.piparo@cern.ch>

    Hi Ben,

    thanks for your trials.
    Intrigued by this behaviour, I tried again my instructions. Indeed, it seems we are in presence of an intermittent bug. Sometimes it works like a charm, some times (20% of the times), I get your exact same output lines.
    I contacted the author with the recipe to reproduce the issue: I'll keep you posted.

    Enjoy your weekend,


    2012/11/30 Ben Couturier <Ben.Couturier@cern.ch>

        HI Danilo,

        I tried again, but this time without the LHCb environment, and everything worked fine.
        Once I setup everything again with our scripts, I get the same problem!

        Let's see on Monday !
        Have a good week-end


        From: piparo.danilo@gmail.com [piparo.danilo@gmail.com] on behalf of Danilo Piparo [danilo.piparo@cern.ch]
        Sent: 30 November 2012 18:58
        To: Ben Couturier
        Subject: Re: IgProf usage with LHCb Software

        Hi Ben,

        thank you for trying this out.
        This is strange. I re-tried the recipe on a random lxplys node with slc5 as is and the procedure was successful.
        I re-tried the the same job option file multiple times and others as well without encountering the problem you have shown me.
        Maybe we can shortly meet on Monday to solve this in front of a screen?


        2012/11/30 Ben Couturier <Ben.Couturier@cern.ch>

            Hi Danilo !

            thanks very much !
            I tried the approach but in all cases, I don't seem to actually run the job but exit early with an output like the following:

            *** IgProf(19609, 1354275850.548): profiler activated in /usr/bin/env, main thread id 0x2b27da4e8460
            *** IgProf(19609, 1354275850.548): profiler options: igprof:out='|gzip -c>igprof.pp.gz' perf
            *** IgProf(19609, 1354275850.548): timing resolution is 0.005999 s
            *** IgProf(19609, 1354275850.549): exit (0x2b27d93e7430): instrumenting 5 bytes into 0x2b27dd6ec004
            *** IgProf(19609, 1354275850.549): _exit (0x2b27d944dee0): instrumenting 7 bytes into 0x2b27dd6ed004
            *** IgProf(19609, 1354275850.550): kill (0x2b27d93e4710): instrumenting 5 bytes into 0x2b27dd6ee004
            *** IgProf(19609, 1354275850.550): pthread_create (0x2b27d9b349e0): instrumenting 9 bytes into 0x2b27dd6ef004
            *** IgProf(19609, 1354275850.550): exit (0x2b27d93e7430): hook trampoline already installed, ignoring
            *** IgProf(19609, 1354275850.550): _exit (0x2b27d944dee0): hook trampoline already installed, ignoring
            *** IgProf(19609, 1354275850.551): kill (0x2b27d93e4710): hook trampoline already installed, ignoring
            *** IgProf(19609, 1354275850.551): dlsym(self, 'pthread_create'): /usr/bin/env: undefined symbol: pthread_create, version GLIBC_2.0
            *** IgProf(19609, 1354275850.551): dlsym(self, 'pthread_create'): /usr/bin/env: undefined symbol: pthread_create, version GLIBC_2.1
            *** IgProf(19609, 1354275850.551): performance profiler: measuring process cpu time
            *** IgProf(19609, 1354275850.551): fork (0x2b27d944dc20): instrumenting 6 bytes into 0x2b27dd6f0004
            *** IgProf(19609, 1354275850.552): system (0x2b27d93f0a50): instrumenting 5 bytes into 0x2b27dd6f1004
            *** IgProf(19609, 1354275850.552): pthread_sigmask (0x2b27d9b39c40): instrumenting 7 bytes into 0x2b27dd6f2004
            *** IgProf(19609, 1354275850.553): sigaction (0x2b27d93e44e0): instrumenting 7 bytes into 0x2b27dd6f3004
            *** IgProf(19609, 1354275850.553): performance profiler enabled

            Have I missed something ?


            From: piparo.danilo@gmail.com [piparo.danilo@gmail.com] on behalf of Danilo Piparo [danilo.piparo@cern.ch]
            Sent: 30 November 2012 10:06
            To: Ben Couturier
            Cc: Benedikt Hegner
            Subject: IgProf usage with LHCb Software

            Hi Ben,

            I cooked some instructions for you to get started with the profiling of the LHCb software*.
            Basically they allow you to arrive to a browsable web report from scratch. The final result should look like:

            The instructions are divided in steps:
              o Setup the lcg release of IgProf
              o Get and install Gaudi (not necessary in general but needed to create a complete example)
              o Produce a report and make it browsable in your web area

            As we briefly discussed, there is a little twist to be able to set up the environment, running gaudi and let IgProf recognise which process it is actually supposed to profile.
            Complete instructions about the tool itself can be found here: http://igprof.org/
            Do not hesitate to pop in my office if you want to discuss something.

            Have fun,

            export CMTCONFIG=x86_64-slc5-gcc46-opt
            export LCGEXTERNAL=/afs/cern.ch/sw/lcg/external/
            export LIBUNWINDDIR=$LCGEXTERNAL/libunwind/5c2cade/$CMTCONFIG/
            export IGPROF_DIR=$LCGEXTERNAL/igprof/5.9.2/$CMTCONFIG/
            export PATH=$PATH:$IGPROF_DIR/bin/

            # copy the navigator into your web area on afs and create the reports dir
            export CGIBINDIR=~/www/cgi-bin
            mkdir -p $CGIBINDIR/data
            cp $IGPROF_DIR/bin/igprof-navigator $CGIBINDIR

            # Get Gaudi
            # set up the minimal environment
            export CMTPROJECTPATH=/afs/cern.ch/sw/lcg/app/releases
            export PATH=/afs/cern.ch/sw/lcg/external/CMake/2.8.8/$CMTCONFIG/bin:$PATH
            export PATH=/afs/cern.ch/sw/lcg/external/Python/2.6.5p2/$CMTCONFIG/bin:$PATH
            source /afs/cern.ch/sw/lcg/external/gcc/4.6.2/x86_64-slc5/setup.sh

            # clone
            git clone http://cern.ch/gaudi/GaudiMC.git Gaudi
            cd Gaudi

            # Build!
            cd build."$CMTCONFIG"
            make -j20 install
            cd ..

            # Run Igprof from within a python interpreter where the environment is correctly set up
            # This way gaudi can run and IgProf can profile the correct process
            build.x86_64-slc5-gcc46-opt/run  python -c "import subprocess; subprocess.call('igprof -d -pp -z -o igprof.pp.gz InstallArea/x86_64-slc5-gcc46-opt/scripts/gaudirun.py GaudiExamples/options/Timing.py',shell=True)"

            # convert the report to an sql3 file
            igprof-analyse --sqlite -d -v -g igprof.pp.gz | sqlite3 igreport_perf.sql3

            # move it into the web area
            mv igreport_perf.sql3 $CGIBINDIR/data

            # browse it!
            firefox https://$USER.web.cern.ch/$USER/cgi-bin/igprof-navigator/igreport_perf

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2012-12-19 - BenjaminCouturier
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback