Implementation of OffloadingSvc in Muon Trigger Algorithms (GPU Demonstrator)

Setup of Athena

Current tests has been performed using Athena 20.0.0, minor bug-fixing is ongoing to adapt that to the dev version (the one used for the GPU Demonstrator plans).

Simple command to setup the environment (in the main project directory) which can in case collected in a script:

source /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/user/atlasLocalSetup.sh (setupATLAS #if you setup that)
asetup AtlasHLT,20.0.0,gcc48,here
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/afs/cern.ch/atlas/offline/external/APE/1.0/x86_64-slc6-gcc48-opt/lib

then you can proceed to setup a workarea and compile what is included. Of course you can first add the packages you need, or do it later and repeat the configuration and compilation

setupWorkArea.py
cd WorkArea/cmt
cmt br cmt config
source setup.(c)sh
cmt br cmt make

SVN repositories for the related packages

The code for Muon Offloading Tools packages is available here.

from your main working directory you can download the latest tag of the package with the following command:

cmt co -r TrigMuonAccelTools-00-00-01 Trigger/TrigTools/TrigAccelTools/TrigMuonAccelTools

or with the analogous svn command

svn co svn+ssh://svn.cern.ch/reps/atlasoff/Trigger/TrigTools/TrigAccelTools/TrigMuonAccelTools/tags/TrigMuonAccelTools-00-00-01 Trigger/TrigTools/TrigAccelTools/TrigMuonAccelTools

this package needs the OffloadingSvc package available in Athena (see TrigMuonAccelTools /cmt/requirements for all the details) which can be downloaded typing:

cmt co -r Offloading/OffloadSvc

and can be seen here.

Quick recipe for setting up APE in stand alone fashion

After setting up Athena as above, check out the following packages:

cmt co Offloading/APE
cmt co Offloading/APEModules
and edit the file
Offloading/APE/CMakeLists.txt
as follows: search for the line
if( NOT  "$ENV{ATLAS_EXTERNAL}" STREQUAL "")
(line 75) and change it to
if( NOT  "$ENV{ATLAS_EXTERNAL}" STREQUAL "" AND NOT CMAKE_INSTALL_PREFIX)

Then within your Athena TestArea:

mkdir BuildArea
cd BuildArea
ccmake ../Offloading/APE

then abide to the following flag/value settings (the values are related to a typical Athena setup with 20.0.0 or 20.1.0) which are meant to be set by steps:

  • 1st step (then configure)
CMAKE_CXX_COMPILER = $(which g++)
CMAKE_C_COMPILER = $(which gcc)
  • 2nd step (then configure)
APE_MODULES_DIR = $TestArea/Offloading/APEModules
CMAKE_INSTALL_PREFIX = $TestArea/InstallArea/x86_64-slc6-gcc48-opt
  • 3rd step (then configure)
TBB_INCLUDE_DIR = /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc48-opt/20.1.0/AtlasCore/20.1.0/InstallArea/x86_64-slc6-gcc48-opt/include
TBB_INCLUDE_DIRS = /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc48-opt/20.1.0/AtlasCore/20.1.0/InstallArea/x86_64-slc6-gcc48-opt/include
TBB_INSTALL_DIR = /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc48-opt/20.1.0/AtlasCore/20.1.0/InstallArea/x86_64-slc6-gcc48-opt
TBB_LIBRARY = /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc48-opt/20.1.0/AtlasCore/20.1.0/InstallArea/x86_64-slc6-gcc48-opt/lib/libtbb.so
TBB_LIBRARY_DIRS = /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc48-opt/20.1.0/AtlasCore/20.1.0/InstallArea/x86_64-slc6-gcc48-opt/lib
TBB_MALLOC_LIBRARY = /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc48-opt/20.1.0/AtlasCore/20.1.0/InstallArea/x86_64-slc6-gcc48-opt/lib/libtbbmalloc.so
  • 4th step (then configure)
BUILD_EXAMPLES = ON
BUILD_MODULES = ON
BUILD_ECHO_MODULE = ON
BUILD_CUDA_EXAMPLE = ON
CUDA_TOOLKIT_INCLUDE = /usr/local/cuda/include
CUDA_TOOLKIT_ROOT_DIR = /usr/local/cuda

Notice that the settings for yampl include and libraries should be already correctly set. The above is a standard setting for a node with a GPU on it. In case no GPU is available set OFF all the flags containing CUDA. For finer details (like debugging or other features) you may want to set as well:

BUILD_PARALLELCUDA = ON
CUDA_NVCC_FLAGS_DEBUG = -g -G

After all the configuring step, generate the makefiles, exit ccmake then compile and install:

make && make install

Then you're done.

Setup of the APE server

To properly setup an APE server you can follow the instructions reported here.

This can be done also in a separate directory.

If you download the tarball (SERVER.tar.gz), extract it, read the README.txt file to compile it.

The dedicated Muon Trigger Modules and Works are available here; the corresponding EDM Data Structures are available here, instead.

You can download them (from the main APE Server folder by doing

cmt co Offloading/Modules/TrigMuonModules/
cmt co Offloading/DataStructures/TrigMuonDataStructs/

and compile them

cd Offloading/Modules/TrigMuonModules/
make

Alternatively to TrigMuonModules checking out, you can check out a preliminary module with working kernels for GPU execution used for the Lisbon Sprint Workshop by typing from your TestArea:

mkdir Offloading/Modules
cd Offloading/Modules
svn co svn+ssh://svn.cern.ch/reps/atlasoff/Offloading/Modules/TrigMuonModules/branches/TrigMuonModules-00-00-01-branch
mv TrigMuonModules-00-00-01-branch TrigMuonModules

and then compile.

If everything goes fine you will have a generated library called

libmuonTriggerModule.so

, which has to be given as input module when running the APE Server.

This procedure (download and compilation) has been succesfully tested on gap01.roma1.infn.it.

Modifications to the MuonHoughPatternTools algorithm to offload HT calculation to APE

First of all you should download locally a working version ot the algorithm considered. I'm currently working on the MuonHoughPatternTools but a similar approach can be applied to different ones. Latest tag considered is MuonHoughPatternTools -01-00-16, hence:

cmt co -r MuonHoughPatternTools-01-00-16 MuonSpectrometer/MuonReconstruction/MuonPatternFinders/MuonPatternFinderTools/MuonHoughPatternTools

The files that you need to edit are in the cmt/, MuonHoughPatternTools / and src/ folders.

Add to MuonHoughPatternTools /cmt/requirements the line

use TrigMuonAccelTools          TrigMuonAccelTools-* Trigger/TrigTools/TrigAccelTools

Add at the top of MuonHoughPatternTools/MuonHoughPatternTool.h the line

class ITrigMuonAccelerationSvc;

and in the class definition (private part) add a useful flag and the handle to the service

bool m_useAccelerator;
ITrigMuonAccelerationSvc* m_accelerationService;

In the file src/TrigMuSuperEF.cxx include at the top

#include "TrigMuonAccelTools/ITrigMuonAccelerationSvc.h"

and as the default configuration in the class instantiation

m_useAccelerator(true)

and among the property declaration the useful

declareProperty("UseAccelerator", m_useAccelerator);

Inside the function MuonHoughPatternTool::initialize() add

if(m_useAccelerator) {
 StatusCode sc = service("TrigMuonAccelerationSvc", m_accelerationService);
 if (sc.isFailure() || m_accelerationService == 0) {
    m_accelerationService = 0;
    msg() << MSG::WARNING << "Could not retrieve TrigMuonAccelerationSvc Service" << endreq;
    m_useAccelerator=false;
 } else {
    msg() << MSG::INFO<< "TrigMuonAccelerationSvc retrieved" << endreq;
 }
}

if(!m_useAccelerator) {
msg() << MSG::INFO<< "Running without accelerator support" << endreq;
}

which will try to retrieve the service and initialize it if needed. This is the first interaction between Athena and APE during the run.

After that should be sent any configuration for the server algorithm execution, if needed. A prototype is present and can be used like the following:

if(m_useAccelerator) {
 //send configurations here
 std::cout << "MBDEVEL: Using Accelerator in initialize (MuonHoughPatternTool)" << std::endl;
 std::cout << "MBDEVEL: Sending tool configuration parameters" << std::endl;
 StatusCode scConfig = m_accelerationService->sendToolConfiguration(m_number_of_ids,
                                                                    m_number_of_maxima,
                                                                    m_use_rpc_measures_phi,
                                                                    m_use_rpc_measures_eta,
                                                                    m_use_ip,
                                                                    m_thresholdpattern_xyz,
                                                                    m_thresholdpattern_rz,
                                                                    m_maximum_residu_mm,
                                                                    m_maximum_residu_mm_cosmics,
                                                                    m_maximum_residu_angle,
                                                                    m_maximum_level,
                                                                    m_use_hough_correction,
                                                                    m_z_cor_constant,
                                                                    m_z_cor_constant2,
                                                                    m_theta_cor_constant,
                                                                    m_theta_cor_constant2,
                                                                    m_detectorsize_xy_full,
                                                                    m_detectorsize_yz_full,
                                                                    m_detectorsize_rz_full,
                                                                    m_detectorsize_xy_ip,
                                                                    m_detectorsize_yz_ip,
                                                                    m_detectorsize_rz_ip,
                                                                    m_detectorsize_angle_xyz,
                                                                    m_detectorsize_angle_rz,
                                                                    m_detectorsize_inv_sqrt_curvature,
                                                                    m_stepsize_xy,
                                                                    m_stepsize_yz,
                                                                    m_stepsize_rz,
                                                                    m_stepsize_xy_cosmics,
                                                                    m_stepsize_rz_cosmics,
                                                                    m_stepsize_per_angle_xyz,
                                                                    m_stepsize_per_angle_rz,
                                                                    m_stepsize_per_angle_xy_cosmics,
                                                                    m_stepsize_per_angle_rz_cosmics,
                                                                    m_stepsize_per_inv_sqrt_curvature,
                                                                    m_nbins_curved,
                                                                    m_weightmdt,
                                                                    m_thresholdhisto_xyz,
                                                                    m_thresholdhisto_rz,
                                                                    m_number_of_sectors_xyz,
                                                                    m_number_of_sectors_rz,
                                                                    m_number_of_sectors_rz_cosmics,
                                                                    m_printlevel,
                                                                    m_ncalls);

 if(scConfig.isRecoverable()) {
    //ask for errors here ...
    std::cout << "Recoverable errors encountered usign APE" << std::endl;
 }
 else {
    if(scConfig.isFailure()) std::cout << "Failed Usign APE" << std::endl;
 }
}

The rest of modifications can be placed in the proper place you want to use the offload to send data to an external device. For this algorithm I added the following at the beginning of the MuonHoughPatternTool::makePatterns() function

if(m_useAccelerator) {
 std::cout << "MBDEVEL: Using Accelerator in makePattens (MuonHoughPatternTool)" << std::endl;
 std::cout << "muonHoughHitContainer size: " << sizeof(MuonHoughHitContainer) << std::endl;
 std::vector<IdentifierHash> listOfHits;
 listOfHits.clear();
 StatusCode scAPE = m_accelerationService->processMSHits(m_event, listOfHits);
 if(scAPE.isRecoverable()) {
    //ask for errors here...
    std::cout << "Recoverable errors encountered usign APE" << std::endl;
 }
 else {
    if(scAPE.isFailure()) std::cout << "Failed Usign APE" << std::endl;
 }
}//end m_useAccelerator

Take a look at the TrigMuonAccelTools to see what is done with the data sent from Athena using processMSHits().

Basic Tests

If you setup both client and server side you should be able to run such machinery to perform come communication tests (I tested succesfully). Remeber to lauch Athena (client) and APE (server) on the same physical machine; at the moment that's the only communication way implemented.

Once you have everything in place and compiled follow the following steps:

1) launch the APE server from the ServerApp/ folder

./apeMod -m ../Offloading/Modules/TrigMuonModules/libmySimpleModule.so

assuming that libmySimpleModule.so contains also the muonSimpleWork.cxx library.

2) launch Athena from WorkArea /run/

athena.py -c 'enableCostMonitoring=TRUE;RunningRTT=TRUE;testPhysicsV5=True;' -b ./mytestCommonSliceAthenaTrigRDO.py

where the jobOption file ( mytestCommonSliceAthenaTrigRDO.py.txt, rename removing the .txt at the end) can be the one attached to this webpage or a similar one.

If the test is working properly you should see some messages both in Athena and in the APE log.

NB: You should carefully check that the channel created by the server is the same that client is looking for. This is slightly different due to a bug in some versions: the two channels should appear in the /tmp/ folder as /tmp/apeSock_xxxxx_announce and /tmp/apeSock_xxxxx_upload_announce, where xxxxx = uid, the user id (this is the default from OffloadingSvc you use in the client side). You can modify this name as you like, allthough this convention is a good one. By default if you get SERVER.tar.gz the APE server application, you'll see that this instead is missing the xxxxx part and will create a different communication channel which the client will not found (mentioned in the log) and the run will proceed without using the accelerators.

You can fix this by replacing the two lines in ServerApp/src/AccelerationServer.cxx

yampl::Channel channelS(m_channelName.c_str(),yampl::LOCAL_SHM);
yampl::Channel channelU((m_channelName+"_upload").c_str(),yampl::LOCAL_SHM);

with

char buff[1000];
uid_t userUID=geteuid();
snprintf(buff,1000,"%s_%u",m_channelName.c_str(),userUID);
yampl::Channel channelS(buff,yampl::LOCAL_SHM);
snprintf(buff,1000,"%s_%u_upload",m_channelName.c_str(),userUID);
yampl::Channel channelU(buff,yampl::LOCAL_SHM);

and recompile

rm apeMod; make;

TODO List

  • Algo settings: sparse list of parameters copied into a structure (HTAlgoConfiguration), looking for an easier way to code the copy
  • Need to create output result structures and corresponding DataExportTool functions to convert back into Athena
  • Mdt hits serialization not optimized for algorithm, can reduce the structure of data sent to GPU
  • Finalize standalone CPU version of HT, currently under development
  • Need to implement all the functionalities of the Athena one (cosmic, different magnetic field configuration, ... )
  • Compilation using ccmake under investigation, will wait Tomasz fixes
  • Diagnostic: create a monitorWork for algorithm/result validation (e.g. voting matrix checking)
  • Timing measurement infrastructure: setup that in Athena for easy latency measurements
  • Configuration/JO for FullScan run
  • Creation of Athena Tool including Offloading (MuonHoughPatternTool duplicate)

-- MatteoBauce - 2015-05-11

Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r13 - 2015-05-22 - SalvatoreAlessandroTupputi
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback