Asynchronous prefetching of input data on the Event Service worker nodes

Introduction

In this document we describe two ways of implementing an asynchronous prefetching of input data on the Event Service worker nodes: file-based prefetching in the pilot and the implementation based on the usage of Shared Reader process for AthenaMP.

Prefetching in the Pilot (using local input files)

Architecture

prefetching-v01.png

This picture shows a schematic view of asynchronous data prefetching by the pilot using a special prefetcher process.

  • At the beginning pilot starts an instance of AthenaMP (as in conventional Event Service configuration) and an instance of the prefetcher;
  • Once both AthenaMP and the prefetcher are ready, pilot starts receiving event ranges from PanDA/JEDI;
  • Instead of sending an event range directly to AthenaMP, pilot first sends it to the prefetcher;
  • The prefetcher retrieves input data corresponding to given event range from (possibly remote) input file and writes it into a local file;
    • New local file is created for each event range!
  • After that the prefetcher inserts absolute path (Physical File Name, PFN) to the local input file into the event range (adds new field to it) and sends the modified range back to the pilot;
    • The prefetcher finalizes a file and reports it back to the pilot when one of the following happens
      • Prefetcher receives the next event range from the pilot (at this point the prefetcher finalizes a file corresponding to the previous range)
      • Prefetcher receives "No more events" (at this point the prefetcher finalizes a file corresponding to the last event range it was processing)
  • Pilot delivers the modified event range to AthenaMP on demand;
  • When AthenaMP workers see the PFN field inside event range, then they use the local file for reading the data.

Note In this schema there are no changes for output handling wrt conventional Event Service configuration.

Prefetcher

Given the current implementation of the Event Service can run only G4 simulation, a first implementation of the prefetcher is basically an instance of EVNTMerge_tf.py running single-core Event Service configuration. The pilot and the prefetcher communicate to each other using Yampl channel (which is different from the Yampl communication channel between the Pilot and AthenaMP). Here is an example command for starting the prefetcher:

ATHENA_PROC_NUMBER=1 EVNTMerge_tf.py '--inputEVNTFile' 'path-to-the-remote-input' '--outputEVNT_MRGFile' 'prefix-of-the-local-file-names' '--eventService=True' '--preExec' 'from AthenaMP.AthenaMPFlags import jobproperties as jps;jps.AthenaMPFlags.EventRangeChannel="EventService_Prefetcher"'

Note This example uses "EventService_Prefetcher" for naming the communication channel between the pilot and the prefetcher. In real examples we can use any name as long as it is different from the name of the communication channel between the pilot and the payload instance (AthenaMP).

Code changes

AthenaMP. All code changes required to deal with PFN fields of event ranges inside AthenaMP workers have been developed and tested in various release branches. The code is available in 20.3.X.Y release branch as of release 20.3.7.5.

Pilot. In progress. Discussing details with Vakho. [Dec 1]

Testing

The testing so far has been done by running interactive jobs on a machine at CERN, using a toy implementation of the pilot (local input file, no connection to PanDA/JEDI).

For running "real life" examples with this mechanism we should use software releases in the 20.3.X.Y branch, as this is the release branch for running simulation productions in the coming months. The first 20.3.X.Y series release which can be used for testing is 20.3.7.5.

-- VakhoTsulaia - 2016-11-03

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng prefetching-v01.png r1 manage 657.0 K 2016-11-07 - 23:57 VakhoTsulaia  
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2017-03-23 - VakhoTsulaia
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback