CMSSW I/O in asynchronous mode

This page discusses CMSSW I/O under the following conditions:
  • Running PAT tuple creation.
  • With the CMS I/O patches found here.
  • cacheSize set to 20MB.
  • readHint mode set to "storage-only".
  • Reading out 40 events; after the first 20 events, the TTreeCache is activated.
We assume that you are familiar with the readHint=application-only page already.

Below are a few graphs showing the I/O activity, with the zoomed-in picture focusing on the area where TTreeCache fills its buffer (TTreeCache::!FillBuffer):

  • I/O activity on analyzed file over the whole run of the application:
    trace_storagecache2.png
    The picture here looks remarkably similar to the application-only mode. We'll need to zoom in on the FillBuffer area to understand the difference.
  • Focus on the time where ROOT was filling the TTreeCache buffers:
    trace_storagecache2_zoom.png
    Here, we can see the activity as CMSSW declares its future read pattern using posix_fadvise.
  • Notice there's a huge amount of disk-level parallelism here. There are up to ~400 requests in the block queue at a given time, and up to ~30 requests handed off to the device driver.
  • Notice the posix_fadvise call can block! This is due to the sheer number of requests we hand to the operating system overflowing its queue.
    • Because of this, we don't really overlap any I/O and CPU.
  • Overall, calling FillBuffer using asynchronous I/O is about 2 times faster than synchronous I/O.
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng trace_storagecache2.png r1 manage 63.5 K 2010-03-16 - 17:54 BrianBockelman  
PNGpng trace_storagecache2_zoom.png r1 manage 84.2 K 2010-03-16 - 17:54 BrianBockelman  
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2010-05-13 - BrianBockelman
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback