(r23) CpuBenchmark < HEPIX

HEPIX Web>CpuBenchmark (revision 23) (raw view)~~EditAttachPDF~~
%DASHBOARD{ section="dashboard_start" }%
%DASHBOARD{ section="banner"
 image="%PUBURL%/%SYSTEMWEB%/TWikiDashboardImages/computer-wires.jpg"
title="HEPIX CPU Benchmarking Working Group"
 titlestyle="color:white;"
}%


%DASHBOARD{ section="box_start" title="" width="999"}%
The working group was formed in 2007 and re-launched in 2016 with the following aims:
   * Working on the next generation HEPiX CPU benchmark (successor of HS06)
   * Development and proposal of a fast benchmark to evaluate the performance of a provided job slot (or VM instance)

If you would like to participate with this activity, please contact manfred.alef@kit.edu , domenico.giordano@cern.ch, michele.michelotto@pd.infn.it

*Mailing list*
   * hepix-cpu-benchmark@hepix.org
   * Archive available at http://listserv.in2p3.fr

*NB: page under construction*
%DASHBOARD{ section="box_end" }%

%DASHBOARD{ section="box_start" title="Meetings and Minutes"  width="470" }%


   * [[https://twiki.cern.ch/twiki/pub/HEPIX/CpuBenchmark/minutes-2016-04-21.pdf][2016-04-21 Informal meeting at HEPiX Spring 2016 (Zeuthen)]]
   * [[https://indico.cern.ch/event/535458/][2016-06-03 Kickoff meeting]]
   * [[https://indico.cern.ch/category/1806/][Bi-weekly meeting Indico category]]  
   * [[https://indico.cern.ch/event/578967/][pre-GDB Benchmarking]]
   
%DASHBOARD{ section="box_end" }%

%DASHBOARD{ section="box_start" title="Documentation"  width="470" height="300"}%
   * HS06   
      * [[https://w3.hepix.org/benchmarks/][HS06 WIKI]]
      * [[http://iopscience.iop.org/article/10.1088/1742-6596/219/5/052009?fromSearchPage=true][A comparison of HEP code with SPEC1 benchmarks on multi-core worker nodes]]
      * [[http://iopscience.iop.org/article/10.1088/1742-6596/219/5/052016/meta][The CMSSW benchmarking suite: using HEP code to measure CPU performance ]]

   * DB12
      * [[https://github.com/DIRACGrid/DB12][Code repository]]
      * [[http://linuxsoft.cern.ch/wlcg/sl6/x86_64/][RPM DB12 are available for SLC6 in the WLCG repository]]

   * KV
      * [[https://sdccloud.web.cern.ch/sdccloud/KV/IT-4143/kv-script.sh][Script used in several CERN commercial cloud activities]]

   * Benchmarking Suite 
      * [[https://bmkwg.web.cern.ch/bmkwg/][A toolkit to run and collect benchmark tests (DB12, KV, Whetstone)]]

%DASHBOARD{ section="box_end" }%
%DASHBOARD{ section="dashboard_end" }%

---+!!Table of Content
%TOC%

---+ Subjects for Studies

---++ HS06
   * icon:led-red Shall HS06 be still run in 32-bit or in 64-bit (-m32 Vs -m64)
      *   Discussion started in the mailing list. Motivations
          * New architectures can be only tested in 64-bit
          * The experiment applications are in 64-bit
          * Scattered studies have reported a ratio of 20% among -m32 and -m64. Is this ratio constant for all CPU models? 

   * icon:led-red HS06 variation with OS

   * icon:led-red HS06 correlation with Experiment workloads
      *  HS06 doesn't scale anymore (in new Intel CPU models) with simulation workloads. 
      * Lack of "magic boost" seen for experiment applications.
         * Recent reports from  [[https://indico.cern.ch/event/578967/contributions/2455906/attachments/1408419/2153502/20170207-mcnab-db12-lhcb.pdf][LHCb]] and [[https://docs.google.com/presentation/d/10nQTZo8C3mRu82Dtjp0WhJGNi8pNZVnIxzRDZtJt_fU/edit#slide=id.p][Alice]]
      * What's the situation for Reconstruction workloads? 
      * What's the situation for Atlas and CMS workloads? 


 
---++ DB12
   * icon:led-green DB12 boost in Haswell and Broadwell 
      * Investigated by M. Guerri. Reason found to be due to the better branch prediction 
      * [[https://indico.cern.ch/event/578967/contributions/2455916/attachments/1408471/2153228/PreGDB_MG.pdf][pre-GDB]]
      * [[https://mguerri.web.cern.ch/mguerri/Benchmarking/SandyBridgeVSHaswell.html][notebook]]

   * icon:led-green  DB12 variation with different OS and python versions 
      * Is DB12 affected by different python or OS versions, on the same CPU model?
      * [[DB12VsPythonVersion][Studies here]]

   * icon:led-green  DB12 performance with SMT ON/OFF 
      * Respect to HS06 DB12 doesn't seem to benefit from SMT enabled respect to the 20% seen in HS06
         * Studies on VMs benchmarking full HW node
            * [[%ATTACHURLPATH%/bmk-scaling-in-VM.png][Plot showing the Relative performance of DB12, WSN,KV and HS06 across different VM configurations. The performance of VM-83A is used as reference]]
            * [[https://indico.cern.ch/event/578967/contributions/2455910/attachments/1408475/2153234/pre-GDB_benchmarking_giordano.pdf][Full presentation]]
         * [[DB12VsPythonVersion][Tables here]]

   * icon:led-red  DB12 Vs multi-core jobs performance 
      * Is DB12 well correlated with the execution time of multi-core jobs, such as the ones running in ATLAS and CMS?

---++ KV 
   * icon:led-red KV double peak effect 
      * Investigated by M. Guerri. Is related to different performance of the first core of a dual-socket server respect to all the other cores.
      * [[https://indico.cern.ch/event/578967/contributions/2455916/attachments/1408471/2153228/PreGDB_MG.pdf][pre-GDB]]
      * [[http://cds.cern.ch/record/2257973?ln=en][CERN internal note: Profiling CPU-bound workloads on Intel Haswell-EP platforms]]

   * icon:led-green Reduce initialisation time for KV 
      * the athena applications runs in ~2 mins to process 100 single muon events, but the initialization time (sw-mgr application) can take up to 3 additional minutes. Can initialization be reduced?
         * A slim implementation of the KV benchmark is available in Docker container 
            * To run docker run -it --rm  gitlab-registry.cern.ch/giordano/hep-workloads:atlas-kv-bmk-v17.8.0.9
            * [[https://gitlab.cern.ch/giordano/hep-workloads/tree/master][gitlab repository]] 
            * Further details described in [[https://indico.cern.ch/event/624831/#2-hep-reference-workloads-in-c][this talk]]


   * icon:led-red KV License
      * License aspects need to be sorted out


#SiteAnchor
---+ Resources Available to Run Benchmarks

---++ GridKa

GridKa has reconfigured its compute farm to enable special benchmarking tasks:

   * An open issue is the correlation of static benchmark results (like HS06, or DB12-at-boot) with applications, depending on the number of configured job slots. Therefore there are several flavors of worker nodes, for instance:
      * Intel Xeon E5-2630v4 (Broadwell, 10-core, Hyperthreading enabled):
         * 20 job slots (1.0 slots per physical core)
         * 32 job slots (1.6 slots per physical core)
         * 40 job slots (2.0 slots per physical core)
      * Intel Xeon E5-2630v3 (Haswell, 8-core, Hyperthreading enabled):
         * 24 job slots (1.5 slots per physical core)
         * 32 job slots (2.0 slots per physical core)
      * Intel Xeon E5-2665 (Sandy Bridge, 8-core, Hyperthreading enabled):
         * 16 job slots (1.0 slots per physical core)
         * 24 job slots (1.5 slots per physical core)
   * The static benchmark scores are available to all batch jobs (submitted to either arc-1-kit.gridka.de, arc-2-kit.gridka.de, or arc-3-kit.gridka.de) using the machine job features (MJF):
      * $JOBFEATURES/hs06_job: HS06 score available to the job
      * $JOBFEATURES/db12_job: DB12 score available to the job
      * $JOBFEATURES/allocated_cpu: number of single-core job slots provided to the job
   * Manfred Alef at KIT can provide static benchmark scores afterwards; please send a CVS (or Excel or ODF spreadsheet) file which contains at least the worker node hostnames and the individual performance (events/s) of the jobs

---++ CERN


A number of resources can be made available for testing, based on bare metal servers or whole node VMs.
Access,  based on ssh public key, can be provided on demand. 

  * List of available resources (this list can change following the needs of Tier-0 resources)
| *Type* | *CPU model* | *OS* | *N cores* |  *N machines* |
|Bare-metal|  Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (Ivy Bridge)  | SLC6.8 |  32	|  2 |
|VM|  Intel Xeon E5-2630v3 (Haswell) | CC7 - x86_64  |  32	|  2 |  
|VM|  Intel Xeon E5-2630v3 (Haswell) | SLC6 - x86_64  |  32	|  2 |  
|VM|  Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (Broadwell) | SLC6 - x86_64  |  40	|  2 |  


---++ Other sites that would like to join
_TBD: please describe the kind of resources available, the configuration and how it's possible to access them_

#ExpAnchor
---+  Recipes to Run Experiment Workloads
Collect here the information about how to run experiment workloads. 
Possibly, provide instructions and setup (VM/containers , access from cvmfs) in order to allow execution by other members  of the working group.

   * ALICE
      * Contact person
      * Version of the experiment application (details about compiler flags)
      * Event Generation 
      * Simulation
      * Digitization
      * Reconstruction

   * ATLAS
      * Contact person
      * Version of the experiment application (details about compiler flags)
      * Event Generation 
      * Simulation
      * Digitization
      * Reconstruction

   * CMS   
      * Contact person
      * Version of the experiment application (details about compiler flags)
      * Event Generation 
      * Simulation
      * Digitization
      * Reconstruction

   * LHCb
      * Contact person
      * Version of the experiment application (details about compiler flags)
      * Event Generation 
      * Simulation
      * Digitization
      * Reconstruction
     
#PassiveBmkAnchor
---+  Passive Benchmark
   * A method to compare server performance using the experiment job information
   * Responsible: Andrea Sciaba (andrea.sciaba@cern.ch)
   * Description of the approach and results at [[https://indico.cern.ch/event/578967/contributions/2455901/attachments/1408462/2153742/CPU_benchmarking_with_production_jobs.pdf][pre-GDB]] and [[https://indico.cern.ch/event/614359/contributions/2510792/attachments/1425728/2187149/Passive_benchmarking_of_ATLAS_Tier-0_CPUs.pdf][WG meeting]]
   * Some results:
      * Speed factor k Vs HS06 correlation for ATLAS T0 jobs:  
      <a href="%ATTACHURLPATH%/Passive_benchmarking_of_ATLAS_Tier-0_CPUs.png"> <img src="%ATTACHURLPATH%/Passive_benchmarking_of_ATLAS_Tier-0_CPUs.png" alt="Passive_benchmarking_of_ATLAS_Tier-0_CPUs.png" width="200"  /></a>

   * Data required to run the passive benchmark
| *Quantity* | *CMS variable* | *ATLAS Grid jobs variable* | *ATLAS T0 variable* |
| CPU time | !CpuTimeHr | cpuconsumptiontime | cpuTime |
| Number of events in job | !KEvents | nevents | nevents |
| Job status | Status | jobstatus | n/a |
| Job type | !TaskType | processingtype | n/a |
| Site name | Site | computingsite | n/a |
| Task | !WMAgent_SubTaskName | jeditaskid | taskid |
| CPU model | n/a | cpuconsumptionunit | machine.model_name |

---+ Actions List

---+++ 2017-03-10
   * icon:led-green For the site representatives: to fill the information in this [[#SiteAnchor][section]]
   * icon:led-red For the experiment representatives: to fill the information in this [[#ExpAnchor][section]]
   * icon:led-green For Andrea Sciaba': to fill the information in this [[#PassiveBmkAnchor][section]]

---+++ 2017-04-19
   * icon:led-red WLCG workshop: [[WLCGWorkshopBenchmarking][preparation]] of topics for discussion 


-- Main.ManfredAlef - 2016-06-03
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	Passive_benchmarking_of_ATLAS_Tier-0_CPUs.png	r1	manage	357.1 K	2017-03-15 - 12:20	DomenicoGiordano	Speed factor k â€“ HS06 correlation for ATLAS T0 jobs
png	bmk-scaling-in-VM.png	r1	manage	215.9 K	2017-03-09 - 12:00	DomenicoGiordano
pdf	minutes-2016-04-21.pdf	r1	manage	51.2 K	2016-06-03 - 10:18	ManfredAlefExternal
Topic revision: r23 - 2017-06-12 - ManfredAlefExternal
HEPIX
Public webs
- Cern Search
- TWiki Search
- Google Search
HEPIX All webs
Copyright &© 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback