WLCG Multicore Deployment Task Force

Mandate and goals

In the scope of WLCG Operations Coordination, the Multicore Deployment Task Force pursued these objectives:

• collect information from the experiments on their model for requesting and using multicore resources accessible at WLCG sites via traditional computing elements
• agree on a simple but efficient (set of) multicore resource provisioning model(s) that sites can adopt with reasonable effort. Pilot the model(s) at some selected sites and evaluate them based on metrics taking into account experiment workflow efficiencies and impact on sites.
• define a deployment strategy for the migration of an increasing amount of resources to multicore queues according to the needs and priorities of the experiments and the sites
• drive the deployment process

The task force in addition investigated accounting issues on multicore resources in collaboration with the Machine/Job features task force.

Although the task force is not asked to provide a solution for cloud resources, the proposed solution for Grid resources have the potential to be extended to cloud resources.

In addition, this TF should ensure proper coordination with the work done in WLCG Cloud Working Group and WLCG Operations Coordination Machine/Job Feature TF on aspects of its work that may overlap.

The coordination of the task force will be shared by Alessandra Forti and Antonio Pérez-Calero Yzquierdo.

Contact

All members of the task force can be contacted at wlcg-ops-coord-tf-mc@cernSPAMNOTNOSPAMPLEASE.ch

wlcg-ops-coord-tf-mc egroup page and membership

Task Tracking and Timeline

The task force coordination has proposed to adopt October 2014 as the aim date for a working system to be fully deployed. This should comply with the need to make sure that everything is ready for the LHC re-start on 2015. The involved experiments, ATLAS and CMS so far, agree on this schedule. The table below presents a more detailed plan of tasks

Documentation

ATLAS

• Multicore job submission and grid resource scheduling for ATLAS AthenaMP: CHEP '12 presentation, Paper
• Atlas multi core twiki page

CMS

• CMS preliminary model and tests for a multicore scheduling strategy: contribution to CHEP2013.

CE related information

• CREAM example from NIKHEF
• If using a different batch queue, remember to add the queue info to the CE information system, e.g. if using YAIM update QUEUES and QUEUENAME_GROUP_ENABLE
• Concerning accounting:
  • Publishing multicore accounting to APEL works. ARC CEs publish correctly. For CREAM CEs to make it work it has to be an EMI-3 CE and it has to be enabled in the configuration.
  • Edit /etc/apel/parser.cfg and set the attribute parallel=true.
  • If the site was running multicore already, before upgrading and/or applying this modification, they need to reparse and republish the corrected accounts.
  • See slides from APEL team on December 10th GDB

Batch system related information

• HTCondor
• Torque/Maui
• SGE

• Passing parameters to the batch systems

Deployment table

Recurring Questions

As exposed in Thomas Hartmann presentation

• If scheduler going to utilize wall time prediction needs HEP-SPEC06 secs how exact have HS06 to be for a VO/for a site?
• HS06 scores are designed to scale with the average performance of typical HEP job mix. Be aware that there is absolutely no warranty that it scales with every individual job!
• How are inefficiencies being accounted? no official requirements on WLCG sites
• Is it reasonable to account a VO when a job’s run time deviates by from the prediction and spoiling the scheduler?
• Currently no wall time prediction provided per WLCG job no duty of VOs to supply wall time predictions precise enough to avoid gaps/optimize scheduling
• Is it accountable to the VO submitting a mcore job?
• Is it solemnly a site issue?
• How large is the effect in the end?
• How many ramp-up periods for how long with steady stream of mcores negligible after x?
• System states with efficient utilization of bare metal?
  • High entropy: many(?) short(m, h, ?) jobs filling free slots what max. wall time prediction variance necessary for good scheduling? (necessary at all?)
  • Low entropy: long(h, d, ?) with accurate wall time estimation what max. wall time prediction variance necessary for good scheduling?
• Sites with mixed VO users
  • Stable mixed state possible? or implicit/explicit segmentation inevitable?
  • How efficient could an ideal scheduler be under which variances in which state? i.e. how large would the inefficiency in node utilization become under which variance? if the wall time prediction is binned, i.e., in finite numbers of attainable run times/queues, how does the efficiency evolve with the number of prediction time slots

Task Force and related meetings:

• Tue. 04/12/2016, slides on multicore accounting for the April pre-GDB
• Thu. 18/02/2016, Slides to the WLCG Ops. Coord. meeting
• Wed. 12/09/2015, Accounting update. Slides
• Tue. 07/14/2015, Accounting for multicore. Slides
• Tue. 04/14/2015, Contribution from the Multicore TF to CHEP2015. Slides
• Tue. 04/14/2015, Contribution to CHEP2015 on the multicore-float algorithm. Slides
• Tue. 04/14/2015, Contribution to CHEP2015 on provisioning for multicore applications with LSF. Poster
• Fri. 03/27/2015, Contribution to HEPIX Spring 2015 on drain-boss algorithm. Slides
• Tue. 03/17/2015, Accounting for multicore. Slides
• Tue. 11/18/2014, Report on Multicore usage accounting for WLCG MB Slides
• Wed. 11/12/2014, Report from the TF to the November GDB. Slides
• Tue. 09/16/2014, Review of LSF setup at CNAF. Indico agenda, Minutes
• Tue. 09/09/2014, Review of UGE setup at CCIN2P3, updates from KIT and RAL, standardization of blah scripts. Indico agenda, Minutes
• Mon. 07/07/2014, Contribution to the 2014 WLCG Collaboration Workshop. Slides
• Tue. 07/01/2014, Review of CMS activity from KIT, PIC and RAL. Indico agenda, Minutes
• Tue. 05/20/2014, Contribution with the Task Force activities to the EGI Community Forum 2014.Indico entry for EGI 2014
• Tue. 04/15/2014, Discussion on SLURM setup at CSCS. Indico agenda, Minutes
• Tue. 04/08/2014, First Phase wrap-up session. Indico agenda, Minutes
• Tue. 04/01/2014, Discussion on Torque/Maui setup at NIKHEF Indico agenda, Minutes
• Tue. 03/11/2014, Bologna pre-GDG meeting dedicated to batch systems. Agenda, Minutes
• Tue. 03/04/2014, Discussion on SGE setup at KIT Indico agenda, Minutes
• Tue. 02/25/2014, Discussion on HTCondor setup at RAL Indico agenda, Minutes
• Tue. 02/18/2014, Reports from PIC and KIT Indico agenda, Minutes
• Tue. 02/11/2014, Contribution to the WLCG Operations Coordination F2F Meeting. Agenda
• Tue. 02/04/2014, Initial reports from CMS, PIC and KIT Indico agenda, Minutes
• Tue. 01/28/2014, Initial reports from ATLAS and NIKHEF Indico agenda, Minutes
• Tue. 01/21/2014, First meeting TF presentation. Indico agenda, Minutes

• WLCG Operations Coordination Reports

Additional sources of related material

• Multicore TEG twiki
• MachineJobFeatures

-- SimoneCampana - 27 Nov 2013