WLCG Operations and Tools TEG - WG3 Software Mangement, Software Configuration and Deployment Management

Application Software Management and Configuration Management

The software stack for LHC experiments is layered into several levels. The bottom being the operating system (e.g. Scientific Linux), on top of it being a set of common libraries which are used together by the LHC experiments (e.g. ROOT, COOL, CORAL, Boost, mysql, Python, etc) and the top most layer being the experiment specific applications for data reconstruction, analysis and simulation. Essential needs for the experiments are

  • possibility to have a fast turnaround of package versions and their grid wide deployment
  • package version decoupling from operating system to allow later versions to be used than natively provided
  • provide the same package version on different OS, e.g. on other development platforms (Mac)
  • possibility to fix the package version, essential for reproducibility

The widest used operating system being "Scientific Linux (SL)" is provided by teams at Cern and Fermilab. The common software layer for LHC experiments is being steered by PH/SFT through the "LCG Applications Area" where the set of common libraries and their versions is being discussed and defined. The topmost layer, the experiment specific applications, are being developed individually within experiments. The software management and configuration of these layers for experiments is being driven by several tools, some being used in common by multiple experiments.

... missing paragraph on grid middleware ...

CMT

Cmt (http://www.cmtsite.org) is the main tool currently in use by Atlas, LHCb and PH/SFT. Cmt implements its own language through which it allows package versioning, package building and the setup of the corresponding runtime environment.

Usually 2 to 3 times a year the PH/SFT group is providing a so called "LCGCMT configuration" which denotes a baseline of ~ 120 common packages provided to experiments. The packages within such a configuration include the "self developed" projects (ROOT, COOL, CORAL), "external packages" - recompiled from source (e.g. gcc, Boost, Python, mysql, frontier) and grid middleware clients provided so far by gLite. These configurations are augmented by fix releases if needed (usually 5-6). Experiments using CMT are taking these LCGCMT configurations to build their specific applications on top. To allow an even faster turnaround of packages, LHCb has invented an experiment specific "LHCbGrid" configuration, which can overwrite the package versions and instructions provided by LCGCMT for grid middleware packages.

What works well

  • LCGCMT configurations provide a common baseline for LHC experiments with immutable package versions.
  • Decoupling of packages from operating system, allowing usually a faster turnaround of new package versions and also usage of versions later than provided by the underlying OS
  • Fixing package versions up to the patch level, not relying on OS package deployment which could be different for sites. Facilitates also reproducing of problems.
  • "External packages" can be patched if needed, not relying on OS vendor.

Problems / Concerns

  • At the moment only a limited set of operating systems is available and certified. With more manpower more OS versions could be provided.
  • What is the future of the grid middleware packages provided to PH/SFT after the move to EMI?

SCRAM

What works well

Problems / Concerns

Gnu Make

What works well

Problems / Concerns

CMake

currently under investigation by PH/SFT + experiments

Deployment Management

CVMFS

What works well

Problems / Concerns

Tarballs

What works well

Problems / Concerns

Packman

What works well

Problems / Concerns

RPM / apt

What works well

Problems / Concerns

Torrent

What works well

Problems / Concerns

Additional Info

Contributors

Jakob Blomer, Marco Clemencic, Andreas Pfeiffer, Marco Cattaneo,

-- StefanRoiser - 09-Nov-2011

Edit | Attach | Watch | Print version | History: r18 | r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r3 - 2011-11-11 - StefanRoiser
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback