Facilities & Services, Documentation

Site Support Documents:

Central Services Documents:


  • Host and service certificates need to chain back to a root CA that is part of the International Grid Trust Federation, IGTF


Compute Elements:

  • CREAM-CE support ends December 31st, 2019. Alternatives are ARC or HTCondor CEs. The recommendation for existing CREAM-CE sites is to setup a new ARC or HTCondor CE, commission it with factory operations, and then decommission the old CREAM CE. Single VO sites may want to setup the new CE to schedule whole-nodes instead of eight-core pilots. (In this case please configure one extra short, i.e. 15 or 30 minute, slot for SAM tests.)
  • ARC CE installation and configuration instructions are here.
  • HTCondor CE documentation and OSG installation and configuration instructions.


  • CMS plans to prepare the 2018 data processing to run on SL/CentOS 7. Analysis of existing data will continue to use SL 6. Sites are asked to provide Singularity before commissioning starts in early 2018. Singularity will become mandatory for CMS sites in March 2018, i.e. SAM availability will drop to zero for sites without Singularity!
  • Singularity allows CMS to select the OS on a per job basis and decouples the OS of worker nodes from that required by experiments. Sites can setup worker nodes with a Singularity supported OS and CMS will choose the appropriate OS image for each job.
  • CMS recommendation is to setup Singularity on SL/CentOS 7 worker nodes, i.e. run the CMS pilot directly on the SL/CentOS 7 worker node and let it start Singularity (no need for a container/VM in between). The Singularity images of CMS will come from CVMFS. Please forsee sufficient CVMFS cache space (20 GB, more for systems with large core count).
  • Future versions of Singularity will provide an unpriviledged mode on SL/CentOS 7. The current version of Singularity has three setuid executables (detailed Singularity security information) and can be installed/is fully supported on both SL 6 and 7. Singularity improves/simplifies job isolation compared to the current glExec setup (which is then no longer needed/used by CMS). (If your site is interested/needs container type restrictions, please take a look at the OSG documentation.)
  • For a most minimal setup, only the singularity-runtime RPM could be deployed.
  • Posix storage needs to be at /cms and gfal2/xrdcp are the supported stage-out plugin methods. If your site does not currently use gfal2 or xrdcp, you need to switch the stage-out (and fallback) plugin!
  • Additional bind path can be specified via the environmental variable SINGULARITY_BINDPATH . (BUT the file/directoy path must exist in the image, i.e. it cannot be used to, for instance, add /mystorage.)
  • The grid environment inside Singularity is taken from OSG, so the OSG client software CVMFS needs to be available, i.e. /cvmfs/oasis.opensciencegrid.org/osg-software/osg-wn-client. (You may need to add the CVMFS.)
  • For sites that want to run CMS pilots inside a Docker container, the Docker container needs the kernel capabilities DAC_OVERRIDE, SETUID, SETGID, SYS_ADMIN, SYS_CHROOT, SYS_PTRACE. and the timeout of autofs needs to be set to 0 to prevent CVMFS from unmounting inside the containers. (The Docker container should be SL 6 or better SL/CentOS 7 and contain at a minimum singularity-runtime plus other RPMs to run the CMS pilot script.)
  • There are no known/outstanding Singurarity issues on CMS site. Please contact SI/factory operations and arrange with them to enable Singularity on your compute element(s), i.e. to set OS to "any" and glexec to "NONE".
  • details on Brian's Singularity How-To
  • OSG Singularity documentation, LBL Singularity documentation
  • In case your worker nodes run currently SL 6, you don't want to upgrade them right now, and want the minimal setup, please opt for the singularity-runtime installation (yum install of one RPM).
  • Pilots per site with singularity vs no singularity
  • glide-inWMS validation script SAM probe

IPv6 Information:

Edit | Attach | Watch | Print version | History: r55 < r54 < r53 < r52 < r51 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r55 - 2020-10-16 - StephanLammel
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback