5.7 Data Analysis with CMS Connect


Introduction

CMS Connect is a service designed to provide a Tier3-like environment for condor analysis jobs and enables users to submit to all resources available in the CMS Global Pool. It is a complementary service to CRAB.

  • Use CRAB for e.g large scale datasets processing via cmsRun
  • Use CMS Connect for user-defined scripts via condor for late-analysis non-cmsRun jobs like e.g: making histograms, plots, analyzing trees, etc.
For users, interacting with CMS Connect is similar to interacting with a private cluster or e.g the LPC CAF via condor, the main difference is that jobs do not run on a local pool but are sent to Tier Resources available in the CMS Global Pool, hence it does not provide a shared-filesystem infrastructure or include your CERN/FNAL AFS Home area for example.

Service Website:

CMS Connect website: https://connect.uscms.org/

Tutorials

CMS Connect Tutorial - 17 June 2016 at Fermilab, U.S.A.

Running gridpacks with CMS Connect

Documentation

Prerequisites

Just like CRAB, CMS Connect submits to the Grid (LCG), so you need to apply for certificates and sign it to the CMS VOMS server. You can follow Section 5.1 to get a CERN CA and register to the CMS VO.

QuickStart

This is a quick start page which should take only a few minutes to complete. It will show how to:

  • Sign-up to the service
  • Check and setup your proxy certificates
  • Submit your jobs

Sign-up to CMS Connect

First, you will need to register to CMS Connect in order to get an account to the submission machine. Go to the registration site and follow instructions there. Once registered you will be authorized to use login.uscms.org (the Condor submit host) authenticating with your User ID (netid).

After approval, you will need to create and upload your public ssh-key to your account. You can follow the steps for that HERE. Once your public key is uploaded, please allow 3 to 4 of hours for the system to create your account in the submit node. You can then log in into login.uscms.org with your username via ssh (ssh your_username@loginNOSPAMPLEASE.uscms.org).

To verify access


$ ssh netid@login.uscms.org 

Setting up your proxy certificates

You will need a valid VO CMS grid proxy certificate in order to be able to submit your jobs.

  • If you don't have a valid grid proxy certificate, you need to apply for it. You can find more information on how to setup your proxy certificates here.
  • If you already have a valid grid proxy certificate installed another remote machine like lxplus.cern.ch, you can simply copy them to the CMS Connect login machine. Just execute the following command from login.uscms.org and enter the username and password to access that remote machine.
$ copy_certificates

<b>=============================================================================</b>

This script checks if you have globus certificates or lets you copy them from another machine otherwise (default: lxplus.cern.ch)

      
         $ NOTE: New certificates need to be requested first. Follow this Twiki for that:

https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookStartingGrid#ObtainingCert

<b>=============================================================================</b>

Check for certificates in /home/yourusername/.globus

...

Couldn't find any certificates. Copying certificates from another machine Note: This requires certificates to be under the standard $HOME/.globus location ...

Enter hostname of machine to login: lxplus.cern.ch Enter username for lxplus.cern.ch: yourusername

Warning: Permanently added the RSA host key for IP address '188.184.70.205' to the list of known hosts.

Password:

usercert.pem 100% 3526 3.4KB/s 00:00 userkey.pem 100% 2009 2.0KB/s 00:01

All Done... You can execute the following to initialize your proxy:

voms-proxy-init -voms cms -valid 192:00 

Submitting your Jobs

  • You can now start submitting your condor jobs. Your jobs will be sent to all available CMS Tier Sites by default, but you can specify the CMS Sites you desire to use following this section. Jobs will be reported to the CMS Dashboard.
  • If you don't have experience submitting condor jobs, you can read the following Quick Tutorial.
  • To learn how to pack a CMSSW release and use it in your workflow, a CMSSW analysis example can be found here: CMSSW Analysis Example.

For example, you can do the following to submit an example job to all US and non-US Tier Sites in the Global Pool:


$ tutorial quickstart

Installing quickstart (master)...

Tutorial files installed in ./tutorial-quickstart.

Running setup in ./tutorial-quickstart...

$ cd tutorial-quickstart

$ condor_submit tutorial01.submit 

Additional features

Requesting different Operative Systems

The CMS Global Pool is integrated with singularity, a container solution that allows requesting different operative systems via containers if necessary. For example, you can add:

+REQUIRED_OS = "rhel7"

to your submit file to run jobs under RedHat 7 on Sites supporting Singularity.

For more information on how to use singularity on CMS Connect, click here.

Submitting GPU jobs

GPU resources are available in the Global Pool.

For information about submitting GPU jobs to the Global Pool via CMS Connect, see think link.

Using TensorFlow (CPU and GPU)

OSG provides ready-to-use singularity containers for TensorFlow workflows for both CPU and GPU jobs that can be used in CMS Connect. Use the following link to learn how to use TensorFlow via singularity and submit CPU and GPU jobs or how to access these resources interactively via condor_ssh_to_job.

Reporting to CMS Dashboard

Condor jobs submitted via CMS-Connect will be automatically reported to CMS Dashboard, in a similar way CRAB does. A basic report that doesn't require any particular action from the user is done by default, but users are encouraged to provide a few parameters in their submission workflows in order to do handle e.g: stage-in, stage-out and full error code management in the report.

The reporting procedure is done in 2 steps:

  1. Report from the Submission Machine:
    The whole task is registered and sent to Dashboard from the submission machine while using condor_submit.
  2. Report from the Worker Node
    Each job is reported once it is assigned to an available machine and executed from it.
    As opposed to regular CRAB workflows, users define their own submission scripts in CMS-Connect (as in any regular condor workflow). Due to this fact, tasks like stage-out, stage-in and error code management are implemented and handled by each user. For this reason, only a few parameters are reported by default, without the need of any further action from the user.

Basic Report (Default)

The basic report is handled by CMS-Connect wrappers and there no user-side action is required for it. This report includes the following:

  • Start and End time of report
  • Executable CPU and WallClock time
  • Executable exit code
    Please, notice that if the user submits a wrapper on top of the executable, the wrapper exit code and times will be reported, unless the user specifies such values (see Advanced Report).
  • Hostname of machine where the job was executed
  • Computing Element Name
Please, see the Advanced Report in order to report stage-in/stage-out times and exit codes, number of events in the job or to override some of the default parameters.

Full Report

The following parameters can be specified by the user in order to report more advanced parameters from the worker node to the CMS Dashboard. The only requirement is to print out such parameters in the format:

Parameters report format

PARAMETER = VALUE

# Example: Print this out at the end of your job to report the number of events on it.

CMS_DASHBOARD_N_EVENTS = 5000

The following table provides a list of the parameters than can be reported from the user side and the default values for the basic report case.

Parameters Description
Parameters Description
CMS_DASHBOARD_N_EVENTS Number of events in the job. Default: 0
CMS_DASHBOARD_EXE_WC_TIME Executable wall clock time. Default: Condor executable WC time.
CMS_DASHBOARD_EXE_CPU_TIME Executable CPU time. Default: Condor executable CPU time.
CMS_DASHBOARD_EXE_EXIT_CODE

Executable exit code. Default: Condor Executable exit code.

Note: The user might want to override the default values for EXE_WC_TIME, EXE_CPU_TIME and EXE_EXIT_CODE in cases where e.g the Condor Executable is just a user wrapper running the actual executable.
CMS_DASHBOARD_STAGEOUT_SE Storage Element name. Default: unknown.
CMS_DASHBOARD_STAGEOUT_EXIT_CODE Stage out exit code.
CMS_DASHBOARD_STAGEOUT_TIME Stage out exit time.
CMS_DASHBOARD_JOB_EXIT_CODE Job Exit code. Default: Executable exit code.
User can report their own job exit codes to handle the overall completion state of the job.
CMS_DASHBOARD_JOB_EXIT_REASON Job Exit Reason. Default: Empty
You can follow this Twiki link to find more information about job monitoring with CMS Dashboard.

Historical View Example

Example CMS-Connect jobs reported to Dashboard.

Using the Connect client

The connect client allows you to submit jobs from your laptop or local cluster to the Global Pool via CMS Connect. This is provided by OSG and the full documentation can be found here.

Note: You don't need to install the connect client on login.uscms.org to submit jobs. The client is only useful if you want to submit from e.g: cmslpc-sl6.fnal.gov, lxplus.cern.ch, your laptop, etc.

Option 1: Using the client from CVMFS

To use the client from cvmfs, download and source the client script for your shell (tcsh and bash/zsh supported), as shown below:

* Important note: The client uses python modules that might be incompatible with your CMSSW release. * If for any reason you need to cmsenv a CMSSW release before submitting jobs, please use install the client instead.


git clone https://github.com/CMSConnect/cmsconnect-client

cd cmsconnect-client

source cmsconnect_client.sh

# or for tcsh

source cmsconnect_client.tcsh 

Option 2: Installing the client locally

A CMSSW environment is not needed for submitting jobs, unless you are planning to e.g: inherit the environment from the submit node to the worker nodes. In order to use the CMS Connect client within the environment of a CMSSW release, the client needs to be installed under that release to guarantee python compatibility. You can follow the commands below for that:


# First, cmsenv a CMSSW release providing python2.7. For example:

$ source /cvmfs/cms.cern.ch/cmsset_default.csh

cmsrel CMSSW_7_1_28; cd CMSSW_7_1_28; cmsenv

# Now, install the client

$ wget -O - https://raw.githubusercontent.com/CMSConnect/cmsconnect-client/master/cmsconnect_client_install.sh | bash 

Please notice that if you change to a different CMSSW release (e.g from 8.x to 9.x), you might need to reinstall the client.

Using the client locally

Once installed, you just need to source the client script (examples for tcsh and bash):


# For tcsh

source ~/software/connect-client/cmsconnect_client.tcsh

# For bash:

source ~/software/connect-client/cmsconnect_client.sh 

Setting up the account the first time

To setup the connect client with cms connect the, you can type:


$ connect setup username@login.uscms.org 

Initializing proxy certificates

We will need to initialize our proxy certificates in the CMS Connect node in order to submit jobs.

$ connect shell
$ export HOME=/home/$USER
$ voms-proxy-init -voms cms -valid 192:00
$ exit

Submitting jobs

To verify your proxy life, you can then type:


$ connect shell voms-proxy-info -all 

Use the git repo to get a submit example from the tutorial:


$ git clone https://github.com/CMSConnect/tutorial-quickstart
$ cd tutorial-quickstart

You will need to add your project name to the submit file (tutorial01.submit).

+ProjectName="cms.org.yourinstitution"

to see your default project you can type:


$ connect shell cat /home/\$USER/.ciconnect/defaultproject 

To submit your job, you can type:


$ connect submit tutorial01.submit

To see the queue:


$ connect q 

Once your job is done in the queue, you can pull the output via:


$ connect pull 

Note: In case of the tutorial01 example, you will see "job.output, job.err" being transferred.

Additional Documentation

You can find additional documentation at:

http://docs.uscms.org/

Contacts

Review status

<!-- Add your review status in this table structure with 2 columns delineated by three vertical bars -->

Reviewer/Editor and Date (copy from screen) Comments
Kenyi Hurtado - 10 June 2016 created documentation v1
<!-- In the following line, be sure to put a blank space AFTER your name; otherwise the Summary doesn't come out right. -->


Responsible: KenyiHurtado

Last reviewed by: Most recent reviewer

Edit | Attach | Watch | Print version | History: r23 < r22 < r21 < r20 < r19 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r23 - 2018-11-27 - KenyiPaoloHurtadoAnampa
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback