Job running time, Memory and local disk limits to protect workernodes

Introduction

• Purpose: define limits for job running time, memory and local disk space consumption for GRID jobs/applications to protect workernodes
• Limits can be imposed on 4 different levels ordered by distance to the actual processes on the workernode
  1. site protections on the batch system level
  2. pilot protections in the pilot
  3. protections in the WMAgent/CRAB job wrapper
  4. protections directly in CMSSW
• limits should be stricter the larger the distance to the actual process on the workernde
• Advice
  • Job termination should be tried first gracefully (finish processing current event) when possible and return output for debugging, next a hard kill is ok

Proposed limits

• Comments:
  • 40 hour limit for CMSSW should allow for stageout and close out of job
    • Need to think about chained workflows (multiple steps run on the same workernode)
  • local disk space limitation not applicable to CMSSW limit level
  • memory limit specified in RSS, on SL6 and later PSS should be used
  • batch system limits:
    • if sites have more resources per core, limits should be adapted to support the additional resources
      • Ops should be informed and we should maybe setup special queues

Analysis April 2012: local disk usage from glideIn schedd used for analysis and production

Analysis April 2012: information from DashBoard for analysis and production jobs

Analysis April 2012: information from CouchDB only for production jobs