CMS Job Monitoring collectors

Contents:

Overview

New implementation of CMS job monitoring collector for the Experiment Dashboard monitoring project.

This page at the moment intend to list all the received messages (at the current status) grouped by message type. First we want to undrestand/define where each parameter need to be written, so we can optimize the implementation of bulk queries for update, insert.

Task Meta

This concerns information at task/workflow level. Usually this message implies to insert a new task in the TASK table, or to update existing information if the TASK already exists:

• TaskId: task name that should go in TaskMonitorId
• JSTool: SubmissionTool ; SubmissionToolId foreign key to SubmissionToolId
• JSToolVersion: SubToolVer ; SubmissionToolVerId foreign key to SubmissionToolVerId
• TaskType: TASK_TYPE.Type ; TaskTypeId foreign key to TaskTypeId
• CMSUser: UnixName ; UserId foreign key to UserId
• Workflow: TODO
• AgentName: SubmissionUI ; SubmissionUIId foreign key to SubmissionUIId
• HostIp: TODO

Job Meta

Job meta information is sent as soon as a job is created/submitted, to transmit static information about a job. It mainly is translated on population of the JOB and TASK_JOB table.

• JobId: job name should go in JobMonitorId and also in EventRange but with the removal of the submission counter from the JobId; TaskJobId is the foreign key to TaskJobId
• TaskId: task name that should go in TaskMonitorId; TaskId and TaskId are foreign keys to TaskId
• GridJobID: SchedulerJobId
• RetryCount: TODO
• JobType: JobType
• StatusValue: StatusName; GridStatusId is the foreign key pointing to StatusId (there is at least one trigger when this is added/updated)
• StatusEnterTime: GridStatusTimeStamp (there is at least one trigger when this is added/updated)
• Scheduler: SchedulerName; SchedulerId foreign key to SchedulerId
• HostIp: NodeName; SubNodeIp foreign key to NodeId
• TargetCE: TargetCE (this should be probably moved to TaskMeta)
• TargetSE: TargetCE (this should be probably moved to TaskMeta)
• JobIncrementalId: TODO
• WMSName: RB.name; RbId foreign key to RbId
• StatusDestination: this gets splitted into 3
  • CENameLong: TODO (it seems it is being added in LONG_CE)
  • CENameShort: TODO (it seems it is being added in SHORT_CE)
  • SiteName: TODO
• StatusReason: GridStatusReason ; GridStatusReasonId foreign key to GridStatusReasonId

Job Status

• JobId: job name should go in JobMonitorId and also in EventRange but with the removal of the submission counter from the JobId; TaskJobId is the foreign key to TaskJobId
• TaskId: task name that should go in TaskMonitorId
• GridJobID: SchedulerJobId
• RetryCount: TODO
• StatusValue: StatusName; GridStatusId is the foreign key pointing to StatusId (there is at least one trigger when this is added/updated)
• StatusEnterTime: GridStatusTimeStamp (there is at least one trigger when this is added/updated)
• StatusReason: GridStatusReason ; GridStatusReasonId foreign key to GridStatusReasonId
• JobExitCode
• HostIp: NodeName; SubNodeIp foreign key to NodeId
• SyncCE:
  • CENameLong: TODO (it seems it is being added in LONG_CE)
  • CENameShort: TODO (it seems it is being added in SHORT_CE)
  • SiteName: TODO
• JSTool: SubmissionTool
• JSToolUI: SubmissionUI ; SubmissionUIId foreign key to SubmissionUIId
• NbEvPerRun: NoEventsPerRun
• NSteps: CURRENTLY IGNORED
• RBname: RB.name; RbId foreign key to RbId
• StatusValueReason: GridStatusReason ; GridStatusReasonId foreign key to GridStatusReasonId
• stepName: TOADD
• TaskType: TASK_TYPE.Type, GenericType
• UserName: UnixName ; UserId foreign key to UserId
• VO: VO.VO ; USER.VOId and VOJobId foreign keys to VO.VOId

Runtime

This information is usually sent directly from the WN. It may consists of small messages sent in a short interval time (CRAB-2 case).

• JobId: job name should go in JobMonitorId and also in EventRange but with the removal of the submission counter from the JobId; TaskJobId is the foreign key to TaskJobId
• TaskId: task name that should go in TaskMonitorId
• GridJobID: SchedulerJobId
• RetryCount: TODO
• ExeEnd: CurrProcStep ; TODO?
• ExeStart: CurrProcStep ; TODO?
• HostIp:
• WNHostName: WNHostName
• SyncCE:
  • CENameLong: TODO (it seems it is being added in LONG_CE)
  • CENameShort: TODO (it seems it is being added in SHORT_CE)
  • SiteName: TODO
• StatusValue: StatusName; GridStatusId is the foreign key pointing to StatusId (there is at least one trigger when this is added/updated)
• StatusEnterTime: GridStatusTimeStamp (there is at least one trigger when this is added/updated)
• GridFlavour: TODO

Performance

This is something currently not handled. Idea is to have a new table (JOB_PERFORMANCE) pointing to the JOB table (foreign key), because we want this information to be cleaned periodically (after proper aggregation).

• JobId
• TaskId: task name that should go in TaskMonitorId
• StepName
• GridJobID
• StatusValue
• StatusEnterTime
• WriteTotalMB
• ReadTotalMB
• ReadMBSec
• ReadMaxMSec
• ReadAveragekB
• MinEventTime
• MaxEventTime
• MinEventCPU
• MaxEventCPU
• AvgEventCPU
• TotalEventCPU
• TotalJobCPU
• ReadPercentageOps
• ReadCachePercentageOps
• ReadTotalSecs
• WriteTotalSecs
• PeakValueVsize
• AvgEventTime
• TotalJobTime
• PeakValueRss
• ReadNumOps
• JobExitCode

-- MattiaCinquilli - 27-Jun-2011