Manual install of HTCondor-CE accounting data extraction scripts

From htcondor-ce-3.3.0-1, the data extraction scripts are shipped as an RPM in the standard htcondor-ce yum repositories.

But if you are using a previous version of htcondor-ce, here is the run down on how to do it by hand (BTW: you'll need to update the site and CE names in the following scripts, else Liverpool will get the accounting credit!)

Every day, at some fixed time using a cron, we run the data extraction script, condor_ce_blah.sh to get the CE data.

#!/bin/bash

# "timestamp=1544614059" "userDN=/C=UK/O=eScience/OU=Liverpool/L=CSD/CN=stephen jones"
# ... "userFQAN=/dteam/Role=NULL/Capability=NULL" "ceID=hepgrid6.ph.liv.ac.uk:9619/hepgrid6.ph.liv.ac.uk-condor" "jobID=1.0_hepgrid6.ph.liv.ac.uk"
# ... ... "lrmsID=1_hepgrid6.ph.liv.ac.uk" "localUser=dteam001"

convertEpochToTextDate()
{
  while read line; do

    regex="(.*timestamp=)([0-9][0-9]*)(.*)"
    if [[ $line =~ $regex ]]
    then
      leading="${BASH_REMATCH[1]}"
      epoch="${BASH_REMATCH[2]}"
      trailing="${BASH_REMATCH[3]}"
      textdate=$(date -u +"%Y-%m-%d %H:%M:%S" -d@$epoch)
      echo "${leading}${textdate}${trailing}"
    else
      echo $line
    fi
  done;
}

today=$(date -u --date='00:00:00 today' +%s)
yesterday=$(date -u --date='00:00:00 yesterday' +%s)
output=blah-$(date -u --date='yesterday' +%Y%m%d )-$(hostname -s)
CONST="EnteredCurrentStatus >= $yesterday && EnteredCurrentStatus < $today && RemoteWallclockTime !=0"

/usr/bin/condor_history -const "$CONST" \
 -format "\"timestamp=%d\" " EnteredCurrentStatus  \
 -format "\"userDN=%s\" " x509userproxysubject \
 -format "\"userFQAN=%s\" " x509UserProxyFirstFQAN \
 -format "\"ceID=hepgrid6.ph.liv.ac.uk:9619/hepgrid6.ph.liv.ac.uk-condor\" " EMPTY \
 -format "\"jobID=%v_hepgrid6.ph.liv.ac.uk\" " RoutedFromJobId \
 -format "\"lrmsID=%v_hepgrid6.ph.liv.ac.uk\" " ClusterId \
 -format "\"localUser=%s\"\n"  Owner  | convertEpochToTextDate \
   > /var/lib/condor/accounting/$output

And once that's completed, in the same cron, we run the other data extraction script, condor_batch.sh to get the batch system data.

#!/bin/sh

CONDOR_LOCATION=/usr
OUTPUT_LOCATION=/var/lib/condor/accounting/

# Create a temporary accounting file name
today=$(date -u --date='00:00:00 today' +%s)
yesterday=$(date -u --date='00:00:00 yesterday' +%s)
output=batch-$(date -u --date='yesterday' +%Y%m%d )-$(hostname -s)

OUTPUT_FILE=$OUTPUT_LOCATION/$output

# Build the filter for the history command
CONSTR="EnteredCurrentStatus >= $yesterday && EnteredCurrentStatus < $today && RemoteWallclockTime !=0"

$CONDOR_LOCATION/bin/condor_history -constraint "$CONSTR" \
    -format "%s_hepgrid6.ph.liv.ac.uk|" ClusterId \
    -format "%s|" Owner \
    -format "%d|" RemoteWallClockTime \
    -format "%d|" RemoteUserCpu \
    -format "%d|" RemoteSysCpu \
    -format "%d|" JobStartDate \
    -format "%d|" EnteredCurrentStatus \
    -format "%d|" ResidentSetSize_RAW \
    -format "%d|" ImageSize_RAW \
    -format "%d|" RequestCpus \
    -format "%v|" MachineAttrRalScaling0 \
    -format "\n" EMPTY  | sed -e "s/undefined|$/1.0|/" > $OUTPUT_FILE

At this point, we'll have two fresh files under

/var/lib/condor/accounting

-- SteveJones - 2019-08-08