Manual install of HTCondor-CE accounting data extraction scripts
From htcondor-ce-3.3.0-1, the data extraction scripts are shipped as an RPM in the standard htcondor-ce yum repositories.
But if you are using a previous version of htcondor-ce, here is the run down on how to do it by hand (BTW: you'll need to update the site and CE names in the following scripts, else Liverpool will get the accounting credit!)
Every day, at some fixed time using a cron, we run the data extraction script, condor_ce_blah.sh to get the CE data.
#!/bin/bash
# "timestamp=1544614059" "userDN=/C=UK/O=eScience/OU=Liverpool/L=CSD/CN=stephen jones"
# ... "userFQAN=/dteam/Role=NULL/Capability=NULL" "ceID=hepgrid6.ph.liv.ac.uk:9619/hepgrid6.ph.liv.ac.uk-condor" "jobID=1.0_hepgrid6.ph.liv.ac.uk"
# ... ... "lrmsID=1_hepgrid6.ph.liv.ac.uk" "localUser=dteam001"
convertEpochToTextDate()
{
while read line; do
regex="(.*timestamp=)([0-9][0-9]*)(.*)"
if [[ $line =~ $regex ]]
then
leading="${BASH_REMATCH[1]}"
epoch="${BASH_REMATCH[2]}"
trailing="${BASH_REMATCH[3]}"
textdate=$(date -u +"%Y-%m-%d %H:%M:%S" -d@$epoch)
echo "${leading}${textdate}${trailing}"
else
echo $line
fi
done;
}
today=$(date -u --date='00:00:00 today' +%s)
yesterday=$(date -u --date='00:00:00 yesterday' +%s)
output=blah-$(date -u --date='yesterday' +%Y%m%d )-$(hostname -s)
CONST="EnteredCurrentStatus >= $yesterday && EnteredCurrentStatus < $today && RemoteWallclockTime !=0"
/usr/bin/condor_history -const "$CONST" \
-format "\"timestamp=%d\" " EnteredCurrentStatus \
-format "\"userDN=%s\" " x509userproxysubject \
-format "\"userFQAN=%s\" " x509UserProxyFirstFQAN \
-format "\"ceID=hepgrid6.ph.liv.ac.uk:9619/hepgrid6.ph.liv.ac.uk-condor\" " EMPTY \
-format "\"jobID=%v_hepgrid6.ph.liv.ac.uk\" " RoutedFromJobId \
-format "\"lrmsID=%v_hepgrid6.ph.liv.ac.uk\" " ClusterId \
-format "\"localUser=%s\"\n" Owner | convertEpochToTextDate \
> /var/lib/condor/accounting/$output
And once that's completed, in the same cron, we run the other data extraction script, condor_batch.sh to get the batch system data.
#!/bin/sh
CONDOR_LOCATION=/usr
OUTPUT_LOCATION=/var/lib/condor/accounting/
# Create a temporary accounting file name
today=$(date -u --date='00:00:00 today' +%s)
yesterday=$(date -u --date='00:00:00 yesterday' +%s)
output=batch-$(date -u --date='yesterday' +%Y%m%d )-$(hostname -s)
OUTPUT_FILE=$OUTPUT_LOCATION/$output
# Build the filter for the history command
CONSTR="EnteredCurrentStatus >= $yesterday && EnteredCurrentStatus < $today && RemoteWallclockTime !=0"
$CONDOR_LOCATION/bin/condor_history -constraint "$CONSTR" \
-format "%s_hepgrid6.ph.liv.ac.uk|" ClusterId \
-format "%s|" Owner \
-format "%d|" RemoteWallClockTime \
-format "%d|" RemoteUserCpu \
-format "%d|" RemoteSysCpu \
-format "%d|" JobStartDate \
-format "%d|" EnteredCurrentStatus \
-format "%d|" ResidentSetSize_RAW \
-format "%d|" ImageSize_RAW \
-format "%d|" RequestCpus \
-format "%v|" MachineAttrRalScaling0 \
-format "\n" EMPTY | sed -e "s/undefined|$/1.0|/" > $OUTPUT_FILE
At this point, we'll have two fresh files under
/var/lib/condor/accounting
--
SteveJones - 2019-08-08