Manual install of HTCondor-CE accounting data extraction scripts

From htcondor-ce-3.3.0-1, the data extraction scripts are shipped as an RPM in the standard htcondor-ce yum repositories.

But if you are using a previous version of htcondor-ce, here is the run down on how to do it by hand (BTW: you'll need to update the site and CE names in the following scripts, else Liverpool will get the accounting credit!)

Every day, at some fixed time using a cron, we run the data extraction script, to get the CE data.


# "timestamp=1544614059" "userDN=/C=UK/O=eScience/OU=Liverpool/L=CSD/CN=stephen jones"
# ... "userFQAN=/dteam/Role=NULL/Capability=NULL" "" ""
# ... ... "" "localUser=dteam001"

  while read line; do

    if [[ $line =~ $regex ]]
      textdate=$(date -u +"%Y-%m-%d %H:%M:%S" -d@$epoch)
      echo "${leading}${textdate}${trailing}"
      echo $line

today=$(date -u --date='00:00:00 today' +%s)
yesterday=$(date -u --date='00:00:00 yesterday' +%s)
output=blah-$(date -u --date='yesterday' +%Y%m%d )-$(hostname -s)
CONST="EnteredCurrentStatus >= $yesterday && EnteredCurrentStatus < $today && RemoteWallclockTime !=0"

/usr/bin/condor_history -const "$CONST" \
 -format "\"timestamp=%d\" " EnteredCurrentStatus  \
 -format "\"userDN=%s\" " x509userproxysubject \
 -format "\"userFQAN=%s\" " x509UserProxyFirstFQAN \
 -format "\"\" " EMPTY \
 -format "\"\" " RoutedFromJobId \
 -format "\"\" " ClusterId \
 -format "\"localUser=%s\"\n"  Owner  | convertEpochToTextDate \
   > /var/lib/condor/accounting/$output

And once that's completed, in the same cron, we run the other data extraction script, to get the batch system data.



# Create a temporary accounting file name
today=$(date -u --date='00:00:00 today' +%s)
yesterday=$(date -u --date='00:00:00 yesterday' +%s)
output=batch-$(date -u --date='yesterday' +%Y%m%d )-$(hostname -s)


# Build the filter for the history command
CONSTR="EnteredCurrentStatus >= $yesterday && EnteredCurrentStatus < $today && RemoteWallclockTime !=0"

$CONDOR_LOCATION/bin/condor_history -constraint "$CONSTR" \
    -format "|" ClusterId \
    -format "%s|" Owner \
    -format "%d|" RemoteWallClockTime \
    -format "%d|" RemoteUserCpu \
    -format "%d|" RemoteSysCpu \
    -format "%d|" JobStartDate \
    -format "%d|" EnteredCurrentStatus \
    -format "%d|" ResidentSetSize_RAW \
    -format "%d|" ImageSize_RAW \
    -format "%d|" RequestCpus \
    -format "%v|" MachineAttrRalScaling0 \
    -format "\n" EMPTY  | sed -e "s/undefined|$/1.0|/" > $OUTPUT_FILE

At this point, we'll have two fresh files under


-- SteveJones - 2019-08-08

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2019-08-08 - SteveJones
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback