CRAB Logo

Deployment of HTCondor Schedd for CRAB3

Complete: 5 Go to SWGuideCrab

About CRAB3 Schedd

CRAB3 Schedds are just an HTCondor Schedd with some custom configs.

Deployment of Schedd for CRAB3 operators

Machine details

This machine should be managed by puppet so you need to ask VOC cms-voc@cernNOSPAMPLEASE.ch to provide a machine in CRAB3 openstack project with the following specs:

Table 1
Specs
Flavor m2.3xlarge
Flavor ID 28936
RAM 58.6GB
VCPUs 32 VCPU
Disk 320GB
Metadata
Imagen Name SLC6 Base - x86_64 [2015-09-04]
landb-mainuser CMS-SERVICE-CRAB3HTCONDOR
landb-responsible CMS-SERVICE-CRAB3HTCONDOR-ADMINS
tenant-name CMS CRAB
Volumes Attached
HighIO volume attached To <your-brand-new-vm>_high on /dev/vdb
Standard volume attached To <your-brand-new-vm>_standard on /dev/vdc

Why you cannot create the VM by yourself since you have access to CRAB3 project? Simple, those specs are not available for you and neither the HighIO volumes. The instructions here were tested and they work for a VM which follow the specifications listed on table 1.

Be sure that file volumes.pp on master branch (https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocmsglidein/blob/master/code/manifests/modules/volumes.pp) is up to date with the attached volumes info, the HighIO volume must be mounted on /data and standard volume on /home/grid. If is not updated ask VOC to update it otherwise you will need to do it by youself. Bellow is a code snippet that need to be inserted on volumes.pp by VOC, please pay attention to the block devices used by HighIO volume(/dev/vdb) and standard volume(/dev/vdc). These lines will take care of create a partition table and a ext4 FS.

  elsif $::hostname == 'pvtschedd' {

    exec {'/sbin/mkfs.ext4 -L "STANDARD-VOL" /dev/vdc':
      unless => '/sbin/blkid -t TYPE=ext4 /dev/vdc'
    }

    exec {'/sbin/mkfs.ext4 -L "HIGH-IO-VOL" /dev/vdb':
      unless => '/sbin/blkid -t TYPE=ext4 /dev/vdb'
    }

    mount { "/mnt":
      ensure => absent,
    }

    file{"/data":
      ensure => directory,
      mode => 0755,
    }

    mount {"/data":
      device => "/dev/vdb",
      fstype => "ext4",
      ensure => "mounted",
      options => "defaults",
      atboot => "true",
      require => File['/data']
    }

    file{"/home/grid":
      ensure => directory,
      mode => 0755,
    }

    mount {"/home/grid":
      device => "/dev/vdc",
      fstype => "ext4",
      ensure => "mounted",
      options => "defaults",
      atboot => "true",
      require => File['/home/grid']
    }
  }

Once you get the machine you need to create a yaml file for it, you can just copy and paste from another CRAB3 vm, for instance vocms0106.

Bear in mind that there are 2 kinds of CRAB3 schedds, production schedds(the ones in master branch) and ITB schedds(the ones on qa branch).

To deploy production schedds you should write an yaml file like this:

gwms_type: crabschedd
cernfw_landbset: it_cc_cms_glideinwms
glidecondor_pool: global
condor_version: 8.5.4-1.el6

sssd::interactiveallowusers:
   - cmsprd

sssd::interactiveallowgroups:
   - cms-service-crab
   - cms-service-crab2
   - cms-service-crab3htcondor
   - cms-service-glideinwms

and to deploy an ITB schedds one like this:

gwms_type: crabschedd
cernfw_landbset: it_cc_cms_glideinwms
glidecondor_pool: itb
condor_version: 8.5.4-1.el6

sssd::interactiveallowusers:
   - cmsprd

sssd::interactiveallowgroups:
   - cms-service-crab
   - cms-service-crab2
   - cms-service-crab3htcondor
   - cms-service-glideinwms

In case you are deploying an dev schedd use this yaml file to avoid submissions to FNAL LPC:

gwms_type: crabschedd
cernfw_landbset: it_cc_cms_glideinwms
glidecondor_pool: itb
condor_version: 8.5.4-1.el6
lcp_submit: false

sssd::interactiveallowusers:
   - cmsprd

sssd::interactiveallowgroups:
   - cms-service-crab
   - cms-service-crab2
   - cms-service-crab3htcondor
   - cms-service-glideinwms

Now go to your cloned repo of https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocmsglidein and write the yaml file.

cd it-puppet-hostgroup-vocmsglidein/data/fqdns
git checkout master
git pull 
cp vocms0106.cern.ch.yaml &lt;your-brand-new-vm&gt;.cern.ch.yaml

Do the necessary changes to have either a production or a ITB schedd and commit your changes to gitlab, remember for production schedds push your changes to master branch for ITB schedds push to qa branch.

git add <your-brand-new-vm>.cern.ch.yaml
git commit -m 'a nice description for your commit'
git push origin master

Puppet first run

After create the yaml file you are ready to run puppet for the first time. This will setup the machine, install condor and all default configs, create all users home directories under /home/grid, install cron jobs, install lemon alarms and sensors. Most probably you will need to run puppet at least 3 or 4 times until all dependent services get installed and configured. Please during the puppet run pay attention to all outputs.

sudo -s
puppet agent -tv

Once you dont get any puppet errors, create the /etc/grid-security/condorkey.pem, this file is necessary to allow the schedd to contact argus and get a grid user name for a particular user DN. Run the following command for that:

cp /etc/grid-security/hostkey.pem /data/srv/condorkey.pem.tmp; mv /data/srv/condorkey.pem.tmp /etc/grid-security/condorkey.pem; chown condor:condor /etc/grid-security/condorkey.pem

This command comes from the cron job installed by puppet,

Configuration details

All config files are located at /etc/condor/config.d/ folder but only 82_cms_schedd_crab_generic.config is specific for CRAB. Usually you dont need to touch these files and if you need to apply custom changes for a specific schedd you should write your changes on 99_local_tweaks.config and if after some time you decide that this custom change needs to be applied to all CRAB3 schedds is better to move the change to 82_cms_schedd_crab_generic.config and leave 99_local_tweaks.config much cleaner as possible.

82_cms_schedd_crab_generic:

########################################################
# HTCondor configurations specific to crab schedd

# Adding JobStatus to significant attributes to help Brian sort out
# idle jobs per user
# Adeed 2015-07-29
ADD_SIGNIFICANT_ATTRIBUTES=$(ADD_SIGNIFICANT_ATTRIBUTES),CRAB_UserHN

# Accounting group settings for crab
use_x509userproxy = true
accounting_group = analysis
SUBMIT_EXPRS = $(SUBMIT_EXPRS) use_x509userproxy accounting_group

# To cache Argus results and prevent GSI authorization callout
# related timeouts.
# Added by Farrukh - Nov. 25, 2014
GSS_ASSIST_GRIDMAP_CACHE_EXPIRATION=7200 

# Overwriting the system periodic remove expression for CRAB3
SYSTEM_PERIODIC_REMOVE = ((JobUniverse=!=7)&&(((NumJobStarts>9)=?=True)||((NumShadowStarts>19)=?=True)))
SYSTEM_PERIODIC_REMOVE = ($(SYSTEM_PERIODIC_REMOVE)) || ((DiskUsage>27000000)=?=True)

# Older version of Condor used a lower default
GSI_DELEGATION_KEYBITS = 1024

# Enable Condor-C full  delegation, 
# but keep the delegation to worker nodes limited and short
DELEGATE_FULL_JOB_GSI_CREDENTIALS = True
SHADOW.DELEGATE_FULL_JOB_GSI_CREDENTIALS = False
DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME = 0
SHADOW.DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME = 86400

# Whitelist the CRAB3 servers
SCHEDD.ALLOW_WRITE = */vocms052.cern.ch, */vocms045.cern.ch, */vocms0118.cern.ch, */vocms0119.cern.ch, */$(FULL_HOSTNAME)
SCHEDD.HOSTALLOW_WRITE =

# Limit the number of dagmans
START_SCHEDULER_UNIVERSE = TotalSchedulerJobsRunning < 250

Used_Gatekeeper = "$$(GLIDEIN_Gatekeeper:Unknown)"
JOB_CMSSite = "$$(GLIDEIN_CMSSite:Unknown)"
# the userlog will have Used_Gatekeeper defined at job runtime
# and MATCH_GLIDEIN_Gatekeeper at job termination
# but never both

JOB_Gatekeeper = ifthenelse(\
 substr(Used_Gatekeeper,0,1)=!="$", Used_Gatekeeper, \
   ifthenelse(\
       MATCH_GLIDEIN_Gatekeeper=!=UNDEFINED,MATCH_GLIDEIN_Gatekeeper,\
           "Unknown"))
SUBMIT_EXPRS = $(SUBMIT_EXPRS) JOB_Gatekeeper JOB_CMSSite Used_Gatekeeper

# this assumes noone else is defining the list
job_ad_information_attrs = MATCH_GLIDEIN_Gatekeeper, JOB_Gatekeeper,\
Used_Gatekeeper, JOB_CMSSite

# in the Condor logic, this is a user-provided attribute
# so tell the schedd to treat it as such
SUBMIT_EXPRS = $(SUBMIT_EXPRS) job_ad_information_attrs

# Disabling FSYNC() call for logs
# Added 2015-FEB-09 
CONDOR_FSYNC = false

# CRAB specific job throttle parameters
# Added 2015-FEB-09 
JOB_START_DELAY = 2
JOB_START_COUNT = 10

# Used by CRAB3 for HTTP access to log files
# Added 2015-FEB-16
CRAB_StorageRules = ^/home/grid,http://vocms0106.cern.ch/mon

HISTORY_HELPER=$(LIBEXEC)/condor_history_helper
# Unsetting this so it reverts to default of 10k
# ELOG: https://cms-logbook.cern.ch/elog/GlideInWMS/3413
#HISTORY_HELPER_MAX_HISTORY=1000000

# Unset on 2015-08-04
# libsnoopy.so has been updated, so no longer needed
# ELOG: https://cms-logbook.cern.ch/elog/GlideInWMS/2472 
# USE_CLONE_TO_CREATE_PROCESSES=false

# To help retrieve condor logs for CRAB3
KILLING_TIMEOUT = 300

# Enabling OVERFLOW for CRAB3 machines only
CMS_ALLOW_OVERFLOW = "True"

OVERFLOW_US = ifthenelse(regexp("T[1,2]_US_",DESIRED_Sites),"True",UNDEFINED)
OVERFLOW_IT = ifthenelse(regexp("T[1,2]_IT_",DESIRED_Sites),"True",UNDEFINED)
OVERFLOW_UK = ifthenelse(regexp("T2_UK_London_",DESIRED_Sites),"True",UNDEFINED)

DESIRED_Overflow_Region = strcat( ifthenelse(OVERFLOW_US=?="True","US", "none"), ",",ifthenelse(OVERFLOW_IT=?="True","IT", "none"), ",", ifthenelse(OVERFLOW_UK=?="True","UK", "none") )

SUBMIT_ATTRS = $(SUBMIT_ATTRS) OVERFLOW_US OVERFLOW_IT OVERFLOW_UK CMS_ALLOW_OVERFLOW DESIRED_Overflow_Region

# To monitor the number of overflow jobs
OVERFLOW_CHECK = ifthenelse(MATCH_GLIDEIN_CMSSite isnt undefined, ifthenelse(stringListMember(MATCH_GLIDEIN_CMSSite,DESIRED_Sites),FALSE,TRUE), FALSE)
SUBMIT_ATTRS = $(SUBMIT_ATTRS) OVERFLOW_CHECK

# Adding PER_JOB_HISTORY_DIR feature
PER_JOB_HISTORY_DIR=/data/srv/glidecondor/condor_local/job_history/

# Number of jobs Dagman will submit to condor_q
DAGMAN_MAX_SUBMITS_PER_INTERVAL = 100

99_local_tweaks.config

############################################
#
# 99_local_tweaks.config
# 
# This config file should be used for all manual changes
# on this machine. The other config files should not
# be modified, as they will be overwritten by puppet. 
# Include permanent changes in puppet, and temporary 
# changes or machine-specific changes here.
#
############################################

# To avoid spamming others while debugging issues
CONDOR_ADMIN = jadir.silva13@gmail.com

As was said before this file should be kept as cleaner as possible, but if you want to change one particular schedd please put your customizations here. Bellow you can find an example of custom schedd config.

############################################
#
# 99_local_tweaks.config
# 
# This config file should be used for all manual changes
# on this machine. The other config files should not
# be modified, as they will be overwritten by puppet. 
# Include permanent changes in puppet, and temporary 
# changes or machine-specific changes here.
#
############################################

# To avoid spamming others while debugging issues
CONDOR_ADMIN = jadir.silva13@gmail.com

SCHEDD_NAME = crab3@vocms0106.cern.ch

SCHEDD.ALLOW_WRITE= $(SCHEDD.ALLOW_WRITE), balcas-crab2.cern.ch, */brazil.accre.vanderbilt.edu

# changed on Jun 21, https://cms-logbook.cern.ch/elog/Analysis+Operations/956 
START_SCHEDULER_UNIVERSE = (TotalSchedulerJobsRunning < 300) && ((ShadowsRunning is undefined) || (ShadowsRunning < .95*$(MAX_JOBS_RUNNING))) 

# debug configs
SHARED_PORT_DEBUG = D_ALL:2

MAX_JOBS_RUNNING = 12000

In this case we have this configurations

knob Effect
CONDOR_ADMIN overwrite the default condor admin email, useful during the tests to avoid flooding the others operators inbox with debugging msg
SCHEDD_NAME choose a different schedd name, the current naming convertion is crab3@$(FULLHOSTNAME)
SCHEDD.ALLOW_WRITE useful for developers to allow this schedd to work with non default TW
START_SCHEDULER_UNIVERSE define how many DAGs the schedd can run
SHARED_PORT_DEBUG enable full debugging for shared_port daemon
MAX_JOBS_RUNNING define how many jobs the schedd can run

There are plenty of possibilities here, if you want to see more options go to HTCondor configurations page http://research.cs.wisc.edu/htcondor/manual/v8.5/3_3Configuration.html and have fun.

Configurations changes on Central Managers and Frontends

All schedds needs to be added to Global Pool central managers and Frontends and also to tier0 Central Managers and Frontends for that is sufficient to create an elog on https://cms-logbook.cern.ch/elog/GlideInWMS with the following content:

Please add the schedd <your-brand-new-vm>.cern.ch to global pool central managers and frontends and also to tier0 central managers and frontends.

Cheers,
Your Name here

Configurations changes on LPC firewall

Since now all scheeds run job router they need to be whitelisted in the FNAL firewall, so write a email to Krista<klarson1@fnal.gov> and Tony<tiradani@fnal.gov> asking then to include the new schedd on the firewall whitelist.

Redirection rules for CMSWEB

This is needed to allow users outside CERN to access log files on CERN schedds. For every new schedd you should add a rule on https://github.com/dmwm/deployment/blob/master/frontend/backends-prod.txt.

^/auth/complete/dqm/(?:online|online-playback|online-test)(?:/|$) cmsdaq0.cern.ch
^/auth/complete/dqm/offline(?:/|$) vocms0138.cern.ch
^/auth/complete/dqm/relval(?:/|$) vocms0139.cern.ch
^/auth/complete/dqm/(?:dev|offline-test|relval-test)(?:/|$) vocms0131.cern.ch
^/auth/complete/couchdb2/asynctransfer1(?:/|$) vocms0306.cern.ch  :affinity
^/auth/complete/(?:aso-monitor|couchdb2)(?:/|$) vocms0141.cern.ch :affinity
^/auth/complete/(?:couchdb/wmstats|wmstats)(?:/|$) vocms0307.cern.ch :affinity
^/auth/complete/(?:c?)(?:couchdb|workqueue|workloadsummary|alertscollector|analysis_workqueue|analysis_wmstats|tier0_wmstats|t0_workloadsummary|acdcserver)(?:/|$) vocms0140.cern.ch :affinity
^/auth/complete/das(?:/|$) vocms0141.cern.ch|vocms0307.cern.ch :affinity
^/auth/complete/crabcache(?:/|$) vocms0141.cern.ch
^/auth/complete/confdb(?:/|$) vocms0307.cern.ch
^/auth/complete/wmarchive(?:/|$) vocms071.cern.ch
^/auth/complete/scheddmon/021/ vocms021.cern.ch
^/auth/complete/scheddmon/095/ vocms095.cern.ch
^/auth/complete/scheddmon/096/ vocms096.cern.ch
^/auth/complete/scheddmon/0106/ vocms0106.cern.ch
^/auth/complete/scheddmon/0109/ vocms0109.cern.ch
^/auth/complete/scheddmon/0112/ vocms0112.cern.ch
^/auth/complete/scheddmon/0114/ vocms0114.cern.ch
^/auth/complete/scheddmon/066/ vocms066.cern.ch
^/auth/complete/scheddmon/059/ vocms059.cern.ch
^ vocms0136.cern.ch|vocms0161.cern.ch|vocms0163.cern.ch|vocms0165.cern.ch

to proceed you need to fork https://github.com/dmwm/deployment repo and then clone your forked repo and create a pull request with your changes.

git clone https://github.com/jmarra13/deployment.git
cd deployment
git remote -v
git remote add upstream https://github.com/dmwm/deployment
git remote -v

In this example I cloned my own fork, you should adjust the url https://github.com/jmarra13/deployment.git to your own.

git checkout -b redirect-rule-vocms0121
vim frontend/backends-prod.txt

Then add a line with the redirect rule, like this one:

^/auth/complete/scheddmon/0121/ vocms0121.cern.ch

then save the file and push your changes to your newly created branch.

git add frontend/backends-prod.txt
git commit -m 'some description of your changes'
git push origin redirect-rule-vocms0121

Then go to your fork, in my case https://github.com/jmarra13/deployment, and click on pull_request_test_highlighted.png

Free memory script for populating a schedd ClassAd TotalFreeMemoryMB

On each schedd is a condor cronjob (much like a regular cronjob, except it's managed by condor and defined in condor syntax) which reports how much free memory is available on the machine. This is mainly used for load balancing across different schedds in the schedd picker function which should select the least loaded schedd to submit a task to.

Currently, it is defined in the /etc/condor/config.d/99_local_tweaks.config file like this:

SCHEDD_CRON_CONFIG_VAL = $(RELEASE_DIR)/bin/condor_config_val
SCHEDD_CRON_JOBLIST = $(SCHEDD_CRON_JOBLIST) TOTAL_FREE_MEMORY_MB
SCHEDD_CRON_TOTAL_FREE_MEMORY_MB_EXECUTABLE = /data/srv/SubmissionInfrastructureScripts/total_free_memory.sh
SCHEDD_CRON_TOTAL_FREE_MEMORY_MB_PERIOD = 300s

The script itself is puppetized, it is located in this repo https://gitlab.cern.ch/CMSSI/SubmissionInfrastructureScripts/blob/master/total_free_memory.sh. Note that enabling this cron requires a condor restart. After it is defined in a config file (for example in the 99_local_tweaks), condor_reconfig is not enough for it to start running.

-- JadirSilva - 2016-08-09

Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r11 - 2017-03-06 - EmilisAntanasRupeika



 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback