CRAB Logo

Deployment and Maintenance of HTCondor Schedd for CRAB3

Complete: 5 Go to SWGuideCrab

About CRAB3 Schedd

CRAB3 Schedds are just an HTCondor Schedd with some custom configs.

Deployment of Schedd for CRAB3 operators

Machine details

This machine should be managed by puppet so you need to ask VOC cms-voc@cernNOSPAMPLEASE.ch to provide a machine in CRAB3 openstack project with the following specs:

Table 1
Specs
Flavor m2.3xlarge
Flavor ID 28936
RAM 58.6GB
VCPUs 32 VCPU
Disk 320GB
Metadata
Imagen Name SLC6 Base - x86_64 [2015-09-04]
landb-mainuser CMS-SERVICE-CRAB3HTCONDOR
landb-responsible CMS-SERVICE-CRAB3HTCONDOR-ADMINS
tenant-name CMS CRAB
Volumes Attached
HighIO volume attached To <your-brand-new-vm>_high on /dev/vdb
Standard volume attached To <your-brand-new-vm>_standard on /dev/vdc

Why you cannot create the VM by yourself since you have access to CRAB3 project? Simple, those specs are not available for you neither the HighIO volumes. The instructions here were tested and they work for a VM which follow the specifications listed on table 1.

Once you get the machine get sure that the volumes for /data and /home/grid were attached. Bear in mind that there are 2 kinds of CRAB3 schedds, production schedds(the ones in master branch) and ITB schedds(the ones on qa branch) what differs one to another is that on ITB we disable submissions to FNAL LPC by default, this is done at https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocms/tree/qa/data/hostgroup/vocms/crab/schedd.yaml

cernfw_landbset: it_cc_cms_glideinwms

vocmsgwmshtcondor::condor_version: 8.5.7-1.el6
vocmshtcondor::condor_pool: itb
vocmshtcondor::machine_type: crabschedd
vocmshtcondor::lpc_submission: false
vocmshtcondor::condor_admin: jadir.silva13@gmail.com

sssd::interactiveallowusers:
   - cmsprd

sssd::interactiveallowgroups:
   - cms-service-crab
   - cms-service-crab2
   - cms-service-crab3htcondor
   - cms-service-glideinwms

To have a production schedd tell VOC to put it on master branch of puppet and to have an ITB schedd ask to put it on qa branch, you can move then from one branch to another editing the host details at https://judy.cern.ch. The sub hostgroup for both kind of schedds needs to be vocms/crab/schedd.

The schedd deployment is done py puppet. To change anything in the configuration you should first clone the repo for vocms hostgroup.

git clone https://:@gitlab.cern.ch:8443/ai/it-puppet-hostgroup-vocms.git

Made your changes and push to the proper branch qa for ITB and master for production.

CRAB3 configuration structure on puppet:

https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocms/tree/qa/code/manifests/crab
├── schedd.pp
└── schedd
    ├── cron.pp
    ├── directories.pp
    ├── firewall.pp
    ├── monitor.pp
    ├── repos.pp
    ├── rpms.pp
    ├── services.pp
    ├── sudoers.pp
    ├── tunning.pp
    └── volumes.pp

An explanation of each file will follow.

schedd.pp

This is the main file which includes all others and is this the one responsable to install condor. If you need to install a new puppet class which should be common for all schedds here is the place.

class hg_vocms::crab::schedd {

  # Ganglia
  class {'hg_vocms::modules::ganglia':}

  # GlideinMon
  class {'hg_vocms::modules::glideinmon':}

  # GSI Server
  class {'hg_vocms::modules::gsiserver':}

  # VOMS Config
  include ('voms::cms')

  # Firewall rules
  class {'hg_vocms::crab::schedd::firewall':}

  # Linux tweaks
  class {'hg_vocms::crab::schedd::tunning':}

  # Monitoring metrics
  class {'hg_vocms::crab::schedd::monitor':}

  # Volumes configurations
  class {'hg_vocms::crab::schedd::volumes':}

  # Files and directory config
  class {'hg_vocms::crab::schedd::directories':}

  # Repos
  #class {'hg_vocms::crab::schedd::repos':}

  # Packages/Dependencies
  class {'hg_vocms::crab::schedd::rpms':}

  # Service management
  class {'hg_vocms::crab::schedd::services':}

  # Sudoers
  class {'hg_vocms::crab::schedd::sudoers':}

  # Cron jobs
  class {'hg_vocms::crab::schedd::cron':}

  # Including the condor module
  include ('vocmshtcondor')

}

cron.pp

Here you will find all crontab entries that needs to run on an schedd. Note that some times the cron job needs a file or script which also needs to be provided by puppet, in this case you add entry on https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocms/tree/qa/code/manifests/crab/schedd/directories.pp for each file needed.

class hg_vocms::crab::schedd::cron {

  cron { 'Clean user folder CRAB':
    command => "/bin/sh /root/clean_homedir.sh",
    user    => root,
    environment => "MAILTO=jmsilva@cern.ch",
    hour    => 8,
    minute  => 6,
  }

  cron { "clean_per_job_history_dir":
    command  => "/usr/bin/python /root/jobhistoryfeeder.py",
    user     => "root",
    hour     => [13,23],
    minute   => 10,
  }

  cron { "clean_condor_spool_dir":
    command  => "/bin/sh /root/clean_condor_spool_dir.sh",
    user     => "root",
    hour     => [13,23],
    minute   => 10,
  }

  # Crontab to create condorkey.pem, keep it up to date and enforce correct permissions
  # Added by Farrukh - April 23, 2015  
  cron { "create_condor_key":
    command  => "cp /etc/grid-security/hostkey.pem /data/srv/condorkey.pem.tmp; mv /data/srv/condorkey.pem.tmp /etc/grid-security/condorkey.pem; chown condor:condor /etc/grid-security/condorkey.pem",
    user     => "root",
    hour     => 12,
    minute   => 10,
    monthday => 1,
  }

}

directories.pp

This class have the function that will create all home users directories and also all files/scripts that needs to be copied to schedds. For instance this entry in https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocms/tree/qa/code/manifests/crab/schedd/cron.pp:

  cron { "clean_condor_spool_dir":
    command  => "/bin/sh /root/clean_condor_spool_dir.sh",
    user     => "root",
    hour     => [13,23],
    minute   => 10,
  }

To this works we need to copy clean_condor_spool_dir.sh script to /root/clean_condor_spool_dir.sh what can be done by an entry like this one in directories.pp:

   '/root/clean_condor_spool_dir.sh':
    ensure => file,
    owner  => root, group => root, mode => 644,
    source => 'puppet:///modules/hg_vocms/profiles/crab/schedd/clean_condor_spool_dir.sh',
    ;

class hg_vocms::crab::schedd::directories {

  require hg_vocms::crab::schedd::volumes

  # Function to create home directories of users
  define createHomeDirs ($count, $limit)
  {
    if ($count < 100) and ($count >= 10) {
      exec{"create_cms0${count}":
        command => "/sbin/mkhomedir_helper cms0${count}",
        creates  => "/home/grid/cms0${count}",
      }
    }
    elsif ($count < 10) {
      exec{"create_cms00${count}":
        command => "/sbin/mkhomedir_helper cms00${count}",
        creates => "/home/grid/cms00${count}",
      }
    }
    else {
      exec{"create_cms${count}":
        command => "/sbin/mkhomedir_helper cms${count}",
        creates  => "/home/grid/cms${count}",
      }
    }
    $next = $count - 1

    if $next == $limit {
      notify {"Finished creating directories upto ${limit}":}
    }
    else {
      createHomeDirs {"create${next}":
        count => $next,
        limit => $limit,
      }
    }
  }


  # Function calls
  # Had to make multiple calls since puppet
  # couldn't handle all 2000 at once

  createHomeDirs {'create_first_set':
    count => 1999,
    limit => 1500,
  }
  createHomeDirs {'create_2nd_set':
    count => 1500,
    limit => 1000,
  }
  createHomeDirs {'create_3rd_set':
    count => 1000,
    limit => 500,
  }
  createHomeDirs {'create_4th_set':
    count => 500,
    limit => 0,
  }
  exec{"create_cmsprd":
    command => "/sbin/mkhomedir_helper cmsprd",
    creates => "/home/grid/cmsprd",
  }


  file {
#   "/data/srv/glidecondor/condor_local/job_history/":
#    ensure => directory,
#    mode => 0755,
#    owner => "condor",
#    group => "condor",
#    require => Class["hg_vocms::modules::condor"],
#   ;
   "/data/poolhome":
    ensure => link,
    target => '/home/grid',
    force => true,
    owner => "root",
    group => "root",
    require => Package['httpd'],
   ;
   '/etc/httpd/conf.d/welcome.conf':
    ensure => absent,
   ;
   '/etc/httpd/conf/httpd.conf':
    ensure => file,
    owner  => root, group => root, mode => 644,
    source => 'puppet:///modules/hg_vocms/profiles/crab/schedd/httpd.conf',
    notify => Service['httpd'],
    ;
   '/usr/libexec/sensors/glexec_errors.py':
    ensure => file,
    owner  => root, group => root, mode => 644,
    source => 'puppet:///modules/hg_vocms/profiles/crab/schedd/glexec_errors.py',
    ;
   '/usr/libexec/sensors/check_maxdags.py':
    ensure => file,
    owner  => root, group => root, mode => 644,
    source => 'puppet:///modules/hg_vocms/profiles/crab/schedd/check_maxdags.py',
    ;
   '/root/jobhistoryfeeder.py':
    ensure => file,
    owner  => root, group => root, mode => 644,
    source => 'puppet:///modules/hg_vocms/profiles/crab/schedd/jobhistoryfeeder.py',
    ;
   '/root/clean_homedir.sh':
    ensure => file,
    owner  => root, group => root, mode => 644,
    source => 'puppet:///modules/hg_vocms/profiles/crab/schedd/clean_homedir.sh',
    ;
   '/root/clean_condor_spool_dir.sh':
    ensure => file,
    owner  => root, group => root, mode => 644,
    source => 'puppet:///modules/hg_vocms/profiles/crab/schedd/clean_condor_spool_dir.sh',
    ;
   '/usr/libexec/sensors/htcondor_ads.py':
    ensure => file,
    owner  => root, group => root, mode => 644,
    source => 'puppet:///modules/hg_vocms/profiles/crab/schedd/htcondor_ads.py',
    ;
   '/etc/libreport/events.d/mailx_event.conf':
    ensure => file,
    owner  => root, group => root, mode => 644,
    source => 'puppet:///modules/hg_vocms/profiles/crab/schedd/etc/libreport/events.d/mailx_event.conf',
    ;
  }

}

firewall.pp

class hg_vocms::crab::schedd::firewall {

  firewall { '1 CMS CRAB schedd ssh/gsissh/shared_deamon on port 2222/22/4080':
    chain => 'INPUT',
    proto   => 'tcp',
    dport   => ['22','2222','4080','80'],
    state   => 'NEW',
    action  => 'accept',
  }

 firewall { '1 CMS CRAB schedd ssh/gsissh/shared_deamon on port 2222/22/4080 (IPv6)':
    chain => 'INPUT',
    proto   => 'tcp',
    dport   => ['22','2222','4080','80'],
    provider => 'ip6tables',
    state   => 'NEW',
    action  => 'accept',
  }
}

monitor.pp

All VMs have a default set of alarms/metrics installed but you can add more, this class is here for this. If you want to install a new alarm/metric you need to pick the metric ID and include a new line with the new metric id. If you are installing a custom metric/alarm you should update https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocms/tree/qa/code/manifests/crab/schedd/directories.pp to include the code for new alarm/metric.

class hg_vocms::crab::schedd::monitor {

  # SLS monitoring
  lemon::metric {"13107":
    params => {
      'xml_location' => '/data/srv/service_report/',
    }
  }

  lemon::metric{'13242':} # glexec_sensor
  lemon::metric{'13246':} # htcondor_ads sensor
  lemon::metric{'13275':} # monitor the number of postjob
  lemon::metric{'13276':} # memory usage of postjob processes
  lemon::metric{'30134':} # monitor /data to prevent it to be fullfilled
  lemon::metric{'33636':} # monitor /home/grid to prevent it to be fullfilled
  lemon::metric{'13353':} # publish the maximun number of DAGs/Tasks per schedd

}

repos.pp


class hg_vocms::crab::schedd::repos {

  # Please keep this file empty and let the individual
  # modules set up their desired repositories to avoid
  # conflicts
  # htcondor-stable

}

rpms.pp

class hg_vocms::crab::schedd::rpms {
  
  $pkgs_admin = ['augeas']
  $pkgs_user = ['git']
  $pkgs_crab = [ 'gsi-openssh-server','argus-gsi-pep-callout','httpd','libconfuse']
  $pkgs_all = unique(flatten([$pkgs_admin,$pkgs_user,$pkgs_crab]))
  ensure_packages($pkgs_all)
  
}

services.pp

class hg_vocms::crab::schedd::services {


}

sudoers.pp

class hg_vocms::crab::schedd::sudoers {
  
  if ! defined(Sudo::Directive['belforte']) {
    sudo::directive {'belforte':
      ensure  => present,
      content => 'belforte ALL= (ALL)  NOPASSWD:  ALL',
    }
  }

  if ! defined(Sudo::Directive['jletts']) {
    sudo::directive {'jletts':
      ensure  => present,
      content => 'jletts ALL= (ALL)  NOPASSWD:  ALL',
    }
  }


}

tunning.pp

class hg_vocms::crab::schedd::tunning {

  # Making sure SE Linux is disabled
  # Added by Farrukh - Nov. 25, 2014
  exec { "setenforce 0":
    path => "/usr/sbin/:/bin/",
    command => "setenforce 0",
    unless => "sestatus | grep -q 'Current mode:.*permissive|SELinux.*disabled'",
  }

  # Make the ssh server listening on port 2222
  file_line { 'sshd_port_2222':
    path => '/etc/ssh/sshd_config',
    line => 'Port 2222',
    notify => Service["sshd"]
  }

}

volumes.pp

class hg_vocms::crab::schedd::volumes {

    # partition to be used for /home/grid
    exec {'/sbin/mkfs.ext4 -L "STANDARD-VOL" /dev/vdb':
      unless => '/sbin/blkid -t LABEL=STANDARD-VOL /dev/vdb'
    }

    # partition to be used by /data
    exec {'/sbin/mkfs.ext4 -L "HIGH-IO-VOL" /dev/vdc':
      unless => '/sbin/blkid -t LABEL=HIGH-IO-VOL /dev/vdc'
    }

    mount { "/mnt":
      ensure => absent,
    }

    file{"/data":
      ensure => directory,
      mode => 0775,
          owner => root,
          group => root,
    }
    file{"/home/grid":
      ensure => directory,
      mode => 0775,
          owner => root,
          group => root,
    }

    mount {"/data":
      device => 'LABEL="HIGH-IO-VOL"',
      fstype => "ext4",
      ensure => "mounted",
      options => "defaults",
      atboot => "true",
      require => File['/data']
    }

    mount {"/home/grid":
      device => 'LABEL="STANDARD-VOL"',
      fstype => "ext4",
      ensure => "mounted",
      options => "defaults",
      atboot => "true",
      require => File['/home/grid']
    }

}

Puppet first run

This will setup the machine, install condor and all default configs, create all users home directories under /home/grid, install cron jobs, install lemon alarms and sensors. Most probably you will need to run puppet at least 3 or 4 times until all dependent services get installed and configured. Please during the puppet run pay attention to all outputs.

sudo -s
puppet agent -tv

Once you dont get any puppet errors, create the /etc/grid-security/condorkey.pem, this file is necessary to allow the schedd to contact argus and get a grid user name for a particular user DN. Run the following command for that:

cp /etc/grid-security/hostkey.pem /data/srv/condorkey.pem.tmp; mv /data/srv/condorkey.pem.tmp /etc/grid-security/condorkey.pem; chown condor:condor /etc/grid-security/condorkey.pem

This command comes from the cron job installed by puppet,

HTCondor Configuration Details

All config files are located at /etc/condor/config.d/ folder but only 82_cms_schedd_crab_generic.config is specific for CRAB. Usually you dont need to touch these files and if you need to apply custom changes for a specific schedd you should write your changes on 99_local_tweaks.config and if after some time you decide that this custom change needs to be applied to all CRAB3 schedds is better to move the change to 82_cms_schedd_crab_generic.config and leave 99_local_tweaks.config much cleaner as possible.

82_cms_schedd_crab_generic:

########################################################
# HTCondor configurations specific to crab schedd

# Adding JobStatus to significant attributes to help Brian sort out
# idle jobs per user
# Adeed 2015-07-29
ADD_SIGNIFICANT_ATTRIBUTES=$(ADD_SIGNIFICANT_ATTRIBUTES),CRAB_UserHN

# Accounting group settings for crab
use_x509userproxy = true
accounting_group = analysis
SUBMIT_EXPRS = $(SUBMIT_EXPRS) use_x509userproxy accounting_group

# To cache Argus results and prevent GSI authorization callout
# related timeouts.
# Added by Farrukh - Nov. 25, 2014
GSS_ASSIST_GRIDMAP_CACHE_EXPIRATION=7200 

# Overwriting the system periodic remove expression for CRAB3
SYSTEM_PERIODIC_REMOVE = ((JobUniverse=!=7)&&(((NumJobStarts>9)=?=True)||((NumShadowStarts>19)=?=True)))
SYSTEM_PERIODIC_REMOVE = ($(SYSTEM_PERIODIC_REMOVE)) || ((DiskUsage>27000000)=?=True)

# Older version of Condor used a lower default
GSI_DELEGATION_KEYBITS = 1024

# Enable Condor-C full  delegation, 
# but keep the delegation to worker nodes limited and short
DELEGATE_FULL_JOB_GSI_CREDENTIALS = True
SHADOW.DELEGATE_FULL_JOB_GSI_CREDENTIALS = False
DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME = 0
SHADOW.DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME = 86400

# Whitelist the CRAB3 servers
SCHEDD.ALLOW_WRITE = */vocms052.cern.ch, */vocms045.cern.ch, */vocms0118.cern.ch, */vocms0119.cern.ch, */$(FULL_HOSTNAME)
SCHEDD.HOSTALLOW_WRITE =

# Limit the number of dagmans
START_SCHEDULER_UNIVERSE = TotalSchedulerJobsRunning < 250

Used_Gatekeeper = "$$(GLIDEIN_Gatekeeper:Unknown)"
JOB_CMSSite = "$$(GLIDEIN_CMSSite:Unknown)"
# the userlog will have Used_Gatekeeper defined at job runtime
# and MATCH_GLIDEIN_Gatekeeper at job termination
# but never both

JOB_Gatekeeper = ifthenelse(\
 substr(Used_Gatekeeper,0,1)=!="$", Used_Gatekeeper, \
   ifthenelse(\
       MATCH_GLIDEIN_Gatekeeper=!=UNDEFINED,MATCH_GLIDEIN_Gatekeeper,\
           "Unknown"))
SUBMIT_EXPRS = $(SUBMIT_EXPRS) JOB_Gatekeeper JOB_CMSSite Used_Gatekeeper

# this assumes noone else is defining the list
job_ad_information_attrs = MATCH_GLIDEIN_Gatekeeper, JOB_Gatekeeper,\
Used_Gatekeeper, JOB_CMSSite

# in the Condor logic, this is a user-provided attribute
# so tell the schedd to treat it as such
SUBMIT_EXPRS = $(SUBMIT_EXPRS) job_ad_information_attrs

# Disabling FSYNC() call for logs
# Added 2015-FEB-09 
CONDOR_FSYNC = false

# CRAB specific job throttle parameters
# Added 2015-FEB-09 
JOB_START_DELAY = 2
JOB_START_COUNT = 10

# Used by CRAB3 for HTTP access to log files
# Added 2015-FEB-16
CRAB_StorageRules = ^/home/grid,http://vocms0106.cern.ch/mon

HISTORY_HELPER=$(LIBEXEC)/condor_history_helper
# Unsetting this so it reverts to default of 10k
# ELOG: https://cms-logbook.cern.ch/elog/GlideInWMS/3413
#HISTORY_HELPER_MAX_HISTORY=1000000

# Unset on 2015-08-04
# libsnoopy.so has been updated, so no longer needed
# ELOG: https://cms-logbook.cern.ch/elog/GlideInWMS/2472 
# USE_CLONE_TO_CREATE_PROCESSES=false

# To help retrieve condor logs for CRAB3
KILLING_TIMEOUT = 300

# Enabling OVERFLOW for CRAB3 machines only
CMS_ALLOW_OVERFLOW = "True"

OVERFLOW_US = ifthenelse(regexp("T[1,2]_US_",DESIRED_Sites),"True",UNDEFINED)
OVERFLOW_IT = ifthenelse(regexp("T[1,2]_IT_",DESIRED_Sites),"True",UNDEFINED)
OVERFLOW_UK = ifthenelse(regexp("T2_UK_London_",DESIRED_Sites),"True",UNDEFINED)

DESIRED_Overflow_Region = strcat( ifthenelse(OVERFLOW_US=?="True","US", "none"), ",",ifthenelse(OVERFLOW_IT=?="True","IT", "none"), ",", ifthenelse(OVERFLOW_UK=?="True","UK", "none") )

SUBMIT_ATTRS = $(SUBMIT_ATTRS) OVERFLOW_US OVERFLOW_IT OVERFLOW_UK CMS_ALLOW_OVERFLOW DESIRED_Overflow_Region

# To monitor the number of overflow jobs
OVERFLOW_CHECK = ifthenelse(MATCH_GLIDEIN_CMSSite isnt undefined, ifthenelse(stringListMember(MATCH_GLIDEIN_CMSSite,DESIRED_Sites),FALSE,TRUE), FALSE)
SUBMIT_ATTRS = $(SUBMIT_ATTRS) OVERFLOW_CHECK

# Adding PER_JOB_HISTORY_DIR feature
PER_JOB_HISTORY_DIR=/data/srv/glidecondor/condor_local/job_history/

# Number of jobs Dagman will submit to condor_q
DAGMAN_MAX_SUBMITS_PER_INTERVAL = 100

99_local_tweaks.config

############################################
#
# 99_local_tweaks.config
# 
# This config file should be used for all manual changes
# on this machine. The other config files should not
# be modified, as they will be overwritten by puppet. 
# Include permanent changes in puppet, and temporary 
# changes or machine-specific changes here.
#
############################################

# To avoid spamming others while debugging issues
CONDOR_ADMIN = jadir.silva13@gmail.com

As was said before this file should be kept as cleaner as possible, but if you want to change one particular schedd please put your customizations here. Bellow you can find an example of custom schedd config.

############################################
#
# 99_local_tweaks.config
# 
# This config file should be used for all manual changes
# on this machine. The other config files should not
# be modified, as they will be overwritten by puppet. 
# Include permanent changes in puppet, and temporary 
# changes or machine-specific changes here.
#
############################################

# To avoid spamming others while debugging issues
CONDOR_ADMIN = jadir.silva13@gmail.com

SCHEDD_NAME = crab3@vocms0106.cern.ch

SCHEDD.ALLOW_WRITE= $(SCHEDD.ALLOW_WRITE), balcas-crab2.cern.ch, */brazil.accre.vanderbilt.edu

# changed on Jun 21, https://cms-logbook.cern.ch/elog/Analysis+Operations/956 
START_SCHEDULER_UNIVERSE = (TotalSchedulerJobsRunning < 300) && ((ShadowsRunning is undefined) || (ShadowsRunning < .95*$(MAX_JOBS_RUNNING))) 

# debug configs
SHARED_PORT_DEBUG = D_ALL:2

MAX_JOBS_RUNNING = 12000

In this case we have this configurations

knob Effect
CONDOR_ADMIN overwrite the default condor admin email, useful during the tests to avoid flooding the others operators inbox with debugging msg
SCHEDD_NAME choose a different schedd name, the current naming convertion is crab3@$(FULLHOSTNAME)
SCHEDD.ALLOW_WRITE useful for developers to allow this schedd to work with non default TW
START_SCHEDULER_UNIVERSE define how many DAGs the schedd can run
SHARED_PORT_DEBUG enable full debugging for shared_port daemon
MAX_JOBS_RUNNING define how many jobs the schedd can run

There are plenty of possibilities here, if you want to see more options go to HTCondor configurations page http://research.cs.wisc.edu/htcondor/manual/v8.5/3_3Configuration.html and have fun.

Configurations changes on Central Managers and Frontends

All schedds needs to be added to Global Pool central managers and Frontends and also to tier0 Central Managers and Frontends for that is sufficient to create an elog on https://cms-logbook.cern.ch/elog/GlideInWMS with the following content:

Please add the schedd <your-brand-new-vm>.cern.ch to global pool central managers and frontends and also to tier0 central managers and frontends.

Cheers,
Your Name here

Configurations changes on LPC firewall

Since now all scheeds run job router they need to be whitelisted in the FNAL firewall, so write a email to Krista<klarson1@fnal.gov> and Tony<tiradani@fnal.gov> asking then to include the new schedd on the firewall whitelist.

Redirection rules for CMSWEB

This is needed to allow users outside CERN to access log files on CERN schedds. For every new schedd you should add a rule on https://github.com/dmwm/deployment/blob/master/frontend/backends-prod.txt.

^/auth/complete/dqm/(?:online|online-playback|online-test)(?:/|$) cmsdaq0.cern.ch
^/auth/complete/dqm/offline(?:/|$) vocms0138.cern.ch
^/auth/complete/dqm/relval(?:/|$) vocms0139.cern.ch
^/auth/complete/dqm/(?:dev|offline-test|relval-test)(?:/|$) vocms0131.cern.ch
^/auth/complete/couchdb2/asynctransfer1(?:/|$) vocms0306.cern.ch  :affinity
^/auth/complete/(?:aso-monitor|couchdb2)(?:/|$) vocms0141.cern.ch :affinity
^/auth/complete/(?:couchdb/wmstats|wmstats)(?:/|$) vocms0307.cern.ch :affinity
^/auth/complete/(?:c?)(?:couchdb|workqueue|workloadsummary|alertscollector|analysis_workqueue|analysis_wmstats|tier0_wmstats|t0_workloadsummary|acdcserver)(?:/|$) vocms0140.cern.ch :affinity
^/auth/complete/das(?:/|$) vocms0141.cern.ch|vocms0307.cern.ch :affinity
^/auth/complete/crabcache(?:/|$) vocms0141.cern.ch
^/auth/complete/confdb(?:/|$) vocms0307.cern.ch
^/auth/complete/wmarchive(?:/|$) vocms071.cern.ch
^/auth/complete/scheddmon/021/ vocms021.cern.ch
^/auth/complete/scheddmon/095/ vocms095.cern.ch
^/auth/complete/scheddmon/096/ vocms096.cern.ch
^/auth/complete/scheddmon/0106/ vocms0106.cern.ch
^/auth/complete/scheddmon/0109/ vocms0109.cern.ch
^/auth/complete/scheddmon/0112/ vocms0112.cern.ch
^/auth/complete/scheddmon/0114/ vocms0114.cern.ch
^/auth/complete/scheddmon/066/ vocms066.cern.ch
^/auth/complete/scheddmon/059/ vocms059.cern.ch
^ vocms0136.cern.ch|vocms0161.cern.ch|vocms0163.cern.ch|vocms0165.cern.ch

to proceed you need to fork https://github.com/dmwm/deployment repo and then clone your forked repo and create a pull request with your changes.

git clone https://github.com/jmarra13/deployment.git
cd deployment
git remote -v
git remote add upstream https://github.com/dmwm/deployment
git remote -v

In this example I cloned my own fork, you should adjust the url https://github.com/jmarra13/deployment.git to your own.

git checkout -b redirect-rule-vocms0121
vim frontend/backends-prod.txt

Then add a line with the redirect rule, like this one:

^/auth/complete/scheddmon/0121/ vocms0121.cern.ch

then save the file and push your changes to your newly created branch.

git add frontend/backends-prod.txt
git commit -m 'some description of your changes'
git push origin redirect-rule-vocms0121

Then go to your fork, in my case https://github.com/jmarra13/deployment, and click on pull_request_test_highlighted.png

-- JadirSilva - 2016-08-09

Edit | Attach | Watch | Print version | History: r17 < r16 < r15 < r14 < r13 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r17 - 2016-10-21 - JadirSilva
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback