WMAgent Resource Control

User's Manual

The tool for users to manage the resources in the agent is the wmagent-resource-control script.

Its usage can be obtained by doing:

$manage execute-agent wmagent-resource-control -h
Executing wmagent-resource-control -h ...
Usage: wmagent-resource-control [options]

Options:
  -h, --help            show this help message and exit
  -p, --thresholds      Print out all known thresholds and site information.
  --add-T1s             Add all of the CMS T1 sites to resource control.
  --add-T2s             Add all of the CMS T2 sites to resource control.
  --add-all-sites       Add all of the CMS sites to resource control.
  --site-name=SITENAME  Specify the unique name of the location
  --pending-slots=PENDINGSLOTS
                        Specify the maximum number of pending slots to use at
                        the site
  --running-slots=RUNNINGSLOTS
                        Specify the maximum number of running slots to use at
                        the site
  --ce-name=CENAME      Specify the CEName for the site
  --se-name=SENAME      Specify the SEName for the site
  --cms-name=CMSNAME    Specify the name of the site in SiteDB
  --task-type=TASKTYPE  Specify the name of the task to add/modify
  --task-slots=MAXSLOTS
                        Specify the maximum number of slots for the task type
  --priority=PRIORITY   Specify the priority of the threshold at the site
                        (higher is better)
  --plugin=PLUGIN       Specify submit plugin to use for specific site
  --drain               Drain the site.
  --finalize            Finalize workflows at the site.
  --down                Stop submission to the site.
  --normal              Normal operation for the site.

Common use cases are:

  • Add all T1 sites (according to the information in SiteDB) with default thresholds (10 pending slots, 10 running slots and 10 max slots for each task type) and condor submission (plugin must be specified):
         $manage execute-agent  wmagent-resource-control --add-T1s --plugin=CondorPlugin
         
  • Change the pending slots of a site to allow more jobs to be submitted:
         $manage execute-agent wmagent-resource-control --site-name=T1_US_FNAL --pending-slots=2000
         
  • Remove the limit on running processing jobs of a site
         $manage execute-agent wmagent-resource-control --site-name=T1_US_FNAL --running-slots=-1
         $manage execute-agent wmagent-resource-control --site-name=T1_US_FNAL --task-type=Processing --task-slots=-1
         
  • Suspend submission of Production jobs to a site
         $manage execute-agent wmagent-resource-control --site-name=T1_US_FNAL --task-type=Processing --task-slots=0
         
  • Set site to drain state
         $manage execute-agent wmagent-resource-control --site-name=T1_US_FNAL --drain
         
  • Set site to normal operation
         $manage execute-agent wmagent-resource-control --site-name=T1_US_FNAL --normal
         

System description

In this section we describe the role of the different variables in the resource allocation schema of the agent.

Input information

Each agent holds a local SQL database which contains the information about the resources for every site it can submit work to. This information is divided in two tables:

Locations (wmbs_location)

This table contains the general information about each site the agent knows of, it has the following columns related to resource allocation:

  • pending_slots : An integer which defines the maximum number of pending jobs.
  • running_slots : An integer which defines the maximum number of running jobs. A negative number indicates no limit.
  • state : A string which describes the current state of the site, can be one of the following: Normal, Draining, Finalizing, Down.

Resource control (rc_threshold)

This table contains thresholds and priorities for each site and type of task:

  • max_slots : An integer which defines the maximum number of running jobs for each task. A negative number indicates no limit.
  • priority : Priority of each type of task at the time of submission.

Operation

The two components involved in the resource allocation decisions are the JobSubmitter and the WorkQueueManager. First we describe the WorkQueueManager's operation in the different modes and thereafter the JobSubmitter's.

WorkQueueManager

The WorkQueueManager takes decisions based on the resource information in two phases of its operation:

  • To pull new elements of work from the workqueue, which can be from a new request or more input data from an existing workflow in the agent.
  • To inject files from obtained elements into the local SQL database, this allows the JobCreator component to create jobs and therefore start processing the acquired work.

For each of these, the WorkQueueManager obtains information from the database about the resources available in each site and aggregates this information before deciding whether to inject/pull more work.

  • In the first case, only pending_slots is considered but without diminishing by the jobs already pending in the agent. However it takes into account the work available in the local queue which has not been injected into the local SQL database.
    • Example: A request with site A as its whitelist was divided in 4 elements with 20 jobs each in the global workqueue, the pending_slots for site A is 40 and there are already 40 jobs pending in the local batch system. Then, in each polling cycle, the WorkQueueManager will fetch 1 element of work. This is because the number of pending jobs in the batch system doesn't matter and the WorkQueueManager only consider half of the pending_slots for each site in this case. However if there is already an element of work with 25 jobs in the local queue in available state, which means it has not been injected into the SQL database, no more work will be pulled for the site until that element is injected.
  • In the second case, only pending_slots is considered but in this case the number of jobs that are not running in the batch system is subtracted from this number, note that "jobs that are not running..." includes not only pending jobs, but jobs which have not been submitted and jobs in cooloff.
    • Example: The local workqueue holds an element of work with 15 jobs that can run at site A, the pending_slots for site A is 20 and there are 12 jobs not running which can run at site A. Then, the element would be injected because there are still 3 pending slots available that the new element can start using. Note that we don't have to wait to 15 jobs to be available, but if there was another element of work it would not be injected because the first element occupied those 3 slots.

Now we describe the differences for each state of the site:

Normal state

The site's resources are considered for pulling work from new and existing requests, and also for injection of elements into the database.

Draining state

The site's resources are considered for pulling work only on existing requests and for injection of elements into the database

Finalizing, Down states

The site's resources are ignored for pulling or injecting work.

JobSubmitter

In each polling cycle, the JobSubmitter polls the database for created jobs and submits them to the different sites based on the available resources and the defined thresholds.

A simplified version of the algorithm is:

  1. Fetch new jobs from the database
  2. Apply site white and black lists to determine the possible locations
  3. Obtain the information for all sites, this includes:
    • Number of jobs pending and running for each type of task
    • Pending and running slots
    • Site state
    • Maximum slots for each type of task
  4. This information is aggregated by site and task, each site has a list ordered according to the priority of each task. Afterwards, the list of sites is ordered by the following criteria:
    • First T1s (or T0 if available), T2s and T3s.
    • Inside these categories, the sites are ordered according to the number of pending slots
  5. For each site do the following:
    1. For each task type do the following:
      1. If the number of running jobs for the site is higher than the running slots, then skip the site. Unless running_slots is negative, in which case there is no limit on running jobs for the site
      2. If the number of running jobs for the given task is higher than the running slots for the task in this site, then skip this task type. Unless max_slots is negative, in which case there is no limit on running jobs for the task.
      3. The maximum number of jobs that can be submitted for the task is equal to the number of pending slots minus the number of pending jobs for the site
      4. For each job that matches the task and site:
        1. Add it to the list of jobs to be submitted.
        2. Add one to the number of pending jobs for the site.
        3. Stop if there are no more jobs for this task and site or when the maximum number of allowed jobs defined in the previous step was reached.

Now we present how the different states affect this algorithm:

Normal state

The algorithm is as described above.

Draining state

In step 2, if a job can run in at least one location in Normal state then a site in Draining state will be removed from its possible locations.

Finalizing state

Only job from merge, cleanup and logCollect tasks will be submitted to the site.

Down state

No job will be submitted to site.

-- DiegoBallesterosVillamizar - 26-Jun-2012

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2012-07-02 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback