Experimental Throttles for Xrootd

We have been developing throttles for Xrootd, a capability commonly requested by sysadmins. This page covers installation and configuration of the throttles.

While this code has been tested at a production site, it is not yet part of any Xrootd release; we cannot provide any guarantees of correctness.

Installation

The latest build is located here:

http://koji.hep.caltech.edu/koji/buildinfo?buildID=933

Download and install the xrootd-server, xrootd-client, and xrootd-libs RPMs. Alternately, you can add the nebraska-testing yum repository to your node:

http://t2.unl.edu/store/repos/nebraska/6/nebraska-el6-testing/x86_64/

This RPM follows the developer versioning, not the release versioning. To switch back to an official release, you will need to completely remove the RPMs.

Configuration

To enable the Xrootd plugin, you must load it as the primary filesystem library. Add the following line to your configuration:

xrootd.fslib ? /usr/lib64/libXrdThrottle.so

The following directives control the behavior of the throttle plugin.


throttle.throttle [data dlimit] [concurrency climit] [iops ilimit]

Each option in this directive controls a different aspect of the throttling:

  • dlimit: The maximum number of bytes per second the server may read and write to clients. You may use the suffixes "k", "m", or "g" to indicate values in kilobytes, megabytes, or gigabytes, respectively.
  • ilimit: The maximum number of read or write operations per second. A read() call is counted as one read; a readv() call with 50 chunks is counted as 50 reads.
  • climit: The maximum number of IO operations in-progress at any given time.
  • A read or write counts equally toward the limit; for example, if dlimit is set to 10M, clients may perform 5MB/s of reads and 5MB/s of writes, but not 10MB/s of reads and 10MB/s of writes.
  • For the dlimit and ilimit limits, a fairshare is applied. Clients mapped to the same username utilize the same fairshare. If Bob runs 50 clients and Alice runs 1 client, Bob and Alice should still get the same amount of aggregate bandwidth.
  • Typically, there is no good way to determine the correct value of ilimit; most sites will really care about the climit, as that is a more direct measure of filesystem load. When the filesystem starts slowing down due to overload, this will eventually increase up to the number of connected clients.
    • This is roughly analogous to the Linux "load" metric as reported by "uptime".

Example usage:

throttle.throttle data 25M concurrency 10


throttle.loadshed [host hostname] [port  portnum ] [frequency  freq ]

This directive controls the behavior of the throttle plugin when the limits in the throttle.throttle directive have been hit. It has the ability to redirect clients to another server (such as a redirector) as opposed to servicing the read or write request.

Notes:

  • hostname and portnum: The client will be redirected to this hostname and port. The default for portnum is 1094.
  • freq: The frequency of client redirects. This is the percent chance the client will be redirected per 100MB of data read while the server is shedding load. For example, if this is set to 10 and the client performs a 100MB read when the server is under heavy load, there is a 10% chance the read will end in a redirection.
  • Not all clients handle a redirect response after a read request. In particular, it appears the ROOT client respects this while xrdcp does not.
  • In order to prevent loops, throttle.loadshed=1 is added to the client's opaque info and the throttle will not loadshed any client with this directive present.

Example usage:

throttle.loadshed host xrootd.unl.edu port 1094 frequency 10


throttle.trace [loadshed] [ioload] [bandwidth] [debug]
This directive controls the logging behavior of the throttle. In general, each option prints out a status report per second. When combined with debug, pertinent debugging information is printed out once per IO request.

Notes:

  • loadshed: Note when a client is loadshed.
  • ioload: Print out load summaries once a second.
  • bandwidth: Print out bandwidth usage summaries once a second.
  • iops: Print out information about the IO operations per second.
  • debug: Print out sufficient debugging information to determine how throttle calculations are done for each request.

Full Throttle

This is the configuration used at Nebraska for testing the throttles

xrootd.fslib ? /usr/lib64/libXrdThrottle.so
throttle.throttle data 25M concurrency 10
throttle.loadshed host xrootd.unl.edu port 1094 frequency 10
throttle.trace loadshed ioload

Typically, a site will only want to implement a concurrency limit. Bandwidth limits are usually only useful if you want to limit Xrootd under 1Gbps.

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r2 - 2013-02-13 - BrianBockelman
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback