xrootd scalability tests

All the test plans and results of the xrootd scalability test will be reported in this wiki page.

Test plans

first series of tests

Test results :

Test Series 1: write <N> clients concurrently 1 Mb files with xrdcp (100 clients per client host) using no authentication mechanism

In Test Series 1 we have deployed CVS version 08.01.2008 with one manual bug fix in XrdCmsClientMan.
We use 2000 clients (20 hosts a 100 clients).
The setup needs 2 RPMs and one configuration file on each machine:

xrootd.fslib /opt/xrootd/lib/libXrdOfs.so
all.export /xns/ r/w

if named server
oss.localroot /var/spool/xroot/
oss.cache public /shift/xns/

sec.protocol /opt/xrootd/lib unix
all.role server
all.role manager if lxb8971.cern.ch

all.manager lxb8971.cern.ch:1213

all.adminpath /var/spool/xroot/

cms.allow host *.cern.ch
cms.delay startup 1 suspend 10
cms.space linger 0 recalc 10 min 1g 2g
cms.request repwait 5

xrd.sched mint 8 maxt 1024 avtl 128 idle 256

#xrootd.async off

oss.fdlimit 16384 32768

The limitiation to 270 files/s is induced by the forced delay for every file creation of 5s per write. Theoretical maximum is for this test 400 files/s.

Test Series 2: read <N> clients concurrently 1 Mb files with xrdcp (100 clients per client host) using no authentication mechanism

Same configuration as for write, but reading back all files this time with 1000 clients .
We can read at a maximum rate of 1200 files/s. The performance was decreasing in further tests to 800 files/s (due to limitations on the disk server (random) io bandwidth ?).
The file locations are in this case already cached in the redirector memory.

The average time to read a file is 1.128 s per 1Mb file. The tail to longer transfer times is quite steep and narrow.

Test Series 2: tests with xrootd and namespace plugin + kerberos authentication

We have configured on the head node a catalogfs plugin. The namespace is stored in an XFS filesystem, which was created inside a memory file residing inside a tmpfs filesystem.
For a reallive setup we would need a SSD or Flash drive. File metadata (like user/group/size/permissions/mtime etc.) is stored as extended attributes on the XFS filesystem.

The kerberos authentication rate is 25 authentications/s. E.g. for xrdcp we can handle 25 requests per second with 10% usage on the head node. Unfortunately the authentication plugin has a kind of
global mutex, therefore we see a hard limit although there is plenty of CPU left.
Using ROOT applications as clients (where one has only 1 authentication in the beginning) we can handle 170 reads/s with a full security framework. Disabling headnode to diskserver token signing we can reach 900 read/s. Although in this encryption is a global mutex which could be removed after careful modification of the code.
A single ROOT client can open with the full authentication/authorization chain 50 files/s. WIthout head-node to disk-server signature 100 files/s.
The CPU on the headnode never exceeds 20%.
The inmemory filesystem was limited to 2GB which could store 435.000 files.

Test Series 3: kerberos authentication measurements

8 individual clients -> 8 server on 8core machine
0-copy-operation no auth: 8 x 40 files/s = 320 files/s - server 7% CPU
0-copy-operation auth :8 x 5 files/s = 40 files/s - server 80% CPU

8 individual clients -> 1 server on 8core machine
0-copy-operation no auth: 8 x 40 files/s = 320 files/s - server 7% CPU
0-copy-operation auth : 8 x 3 files/s = 24 files/s - server 14% CPU

There is a very restrictive mutex in the code for the krb5 authentication which results that for 1 and <n> clients the rate is the same since the requests are serialized by the mutex.
The high overhead for 8 servers comes from access to the same replay cache in /var/tmp/rc_host_0.

Test Series 4: GSI authentication measurements

8 individual clients-> 8 server on 8core machine
0-copy-operation auth : 8 x 7 files/s = 56 files/s - server 60% CPU

8 individual clients-> 1 server on 8core machine
0-copy-operation auth : 8 x 1 file/s = 8 files/s - server 32% CPU

1 client -> 1 server on 8core machine
0-copy-operation auth: 1 x 7 files/s = 7 files/s - server 9% CPU

Follow-up on Kerberos V authentication

The kerberos performance is limited by the use of the replay cache file which resides in /var/tmp/rc_host_0 (for root). With concurrent authentication the size of the replay cache file grows
until the lifetime window of tickets is reached (2min?). Individual processes share the same replay cache file - therefore the authentication rate does not scale with an increase of individual processes handling authentication (not threads!). Doing a trick in the code and setting an individual cache for each process (additionally deleting the cache after each authentication) allowed
280 krb5 authentications/s. I couldn't find out how to use a memory based cache in MIT krb5.

Plain rate Test with default xrootd setup (one XFS filesystem 4.5 TB - 24 disk server 60 client machines)

Write: 2.5 GB/s (600 clients)
Read: 1.9 GB/s (600 clients)
Read + Write combined: Read ~1 GB/s Write ~2.3 GB/s (600 + 600 clients)



-- LanaAbadie - 28 Feb 2008

Topic attachments
I Attachment History Action Size Date Who Comment
JPEGjpg read-cpu-iowait.jpg r1 manage 55.4 K 2008-03-11 - 17:00 AndreasPeters  
JPEGjpg read-write-rw-xrootd.jpg r1 manage 65.5 K 2008-03-11 - 16:43 AndreasPeters  
JPEGjpg read-write-xrootd.jpg r1 manage 45.8 K 2008-03-11 - 16:43 AndreasPeters  
JPEGjpg read-write-xrootd2.jpg r1 manage 46.9 K 2008-03-11 - 16:52 AndreasPeters  
JPEGjpg readratex.jpg r1 manage 20.1 K 2008-02-28 - 13:57 AndreasPeters  
JPEGjpg transfertimex.jpg r1 manage 39.3 K 2008-02-28 - 14:00 AndreasPeters  
JPEGjpg writerate.jpg r1 manage 26.2 K 2008-02-28 - 13:50 AndreasPeters  
Microsoft Word filedoc xrootd_scalability_test_plan_v3.doc r1 manage 30.5 K 2008-02-28 - 11:37 LanaAbadie  
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2008-03-11 - AndreasPeters
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback