Disk Server Test System using 'xrootd'

Quickstart

Installation on Client - Master - Evaluation Machine

# one SL5 machines
wget https://twiki.cern.ch/twiki/pub/DataManagement/XDiskServer/xrootd-ds-test-1.0.0sl5-1.x86_64.rpm
rpm -i xrootd-ds-test-1.0.0sl5-1.x86_64.rpm

# on SL4 machines
wget https://twiki.cern.ch/twiki/pub/DataManagement/XDiskServer/xrootd-ds-test-1.0.0sl4-1.x86_64.rpm
rpm -i xrootd-ds-test-1.0.0sl4-1.x86_64.rpm

Client(s) - Start Test

cd /opt/xrootd/ds-test
./ds-client <master-ip> 1

Master - Start Test

cd /opt/xrootd/ds-test
# now edit ds-config and set your ethernet device and filesystems correctly here - don't change timegap
./ds-master
# the first run takes very long since it will create 100 files of the size of the memory
# wait until the test is over and an archive file ds-test-<hostname>-<timestamp>.tgz

Evaluation of Test

# copy/use the lastest created  ds-test-<hostname>-<timestamp>.tgzfile created on the master machine
/opt/xrootd/parse-ds.pl ds-test-<hostname>-<timestamp>.tgz mytest
# find the peformance plots as /tmp/ds-box/mytest/*.gif

Introduction

This test system can be used to evaluate disk server and network performance using a single disk server and 1-4 machines acting as clients. The server is running a standard 'xrootd' daemon, the clients use 'TestXrdClient_read' which is part of the 'xrootd' package. This client allows to 'play' certain read or write patterns towards a disk server. The disk server runs the 'master' process, which creates testfiles for 'reads' on defined filesystems and tracks the network, disk and cpu utilization on the disk server during a test. The 'master' process starts with a complete flush of the buffer cache to guarantee identical start conditions. Afterwards it set's a 'run' file which signals the 'client' processes to start to play synchronized a predefined test pattern. Once the test pattern has finished the master process creates a tar ball containing all the test information and configuration files.

Setup Requirements & Software Location

One disk server and 1-4 client machines. Installation of the XDiskServer RPM on all machines. TCP port 2000 open on disk server machine for incoming connections. The disk server needs 'vmstat' & 'iostat' installed. The RPM installs files under /opt/xrootd. All files referenced in the following can be found in /opt/xrootd/ds-test. You should make this directory the working directory for the master and client processes. The RPMs require procps and sysstat and ethtool (vmstat,iostat,ethtool).

Master Configuration

The master 'process' ds-master reads two configuration files:

• ds-config
• ds-pattern

ds-config

This file defines the filesystems and devices to use for a test. It can be understood easily with this example:

filesystem /cfs/fs80/dir1
filesystem /cfs/fs80/dir2
filesystem /cfs/fs80/dir3
filesystem /cfs/fs80/dir4
iodevice sdb
iodevice sdc
iodevice sdd
iodevice sde
netdevice eth0
timegap 30
nclients 4

Each filesystem, iodevice or netdevice has to be listed in a seperate line . The number of clients used has to be set correctly otherwise cache effects are not compensated properly. The timegap value defines the running time of a single test pattern. This interval is automatically multiplied by the number of clients! Good values are 120s for 1 client and 30s for 4 clients. If you use a 'sync' pattern in your pattern configuration file, the timegap must be long enough to allow to write flush files of the size of the machine memory and to sync the disk. If you set the timegap to low the test program will discover this during the initial flush operation and will terminate with an error message asking you to increase the timegap value.

ds-pattern

This file contains the test pattern to run. An example test pattern is shown here for easy understanding:

#################################################################################################################################
# <testname> <pattern for read> <# reader> <chunksize> <pattern for write> <#writer> <chunksize> 
#################################################################################################################################

# Valid patterns are: 'sequential','random','sparse'
# <#reader|writer> must be an integer number in the range 1 - 1024
# [chunksizes] are given in kb
###################################################################
# read-only sequential
sr25    sequential      25      128     sequential      0       128
sr10    sequential      10      128     sequential      0       128
sr1     sequential      1       128     sequential      0       128
###################################################################
# write-only sequential
sw1     sequential      0       128     sequential      1       128
sw10    sequential      0       128     sequential      10      128
sw25    sequential      0       128     sequential      25      128
###################################################################
#C Sequential Read Patterns

It is important for the later evaluation macro that the testnames are unique. The useable patterns are sequential, sparse and random. In some cases you have to choose a pattern sequence which guarantees that there are no caching effects in consecutive tests. If you want to make sure that there are no caching effects you can add a line like that:

sync1  sync            0       0       sync            0       0

This will generate a complute flush of the buffer cache and sync the disks afterwards.
Lines starting with '#' or empty lines are ignored. Lines starting with ' #C <text> ' have only a meaning for the evaluation macro. All tests before this pragma are put into one plot and labeled with <text >. Test names starting with 'flush' will not be shown in the graphical representation and can be inserted to flush the buffer cache on server side doing sequential writes.

Startup Procedure

You should first startup the clients, configure the master and then start the master process. The clients wait until the master process signals the start of the test and they will download the current configuration for the test pattern to be used. When the test pattern is completed, master and client(s) will terminate automatically.

Master Startup

The master is started via:

./ds-master

Client Startup

The clients are started via:

./ds-client <master-hostname> <client-id>

<master-hostname> is the host name where the master process will be/is running. <client-id> must be unique number in the range [1-4]. If you use only a single client the id to be used is 1.

Test Results & Evaluation

Result Files

In the working directory of the master process there are four files produced:

• ds-info
• ds-server.out
• ds-test-<hostname>-<unixtime>.tgz
• ds-master.log
  

ds-info contains information about the configuration of the master server (cpu/memory/disk/fs/raid/network etc.) ds-server.out contains the 1Hz samples of network, cpu & disk activity during the test processing. ds-test-<hostname>-<unixtime>.tgz contains for each test ds-config, ds-pattern, ds-info, ds-server.out in a zipped tar archive. ds-master.log contains log information about the test progress. The STDOUT of the ds-master process displays the time left for the test to finish.

Evaluation

You need any ROOT version from http://root.cern.ch to run the evaluation macro 'parser.C'. If you don't have any ROOT in the path, but you have CERN AFS available, the perl evaluation script parse-ds.pl will use ROOT 5.22 from CERN AFS.
Run the evaluation script:

/opt/xrootd/ds-test/ds-parse.pl  ds-test-<hostname>-<timestamp>.tgz<testtag>

1st argument is the archive file which was created in the working directory of the ds-master process. 2nd argument is used as a label for the histograms produced by the test and as the output box directory.

The macro will print out for each analyzed test some average values of interest:

=============================================================================
Test 005 [0-25] : sw25  sequential      0       128     sequential      25      128
=============================================================================
eth-out:         2.56 Mb/s       eth-in:         118.40 Mb/s
 io-read:        0.05 Mb/s       io-write:       112.72 Mb/s
cpu-usr:         0.26%           sys:            11.42%          wait: 3.76%

If you just use the default pattern the macro will create 3 x 2 GIF images containing a graphical representation about the 3 different test pattern:

• sequential tests ds-disk-mytest-0.pdf ds-cpu-mytest-0.pdf
  
• sparse tests ds-disk-mytest-1.pdf ds-cpu-mytest-1.pdf
• random tests ds-disk-mytest-2.pdf ds-cpu-mytest-2.pdf

These files can be found in the output box directory of the parsing script /tmp/ds-box/<testtag>/

ds-disk histogram

Example histogram for the sequential pattern tests.

The x-axis shows the short lable of each test. The y-axis shows:

• blue line: avg. disk read rate in MB/s
• blue markers: avg. network output rate in MB/s
• red line: avg. disk write rate in MB/s
• red markers: avg. network input rate in MB/s
• green line: avg. disk utilization in % as reported by iostat
  

ds-cpu stacked histogram

Example histogram for the sequential pattern tests.

This histogram shows for each test pattern the average user(green), system(blue) and iowait (red) CPU time as a stacked histogram.

10 Gbit / 1Gbit Test Results

This plots show results from 1-1 tests (1 client - 1 server) on a 10 Gbit test setup and 1-1 tests on a 1 Gbit it test setup. The 10 GBit disk server was configured with 4 seperate RAID-5 file systems with 3 disks, while the 1 GBit disk server had a software RAID0 over 4 RAID-5 Arrays with 5 disks each.

The used default test pattern is:

##################################################################
# read-only sequential
# write-only sequential
sr25    sequential      25      128     sequential      0       128
sw25    sequential      0       128     sequential      25      128

sync1   sync            0       0       sync            0       0
sr10    sequential      10      128     sequential      0       128
sw10    sequential      0       128     sequential      10      182

sync2   sync            0       0       sync            0       0
sr1     sequential      1       128     sequential      0       128
sw1     sequential      0       128     sequential      1       128
###################################################################
# read-write sequential
sync3   sync            0       0       sync            0       0
sm25-1  sequential      25      128     sequential      1       128
sm10-1  sequential      10      128     sequential      1       128
sm1-1   sequential      1       128     sequential      1       128
#
sm25-10 sequential      25      128     sequential      10      128
sm10-10 sequential      10      128     sequential      10      128
sm1-10  sequential      1       128     sequential      10      128
#
sm25-25 sequential      25      128     sequential      25      128
sm10-25 sequential      10      128     sequential      25      128
sm1-25  sequential      1       128     sequential      25      128
#C Sequential RW
###################################################################
# sparse read + sequential write
sync4   sync            0       0       sync            0       0
sp25-0  sparse          25      128     sequential      0       128
sp25-25 sparse          25      128     sequential      25      128
sp25-1  sparse          25      128     sequential      1       128
sp25-10 sparse          25      128     sequential      10      128
sync5   sync            0       0       sync            0       0
sp10-0  sparse          10      128     sequential      0       128
sp10-25 sparse          10      128     sequential      25      128
sp10-1  sparse          10      128     sequential      1       128
sp10-10 sparse          10      128     sequential      10      128
sync6   sync            0       0       sync            0       0
sp1-0   sparse          1       128     sequential      0       128
sp1-25  sparse          1       128     sequential      25      128
sp1-1   sparse          1       128     sequential      1       128
sp1-10  sparse          1       128     sequential      10      12
###################################################################
# sparse read varying block sizes
wq10-512        sequential      0       256     sequential      10      512
sp10-256        sparse          10      256     sequential      0       256
wq10-256        sequential      0       256     sequential      10      256
sp10-128        sparse          10      128     sequential      0       128
wq10-128        sequential      0       128     sequential      10      128
sp10-64 sparse          10      64      sequential      0       64
wq10-64 sequential      0       64      sequential      10      64
sp10-32 sparse          10      32      sequential      0       32
wq10-32 sequential      0       32      sequential      10      32
sp10-4  sparse          10      4       sequential      0       4
wq10-4  sparse          0       4       sequential      10      4
#C Sparse RW
###################################################################
# random read only
sync7   sync            0       0       sync            0       0
rd25-0  random          25      128     sequential      0       128
rd10-0  random          10      128     sequential      0       128
rd1-0   random          1       128     sequential      0       128
###################################################################
# random write only
rd0-25  random          0       128     random          25      128
rd0-10  random          0       128     random          10      128
rd0-1   random          0       128     random          1       128
###################################################################
# random read + sequential write
rd1-25  random          1       128     sequential      25      128
rd1-10  random          1       128     sequential      10      128
rd1-1   random          1       128     sequential      1       128
rd10-25 random          10      128     sequential      25      128
rd10-10 random          10      128     sequential      10      128
rd10-1  random          10      128     sequential      1       128
rd25-25 random          25      128     sequential      25      128
rd25-10 random          25      128     sequential      10      128
rd25-1  random          25      128     sequential      1       128
##################################################################
sync8   sync            0       0       sync            0       0
# random read varying block sizes
rd10-256        random          10      256     sequential      0       128
rd10-128        random          10      128     sequential      0       128
rd10-64 random          10      64      sequential      0       128
rd10-32 random          10      32      sequential      0       128
rd10-4  random          10      4       sequential      0       128
###################################################################
#C Random RW

10 Gbit lxfsrl4701

Sequential

Sparse

Random

1 Gbit lxfsrd1804

Sequential

Sparse

Random

-- AndreasPeters - 18 Feb 2009

Downloads

• xrootd-ds-test-1.0.0sl4-1.x86_64.rpm: SL4 RPM

• xrootd-ds-test-1.0.0sl5-1.x86_64.rpm: SL5 RPM

• xrootd-ds-test.tgz: Archive containing Test & Analysis Scripts (without xrootd binary)