Transfer Load Generator

The latest test results, for FTS v. 2.2, are at LCG/Fts22Testing.

Configuration Files

properties.xml

Configuration Parameters

log-properties

submissions.xml

@configDir

Xml file containing endpoints, channels and submissions information.

<?xml version="1.0"?>
<submissions>
  <!-- List of endpoints -->
  <endpoints>
    <data>        
      <name>cern</name> <!-- the name must be unique -->
      <endpoint>
        <host>srm-dteam.cern.ch:8443/srm/managerv2</host>
        <version>2.2</version>
        <source path="/castor/cern.ch/c3/fts-test/source/castorpublic">rand%{random(1,50)}.dat</source>
        <dest path="/castor/cern.ch/c3/fts-test/dest/castorpublic/fts-load-generator">%{date}/%{guid}.dat</dest>
      </endpoint>
    </data>
  </endpoints>
  <!-- List of channels -->
  <channels>
    <channel name="CERN-CERN"> <!-- channel name is used to count active transfers (don't overload channels) -->
      <loads>
        <load name="cern_load_1"> <!-- load name is used just for logging -->
          <source endpoint="cern"/> <!-- must match the name of one of the endpoints in the endpoints list -->
          <dest endpoint="cern">%{date}/%{random(1,10)}/%{guid}.dat</dest> <!-- specifying a new value overrides the default one specified in the endpoints list -->
        </load>
      </loads>
    </channel>
  </channels>
</submissions>

In the example above, we define one endpoint, "cern", and one channel, "CERN-CERN", with one submission, "cern_load_1".

The submitted job for cern_load_1 in this case would be something like:

source: srm:// srm-dteam.cern.ch:8443/srm/managerv2 ?SFN= /castor/cern.ch/c3/fts-test/source/castorpublic / rand023.dat

destination: srm:// srm-dteam.cern.ch:8443/srm/managerv2 ?SFN= /castor/cern.ch/c3/fts-test/dest/castorpublic / 2008-10-07/005/69a58618-2e63-47f3-b06d-4add2f3eafda.dat

The %{something} parts are replaced with results of calls to functions in the python script (see below).

generate_load.py

@configDir

Python script for names generation.

Installation on fts116

Transfer load generator is currently running on fts116, under /opt/load-generator. Three instances are configured to submit jobs to t0, t2 and pilot FTS services.

Load generator instances can be started and stopped using the load-generator.sh script as in the following example:

~ $ ssh fts116
(fts116) ~ $ cd /opt/load-generator/loadgen-pilot
(fts116) /opt/load-generator/loadgen-pilot $ ./load-generator.sh start
(fts116) /opt/load-generator/loadgen-pilot $ ./load-generator.sh stop

Twiki-formatted reports can be created with the create-loadgen-report.py script. The script requires the cx_Oracle and python-yaml packages.

How to run the loadgenerator

Log into the pilot machine fts116 as root and via lxadm

cd /opt/load-generator
ls

The ls shows you that there are directories for differnent instances: pilot, t0 and t2. Lets focus on the pilot tests although the steps are the same for the other 2 cases.

cd loadgen-pilot/
source config-workspace.sh 
vim stage/share/config/glite-data-transfer-load-generator/submissions.xml

In this file you have the definition of the channels. As you see there is an atribute per channel enable="false" which set the channel to disable. For initial tests we only enable CERN-CERN and if you want a two more but not all of them.First test without disturbing and filling the back ends with useless transfers. So initially you can test with CERN-CERN and a another 2.

The submission-schema.xsd contains the schema and it has been described in previous sections.

Remove the loadgen.log a new one will be created with the log of your tests.

Finally execute: ./load-generator.sh

How to get information from the loadgenerator

In org.glite.data.transfer-load-generator/scripts you have a python script that extract information from the DB and for the time (days) you specified) so if you set the DB url to the one of the pilot you will get a report in a twiki format.

The report.cfg is the configuration file, here you can set the number of days from which you want to get information DB of the pilot but you can modified, although in this case it is not necessary as the tests are running in the pilot. You need to have the yaml python package as it's used for regular expressions.

And then you can post the results on this twiki https://twiki.cern.ch/twiki/bin/view/LCG/Fts22Testing

Maintainer: PaoloTedesco