Show Children Hide Children

Transfer Load Generator

The load generator is a test program that periodically submits jobs to the File Transfer Service.

The latest test results, for FTS v. 2.2, are at LCG/Fts22Testing.

Configuration Files

properties.xml

Path: ${GLITE_LOCATION}/etc/transfer-load-generator.properties.xml

Configuration Parameters

Parameter Name Type Mandatory Default Value Description
ftsServer true string   The fts server to use.
configDir true string   Path containing the configuration files for submissions (submissions.xml) and names generation (generate_load.py).
vo false string dteam The vo (used to filter active requests).
jobsThreshold false unsigned int 10 The number of jobs below which new ones will be generated. Must be greater than 0.
maxSubmittedJobs false unsigned int 100 The maximum number of jobs in the 'submitted' state (avoid overloading the server if jobs are not being processed). Set to 0 to disable check.
executeInterval false time_t 60 The interval between two executions, in seconds.
cleanup false boolean true Try to srmRm files for finished jobs. Disable if using stub transfer service.
cleanupInterval false time_t 3600 The interval between two cleanup operations, in seconds.
cleanupMinAge false time_t 3600 Minimum file/directory age to be eligible for cleanup, in seconds.
cleanupLsCount false int 200 Value of count parameter in srmLs requests for cleanup.
httpTimeout false int 30 Http timeout in srm requests.
submitTimeout false unsigned int 10 Timeout for transfer submit operations (0 == wait undefinitely).

log-properties

Path: ${GLITE_LOCATION}/etc/transfer-load-generator.log-properties

submissions.xml

Path: @configDir@/submissions.xml

Xml file containing endpoints, channels and submissions information.

<?xml version="1.0"?>
<submissions>
  <!-- List of endpoints -->
  <endpoints>
    <data>        
      <name>cern</name> <!-- the name must be unique -->
      <endpoint>
        <host>srm-dteam.cern.ch:8443/srm/managerv2</host>
        <version>2.2</version>
        <source path="/castor/cern.ch/c3/fts-test/source/castorpublic">rand%{random(1,50)}.dat</source>
        <dest path="/castor/cern.ch/c3/fts-test/dest/castorpublic/fts-load-generator">%{date}/%{guid}.dat</dest>
      </endpoint>
    </data>
  </endpoints>
  <!-- List of channels -->
  <channels>
    <channel name="CERN-CERN"> <!-- channel name is used to count active transfers (don't overload channels) -->
      <loads>
        <load name="cern_load_1"> <!-- load name is used just for logging -->
          <source endpoint="cern"/> <!-- must match the name of one of the endpoints in the endpoints list -->
          <dest endpoint="cern">%{date}/%{random(1,10)}/%{guid}.dat</dest> <!-- specifying a new value overrides the default one specified in the endpoints list -->
        </load>
      </loads>
    </channel>
  </channels>
</submissions>

In the example above, we define one endpoint, "cern", and one channel, "CERN-CERN", with one submission, "cern_load_1".

The submitted job for cern_load_1 in this case would be something like:

source: srm:// srm-dteam.cern.ch:8443/srm/managerv2 ?SFN= /castor/cern.ch/c3/fts-test/source/castorpublic / rand023.dat

destination: srm:// srm-dteam.cern.ch:8443/srm/managerv2 ?SFN= /castor/cern.ch/c3/fts-test/dest/castorpublic / 2008-10-07/005/69a58618-2e63-47f3-b06d-4add2f3eafda.dat

The %{something} parts are replaced with results of calls to functions in the python script (see below).

generate_load.py

Path: @configDir@/generate_load.py

Python script for names generation.

Installation on fts116

Transfer load generator is currently running on fts116, under /opt/load-generator. Three instances are configured to submit jobs to t0, t2 and pilot FTS services.

Load generator instances can be started and stopped using the load-generator.sh script as in the following example:

~ $ ssh fts116
(fts116) ~ $ cd /opt/load-generator/loadgen-pilot
(fts116) /opt/load-generator/loadgen-pilot $ ./load-generator.sh start
(fts116) /opt/load-generator/loadgen-pilot $ ./load-generator.sh stop

Twiki-formatted reports can be created with the create-loadgen-report.py script. The script requires the cx_Oracle and python-yaml packages.

How to run the loadgenerator

Log into the pilot machine fts116 as root and via lxadm

cd /opt/load-generator
ls

The ls shows you that there are directories for differnent instances: pilot, t0 and t2. Lets focus on the pilot tests although the steps are the same for the other 2 cases.

cd loadgen-pilot/
source config-workspace.sh 
vim stage/share/config/glite-data-transfer-load-generator/submissions.xml

In this file you have the definition of the channels. As you see there is an atribute per channel enable="false" which set the channel to disable. For initial tests we only enable CERN-CERN and if you want a two more but not all of them.First test without disturbing and filling the back ends with useless transfers. So initially you can test with CERN-CERN and a another 2.

The submission-schema.xsd contains the schema and it has been described in previous sections.

Remove the loadgen.log a new one will be created with the log of your tests.

Finally execute: ./load-generator.sh

How to get information from the loadgenerator

In org.glite.data.transfer-load-generator/scripts you have a python script that extract information from the DB and for the time (days) you specified) so if you set the DB url to the one of the pilot you will get a report in a twiki format.

The report.cfg is the configuration file, here you can set the number of days from which you want to get information DB of the pilot but you can modified, although in this case it is not necessary as the tests are running in the pilot. You need to have the yaml python package as it's used for regular expressions.

And then you can post the results on this twiki https://twiki.cern.ch/twiki/bin/view/LCG/Fts22Testing


Last edit: RosaGarciaRioja on 2009-06-17 - 12:04
Number of topics: 1

Maintainer: PaoloTedesco


Edit | Attach | Watch | Print version | History: r14 < r13 < r12 < r11 < r10 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r14 - 2009-06-17 - RosaGarciaRioja
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EGEE All webs login

This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright & by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback