ETICS multinode testing exercises

This page collects some preliminary tests to use the ETICS multinode testing to certify Data Management services.

Sequence diagram for DPM test automation

The general idea is shown in the following sequence diagram:

Yaimgen and test scripts review

Mutlinode testing in ETICS will be used here as a driver and comunication infrastructure built on top of existing script for test automation ([https://svnweb.cern.ch/trac/yaimgen/][Yaimgen]].

Automatic deployment of services: - glite-SE_dpm_mysql:

./vnode-ygen.sh -i sl5-64-dh -n glite-SE_dpm_mysql
[...]

- glite-UI

./vnode-ygen.sh -i sl5-64-dh -n glite-UI

The vnode-ygen.sh script uses the Vnode CLI to create a virtual machine, and then Yaimgen to deploy the service.

The UI deployment also creates and prepare a testuser account generating a voms proxy, so that tests can be immediately run.

Launching DPM client tests using tests-wrapper.py:

 ./tests-wrapper.py -d ../tests/functional lcg

This command will launch the lcgutil functional test using the CTB resources specified in lcgutil-config.

ISSUE #1: How to override the default hostnames in order to use the one just deployed?

SOLUTION #1: The lcgutil-config has been modified in order not to override existing environment variables.

Therefore, before calling tests-wrapper.py one has to set SE_HOST to point the tests to the DPM server just deployed:

export SE_HOST="vtb-generic-91.cern.ch"

ISSUE #2_ The just deployed DPM will not be published in the BDII, causing lcg-* command to fail:

[BDII][][] lxbra2305.cern.ch: No entries for host: vtb-generic-91.cern.ch

SOLUTION #1_A: Do not use the BDII, run all the lcg-* command with the option '--nobdii' and specifying the endpoint type, e.g:

[testuser@vtb-generic-61 scripts]$ lcg-cr -v --nobdii --defaultsetype srmv2 -d vtb-generic-91.cern.ch file:/etc/hosts
Using grid catalog type: lfc
Using grid catalog : lxbra2506v3.cern.ch
Checksum type: None
[GFAL][gfal_init][EINVAL] Invalid request: When BDII checks are disabled, you must provide SURLs and endpoint type
lcg_cr: Invalid argument

Moreover we need to provide the SURL instead of the hostname, like this:

[testuser@vtb-generic-61 scripts]$ lcg-cr --nobdii --defaultsetype srmv2 -d srm://vtb-generic-91.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/gpfile4 file:/etc/hosts
guid:b15c58f8-83a8-432f-bc21-0c7315c9e94b

In order to make this solution work we need to modify modify the script lcg-common in order to use the --nobdii option, which is not working because it relies on a BDII query

[testuser@vtb-generic-61 lcg]$ ./lcg-cr --nobdii --usefullsurl vtb-generic-91.cern.ch

    === lcg-cr, -del test ===    

 ===> --nobdii option will be passed to commands sensitive to it
 ===> Full SURLs will be used whenever possible
 ===> Full SURLs will be used whenever possible
 ===> SE host      : vtb-generic-91.cern.ch
 ===> second storage element (for lcg-rep): 
 ===> [for information]: LCG_GFAL_INFOSYS=
 ===> [for information]: LCG_GFAL_VO=
 ===> [for information]: LFC_HOST=lxbra2506v3.cern.ch
 ===> Trying to determine endpoint for vtb-generic-91.cern.ch
ldap_bind: Can't contact LDAP server (-1)
ldap_bind: Can't contact LDAP server (-1)
ldap_bind: Can't contact LDAP server (-1)
 ===> SE endpoint : 
 ===> SRM endpoint:   
 ===> Could not determine srm endpoint for vtb-generic-91.cern.ch

This solution also has the disadvantage of not testing the BDII interaction.

SOLUTION #2_B: Add a testing BDII that include CTB resources, and where we can publish unstable resources.

Possible interactions among unstable services? Should not be a problem.

The testing BDII is already available, but publishing information automatically (on demand) looks tricky.

SOLUTION #2_C: Create an additional node with an ad-hoc BDII

This way we have control over the machine and would be easier to publish information. However for every test there is going to be an additional node needed.

I will investigate this solution for now.

In order to update the BDII with the newly created DPM host we need to add to /opt/glite/etc/gip/site-urls.conf the string:

DPM  ldap://<dpm hostname>:2170/mds-vo-name=resource,o=grid

and restart the bdii daemon with:

/etc/rc.d/init.d/bdii restart

The new sequence diagram including the BDII node and additional communication messages is:

Tests with vNode machines

BDII node commands

#!/bin/bash

#Install BDII site
cd yaimgen
./yaimgen.sh -n glite-BDII_site

# ETICS client installation (TO SKIP WHEN NOT NECESSARY)
cd
wget "http://eticssoft.web.cern.ch/eticssoft/repository/etics-client-setup.py"
python etics-client-setup.py -r 1.4.15-1 --volatile etics_2_5_0
export ETICS_HOME=/root/etics
export PATH=$ETICS_HOME/bin:$PATH

ETICS_MULTINODE_ID="giannitest6"
export ETICS_INFOSYS_UUID="$ETICS_MULTINODE_ID"
DPM_HOST=`etics-get -b DPM.host`

#Add DPM host to BDII
echo "DPM  ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid" >> /opt/glite/etc/gip/site-urls.conf
/etc/rc.d/init.d/bdii restart
etics-set BDII.host `hostname`

#Wait for the end
etics-get -b TESTS.done

UI node commands

#!/bin/bash

#Install glite-UI
cd yaimgen
./yaimgen.sh -n glite-UI -testuserpass ****

# ETICS client installation
cd
wget "http://eticssoft.web.cern.ch/eticssoft/repository/etics-client-setup.py"
python etics-client-setup.py -r 1.4.15-1 --volatile etics_2_5_0
export ETICS_HOME=/root/etics
export PATH=$ETICS_HOME/bin:$PATH
export ETICS_INFOSYS_UUID="giannitest6"
echo `etics-get -b BDII.host` >> /home/testuser/bdii.host
echo `etics-get -b DPM.host` >> /home/testuser/dpm.host

#prepare test driver script
cat << EOF > /tmp/runtests.sh
#!/bin/bash
svn co http://svnweb.cern.ch/guest/lcgutil/testing/trunk tests
svn co http://svnweb.cern.ch/guest/lcgutil/tests-tools/trunk scripts

export LCG_GFAL_INFOSYS="`cat bdii.host`:2170"
export SE_HOST="`cat dpm.host`"
cd scripts
./tests-wrapper.py -o output.xml -d ../tests/functional lcg

#MV report elsewhere
#exit

EOF
chmod a+x /tmp/runtests.sh

#Launch tests
su - testuser -c /tmp/runtests.sh

etics-set TESTS.done true

DPM node commands

#!/bin/bash

#Install glite-SE_dpm_mysql
cd yaimgen
./yaimgen.sh -n glite-SE_dpm_mysql

# ETICS client installation
cd
wget "http://eticssoft.web.cern.ch/eticssoft/repository/etics-client-setup.py"
python etics-client-setup.py -r 1.4.15-1 --volatile etics_2_5_0
export ETICS_HOME=/root/etics
export PATH=$ETICS_HOME/bin:$PATH
export ETICS_INFOSYS_UUID="giannitest6"

etics-set DPM.host `hostname`

etics-get -b TESTS.done

ETICS integration

TBD

-- GianniPucciani - 08-Dec-2010