Show Children Hide Children FtsServerServicesXml15Issues
Main FTS Pages
FtsRelease22
Install
Configuration
Administration
Procedures
Operations
Development
Previous FTSes
FtsRelease21
FtsRelease21
All FTS Pages
FtsWikiPages
Last Page Update
SteveTraylen
2007-03-26

Information cache file

See also FtsTier0Procedures15ServiceXml for a CERN specific routine.

The FTS requires certain static information from the EGEE.BDII:

  • The endpoints and GOCDB site names of storage elements
  • The endpoint of the MyProxy servers to use

Although the FTA agent server itself has the capability to query the EGEE.BDII directly, the impact of this has not yet been investigated, so we still recommend the use of a static cache file for storing the information from EGEE.BDII. The will contain information about every SRM in EGEE.BDII.

We provide a tool to make the infomation cache snapshot from EGEE.BDII (attached below).

Running the script

The script is called make-services.sh

make-services.sh -h

The script does not need to be run with any options. In this case it will store all SRMs found in the top-level EGEE.BDII lcg-bdii.cern.ch, an entry for the default FTS MyProxy server myproxy-fts.cern.ch and a static service entry for the FTS web-service on the host you are running the script on.

It is recommended to run this script on your web-service node.

It will create a file /opt/glite/etc/services.xml that you should copy to all your FTA agent nodes (in the same place).

You should do this BEFORE starting any of the FTA agents.

Add local definitions

Starting from the version uploaded on 30 May 2006, it is possible to specify local definitions in order include information for services not listed in EGEE.BDII. In order to do that, you need to create a local services.xml like file (named e.g. services.local.xml) and invoke the make-services.sh script with the option --lclxml:

make-services.sh --lclxml services.local.xml

Please note that the local definitions have a higher priority of the ones retrieved from EGEE.BDII, so you can use this feature also to override misconfigured entries.

Keep old definitions

Starting from the version uploaded on 08 June 2006, it is possible to keep the definitions previously retrived from EGEE.BDII in order to be resilient to intermittent problems with the sites. These definitions will be merged with the new (and the local) ones, but with a lower priority, so if the information in EGEE.BDII is updated, these changes will be reflected into the new services.xml file. In order to use this feature, you need to pass the existing, eventually backed up, services.xml file name with the option --oldxml:

make-services.sh --oldxml services.xml.bak

Verify the newly created file

In order to verify that th new file is correct, you can run the follwowing checks:

  • Check that the number of SRM entries is greater or equal to the ones in the former file.
grep "<type>SRM</type>" services.xml.new | wc -l
grep "<type>SRM</type>" services.xml.bak | wc -l 

  • Verify that the services.xml file is well formatted
export GLITE_SD_SERVICES_XML=services.xml.new
export GLITE_SD_PLUGIN=file
glite-sd-query -t SRM

Please note that in order to use the environment variable GLITE_SD_SERVICES_XML you need to have glite-servicediscovery-file-c version >= 2.1.2, otherwise the services.xml file has be to located in $GLITE_LOCATION/etc/services.xml

Restart the agents afterwards

The agents cache the data from the services.xml in memory for 1 hour or so. You should therefore restart them to pick up the new definitions:

service transfer-agents restart

Problems

The service discovery library has some intermittent problems or behaves badly in response to LDAP problems. Since the script makes use of this underlying library, it will sometimes fail. You may have to rerun it a few times to fetch all the information successfully.

As a backup, attached below is a snapshot of EGEE.BDII taken on 7 June 2006. If you shoose to install this, please edit it and update the static FTS web-service hostnames prod-fts-ws.cern.ch to point to your own web-service (there are 3 lines to change, at the top of the file).

Problems II

Older version of the script had some problems with the MyProxy service type and duplicate SRM entries. If you are using an older version of the script ( < 26 May 2006) it is suggested to re-download it. The problems are described in FtsServerServicesXml15Issues.

The script now:

  • Removes all duplicate SRM entries
  • Fixes the MyProxy type to be MyProxy
  • Changes all GOCDB site names to be upper case.

Problems III

Older version of the script had some problems with the generation of the FTS service entries: the hostname was always the host where the script ran, instead of what passed using the --ftshost option.

The script now:

  • Fixes the FTS service entries host, taking correctly the value passed as argument

Problems IV

The file-based ServiceDiscovery plugin does case-sensitive searches, therefore there may be problems in case an SRM endpoint contains uppercase letters, since the FTS VOAgents may fail in resolving the site names from the surls if can the user will use lowecase letters. Starting from the version uploaded on 14 June 2006, a workaround has been added in order to have the hostname part of the endpoint converted to lowercase.

Problems V

We observed that sometimes the FTS contacts the SRM castorsrm.cern.ch instead of srm.cern.ch. This is due to the fact that the ServiceDiscovery library doesn't match perfectly the hostname but performs only a string search. While we're fixing this problem, the workaround consist in having the entry srm.cern.ch before the one of castorsrm.cern.ch and the vo list is ordered in the same way (SevriceDiscovery does a reshuffle based on the VO position in the list). Since EGEE.BDII already returns srm.cern.ch before castorsrm.cern.ch, we modify the make-services.sh script (uploaded the 28 June 2006) in order to order alphabetically the volists.

In addition, this version of the script introduced an additional parameter (--addvo) to include the given vo into all the SRM entries. This was done in order to add ops to the services.xml file while some site are not yet ready.

Be gentle

The script makes heavy queries to the EGEE.BDII, essentially sucking all information for all SRMs from it.

It is NOT recommended to run this script from cron.

You should rerun the script only if you see problems with a newly published or changed endpoint.

Pre-prepared file

A pre-prepared file was generated on 7 June 2006 from lcg-bdii.cern.ch and is attached below.

CERN production

For the CERN production FTS, we run the following command line:

make-services.sh --ftshost prod-fts-ws.cern.ch --oldxml services.xml.bak --serxml services.xml.new --addvo ops --verbose

Please note that the prod-fts-ws.cern.ch in the alias used for the DNS load balanced WebServices nodes.


Maintainer: GavinMcCance


Topic attachments
I Attachment History Action Size Date Who Comment
Unix shell scriptsh make-services.sh r12 r11 r10 r9 r8 manage 17.1 K 2007-02-22 - 16:29 UnknownUser Script to generate services.xml cache file from BDII (remove duplicated VO entries)
XMLxml services.xml r12 r11 r10 r9 r8 manage 225.7 K 2006-12-05 - 09:59 GavinMcCance Sample services.xml file generated from BDII on 28 August 2006
Edit | Attach | Watch | Print version | History: r21 < r20 < r19 < r18 < r17 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r21 - 2007-03-26 - SteveTraylen
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback