EMI Data Client Consolidation Meeting, Aug 24-25, 2011

Present: Adrien, David, Jon, Michail, Zsolt, Zsombor

Where: CERN


Zsolt: we have to clarify the situtation of the command line tools. we should compare the interfaces, and we can think about a potential merging.

Zsolt: lcg_util C interface is only used by the CLI, not by experiments directly but people are using the python interface, which was SWIG generated

Zsolt: we really need a python interface, it can be different than the current one the python interface currently has a little bit more functions than the CLI

Zsolt leaves.

The arclib has also a python API, but the data part is not used currently, but in theory it works.

GFAL2 implemented a way to set parameters in code which were previously only possible to set by environment variables.

We try to compare the python libraries of arclib and lcg_util. ARC does not use information system at all. The logical name and everything is put together into a long URL.

gLite people have to figure out (survey) what the experiments use from the python library then we can see which calls should be mapped to the ARC calls.

current lcg_util python methods -> arc python methods mapping

current lcg_util CLI commands -> ARC CLI commands?

can the lcg_util CLI commands just be python scripts which handle the same command line arguments and translate them to python calls to the ARC python API?

how can we make sure that every possible combination of command line arguments of the lcg_util CLI can be translated to ARC calls?

Going through lcg_utils CLI

  • lcg-bringonline -> ARC API should be changed to handle it
  • lcg-get-checksum -> does it trigger checksum calculation? ARC returns checksum
with arcls, would that trigger anything on the server? checksum of local file?
  • lcg-la -> is there a tool in lfc-? if not, we ask for it
  • lcg-lg -> catalog thing
  • lcg-lr -> c
  • lcg-ra -> c
  • lcg-ls -> bdii? search the missing information to create a complete SURL
  • lcg-rep -> third-party copy, used by FTS, in ARC the data would go through the client
(this should work with HTTP also)
  • lcg-stmd -> ARC does not expose space token metadata

third party copy??? isn't SRM copy enough? what about HTTP? can SRM negotiate HTTP TURLs?

problems with ARC:

  • BDII
  • 3rd party
  • checksum of local file, triggering checksum on remote server
  • bringonline
  • space token metadata

going through the design document

with lcg_cr it's possible to say:

lcg_cr <VO> <SE hostname> <localfilename>
(no protocol, no srm version, no port, no path on the SE, no LFC - it all can come from the BDII)
arccp <localfilename> srm://<SE Hostname>/<remotepath>

ARC would not pass any VO information, so GFAL2

in FTS they use full URLs, not partial URLs

the big VOs probably always use full URLs small VOs can set the environment variable

because currently GFAL2 has no globus plugin, it does not depend on cgsi

FTS3 has already all the libraries ready, only the core (the main logic) is missing, they finished the requirement collection and first designs all it will need is the functionality of lcg_rep

FTS uses the globus gridftp very deeply, getting all kind of statuses and markers while the transfer is going on with srmcopy it does not get anything

how would it be possible that FTS3 would get continous feedback about the transfer while it's going on using libarcdata2?

problems with FTS3 using libarcdata2:

  • no third party transfer
  • GFAL2 has no gridftp plugin
  • no way of getting status/feedback while the transfer is going on

GFAL2 also don't have third-party copy

What we need to test?

  • lcg_util tests are in ETICS
  • arc cli also has test
  • these are functionality tests
  • scalability tests we don't really have
  • performance tests we don't really have
  • stress tests exist for FTS only, may be enough
  • regression tests? comparing the new functionality with the old one

This information should be gathered and put into the separate test plan document. Jon will start it.

Action items

  • need a list of functionality used in the current lcg_util CLI (Zsolt)
  • prepare or conduct analysis what FTS3 requires from a new library (Michail)
  • start writing the test plan (Jon)
  • tell Zsolt everything (Michail)
  • investigate third-party copy in GFAL2 (Adrien)
  • send the URL of the testing page to Jon (Michail)
  • investigate third-party support in ARC (David)
  • summarize the third-party discussion (Zsombor)

-- JonKerrNilsen - 30-Nov-2011

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2011-11-30 - JonKerrNilsen
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback