LCG Management Board

Date/Time:

Tuesday 30 January 2007 at 16:00

Agenda:

http://indico.cern.ch/conferenceDisplay.py?confId=9354

Members:

http://lcg.web.cern.ch/LCG/Boards/MB/mb-members.html

 

(Version 1 - 31.1.2007)

Participants:

A.Aimar (notes), D.Barberis, I.Bird, N.Brook, T.Cass, Ph.Charpentier, F.Donno, M.Litmaath, I.Fisk, D.Foster, B.Gibbard, J.Gordon, C.Grandi, F.Hernandez, M.Lamanna, M.Kasemann, G.Merino, A.Pfeiffer, L.Robertson (chair), J.Shiers, Y.Schutz, O.Smirnova, R.Starink

Action List

https://twiki.cern.ch/twiki/bin/view/LCG/MbActionList

Next Meeting:

Tuesday 6 February 2007 - 16:00-18:00 – Face-to-face Meeting

1.      Minutes and Matters arising (minutes)

 

1.1         Minutes of Previous Meeting

No comments. Minutes approved.

 

2.      Action List Review (list of actions)

Actions that are late are highlighted in RED.

 

  • 15 Jan 2007 - D.Foster will form a group discussing the network setup and performance needed according to the Megatable values (for T1-T1 links on the OPN, etc).

 

Done. The presentation is scheduled in this MB meeting.

  • 19 Dec 2006 - The proposal to ALICE is to consider, as in the TDR, a value of 10**6 for the ALICE ion runs. An email message from Y.Schutz reopened this issue (Y.Schutz’s email).

Done. L.Robertson reported on his discussion with J.Schukraft. The agreement reached will be attached to the minutes.

Additional Information:
The agreement is that in 2010 the entry for ions in the physics beam time column should be "ions: integrated Luminosity =1nb-1   -  (106 seconds at L=1027cm-2sec-1)".

Updated the scenarios on the web:   www.cern.ch/LCG/MB/revised_resources/startup%20scenario.pdf

  • 22 January – All Tier-1 sites should send their site report on December 2006 reliability.

 

Done only by a few sites. Replaced by a new action including also January 2007.

 

Next week the reliability report for January 2007 will be distributed. Replies should be sent before end of the following week.

 

New Action:

  • 9 February 2007 – All Tier-1 Sites should send to the MB their Site Reliability Report for December 06 and January 07.

 

 

  • 22 Jan 2007 - J.Templon should distribute pointers to all information and documentation that can help sites to implement job priorities on their batch systems.

 

R.Starink reported that the information is available in the presentation given at the WLCG Workshop. See presentation.

 

Update: J.Templon distributed an email to the MB mailing list with further information. See email message.

 

Sites now should distribute their plans for the job priorities implementation, for review at the next meeting (5 February).
New Action:

  • 5 February 2007 – All Tier-1 Sites should send to the MB their plans for the implementation of job priorities.

 

 

  • 25 Jan 2007 - F.Donno should report to the MB (by email) the list of people that will participate to the GSSD working group.

 

Done. The list has been distributed to the MB list and a few comments have circulated.

 

  • 31 Jan 2007 - Sites should send to H.Renshall their procurement plans.

 

 

3.      LHCOPN Status and Megatable Network Requirements (Slides) – D.Foster

 

D.Foster presented an update on the status of the OPN and how the network requirements in the Megatable compare with what will be provided by the available and planned infrastructure.

3.1         OPN Status

The Wide Area Networking used for all the aspects of the LHC is composed of many different, and separately managed, infrastructures:

-          National Research Networks (NRENS) Worldwide (NRENS, I2, ESNet, CANARIE etc.)

-          Interconnection of NRENS in Europe (GEANT-2)

-          Transatlantic Connectivity (USLHCNet), funded by DOE and run by Caltech

-          LHC Optical Private Network (LHCOPN)

 

It is important to remember that there is:

-          no centralised funding, no centralised management,

-          independent domains of infrastructure and responsibility.

 

The LHCOPN Started in 2004 to address at least one of the identifiable problems: getting data from CERN (T0) to the T1’s with a predictable performance. This practical goal allowed the GEANT-2 Infrastructure to evolve with the LHCOPN requirements in mind.

The LHCOPN was identified as a “considerable achievement” at the last EU review of GEANT.

 

The LHCOPN Architecture (slide 4) shows that:

-          The OPN will connect the Tier-0 and Tier-1 sites.

-          The Tier-2 sites will communicate with the Tier-1 sites via the general purpose IP infrastructure.
No dedicated links are planned by the OPN group, but in some cases are available if some sites have set them up.

3.2         Comparison with the Megatable

The LCG “Megatable” activity aims to provide a “bottom up” view of the network requirements.

-          Some figures appear to be peak (T0/T1/T1) and some average (T1/T2).

-          All figures have been generated from models of data movement for the “standard” data movement cases.

 

This gives “order of magnitude” requirements for the most basic network needs.

Network provisioning (which takes a long time) needs to work to a model of future predicted behavior, capability and availability. This implies a process in which the network evolves with the needs of the LHC and where new equipment may be required.

3.3         Megatable Rates

Slide 7 shows the data transfer rates expected (MBytes/sec) and the total Gbit/sec (In and Out).

D.Foster stated that the values in the Megatable for Tier-0 and Tier-1 seem to be covered by the OPN.

 

LHC OPN Network

T0-T1 (MB/sec)

T1-T1 In (MB/sec)

Total In     Gb/sec

T1-T1 Out (MB/sec)

Total Out Gb/sec

ASGC

91.3

158.3

2.00

128.8

1.03

BNL

287.2

274.1

4.49

218.5

1.75

CERN

1343.0

208.7

1.67

104.3

11.58

CNAF

136.2

208.0

2.75

209.4

1.68

FNAL

105.0

63.4

1.35

214.9

1.72

FZK

132.6

220.1

2.82

193.6

1.55

IN2P3

157.2

229.5

3.09

263.7

2.11

NDGF

54.4

51.9

0.85

54.2

0.43

NIKHEF

121.7

134.3

2.05

172.2

1.38

PIC

63.7

167.5

1.85

88.2

0.71

RAL

137.2

218.3

2.84

219.4

1.76

TRIUMF

48.3

50.1

0.79

47.4

0.38

ALICE US T1

8.2

10.8

0.15

3.7

0.03

 

Slide 8 shows the values for the Tier-1 and Tier-2 connections.

D.Foster noted that some numbers seem a bit high but should be satisfied by the general purpose infrastructure.

 

General
IP Network

 

T2-T1 In (MB/sec)

Total In      Gb/sec

T1-T2 Out (MB/sec)

Total Out Gb/sec

ASGC

 

54.8

0.44

133.0

1.06

BNL

 

92.4

0.74

225.7

1.81

CERN

 

49.4

0.40

71.4

0.57

CNAF

 

68.1

0.54

155.7

1.25

FNAL

 

30.0

0.24

248.0

1.98

FZK

 

85.4

0.68

191.2

1.53

IN2P3

 

179.0

1.43

215.8

1.73

NDGF

 

3.9

0.03

14.8

0.12

NIKHEF

 

41.0

0.33

69.9

0.56

PIC

 

35.6

0.28

91.6

0.73

RAL

 

94.9

0.76

113.1

0.90

TRIUMF

 

14.3

0.11

35.6

0.28

ALICE US T1

 

32.5

0.26

13.2

0.11

 

3.4         Summary 2007-2008

According to what is known, and what has been tested, the starting situation is:

-          LHCOPN will support the T0-T1 connectivity requirements.

-          LHCOPN will be able to support a (large) fraction of T1-T1 requirements.

 

An “All Hands (Caltech, DOE, CERN, Fermilab, CMS, ESnet, I2, GEANT)” USLHCNet Meeting in October concluded that:

-          Sufficient T1-T2 connectivity will be provided by the general purpose IP infrastructures.

-          Some 20 Gb/sec of little used IP peering (ESnet/I2 to GEANT) is available.

-          An initial extra 5 Gb/sec will be provided by the USLHCNet link from NY to AMS providing the DOE agrees.

 

4.      Other considerations

One should always note that to have a “1Gb connection to CERN” does not mean that there is always 1 Gb/sec available. Often it means a general shared 1Gb/sec connection. Therefore one can only talk of average throughput: there is not exclusive use of the connection, but in general these links had been proven to be sufficient for the experiments’ needs.

 

The current setup is just the starting point and the infrastructure will evolve with the needs of the LHC. As a result of monitoring the situation it will be decided where to intervene with new equipment.

It is important to note that provisioning additional bandwidth intra-Europe or intra-US remains cost effective, while additional transatlantic bandwidth will remain relatively expensive.

 

The GEANT Cost Sharing and AUP policies require caution. LHCOPN services will remain very constrained for the moment, but this is compatible with the stated mission and technical implementation issues. Additional circuits between centers T2/T1/T1 will have to be provisioned as needs arise.

4.1         Issues and Activities

Backup connections remain a major issue. One needs the availability of circuits to make a logical backup feasible and modeling it in a way not to have single point failures. Understanding how single-point physical failures affect the logical model is not easy. Lots of fibers actually occupy the same physical trunk (e.g. both NREN and GEANT fibers) therefore a single trunk failure could lead to multiple simultaneous logical topology failures: the backup physical connection will be cut together with the main connection.

 

Dante with the NRENs are taking the lead in the study of backup links.

 

Operational procedures are still being refined. Monitoring is being set up in the context of the E2ECU/ENOC collaboration. There will be standard processes for all interventions.

 

Capacity planning will derive from real usage. Experience will be important in this context.

 

Requirements for US-ALICE will have to be defined in detail when the ALICE sites are confirmed, with possible implication on the transatlantic link.

 

Y.Schutz asked whether the link to the ALICE Tier-1 site is provisioned or not in the current estimates.

D.Foster replied that currently the official ALICE requirements are not defined. And there is not any 10 Gb/sec link dedicated to US ALICE Tier-1 for now.

 

L.Robertson asked whether the current links have been tested for effective throughput, in particular the Tier-2 connections. D.Foster replied that CMS and ALICE had done some tests but there was no systematic test of all connections. One would need to have a dedicated team to do this. Currently this is done by the experiments and Tier-1 sites.

 

L.Robertson asked whether the experiments expect to take the lead in testing the real performance of the Tier-1 to /from Tier-2 links. All LHC experiments agreed that this is the case.

 

5.      LFC and GFAL client libraries for SLC4 (Slides) – I.Bird

 

5.1         Status of building gLite under SLC4

The gLite build requires that VDT be built with RHEL4 and gcc-3.4. The 1st suitable version of VDT 1.6 was received just the day before the meeting (this is the version that OSG will use). The deployment team still needs to verify that changes to MyProxy are backward compatible and for now cannot build against VDL (missing components). This is being followed up and the build is expects within a few days.

 

This build should provide binaries for SLC4/gcc-3.4 on IA32, IA64, and x86-64.

 

The full build of gLite has only a few remaining issues. Once these are resolved it will have to be tested and certified. LFC and GFAL are part of the more general porting to SLC4. They will be the first ones to be made available.

 

I.Bird will report to the MB next week the progress on SL4 Native building.

5.2         Applications Area vs Middleware configurations

The more general issue is to have uniform software configurations between the Application Area and Middleware. Some middleware is distributed with the App Area software, as an external library, and there may be mismatches between these clients and the versions of the servers deployed at sites.

 

Ph.Charpentier noted that currently the middleware is installed in the external area of the App Area because the experiments must have the exact version of client software they need. The software is not copied in the App Area, but is installed and properly versioned. They cannot wait for the client libraries to be deployed on the sites or run the risk of finding an incompatible (too new or too old) version of the middleware on a given site.

 

I.Bird agreed on that but added that old versions do not work anymore in some cases.

 

D.Barberis added that the real issue is that “new versions must become backward compatible”; otherwise they break the existing experiment software when deployed.

 

The MB agreed that there must be an urgent discussion about software configuration and distribution outside, a working group apparently already exists and should report to the MB in the near future.

 

6.      SRM v2.2: Sites Deployment and Tests (Slides) – F.Donno

 

 

F.Donno summarized the status of the SRM 2.2 implementations and the results of her S2 test suite. 

6.1         Tests Currently Executed

The S2 test suite verifies the availability of endpoints, basic functionality, use cases and boundary conditions, interoperability, exhaustive and stress tests.

More in detail the current tests are:

-          Availability: Ping and full put cycle (putting and retrieving a file)

-          Basic: basic functionality checking only return codes and passing all basic input parameters

-          Use Cases: testing boundary conditions, exceptions, real use cases extracted from the middleware clients and experiment applications.

-          Interoperability: servers acting as clients, cross copy operations

-          Exhaustive: Checking for long strings, strange characters in input arguments, missing mandatory or optional arguments. Output parsed.

-          Stress: Parallel tests for stressing the systems, multiple requests, concurrent colliding requests, space exhaustion, etc.

 

The S2 tests cron job is run automatically 6 times a day (overnight for US sites).

The details are here: http://cern.ch/grid-deployment/flavia

 

In addition, manual tests are performed by the developers of GFAL/lcg-utils, FTS and DPM. The LBNL test suite is also executed daily, at LBNL.

 

Slide 3 shows the results of the tests execution. The method are currently spilt in “MoU needed now”, “MoU needed by end 2007” and “non-MoU” methods.

 

Slide 4 and 5 show the trend of the failures in the last few months. Since a couple of weeks the failures have been reduced considerably and most missing methods have been implemented. The availability tests are mostly always successful, but there are still many failures on the interoperability and use cases tests.

6.2         Status of the SRM Implementations

DPM has been rather stable and in good state for the entire period of testing. Few issues were found (few testing conditions caused crashes in the server) and fixed. All MoU methods implemented beside Copy (not needed at this stage).

 

DRM and StoRM: good interaction. At the moment all MoU methods implemented (Copy in PULL mode not available in StoRM). Implementations rather stable. Some communication issues with DRM need investigation. (These implementations are not required for LCG)

 

DCache: very good improvements in the basic tests in the last weeks. The implementation is rather stable. All MoU methods have been implemented (including Copy that is absolutely needed for dCache) except ExtendFileLifeTime. T.Perelmutov has promised to implement it as soon as he gets back to the US.

 

CASTOR: The implementation has been rather unstable. The problems have been identified in transferring the requests to the back-end server. Therefore, it has been difficult to fully test with the basic test suite the implementation of the SRM interface. A major effort is taking place to fix these problems.

6.3         Plans

The Plans for 1Q 2007 were:

 

Phase 1: From 16 Dec 2006 until end of January 2007:

-          Availability and Basic tests

-          Collect and analyze results, update page with status of endpoints: https://twiki.cern.ch/twiki/bin/view/SRMDev/ImplementationsProblems

-          Plot results per implementation: number of failures/number of tests executed for all SRM MoU methods.

-          Report results to WLCG MB.

 

Currently the issue is that CASTOR does not pass all tests yet and need to be followed-up further.

 

Phase 2: From beginning until end of February 2007: 

-          Perform tests on use-cases (GFAL/lcg-utils/FTS/experiment specific), boundary conditions and open issues in the spec that have been agreed on.

-          Plot results as for phase 1 and report to WLCG MB.

 

Phase 3: From 1 March until “satisfaction or end of March 2007”:

-          Add more SRM 2.2 endpoints (at some Tier-1 sites)

-          Stress testing of the implementation

-          Plot results as for phase 2 and report to WLCG MB. 

 

This plan has been discussed during the WLCG workshop. The developers have agreed to work on this as a matter of priority.

6.4         Grid Storage System Deployment (GSSD)

The Working Group was launched by the GDB to coordinate SRM 2.2 deployment for Tier-1s and Tier-2s.

 

The wiki is here: https://twiki.cern.ch/twiki/bin/view/LCG/GSSD. And this is the mailing list: storage-class-wg@cern.ch

 

The mandate of the working group was distributed and could be summarized as:

-          Establishing a migration plan from SRM v1 to SRM v2 so that the experiments can access the same data from the 2 endpoints transparently.

-          Coordinating with sites, experiments, and developers the deployment of the various 2.2 SRM implementations and the corresponding Storage Classes.

-          Coordinating the Glue Schema v1.3 deployment for the Storage Element making sure that the information published is correct.

-          Ensuring transparency of data access and the functionalities required by the experiments (see Baseline Service report).

 

The people involved are the developers (SRM and clients), providers, site admins, and experiments.

 

The plan of the GSSD working group is:

-          Now: Collecting requirements from the experiments: more details with respect to what is described in TDR. The idea is to understand how to specifically configure a Tier-1 or a Tier-2: storage classes (quality of storage), disk caches (how many, how big and with which access), storage transition patterns, etc.

-          Now: Understand current Tier-1 setup: requirements?

-          Now: Getting hints from developers: manual/guidelines?

 

-          March-July 2007: Selecting production sites as guinea pigs and start testing with experiments.

-          March-July 2007: Assisting experiments and sites during tests (monitoring tools, guidelines in case of failures, cleaning-up, etc.). Define mini SC milestones

 

-          If Necessary: Accommodate new needs, not initially foreseen, if necessary.

 

-          September 2007: Have production SRM 2.2 fully functional (MoU)

 

In practice the next steps are:

-          Detailed requirements already collected from LHCb, ATLAS, and CMS.
Still some reiteration is needed for CMS and ATLAS.

-          Focus on dCache deployment first.
Target sites: GridKA, Lyon, SARA. Experiments: ATLAS and LHCb.

-          Variety of implementations.
Many issues covered by this exercise

-          Compiling specific guidelines targeting an experiment: ATLAS and LHCb.

-          Guidelines reviewed by developers and sites. Covering possibly also some Tier-2 sites.

-          Working then with some of the T2s deploying DPM. Repeat the exercise done for d-Cache. Possible parallel activity.

-          Start working with CASTOR if ready: At CNAF and RAL.

-          Define mini SC milestones with the experiments and test in coordination with FTS and lcg-utils developers.

 

The next steps for the clients will be:

-          Site admins have to run both SRM v1.1 and SRM v2.2
Restrictions: SRM v1 and v2 endpoints must run on the same “host” (or host alias) and use the same file “path”. Endpoints may run on different ports.

-          Experiments will be able to access both old and new data through both the new SRM v2.2 endpoints and the old SRM v1.1.

This is guaranteed when using higher level middleware such as GFAL, lcg-utils, FTS, LFC client tools. The endpoints conversion is performed automatically via the information system.

The SRM type is retrieved from the information system
In case 2 versions found for the same endpoint SRM v2.2 is chosen only if the space token (storage quality) specified. Otherwise SRM v1.1 is the default
FTS can be configured per channel on the version to use; policies can also be specified (“always use SRM 2.2”, “use SRM 2.2 if space token specified”, etc)

 

-          It is the task of the GSSD Working Group to define and coordinate configuration details for mixed-mode operations.

 

It is possible and foreseen to run in mixed-mode with SRM v1.1 and SRM v2.2, until SRM v2.2 is proven stable for all implementations.

6.5         Conclusions

There is now a much clearer description of SRM specifications. All ambiguous behaviors were made explicit. A few issues left out for SRM v3 since they do not affect the SRM MoU.

 

Defined a well established and agreed methodology to check the status of the implementations. Boundary conditions use cases from the upper layer middleware and experiment applications will be the focus of next month’s work.

 

Monthly reports and problems escalation to the WLCG MB.

 

A clear plan has been put in place in order to converge. Developers will work on it as a matter of priority

 

Working with sites and experiments for the deployment of the SRM 2.2 and Storage Classes. Specific guidelines for Tier-1 and Tier-2 sites are being compiled.

 

It is not unreasonable to expect SRM 2.2 in production by September 2007.

 

The migration and backup plans foresee the use of a mixed environment SRM v1 and v2, where the upper layer middleware takes care of hiding the details from the users.

 

Ph.Charpentier stressed the fact that the endpoints must have the same name for SRM 1 and 2. Otherwise the information in the catalogs is invalid and there is a big migration plan to prepare.

F.Donno confirmed that the two SRM services will wait on different ports on the same host. The client can check the version with the “ping” method (available only for SRM 2.2 and returning error in SRM 1.1).

 

L.Robertson asked the experiments which is their “final date for commissioning” of the SRM 2.2 services if they are to be used for the 2007 run. The experiments replied that their cut-off date is about early Summer (June, July) before the Summer holidays. And therefore the tests should start in May. More details must be defined in the GSSD.

 

The backup solution also must to be studied for the cases in which 2.2 cannot be deployed or does not cover the experiments use cases. Although the mixed SRM 1.1 + SRM 2.2 environment will necessarily be demonstrated during the testing and deployment phase (implementations will be rolled out at different sites at different speeds) we will have to have a formal production plan if the mixed environment will be used for the 2007 run.

 

N.Brook asked whether it is true that CASTOR has a 35% failure rate. J.Gordon replied that the average failure rate, on S.De Witt’s test installation, is about 5%. And that all tests have succeeded at some point but sometimes there are still failures. The investigation is going on in order to understand the different behaviours among CASTOR installations.

 

F.Donno and M.Litmaath will come to the face-to-face meetings every month to provide an update on the SRM progress.

 

7.      AOB

 

 

The Overview Board decided that:

-          The formal committees (CB, OB, MB, GDB) should have telephone connections available, even if the primary connection is VRVS. This is to ensure that remote attendees are able to connect even if there is a failure of the network or VRVS system.

-          The experiments requirements should be updated no more often than every 6 months, and not changed irregularly as done until now. The MB should decide on the next date for publishing the update.

 

L.Robertson proposed that we consider whether formal maintenance and support agreements are needed with the providers of software used in the WLCG.

 

Action:

L.Robertson will send a starter list of software involved to the MB.

 

Action:

Sites should report to the face-to-face meeting their progress with their implementation of groups/roles and job priorities. 

 

8.      Summary of New Actions 

 

 

Action:

L.Robertson will send a starter list of software involved to the MB.

 

Action:

Sites should report to the face-to-face meeting their progress with their implementation of groups/roles and job priorities. 

 

 

New Action:

  • 9 February 2007 – All Tier-1 Sites should send to the MB their Site Reliability Report for December 06 and January 07.

 

New Action:

  • 5 February 2007 – All Tier-1 Sites should send to the MB their plans for the implementation of job priorities.

 

 

The full Action List, current and past items, will be in this wiki page before next MB meeting.