LCG Management Board

Date/Time:

Tuesday 7 August 2007 16:00-17:00 - Phone Meeting

Agenda:

http://indico.cern.ch/conferenceDisplay.py?confId=17195

Members:

http://lcg.web.cern.ch/LCG/Boards/MB/mb-members.html

(Version 1 8.8.2007)

Participants:

A.Aimar (notes), D.Barberis, T.Cass, Ph.Charpentier, T.Doyle, M.Ernst, S.Foffano, C.Grandi, M.Kasemann, J.Knobloch, U.Marconi, G.Merino, P.McBride, Di Quing, H.Renshall, L.Robertson (chair), Y.Schutz, O.Smirnova

Action List

https://twiki.cern.ch/twiki/bin/view/LCG/MbActionList

Mailing List Archive:

https://mmm.cern.ch/public/archive-list/w/worldwide-lcg-management-board/

Next Meeting:

Tuesday 21 August 2007 16:00-17:00 - Phone Meeting

1. Minutes and Matters arising (Minutes)

1.1 Minutes of Previous Meeting

No comments received. Minutes of the MB meeting of the 24 July approved.

D.Barberis asked whether there will be an MB meeting on the 28 August.

L.Robertson suggested to wait until next meeting, on the 21 August, and decide then whether to cancel the meeting of the 28^th.

2. Action List Review (List of actions)

Actions that are late are highlighted in RED.

• 10-July 2007 - INFN will send the MB a summary of their findings about HEP applications benchmarking.

Not done. L.Dell’Agnello will distribute a summary by end of July.

• 12 July 2007 - L.Robertson will appoint assessors to review the equivalence of the OSG tests and the WLCG test set agreed in April 2006.

Not done. Waiting for a candidate reviewer to return from holiday.

• 12 July 2007 - Progress of the Job Priorities working group and the progress of the working solution for ATLAS. D.Liko has been asked to provide a summary, on status and plans of the JP Working Group, which will be distributed to the MB by I.Bird.

Not done.

• 24 Jul 2007 - L.Robertson will ask via email to the Experiments to provide their update to the requirements until 2012.

Done. The status of the requirements is discussed in this meeting.

• 24 July 2007 - CNAF and LHCb to specify additional milestones for StoRM to add to the SRM Roll-out plan (i.e. extending milestones SRM-7 and SRM-17 to include testing of StoRM).

Done. Milestone SRM-7A added to the SRM2 Roll-out Plan.

• 31 July 2007 - L.Robertson will present the agreement(s) to propose to the different supplier of external software for the LCG.

Postponed to end of 2007.

• 31 July 2007 – J.Shiers – Analysis of site scheduled and unscheduled interruptions, showing relative impact at sites, frequency and duration of different causes, etc.

Obsolete. J.Shiers presented the analysis of the interruptions in a previous meeting and this will become a standing activity that will be discussed at the Workshop.

3. SRM 2.2 Weekly Update (SRM Update; SRM2 Plan) H.Renshall

H.Renshall summarized the status of the SRM2 Roll-out milestones.

The text below was distributed and read at the meeting by H.Renshall.

 
----------------------------
 Status of SRM v2.2 milestones for MB report on Tuesday August 07 2007
 
 Here is the (brief) status for the MB.
 
Due 11/07 and completed by MB of 17 July:
 
 SRM-01: done (FZK)
 SRM-02: done (IN2P3)
 SRM-05: done (CERN)
 SRM-09: done (Edinburgh)
 
Due 11/07 and not completed by MB of 17 July:
 
 SRM-08: done except storage tokens still to be configured (LAL)
 SRM-03: done. Exposed known dCache bug keeping data as 'precious'
 after migration to hpss. Fix is already available (BNL)
 
Due 18/07:
 
 SRM-04 done (SARA)
 SRM-06 done (NDGF)
 SRM-07 done. Both CASTOR and STORM endpoints published (CNAF)
 
 Summary: all sites meeting the schedule.
 
Due 31/07:
 
 SRM-10 testing experiment scenarios with experiment certificates. In progress.
 - action on LHCb to assign lhcb/lcgprod role to Lana and Flavia. Done.
 - simple tests have been done with ATLAS and LHCB certificates.
 
 For the LHCb exercises:
 -- FZK : ok - srmPing and srmReserveSpace work and even a successful
 transfer via FTS.
 -- IN2P3: srmPing,srmReserveSpace OK but yesterday a problem occured with
 FTS. It seems that it is related with the configuration - there is a
 misunderstanding between the space token and logical path concepts so
 not a full OK
 -- SARA srmPing OK, srmReserveSpace not OK: problem discovered to be
 that LHCb space was reserved using a DTEAM certificate.
 -- CERN/CNAF Castor : The srmPing was OK for both. Static reservation
 is now successfully tested.
 
Due: 31/10
 
 SRM-11: Stress tests of SRMv2.2 development end-points for CASTOR,
 dCache, DPM and StoRM.
 Tests progressing. The following dCache problems found and reported:
 a. Listing of a directory containing 2 files returns only one file
 b. Moving of a file in a directory fails
 c. ReserveSpace operation changes its status with time.
 SRM-13 definition of tests, incl. SRM V1. done.
 - See ATLAS and LHCb plans on GSSD page:
 https://twiki.cern.ch/twiki/bin/view/LCG/GSSD
 - and GSSD team plan at:
 https://twiki.cern.ch/twiki/pub/LCG/GSSD/SRMV2_v2-1.pdf
 
Starting 1/8:
 SRM-14: Testing High-Level Tools/APIs as defined by the plan.
 In progress. The latest version of lcg-utils 1.9.3 is being
 tested with ATLAS and LHCb certificates.
 Few problems reported. Investigating. lcg-utils 1.9.10
 which includes requests coming from ATLAS and CMS
 is about to be released. Preparing to test that as well.
 CMS has asked that the srm* command clients tools be made
 available also for SRM v2.
 Similar command tools exist also for CASTOR, DPM and
 StoRM. We understood that experiments have agreed
 to use gfal and lcg-utils as high level tools and these
 should be the only ones supported by WLCG.
 CMS claims that OSG will never use lcg-utils and
 therefore also the srm* dCache command clients need to be
 supported by WLCG. We would like to point out that at the
 moment the dCache developers are very busy
 fixing the problems with their server software and have
 very little or no time to dedicate to clients. Since these clients
 are distributed by VDT, possibly OSG/VDT should make sure
 that these clients work correctly for OSG ?
 CMS also claims that both the european and the american
 parts of the collaboration have always used the dCache
 client tools and therefore moving to lcg-utils at the
 moment will require a lot of effort on their side.
 SRM-16-A LHCb to provide details and plan for their tests. Done.
 See https://twiki.cern.ch/twiki/bin/view/LCG/GSSDLHCBPLANS
 
Due 31/08:
 
 SRM-16-B ATLAS and CMS to provide details and plan for their tests.
 SRM-17 LHCb transfer exercises using CERN FTS 2.0 production service.
 
 Replies and more details in the web archive:
 
 https://mmm.cern.ch/public/archive-list/s/storage-classes-wg/
----------------------------

All due milestones are done except for SRM-10.

SRM-10 is not complete by all sites, but FZK, IN2P3, SARA, CNAF and CERN are set up. .

Issues to follow

SRM-14 (see issue described above): There is some disagreement with CMS about the wrappers to the high-level tools that need to be made available to the experiments.

L.Robertson and T.Cass said that at the May 2006 FNAL Workshop on SRM it was agreed that the SRM would only be used by experiments through FTS, lcg-utils and GFAL.

M.Kasemann replied that he will investigate inside CMS to clarify the issues with the SRM wrappers and then inform the MB.

Ph.Charpentier asked whether there is any progress about the distinction between the space tokens and logical paths concepts (see above, SRM-10 at IN2P3).

Nobody from IN2P3 was at the MB. This issue is to be discussed at next MB meeting.

4. Experiments Requirements - L.Robertson

L.Robertson said that originally the intention was to have the requirements through 2012 by early-July, in order to inform the funding agencies of clear requirements before the CRRB in October. As we are late he proposed that the MB should now define the process and agree on the date for completing and distributing the requirements until 2012..

ATLAS: The ATLAS requirements until 2012 are already available.

LHCb: The requirements for 2008 and 2009 are unchanged, but those for later years will be able to be made official in September after their Collaboration Meeting. The process should continue noting that the longer term LHCb requirements will be provided later.

CMS: M.Kasemann explained that the requirements are being discussed in the CMS CRB (Computing Resource Board) and a draft is being refined. One more week is needed.

ALICE: Y.Schutz explained that they have collected information inside the Experiment. The ALICE Collaboration Meeting will consider them on 6 September and only after the final approval they will distribute the requirements.

D.Barberis added that ATLAS is re-evaluating their event size with the new software and the changes after the TDR but the current requirements are valid for next few years and any change of requirements will be communicated well in advance.

L.Robertson added that the goal is to have the agencies commit for next year and plan for the following years. Therefore it is urgent to proceed in defining and distributing the requirements as they are now.

Decision of the MB

L.Robertson proposed that the requirements until 2012 for ATLAS and CMS, and until 2009 for LHCB, should be completed within a week and immediately distributed to the funding agencies, with a note that longer term requirements for LHCb and revised requirements for ALICE will be made available later.

The MB approved the proposal.

5.   Sites Reliability and Job Efficiency Data for July 2007 (JE_200707; SR_200707; SR_Reports_200707) - A.Aimar

5.1 Sites Reliability

As agreed in previous meetings A.Aimar collected the data about general (OPS) and VO-specific SAM tests.

All details including daily results are available in the full report attached (SR_200707).

Every month the Reliability Reports will include the VO-specific data.

The summary by site and VO is shown in the table below. One can observe that:

- Most values (general vs. Vo-specific) are consistent. The VO-specific results are better (slightly) than the general tests results.

- ALICE VO-specific tests at 0.0% to 13.5% (except RAL) are either not in production or do not work correctly.

- ATLAS with 0.0% at IN2P3 indicates some problems with the VO-specific tests at the site.

- CMS 19.9% at INFN indicates some problems with the VO-specific tests at the site.

New Action:

28 August 2007 - SAM tests in July 2007: ALICE on all tests, ATLAS@IN2P3 and CMS@INFN will investigate and report on the VO-specific results.

	OPS	ALICE	ATLAS	CMS	LHCb
CERN-PROD	94.9%	0.3%	99.2%	99.5%	93.5%
BNL-LCG2	74.6%		75.3%
FZK-LCG2	74.7%	0.0%	66.2%	78.4%	85.7%
IN2P3-CC	94.0%	3.6%	0.0%	99.5%	96.8%
INFN-T1	85.0%	7.8%	93.7%	19.9%	100.0%
NDGF-T1	96.0%		98.7%
RAL-LCG2	98.2%	85.5%	99.6%	99.1%	92.7%
SARA-MATRIX	92.3%	13.5%	93.6%	98.8%	95.4%
TRIUMF-LCG2	96.6%		97.2%
Taiwan-LCG2	83.2%	0.0%	83.7%	83.7%
USCMS-FNAL-WC1	92.2%			99.9%
pic	95.9%		99.6%	99.6%	95.2%

5.2 Job Efficiency

A.Aimar provided a summary of the Job Efficiency data (by P.Saiz) for July 2007.

All details including daily results are available in the full report (JE_200707).

Summary views are available below, by site and by VO. One can observe that:

- Most values are above 90%, with a few exceptions.

- ATLAS 34.9% at IN2P3

- ALICE 11.5% at RAL

- Only ATLAS Ganga 938 jobs. The jobs accounted are only those that go via the RB.

From next month the Site Reliability and the Job Efficiency table will be combined in order to spot additional correlations.

New Action

10 August 2007 - Sites and Experiments that disagree with the status of the milestones in the table below should send their feedback to A.Aimar before the 10 August 2007

Job Efficiency Summary by SITE
		Successful Jobs	Total Jobs	Success Rate
ASGC	Total	1155	1194	96.73%
	ALICE-AGENT	0	0	n/a
	ATLAS-GANGA	0	0	n/a
	CMS-CRAB	1155	1194	96.73%
	LHCb-PILOT	0	0	n/a
CERN	Total	109700	129006	85.03%
	ALICE-AGENT	50462	63848	79.03%
	ATLAS-GANGA	62	66	93.94%
	CMS-CRAB	30829	34547	89.24%
	LHCb-PILOT	28347	30545	92.80%
IN2P3	Total	11720	13300	88.12%
	ALICE-AGENT	2775	3158	87.87%
	ATLAS-GANGA	94	269	34.94%
	CMS-CRAB	1252	1570	79.75%
	LHCb-PILOT	7599	8303	91.52%
INFN-CNAF	Total	33144	37595	88.16%
	ALICE-AGENT	9049	9992	90.56%
	ATLAS-GANGA	4	6	66.67%
	CMS-CRAB	189	279	67.74%
	LHCb-PILOT	23902	27318	87.50%
FNAL	Total	25706	26864	95.69%
	ALICE-AGENT	0	0	n/a
	ATLAS-GANGA	0	0	n/a
	CMS-CRAB	25706	26864	95.69%
	LHCb-PILOT	0	0	n/a
FZK	Total	28124	39501	71.20%
	ALICE-AGENT	16253	21267	76.42%
	ATLAS-GANGA	302	317	95.27%
	CMS-CRAB	5095	7877	64.68%
	LHCb-PILOT	6474	10040	64.48%
PIC	Total	6831	7117	95.98%
	ALICE-AGENT	0	0	n/a
	ATLAS-GANGA	6	6	100.00%
	CMS-CRAB	1008	1028	98.05%
	LHCb-PILOT	5817	6083	95.63%
RAL	Total	10964	17536	62.52%
	ALICE-AGENT	821	7112	11.54%
	ATLAS-GANGA	10	10	100.00%
	CMS-CRAB	171	179	95.53%
	LHCb-PILOT	9962	10235	97.33%
SARA-NIKHEF	Total	26859	32079	83.73%
	ALICE-AGENT	15916	16949	93.91%
	ATLAS-GANGA	264	264	100.00%
	CMS-CRAB	0	0	n/a
	LHCb-PILOT	10679	14866	71.84%

Job Efficiency Summary by VO
	Successful Jobs	Total Jobs	Success Rate
ALICE-AGENT	95276	122326	77.9%
ASGC	0	0	n/a
CERN	50462	63848	79.0%
IN2P3	2775	3158	87.9%
INFN-CNAF	9049	9992	90.6%
FNAL	0	0	n/a
FZK	16253	21267	76.4%
PIC	0	0	n/a
RAL	821	7112	11.5%
SARA-NIKHEF	15916	16949	93.9%
ATLAS-GANGA	742	938	79.1%
ASGC	0	0	n/a
CERN	62	66	93.9%
IN2P3	94	269	34.9%
INFN-CNAF	4	6	66.7%
FNAL	0	0	n/a
FZK	302	317	95.3%
PIC	6	6	100.0%
RAL	10	10	100.0%
SARA-NIKHEF	264	264	100.0%
CMS-CRAB	65405	73538	88.9%
ASGC	1155	1194	96.7%
CERN	30829	34547	89.2%
IN2P3	1252	1570	79.7%
INFN-CNAF	189	279	67.7%
FNAL	25706	26864	95.7%
FZK	5095	7877	64.7%
PIC	1008	1028	98.1%
RAL	171	179	95.5%
SARA-NIKHEF	0	0	n/a
LHCb-PILOT	92780	107390	86.4%
ASGC	0	0	n/a
CERN	28347	30545	92.8%
IN2P3	7599	8303	91.5%
INFN-CNAF	23902	27318	87.5%
FNAL	0	0	n/a
FZK	6474	10040	64.5%
PIC	5817	6083	95.6%
RAL	9962	10235	97.3%
SARA-NIKHEF	10679	14866	71.8%

6. High Level Milestones July 2007 (HLM_200707) A.Aimar

A.Aimar updated the High-Level Milestones dashboard with the input received via email by the sites (only a few replied).

The milestones on which there was no feedback are considered as “not done” (i.e. orange/red).

WLCG-07-26, -27, 29 have been changed to reflect the milestones on the SRM2 Roll-out. With specific dates for ATLAS (Sept 2007), CMS (Oct 2007) and LHCb (Aug 2007).

WLCG-07-05b: D.Barberis noted that for BNL ATLAS had tested and accepted the SLA for VO Boxes support.

G.Merino asked whether ATLAS intends to run VO Boxes at all Tier-1 sites or only at CERN.

D.Barberis replied that some sites want to have the VO Boxes running locally and the issue is discussed individually with each site.

O.Smirnova observed that the current discussions about Security could cause changes to the VO Boxes SLAs being defined.

L.Robertson replied that the SLAs should be completed and then compared with any future the Security documents and adapted as necessary.make references to its recommendations.

T.Doyle added that common generic guidelines for SLAs of all sites would be very useful.

Ph.Charpentier asked that the sites inform the Experiments when they move their CEs to SL4.

G.Merino noted that the sites update the BDII information and that is the official way of informing the applications. Changes are noted also And at the Operations meetings but often the Experiments are not attending.

Ph.Charpentier nevertheless asked the sites to provide the information in addition via EGEE Broadcast until a better way is defined.

New Action

10 August 2007 - Sites and Experiments that disagree with the status of the milestones in the table below should send their feedback to A.Aimar before the 10 August 2007

7-Aug-07

WLCG High Level Milestones - 2007

Done (green)

Late < 1 month (orange)

Late > 1 month (red)

ID

Date

Milestone

ASGC

CC IN2P3

CERN

FZK GridKa

INFN CNAF

NDGF

PIC

RAL

SARA NIKHEF

TRIUMF

BNL

FNAL

24x7 Support

WLCG-07-01

Feb 2007

24x7 Support Definition Definition of the levels of support and rules to follow, depending on the issue/alarm

Sep 2007

Jun Aug 2007

Sep 2007

WLCG-07-02

Apr 2007

24x7 Support Tested Support and operation scenarios tested via realistic alarms and situations

Oct 2007

WLCG-07-03

Jun 2007

24x7 Support in Operations The sites provides 24x7 support to users as standard operations

Sept 2007

VOBoxes Support

WLCG-07-04

Apr 2007

VOBoxes SLA Defined Sites propose and agree with the VO the level of support (upgrade, backup, restore, etc) of VOBoxes

Jun Aug 2007

WLCG-07-05

May 2007

VOBoxes SLA Implemented VOBoxes service implemented at the site according to the SLA

WLCG-07-05b

Jul 2007

VOBoxes Support Accepted by the Experiments VOBoxes support level agreed by the experiments

ALICE

n/a

n/a

ATLAS

CMS

n/a

n/a

LHCb

n/a

n/a

VOMS Job Priorities

VOMS Milestones below suspended until the VOMS Working Group defines new milestones.

WLCG-07-06b

Jun 2007

New VOMS YAIM Release and Documentation VOMS release and deployment. Documentation on how to configure VOMS for sites not using YAIM

EGEE-SA1

WLCG-07-06

Apr 2007

Job Priorities Available at Site Mapping of the Job priorities on the batch software of the site completed and information published

WLCG-07-07

Jun 2007

Job Priorities of the VOs Implemented at Site Configuration and maintenance of the jobs priorities as defined by the VOs. Job Priorities in use by the VOs.

Accounting

WLCG-07-08

Mar 2007

Accounting Data published in the APEL Repository The site is publishing the accounting data in APEL. Monthly reports extracted from the APEL Repository.

Sept 2007

3D Services

WLCG-07-09

Mar 2007

3D Oracle Service in Production Oracle Service in production, and certified by the Experiments

squid frontier

WLCG-07-10

May 2007

3D Conditions DB in Production Conditions DB in operations for ATLAS, CMS, and LHCb. Tested by the Experiments.

squid frontier

Procurement

WLCG-07-16

1 Jul 2007

MoU 2007 Pledges Installed To fulfill the agreement that all sites procure the 2007 MoU pledged by July 2007

Sept 2007

Sept 2007

Sept 2007

Jan 2008

Aug 2007

Jul Aug 2007

WLCG-07-17

1 Apr 2008

MoU 2008 Pledges Installed To fulfill the agreement that all sites procure they MoU pledged by April of every year

FTS 2.0

WLCG-07-18

Jun 2007

FTS 2.0 Tested and Accepted by the Experiments In production at CERN and accepted tested by each Experiment

ALICE

ATLAS

CMS

LHCb

WLCG-07-19

Jun 2007

Multi-VO Tests Executed and Tested by the Experiments Scheduled at CERN for last week of June

ALICE

ATLAS

CMS

LHCb

WLCG-07-20

Sept 2007

FTS 2.0 Deployed in Production Installed and in production at each Tier-1 Site

BDII

WLCG-07-21

Jun 2007

BDII Guidelines Available On how to install BDII on a separated node

EGEE - SA1

WLCG-07-22

Jun 2007

Top-Level BDII Installed at the Site For each Tier-1 site

glexec

WLCG-07-24

Jul 2007

Decision on Usage of glexec and Guidelines to Follow

GDB

Site Reliability - June 2007

WLCG-07-12

Jun 2007

Site Reliability above 91% Considering each Tier-0 and Tier-1 site

March

April

May

June

WLCG-07-13

Jun 2007

Average of Best 8 Sites above 93% Eight sites should reach a reliability above 93%

MSS Main Storage Systems

WLCG-07-25

Jun 2007

CASTOR 2.1.3 in Production at CERN MSS system supporting SRM 2.2 deployed in production at the site

CERN Tier-0

WLCG-07-26

Aug Oct 2007

SRM: CASTOR 2.1.3 Tested and Accepted by the Experiments From the SRM Roll-Out Plan (SRM-16 to -19)

ALICE n/a

ATLAS Sept 2007

CMS Oct 2007

LHCb Aug 2007

WLCG-07-27

Aug Oct 2007

SRM: dCache 1.8 Tested and Accepted by the Experiments From the SRM Roll-Out Plan (SRM-16 to -19)

ALICE n/a

ATLAS Sept 2007

CMS Oct 2007

LHCb Aug 2007

WLCG-07-28

Sept 2007

Demonstrated Tier-0 Performance Demonstration that the highest throughput (ATLAS 2008) can be reached.

CERN Tier-0

WLCG-07-29

Nov 2007

SRM: CASTOR 2.1.3/dCache in Production at the T1 Site From the SRM Roll-Out Plan (SRM-20 to -21a)

WLCG-07-30

Dec 2007

SRM Implementations with HEP MoU features With full features agreed in the HEP MoU (srmCopy, etc).

CASTOR

DCache

DPM

SL4 Migration

WLCG-07-11

Depl Date + 30d

SL4 Operational at Site (for WN and UI nodes) This has to happen within 30 days after the release from GD.

Replaced by individual milestones for the Middleware components.

WN and UI

WLCG-07-31

Jun 2007

WN Installed in Production at the Tier-1 Sites WN on SL4 installed on each Tier-1 site, with the configuration needed to use SL4 or SL3 nodes

July 2007

July 2007

n/a

July 2007

July 2007

July 2007

WLCG-07-32

Jun 2007

UI Certification and Installation on the PPS Systems

EGEE - SA1-PPS done: Jul 2007

WLCG-07-33

Aug 2007

UI Tested and Accepted by the Experiments

ALICE

ATLAS

CMS

LHCb

gLite CE

WLCG-07-35

Sept 2007

gLite CE Development Completed and Component Released

EGEE - JRA1

WLCG-07-36

+4 weeks

gLite CE Certification and Installation on the PPS Systems

EGEE - PPS

WLCG-07-37

+4 weeks

gLite CE Tested and Accepted by the Experiments

ALICE

ATLAS

CMS

LHCb

WLCG-07-38

+4 weeks

gLite CE Installed in Production at the Tier-1 Sites

n/a

SAM Vo-Specific Tests

WLCG-07-39

Sept 2007

VO-Specific SAM Tests in Place With results included every month in the Site Availability Reports.

ALICE

ATLAS

CMS

LHCb

CAF CERN Analysis Facility

WLCG-07-40

Oct 2008

Experiment provide the Test Setup for the CAF

ALICE

ATLAS

CMS

LHCb

xrootd

WLCG-07-41

Jul 2007

xrootd Interfaces Tested and Accepted by ALICE

ALICE

Site Reliability - Dec 2007

WLCG-07-14

Dec 2007

Site Reliability above 93% Considering each Tier-0 and Tier-1 site

Sept

Oct

Nov

Dec

WLCG-07-15

Dec 2007

Average of Best 8 Sites above 95% Eight sites should reach a reliability above 93%

7. AOB

No AOB.

-

8. Summary of New Actions

New Action

10 August 2007 - Sites and Experiments that disagree with the status of the milestones in the table below should send their feedback to A.Aimar before the 10 August 2007

New Action:

28 August 2007 - SAM tests in July 2007: ALICE on all tests, ATLAS@IN2P3 and CMS@INFN will investigate and report on the VO-specific results.

The full Action List, current and past items, will be in this wiki page before next MB meeting.