dCache upgrade task force

Introduction

The third-party-copy sub-group of the DOMA working group is dedicated to improving bulk transfers between WLCG sites. All WLCG sites are required to upgrade to the releases of storage implementation which support third-party-copy. For dCache storage implementation the recommended versions are those which are equal or higher than 5.2.15. Another important feature which has to be provided by the WLCG storage services is enabling of the Storage Resource Reporting (SRR). The task force will follow up migration of the WLCG dCache sites to version 5.2.15 and higher and enabling of SRR

Initial status

dCache versions of the WLCG dCache sites as of 29.09.2019

• According to CRIC there are 43 sites which are running dCache and are used by the WLCG VOs. The number can be higher since for some of sites CRIC is lacking information about storage implementation and implementation versions the sites are running. In particular it concerns OSG sites, though the number of the OSG sites running dCache might be low.
• Only 6 sites (JINR is counted twice as T1 and T2) are currently running dCache instances with version 5.2.0 and higher. However, not necessarly all instances of the sites are upgraded to 5.2.0.

Current activities

• Ticket sites for upgrade to version 5.2.15 or higher or to enable SRR in case site is already running required versions
• Follow up on progress

Site Instructions

Step by step instructions and useful links are listed below. In case you have questions or comments do not hesitate to contact dcache-upgrade-tf@cernNOSPAMPLEASE.ch

General dCache instructions

• Manual
• Downloads

SRR

• How to enable SRR in latest series of stable dCache releases
• Link for for dCache documentation
• File should be update every half an hour

Update information for your site in CRIC

After changes performed on your service, please, update information in CRIC

Authentication & authorization step

• Go to WLCG cric server , click Core (menu on the top of the page) -> Services. Enable filtering, by clicking on the 'Filter' button and select your site. By default , you won't see implementation and implementation version columns in the table. In order to see this info, you need to click on 'Columns' and then select corresponding columns in the drop down list.

• You should be able to list all CRIC entities (sites (GocDB /OIM and experiment-specific ones), federations, pledges, services, storage protocols and queues) without authentication. However, once you would like to see details of any particular entity, you would be asked to login.

• Those who are registered in the CERN DB, please, use SSO authentication. Authentication with certificate is not yet enabled on this instance, will come soon.

• Those who are not registered in the CERN DB would need to ask for CRIC local account. Please, send a mail to

• As soon as you are logged in, you will be able to see details of any CRIC entity, however in order to edit in information, one would need to get specific privileges.
• As soon as you are authenticated, you will see 'Request privileges' on the top of the page next to your login name. Please, click on it and follow up the request procedure which allows to request global admin privileges, site admin privileges or federation admin privileges. Ask for sites admin privileges for your site. You will be shortly informed that your privileges are enabled. Please re-login.

Editing storage info

• Once you login with appropriate privileges, you should be able to edit information about your site. At the moment we are particularly interested in storage info at your site, namely its implementation, implementation version and SRR URL when it enabled.

• CRIC creates virtual storage service per site/per VO/per media/per implementation. By default it creates 1 disk and 1 tape virtual storage for every VO which is served by a given T1 site and 1 disk visrtual storage for T2 site. However, if for a given VO there are storage instances for the same media but different implementation (for example EOS and dCache instances for disk storage for ATLAS), CRIC should create two different disk virtual storage instances for this VO. Unfortunately, for the moment, there is no reliable primary source for this kind of information, so it is highly likely that only a single virtual storage will be created by CRIC in such cases. Would be great if you could correct it using CRIC UI and add other storage virtual instances with their implementation , implementation versions for your site and SRR URL when it is enabled. In the future we hope to get this information through SRR (Storage Resource Reporting).

• In the service table view, click on a particular service name
• You get a form with detailed information about service
• Click on the 'Edit' button under the first block of information
• You get another form. Please, correct 'Version' of your dCache implementation. In the "Resource Reporting URL" field provide your SRR link
• Click on 'Check input data' and save info

Configuration for Third Party Copy

• Detailed instructions for dCache storage configuration for Third Party Copy

WLCG sites which need upgrade to 5.2.*

Site	dCache Version (11.12.2019)	Upgrade is planned (date)	SRR	Comments	GGUS ticket	Contacts
USCMS-FNAL-WC1	[None, None]			dCache recently upgraded, SRR & CRIC pending. CRIC is not updated with dCache version	GGUS:145964	cms-t1@fnalNOSPAMPLEASE.gov
GoeGrid	[u'5.2.18']		dCache upgraded, SRR is pending	GGUS:144489	goegrid_admin@physikNOSPAMPLEASE.uni-goettingen.de
wuppertalprod	[u'6.2.10']		End of April	Upgrade is done. Waiting for fix in SRR to enable it. Ticket has been updated with a request to upgrade to 5.2.15 and higher	GGUS:144504	lcg-admin@physikNOSPAMPLEASE.uni-wuppertal.de
CIEMAT-LCG2	[u'5.2.27']	DONE			GGUS:144484	lcg.support@ciematNOSPAMPLEASE.es
BNL-ATLAS	[u'6.2.0', u'6.2.0']	Waiting for 2.6 to be released, then will upgrade asap		Upgrade is done. Though the SRR has a problem with empty shares	GGUS:144481	jhover@bnlNOSPAMPLEASE.gov
CSCS-LCG2	[u'5.2.17']			Upgraded to 5.2.17, SRR to be done	GGUS:144485	grid@cscsNOSPAMPLEASE.ch
INFN-LNL-2	[u'5.2.20']	Done		Upgraded to 5.2.20, working on SRR	GGUS:144492	grid-prod@lnlNOSPAMPLEASE.infn.it
INFN-ROMA1-CMS	[u'5.2.31']			SRR has an issue with empty shares section. Upgrade to 5.2.31 is done	GGUS:144493	grid-prod@roma1NOSPAMPLEASE.infn.it
DESY-ZN	[u'5.2.25']	Jan 2020		DONE	GGUS:144487	grid-admin@zeuthenNOSPAMPLEASE.desy.de
CA-VICTORIA-WESTGRID-T2	[u'4.2.31']			No progress. Ticket was updated with the request to upgrade to 5.2.15 and higher	GGUS:144482	rcf-lcg@uvicNOSPAMPLEASE.ca
CA-WATERLOO-T2	[u'5.2.?']			Upgrade to 5.2, SRR is pending. Ticket is updated with the request to upgrade to 5.2.31	GGUS:144483	lcg-support-waterloo@computeontarioNOSPAMPLEASE.ca
UAM-LCG2	[u'5.2.20']			Upgraded to 5.2.20 , no SRR	GGUS:144501	lcg.support@uamNOSPAMPLEASE.es
T3_CH_PSI	[u'5.2.22']			DONE	GGUS:144499	cms-tier3@listsNOSPAMPLEASE.psi.ch
RWTH-Aachen	[u'5.2.15']	Feb 2020		Upgraded 5.2.15, SRR has an issue with empty shares section. End of October 2020: upgraded to 6.2.7	GGUS:144498	lcg-contact@physikNOSPAMPLEASE.rwth-aachen.de
RU-Protvino-IHEP	[u'5.2.31']			DONE	GGUS:144497 GGUS:149015	lcg@ihepNOSPAMPLEASE.ru
ITEP	[u'5.2.31']			Upgrade performed, SRR is not accessible, should be followed in a new ticket	GGUS:144494 Not clear which version site is running, GGUS:149008	lublev@itepNOSPAMPLEASE.ru
praguelcg2	[u'5.2.15']			Upgrade has been performed to 5.2.15, SRR is not enabled to be followed in another ticket	GGUS:144496	ngi-firstline@metacentrumNOSPAMPLEASE.cz
UNI-FREIBURG	[u'5.2.35']			Followed up by Oliver	GGUS:144502, ticket updated	lcg-admin@bfgNOSPAMPLEASE.uni-freiburg.de
BEIJING-LCG2	[u'2.16.22']			Plan to move to DPM	GGUS:144480	lcg-admin@ihepNOSPAMPLEASE.ac.cn
DESY-HH	[u'6.2.0']			Upgrade is performed. Working on SRR	GGUS:144486 Currently upgrading to 6.2	grid@desyNOSPAMPLEASE.de

WLCG dCache sites which upgraded to 5.2.* but need to enable SRR

Site	dCache Version (11.12.2019)	When SRR will be enabled (date)	SRR	Comments	G GUS ticket	Contacts
IEPSAS-Kosice	[u'5.2.28']	2020-03-03	https://se-iep-grid.saske.sk:2880/pnfs/saske.sk/data/srr/storage-descriptor_se-iep-grid.saske.sk.json	DONE	GGUS:144490 GGUS:149006	grid.support@saskeNOSPAMPLEASE.sk
ifae	[u'5.2.16']		https://webdav-dteam.pic.es:8448/storage-descriptor.json	SRR is enabled. But in SRR shares section is empty, only overall space size is provided	GGUS:144491	lcg.support@picNOSPAMPLEASE.es
NDGF-T1	[u'6.2.6']		https://dav.ndgf.org/ops/storage-descriptor.json		GGUS:144516	support@ndgfNOSPAMPLEASE.org
FI_HIP_T2	[u'5.2.31.noarch']			DONE	GGUS:144488 GGUS:149005	finnish-lhc-admins@helsinkiNOSPAMPLEASE.fi
LRZ-LMU	[u'6.0.0']	2020-01-24	http://lcg-lrz-http.grid.lrz.de/pnfs/lrz-muenchen.de/data/atlas/meta/storage-descriptor.json	DONE	GGUS:144495	lcg-admin@listsNOSPAMPLEASE.lrz.de
BEgrid-ULB-VUB	[u'5.2.26']		https://maite.iihe.ac.be:2880/storage-descriptor.json	DONE, but SRR to be checked.	GGUS:144506 GGUS:149003	grid_admin@listservNOSPAMPLEASE.vub.ac.be
JINR-LCG2	[u'5.2.28']			DONE	GGUS:144511 GGUS:149010	lcgadm@lxmailNOSPAMPLEASE.jinr.ru
JINR-T1	[u'5.2.28']			DONE	GGUS:144512 GGUS:149011	t1adm@lxmailNOSPAMPLEASE.jinr.ru
IN2P3-CC	['5.2.22-1']		https://ccdcatli302.in2p3.fr:2880/dteam/SRR/storage-descriptor.json	SRR is enabled, but the SRR storageshares is empty.	GGUS:144510 GGUS:149007	grid.admin@ccNOSPAMPLEASE.in2p3.fr
FZK-LCG2	['5.2.31']		ATLAS, CMS, Compass, LHCb	DONE (pending approval)	GGUS:144509	lcg-admin@listsNOSPAMPLEASE.kit.edu
pic	['5.2.16']		https://webdav-dteam.pic.es:8448/storage-descriptor.json	SRR is enabled. But in SRR shares section is empty, only overall space size is provided	GGUS:144517	lcg.support@picNOSPAMPLEASE.es
KR-KISTI-GSDC-02	[u'5.2.23']			Upgrade DONE. SRR is enabled. But in SRR shares section is empty, only overall space size is provided , similar problem as in IN2P3 , has to be clarified with dCache developers .	GGUS:149012	geonmo@kistiNOSPAMPLEASE.re.kr
IN2P3-CC	['5.2.22-1']		https://ccdcatli302.in2p3.fr:2880/dteam/S	GGUS:144513	grid.admin@ccNOSPAMPLEASE.in2p3.fr
KR-KNU-T3	[u'5,2,26']			Same problem as in IN2P3 and KISTI	GGUS:144514	lcg_knu@knuNOSPAMPLEASE.ac.kr
UKI-LT2-IC-HEP	['5.2.27']				GGUS:144521	lcg-site-admin@imperialNOSPAMPLEASE.ac.uk
MPPMU	[u'5.2.25']	2020-01-23	https://grid-dav.rzg.mpg.de:2880/storage-descriptor.json	DONE SRR is not accessible	GGUS:144515 GGUS:149013	lcg-support@rzgNOSPAMPLEASE.mpg.de
UKI-SOUTHGRID-RALPP	[u'5.2.27']				GGUS:144522	PPD.GRID.Support@stfcNOSPAMPLEASE.ac.uk;pete.gronbech@physics.ox.ac.uk
CA-SFU-T2	[u'5.2.19']		https://lcg-se1.sfu.computecanada.ca:2880/atlas/storage-descriptor.json	DONE, SRR inaccessible via HTTP/WebDAV	GGUS:144507 GGUS:149004	lcg-support@sfuNOSPAMPLEASE.ca
EELA-UTFSM	[u'6.2.3']		https://se.hpc.utfsm.cl:2880/atlas/storage-descriptor.json	DONE	GGUS:144508	grid@hpcNOSPAMPLEASE.utfsm.cl
SE-SNIC-T2	[u'5.2.31']				GGUS:144520 GGUS:149016	wlcg-tier2@listsNOSPAMPLEASE.snic.se
RRC-KI-T1	[u'5.2.31' tape and disk]			DONE , CRIC has to be updated , SRR checked	GGUS:144518 GGUS:149014	gridops@gridNOSPAMPLEASE.kiae.ru
SARA-MATRIX	['6.0.16']			DONE , but SRR has the problem with empty share section	GGUS:144519	eugrid.support@surfsaraNOSPAMPLEASE.nl
TRIUMF-LCG2	['5.2.35']	Enabled	https://webdav.lcg.triumf.ca:2880/ops/SRR/storage-descriptor.json	Done, SRR to be checked	GGUS:144500 GGUS:149017	support@lcgNOSPAMPLEASE.triumf.ca

Performed tasks

Documentation for enabling SRR (DONE)

• Link to final version of SRR documentation: https://www.dcache.org/manuals/Book-5.2/srr.shtml

• Task Force has requested dCache team to provide documentation of how to enable SRR. Jira ticket to follow the progress
• First draft of documentation
• Valery Mitsyn from JINR kindly agreed to validate the documentation. There are few comments from him below:
  • We needed to install additionally libxslt, it was not installed on our dCache nodes.
    • yum -y -d 1 install libxslt
    • before calling the script /usr/sbin/dcache-storage-descriptor http://localhost:2288/info
  • They got the question where/how file can be accessed
    • the file should be accessible through http protocol. URL to the file will be recorded in CRIC
  • With what credentials the file should be accessed
    • WLCG Storage Space accounting collector has robot certificates with the credentials of all 4 LHC experiments. These certificates can be used for this purpose

Membership

• Edward Karavakis (Task Force coordinator)
• Julia Andreeva (WLCG Operations)
• Dimitrios Christidis (Storage Space Accounting expert)
• Christoph Wissing ( CMS)
• Oliver Freyermuth (ATLAS)

dCache team will provide all necessary technical expertise and guidance

Mailing lists

• dcache-upgrade@cernNOSPAMPLEASE.ch
• support@dcacheNOSPAMPLEASE.org

-- JuliaAndreeva - 2019-08-29