File Invalidations

Introduction

Occasionally, files are lost or damaged at tier sites, and the only possible action is to invalidate them in the database. Normally, a Savannah ticket is opened by the site, explaining the reason for requesting the invalidation, and listing the files that need invalidation. A Transfers Team member then does the invalidation, and logs this by sending an email to the e-group: cms-file-invalidations@cernNOSPAMPLEASE.ch

Instruction

General Principle: Local invalidation & Global invalidation

Note : for local invalidations only two steps, for global some additional.

• 1. Take the file(s) and check if it (they) has any replicas available.
• 2. a) if it does have another replica - local invalidation (TMDB invalidate with -node parameter)
  • This is done with local invalidation, which don't need go further with the left steps.
  • After local invalidation, the file is re-transfered to the site automatically.
• 2. b) if it doesn't have other replicas, which means it has the only replica at site which is asking to invalidate -- global invalidation.
• 3. Make invalidation with -invalidate option
• 4. Make DBS invalidation (with -h option you can get help)
• 5. Send an email to cms-file-invalidations@cernNOSPAMPLEASE.ch saying which file(s) and why.

TMDB (PhEDEx) invalidation

• Step1: Set up micro-phedex env
  • For bash shell: source {PhEDEx directory}/etc/profile.d/env.sh
  • For c shell: source {PhEDEx directory}/etc/profile.d/env.csh

• Step2: Test the configuration of phedex
  • {PhEDEx directory}/Utilities/FileDeleteTMDB -h
  • This also produces the instruction to use FileDeleteTMDB.

• Step3: Local invalidation
  • {PhEDEx directory}/Utilities/FileDeleteTMDB -db ~/phedex/info/DBParam:Prod/OPSLIUSI -list lfn:[lfn] -node [site name]
  • For T1 tape nodes, PhEDEx will restart the transfer only if file is invalidated on both Buffer and MSS (eg. -node T1_IT_CNAF_Buffer, T1_IT_CNAF_MSS )

• Step4: Global invalidation
  • {PhEDEx directory}/Utilities/FileDeleteTMDB -db ~/phedex/info/DBParam:Prod/OPSLIUSI -list lfn:[lfn] -invalidate

DBS3 invalidation (Global invalidation only)

• Step1: Set up CMSSW env

• Step2: Set up gLite env
  • source /uscmst1/prod/grid/gLite_SL5.sh

• Step3: Get voms proxy
  • voms-proxy-init -voms cms -cert [your cert] -key [your key]

• Step4: Setup DBS3 env
  • source /opt/cmsdatatransfers/sw/comp.pre/slc5_amd64_gcc461/cms/dbs3-client/3.1.6a/etc/profile.d/init.sh

• Step5: DBS3 invalidation command
  • Copy DBS3SetFileStatus.py script from your DBS3 environment: cp /opt/cmsdatatransfers/sw/comp.pre/slc5_amd64_gcc461/cms/dbs3-client/3.1.5a/examples/DBS3SetFileStatus.py
  • ./DBS3SetFileStatus.py --url=https://cmsweb.cern.ch/dbs/prod/global/DBSWriter --status=invalid --recursive=False --files=LFN

• Step6: Crosscheck file status
  • https://cmsweb.cern.ch/dbs/prod/global/DBSWriter/files?detail=1&logical_file_name=[lfn]

DBS3 invalidation for SL6 (Global invalidation only)

• Step1: Set up CMS Software release environment.
  • cd /afs/cern.ch/user/j/"user_id"/home/cmssw/CMSSW_7_1_4/src
  • cmsenv
  • cd ~

• Step 2: Setup crab3 to get access to DBS3 client
  • source /cvmfs/cms.cern.ch/crab3/crab.sh

• Step 3: Copy DBS3SetFileStatus.py if needed
  • curl https://raw.githubusercontent.com/dmwm/DBS/master/Client/utils/DataOpsScripts/DBS3SetFileStatus.py > DBS3SetFileStatus.py

• Step 4: Grid Proxy
  • voms-proxy-init -voms cms -cert [your cert] -key [your key]

• Step5: DBS3 invalidation command
  • ./DBS3SetFileStatus.py --url=https://cmsweb.cern.ch/dbs/prod/global/DBSWriter --status=invalid --recursive=False --files=LFN

• Step6: Crosscheck file status
  • https://cmsweb.cern.ch/dbs/prod/global/DBSWriter/files?detail=1&logical_file_name=[lfn]

Debug File Invalidation

• Note: use Debug part of DBParam file while invalidating debug test files

• Step1: Find the correct name of the corrupt source file since probably site will give a file like "/LoadTest07/LoadTest07_Estonia_A0"
  • actual LFN: /store/PhEDEx_Debug/LoadTest07Source/SITENAME_XYZ
  • can be found like https://cmsweb.cern.ch/phedex/datasvc/xml/debug/filereplicas?lfn=/store/*/*/*Estonia*_A0
  • /store/PhEDEx_Debug/LoadTest07Source/Estonia_A0

• Step2: Invalidate source file using Debug part of DBParam file
  • ./FileDeleteTMDB -db ~/param/DBParam:Debug/Meric -list [file] -invalidate -keepempty dataset

• Step3: Find all injection files produced with this source file and invalidate them
  • https://cmsweb.cern.ch/phedex/datasvc/xml/debug/filereplicas?lfn=/*/*/*/*/*/LoadTest07_*SITENAME*_XYZ_*
  • e.g. https://cmsweb.cern.ch/phedex/datasvc/xml/debug/filereplicas?lfn=/*/*/*/*/*/LoadTest07_*Estonia*_A0_*
  • You can use datasvc.py script to get lfn list:
    • ~/TransferTeam/commons/datasvc.py --service filereplicas --instance debug --options "lfn=/*/*/*/*/*/LoadTest07_*Estonia*_A0_*" --path /phedex/block/file:name

• Step4: Invalidate injection files using Debug part of DBParam file
  • ./FileDeleteTMDB -db ~/param/DBParam:Debug/Meric -list [file] -invalidate -keepempty dataset

• Since these are PhEDEx test files, they're not registered to DBS, so no action is required in DBS

• If you need to invalidate all source&destination loadtest files to inject new ones, you can check this twiki

• NOTE: Use "-keepempty dataset" option, since you can not delete the entire dataset

Some statistics of invalidations per month, based on the logs in the e-group.