Manual File Re-Injection

Introduction

Some times files are invalidated globally in TMDB, but still exists at storage. In that cases, you may want to re-inject the files. Also, files could exist at TMDB but with no replica, even if they have repica(s) at a given site(s). In those cases, you need to invalidate the files globally (with -keepempty block, to maintain existing subscriptions) and then re-inject them into TMDB

NOTE: When injecting to tape (_Buffer & _MSS), you just need to inject into Buffer. Then, if there is a subscription, the MSS replica will be recreated. If there is no subscription, you need to subscribe the files

Step By Step

Step 1 Generate the injection .xml file. The file should contain the XML structure representing the data to be injected.

  • See requirements of the XML file at phedex::Web::API::Inject
  • Script to generate the XML file from a csv file at: /afs/cern.ch/user/j/jodiazcr/public/csv2xml.py
    • CSV file:
    • dataset_name1,block_name1,file_name1,file_size1,file_cksum1
      dataset_name2,block_name2,file_name2,file_size2,file_cksum2
      ...
      dataset_nameN,block_nameN,file_nameN,file_sizeN,file_cksumN

Step 2 Reactivate the blocks.

  • Command:
    • while read line; do sqlplus -S $(~/home/phedex/PHEDEX/Utilities/OracleConnectId -db ~/private/DBParam.Jorge:Prod/OPSJORGE) @/afs/cern.ch/user/j/jodiazcr/private/reactivateblock.sql $line; done < /afs/cern.ch/user/j/jodiazcr/private/blocks_to_reactivate.list
  • SQL:
    • set role phedex_opsjorge_prod identified by XXXXXXXXXXX;
      insert into t_dps_block_activate (block,time_request,time_until) values ((select id from t_dps_block where name=':1'), (SYSDATE - TO_DATE('1970-01-01', 'YYYY-MM-DD')) *60*60*24, (SYSDATE - TO_DATE('1970-01-01', 'YYYY-MM-DD')) *60*60*24 + 86400);
      commit;
      quit
      /

Step 3 Then wait ~15 mins for the central BlockActivate agent to activate the block. Check mgmt-blockactivate logs for block activation message.

  • Ssh to Phedex Machine
    • ssh phedex@vocms0214
  • Check mgmt-blockactivate
    • cd /data/ProdNodes/Prod_Mgmt/logs
      tail -n 30 mgmt-blockactiv
  • You can also check if the block is active directly in TMDB:
    • select is_active from t_dps_block_replica where block=6414441

Step 4 Reopen the blocks.

  • Commnad:
    • while read line; do sqlplus -S $(~/home/phedex/PHEDEX/Utilities/OracleConnectId -db ~/private/DBParam.Jorge:Prod/OPSJORGE) @/afs/cern.ch/user/j/jodiazcr/private/reopenblock.sql $line; done < /afs/cern.ch/user/j/jodiazcr/private/blocks_to_reopen.list
  • SQL:
    • set role phedex_opsjorge_prod identified by xxxxxxxxxxxxxxxx;
      update t_dps_block set is_open='y', time_update=((SYSDATE - TO_DATE('1970-01-01', 'YYYY-MM-DD')) *60*60*24-3600) where name=':1';
      commit;
      quit
      /

Step 5 Use PhEDEx datasvc inject API to inject the files and close the blocks

  • Command:
    • source ~/PHEDEX/etc/profile.d/env.sh
      ~/PHEDEX/Utilities/phedex --instance prod inject --datafile filestoinject.xml --node TX_XX_XXXXX_XXX --strict

Step 6 (Optional) You can check the replicas of the files to make sure the files have been injected at the correct sites

-- JorgeAlbertoDiazCruz - 2016-02-03

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r9 - 2019-06-27 - DonataMielaikaite
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback