List of exotics RSE:

Note that you will need to have exotics group privileges in order to write on those disks, so set up your certificate accordingly before running the script.

voms-proxy-init -voms=atlas:/atlas/phys-exotics/Role=production -valid 96:00

(this in principle contradicts what is in RucioClientsHowTo: "Long term storage for group datasets: The group datasets are user (possible that this user is working for a group) datasets replicated in group areas. Only the data manager of the group can request the replication of datasets.")

Instructions by Arturos Sanchez Pineda, CxAOD responsible for HSG5. (need to be subscribed to the mailing list to view)



It expects your directory to have the following structure

[yourBaseDir]/group.phys-exotics.[dataset name]/*.root


[yourBaseDir]/user.[dataset name]/*.root

Not clear yet.

Logging work


what happens is that the script tries to read the dataset name from here:

= -bash-4.1$ ls /afs/*/*/*.root* | egrep -E 'group.phys|user.' | sed 's|\.\_00|\n| g' | egrep -E 'group.phys|user.' | sort | uniq | sed 's|\/group\.phys\-higgs\.|\.GRID\.|g' | sed 's|\/group\.phys\-exotics\.|\.GRID\.|g' | sed 's|\/user\.|\.GRID\.|g' | sed "s|\/|\nuser\.doglioni\:|g" | grep \.GRID\. =

and that won't work because these are root files named like this: group.phys-exotics.7955865._000002.tree.root

Check how they are called in Arturos' CxAOD directory:

eos/atlas/atlascerngroupdisk/phys-exotics/CxAODFramework/HIGG5D1_13TeV/CxAOD-00-20-00_mc15b/ -ZZ_Pw/ --group.phys-exotics.mc15_13TeV.361611.PwPy8EGnloME_AZNLOCTEQ6L1_ZZvvqq_mqq20.s2726.HIGG5D1.20-0_CxAOD.root/


so ok, nothing different here, just how many directories this crawls down to. We are looking at the same root files.

Need to change to:

/afs/*/*/*.root* source eos/atlas/atlascerngroupdisk/phys-exotics/jdm/dijet/TLA/Data/TransferTest/TransferTestDS/DataScouting/ doglioni CERN-PROD_PHYS-EXOTICS 12


Provided RSE expression is considered invalid

The problem is when it adds a LOCALGROUPDISK to it, and we don't want that for EXO pledged resources. Changed it in the script.

Problem now: not enough space

rucio add-rule user.doglioni:Data.data15_13TeV.00284484.calibration_DataScouting_05_Jets.v6_AllCalibAllDataScouting_20160316_tree.root.7955865 1 'rse=GOEGRID_PHYS-EXOTICS' 2016-10-09 16:24:08,903 ERROR [There is not enough quota left to fulfil the operation. Details: There is insufficient quota on any of the target RSE's to fullfill the operation.]

and also some trouble with the naming:

rucio upload --rse GOEGRID_PHYS-EXOTICS_SCRATCHDISK user.doglioni:Data.data15_13TeV.00284484.calibration_DataScouting_05_Jets.v6_AllCalibAllDataScouting_20160316_tree.root.7955865 /afs/

2016-10-09 16:27:33,529 WARNING [user.doglioni:Data.data15_13TeV.00284484.calibration_DataScouting_05_Jets.v6_AllCalibAllDataScouting_20160316_tree.root.7955865 cannot be distinguished from scope:datasetname. Skipping it.]


From the hypernews:

mkdir eos eos fuse mount eos

cd /afs/

rucio add-dataset user.wolter:myDataset rucio add-rule user.wolter:myDataset 1 'rse=DESY-HH_SCRATCHDISK' rucio -v upload --rse DESY-HH_SCRATCHDISK user.wolter:myDataset group.phys-higgs.8962162._000001.CxAOD.root group.phys-higgs.8962162._000002.CxAOD.root


on lxplus028

usage: rucio add-rule [-h] [--weight WEIGHT] [--lifetime LIFETIME] [--grouping {DATASET,ALL,NONE}] [--locked] [--source-replica-expression SOURCE_REPLICA_EXPRESSION] [--notify NOTIFY] [--activity ACTIVITY] [--comment COMMENT] [--ask-approval] [--asynchronous] [--account RULE_ACCOUNT] dids [dids ...] copies rse_expression

rucio add-dataset user.doglioni:DSData2015

rucio add-rule user.doglioni:DSData 1 'rse=UKI-NORTHGRID-MAN-HEP_SCRATCHDISK'

rucio -v upload --rse UKI-NORTHGRID-MAN-HEP_SCRATCHDISK user.doglioni:DSData2015.data15_13TeV.00284484.calibration_DataScouting_05_Jets.v6_AllCalibAllDataScouting_20160316_tree.root.7955865 /afs/


Current command line to be used the FIRST time you will upload CxAOD samples:

source <eos-as-local-path-from-$HOME> <grid-user-nickname> <name-of-logical-GRID-site> <lifetime-in-#months>

// eos/atlas/atlascerngroupdisk/phys-higgs/HSG5/Run2/VH/HIGG5D2_13TeV/CxAOD_00-18-00 arturos INFN-NAPOLI-ATLAS-BLAH 24
//source ~/eos/atlas/atlascerngroupdisk/phys-exotics/jdm/dijet/TLA/Data/TransferTest doglioni CERN-PROD_PHYS-EXOTICS 12

To Update files in datasets (and/or their lifetime) already created from a given EOS path:

source <eos-as-local-path-from-$HOME> <grid-user-nickname> <name-of-logical-GRID-site> <lifetime-in-#months>

//Example: eos/atlas/atlascerngroupdisk/phys-higgs/HSG5/Run2/VH/HIGG5D2_13TeV/CxAOD_00-18-00 arturos INFN-NAPOLI-BLAH 24

source /eos/atlas/atlascerngroupdisk/phys-exotics/jdm/dijet/TLA/Data/ doglioni UKI-NORTHGRID-MAN-HEP_SCRATCHDISK 12

-- CaterinaDoglioni - 2016-10-09

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2016-11-10 - CaterinaDoglioni
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback