List of exotics RSE:

http://atlas-agis.cern.ch/agis/ddmendpoint/table_view/?&name=PHYS-EXOTICS&state=ACTIVE

Note that you will need to have exotics group privileges in order to write on those disks, so set up your certificate accordingly before running the script.

voms-proxy-init -voms=atlas:/atlas/phys-exotics/Role=production -valid 96:00

(this in principle contradicts what is in RucioClientsHowTo: "Long term storage for group datasets: The group datasets are user (possible that this user is working for a group) datasets replicated in group areas. Only the data manager of the group can request the replication of datasets.")

Instructions by Arturos Sanchez Pineda, CxAOD responsible for HSG5.

https://its.cern.ch/jira/browse/CXAOD-193 (need to be subscribed to the mailing list to view)

Script:

/afs/cern.ch/user/a/arturos/public/ATLAS/scripts/UploadEOSfilesToGRID.0.1.sh

It expects your directory to have the following structure

[yourBaseDir]/group.phys-exotics.[dataset name]/*.root

or

[yourBaseDir]/user.[dataset name]/*.root

Not clear yet.

Logging work

20161008

what happens is that the script tries to read the dataset name from here:

= -bash-4.1$ ls /afs/cern.ch/user/d/doglioni/eos/atlas/atlascerngroupdisk/phys-exotics/jdm/dijet/TLA/Data/TransferTest/*/*/*.root* | egrep -E 'group.phys|user.' | sed 's|\.\_00|\n| g' | egrep -E 'group.phys|user.' | sort | uniq | sed 's|\/group\.phys\-higgs\.|\.GRID\.|g' | sed 's|\/group\.phys\-exotics\.|\.GRID\.|g' | sed 's|\/user\.|\.GRID\.|g' | sed "s|\/|\nuser\.doglioni\:|g" | grep \.GRID\. =

and that won't work because these are root files named like this: group.phys-exotics.7955865._000002.tree.root

Check how they are called in Arturos' CxAOD directory:

eos/atlas/atlascerngroupdisk/phys-exotics/CxAODFramework/HIGG5D1_13TeV/CxAOD-00-20-00_mc15b/ -ZZ_Pw/ --group.phys-exotics.mc15_13TeV.361611.PwPy8EGnloME_AZNLOCTEQ6L1_ZZvvqq_mqq20.s2726.HIGG5D1.20-0_CxAOD.root/


group.phys-exotics.8251967._000001.CxAOD.root

so ok, nothing different here, just how many directories this crawls down to. We are looking at the same root files.

Need to change to:

/afs/cern.ch/user/d/doglioni/eos/atlas/atlascerngroupdisk/phys-exotics/jdm/dijet/TLA/Data/TransferTest/TransferTestDS/DataScouting/*/*/*.root* source UploadEOSfilesToGRID.0.1.sh eos/atlas/atlascerngroupdisk/phys-exotics/jdm/dijet/TLA/Data/TransferTest/TransferTestDS/DataScouting/ doglioni CERN-PROD_PHYS-EXOTICS 12

then:

Provided RSE expression is considered invalid

The problem is when it adds a LOCALGROUPDISK to it, and we don't want that for EXO pledged resources. Changed it in the script.

Problem now: not enough space

rucio add-rule user.doglioni:Data.data15_13TeV.00284484.calibration_DataScouting_05_Jets.v6_AllCalibAllDataScouting_20160316_tree.root.7955865 1 'rse=GOEGRID_PHYS-EXOTICS' 2016-10-09 16:24:08,903 ERROR [There is not enough quota left to fulfil the operation. Details: There is insufficient quota on any of the target RSE's to fullfill the operation.]

and also some trouble with the naming:

rucio upload --rse GOEGRID_PHYS-EXOTICS_SCRATCHDISK user.doglioni:Data.data15_13TeV.00284484.calibration_DataScouting_05_Jets.v6_AllCalibAllDataScouting_20160316_tree.root.7955865 /afs/cern.ch/user/d/doglioni/eos/atlas/atlascerngroupdisk/phys-exotics/jdm/dijet/TLA/Data/TransferTest/TransferTestDS/DataScouting/Data/group.phys-exotics.data15_13TeV.00284484.calibration_DataScouting_05_Jets.v6_AllCalibAllDataScouting_20160316_tree.root

2016-10-09 16:27:33,529 WARNING [user.doglioni:Data.data15_13TeV.00284484.calibration_DataScouting_05_Jets.v6_AllCalibAllDataScouting_20160316_tree.root.7955865 cannot be distinguished from scope:datasetname. Skipping it.]

20161013

From the hypernews:

mkdir eos eos fuse mount eos

cd /afs/cern.ch/user/w/wolter/eos/atlas/unpledged/group-tokyo/users/yenari/20160719/HIGG5D1_13TeV/CxAOD_00-24-10/data16/group.phys-higgs.data16_13TeV.00297730.physics_Main.HIGG5D1.24-07_CxAOD.root

rucio add-dataset user.wolter:myDataset rucio add-rule user.wolter:myDataset 1 'rse=DESY-HH_SCRATCHDISK' rucio -v upload --rse DESY-HH_SCRATCHDISK user.wolter:myDataset group.phys-higgs.8962162._000001.CxAOD.root group.phys-higgs.8962162._000002.CxAOD.root

20161014

on lxplus028

usage: rucio add-rule [-h] [--weight WEIGHT] [--lifetime LIFETIME] [--grouping {DATASET,ALL,NONE}] [--locked] [--source-replica-expression SOURCE_REPLICA_EXPRESSION] [--notify NOTIFY] [--activity ACTIVITY] [--comment COMMENT] [--ask-approval] [--asynchronous] [--account RULE_ACCOUNT] dids [dids ...] copies rse_expression

rucio add-dataset user.doglioni:DSData2015

rucio add-rule user.doglioni:DSData 1 'rse=UKI-NORTHGRID-MAN-HEP_SCRATCHDISK'

rucio -v upload --rse UKI-NORTHGRID-MAN-HEP_SCRATCHDISK user.doglioni:DSData2015.data15_13TeV.00284484.calibration_DataScouting_05_Jets.v6_AllCalibAllDataScouting_20160316_tree.root.7955865 /afs/cern.ch/user/d/doglioni/eos/atlas/atlascerngroupdisk/phys-exotics/jdm/dijet/TLA/Data/TransferTest/TransferTestDS/DataScouting/Data/group.phys-exotics.data15_13TeV.00284484.calibration_DataScouting_05_Jets.v6_AllCalibAllDataScouting_20160316_tree.root/

%%%

Current command line to be used the FIRST time you will upload CxAOD samples:


source UploadEOSfilesToGRID.0.1.sh <eos-as-local-path-from-$HOME> <grid-user-nickname> <name-of-logical-GRID-site> <lifetime-in-#months>

//Example: 
//sourceUploadEOSfilesToGRID.0.1.sh eos/atlas/atlascerngroupdisk/phys-higgs/HSG5/Run2/VH/HIGG5D2_13TeV/CxAOD_00-18-00 arturos INFN-NAPOLI-ATLAS-BLAH 24
//source UploadEOSfilesToGRID.0.1.sh ~/eos/atlas/atlascerngroupdisk/phys-exotics/jdm/dijet/TLA/Data/TransferTest doglioni CERN-PROD_PHYS-EXOTICS 12

To Update files in datasets (and/or their lifetime) already created from a given EOS path:


source UploadEOSfilesToGRID.0.1.sh <eos-as-local-path-from-$HOME> <grid-user-nickname> <name-of-logical-GRID-site> <lifetime-in-#months>

//Example: sourceUploadEOSfilesToGRID.0.1.sh eos/atlas/atlascerngroupdisk/phys-higgs/HSG5/Run2/VH/HIGG5D2_13TeV/CxAOD_00-18-00 arturos INFN-NAPOLI-BLAH 24


source UploadEOSfilesToGRID.0.1.sh /eos/atlas/atlascerngroupdisk/phys-exotics/jdm/dijet/TLA/Data/ doglioni UKI-NORTHGRID-MAN-HEP_SCRATCHDISK 12


-- CaterinaDoglioni - 2016-10-09

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2016-11-10 - CaterinaDoglioni
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback