CernLbdDiskManagement

This is the LHCb CERN LBD group page describing the management of locally available disk space


Our resources are advertised as volatile and maintained with a fair-use policy. If space becomes limited people should be contacted to voluntarily remove old files, then the manager can if really necessary arbitrarily remove changes on top of that, since the system is advertised as volatile.

The instructions here are in addition to the instructions on CernLbdDiskSpace.

The scripts for management, and other things are kept in backed up afs space: /afs/cern.ch/lhcb/group/lbd/scripts/, which is softlinked to /afs/cern.ch/project/lbcern/scripts

z5:cernlbd

z5:cernlbd holds the access list for the 1TB disk space.

pts mem z5:cernlbd #see the member list
pts adduser <someuser> z5:cernlbd #add someuser to the list
pts removeuser <someuser> z5:cernlbd #remove someone from the list

Group afs space

1TB of space is mounted in 10 volumes in: /afs/cern.ch/lhcb/project/lbcern

afs_admin create -u rlambert -q 100G /afs/cern.ch/project/lbcern/vol2 s.lbcern.vol2 #create a volume
afs_admin set_quota /afs/cern.ch/project/lbcern/vol1 100G #change the quota of a volume
fs sa /afs/cern.ch/project/lbcern/vol2 z5:cernlbd write #give the right permissions
afs-admin list_project lbcern #to list the amount of space for the project
python  /afs/cern.ch/project/lbcern/scripts/check1TB.py #to check the usage and the usage per user

z5:cernlbd is the access list for this volume. All users should make directories with their user name. If there is some top-level directory without a user name it should be moved/deleted.

For more on acl see here http://docs.openafs.org/Reference/1/fs_setacl.html

- Setting more Afs permissions

give the permission to change permissions to a user for their own directory

fs sa /afs/cern.ch/project/lbcern/vol*/<user> <user> rlidwka

To propagate permissions from one place to another use:

fs copy_acl <one_place> <another>

Directories created within a parent directory inherit the parent's permissions

Then if they want to let someone else See/read/write in the directories, the upper directories need to be 'lookup'-able

fs sa /afs/cern.ch/project/lbcern/vol*/<user>/<something> <user-or-group> write 
fs sa /afs/cern.ch/project/lbcern/vol*/<user> <user-or-group> l 
fs sa /afs/cern.ch/project/lbcern/vol* <user-or-group> l 

e.g. <user-or-group> cern:z5

lhcbt3

There are two types of files which can be on the lhcbt3: user files, and central files. In general there are two spying scripts, check30TB.py and castorQuota.py, each with many options to look in specific directories, etc.

- user files

To find out the used quotas of all users:

python  /afs/cern.ch/project/lbcern/scripts/check30TB.py

Then to get the list of files:

stager_qry -M  castor/cern.ch/user/<u>/<uname>

- central files

It is not clear how we can check quickly what is on lhcbt3 from central productions, and which user staged any individual file into the pool is unknown.

python  /afs/cern.ch/project/lbcern/scripts/check30TB.py --data >year< --MC

This will take a very long time, probably.

Then to get the list of files:

stager_qry -M  castor/cern.ch/grid/lhcb/data/<year>/ -M  castor/cern.ch/grid/lhcb/MC

- removal notes:

Once you have the list of files to remove, remove them individually with:

stager_rm -M <existing castor file> -S lhcbt3

Or remove a big list with:

/afs/cern.ch/project/lbcern/scripts/lhcbt3_rm.py <username or directory or file with a list in it>

Removing a user file potentially deletes the file forever, so it's nice to get the users to do that themselves, but any centrally produced or grid-resident file is by definition backed-up and can be removed arbitrarily.


-- RobLambert - 19-Nov-2010

Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r8 - 2010-12-13 - RobLambert
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback