-- ChristopheHaen - 2020-07-28

LHCb wants to reorder its namespace. The desired transformation is as follows (example at GridKa):

/pnfs/gridka.de/lhcb/dirA/dirB/file  -->  /pnfs/gridka.de/lhcb/SPACETOKEN/lhcb/dir-A/dir-B/file

To achieve that, Tigran has developed a script, which performs changes on the dCache chimera and spacemanager DBs. This script is available under this link: https://sas.desy.de/index.php/s/HjKc9dEzTgTe4S9

As requisite it is necessary to install java11 (part of EL7 distribution).

PREREQUISITE. Ensure that all the files present in Chimera belong to a space token.

CAUTION !! THIS PART IS INCOMPLETE (regex list in particular) so check with Chris before doing any change

During the tests performed before going into production, files present in Chimera without having space token information were found. If that is the case, the following rules should apply depending on the following location of the files within the Chimera namespace:

/lhcb/failover/ -> LHCb-Disk
/lhcb/buffer/ -> LHCb_Disk
/lhcb/archive/ -> LHCb-Tape
/lhcb/data/-> LHCb-Tape
lhcb/LHCb/Collision[0-9]+/FULL.DST/ -> LHCb-Tape
/lhcb/MC/ -> LHCb-Disk
/lhcb/user -> LHCb_USER

Include space token information in the srmspacefile table

At GridKa, 2624 files were present in chimera, without having space token information. To fix that, the information was added directly in the srmspacefile table. One real example is shown.

INSERT INTO srmspacefile (vogroup,vorole,spacereservationid,sizeinbytes,creationtime,pnfsid,state) VALUES('/lhcb', 'production', 39930202, 5419533, 1357979169000, '00110000000000001FB2D288', 2);

Please, ensure that the values of vogroup and vorole are correct at your site. As far as I know, the field state represents the access latency of the file. Which means, either the file is online and occupies disk space (state==2) or is offline and doesn't (state == 3), but still belongs to the space reservation. The state==1 probably means "being uploaded right now".

Steps

  • Before running Tigran's script, you need to update the chimera DB with an additional procedure:
CREATE OR REPLACE FUNCTION inumber2path(bigint, bigint) RETURNS varchar AS $$
DECLARE
     inumber bigint := $1;
     iroot bigint := $2;
     path    varchar := '';
     entry   record;
BEGIN
    IF iroot = inumber
    THEN
        return '/';
    END IF;

    LOOP
        IF iroot = inumber
        THEN
           EXIT;
        END IF;

        SELECT * INTO entry FROM t_dirs WHERE ichild = inumber;
        IF FOUND AND entry.iparent != inumber
        THEN
            path := '/' || entry.iname || path;
            inumber := entry.iparent;
        ELSE
            EXIT;
        END IF;
    END LOOP;

    RETURN path;
END;
$$
LANGUAGE 'plpgsql';

  • Second, in migration.properties you will need to point to corresponding databases and adapt the values of path.src and path.dest. Such values at GridKa:
# /src/dir-1/file -> /dest/TOKEN/dir-1/file
path.src = /pnfs/gridka.de/
path.dest = /pnfs/gridka.de/lhcb

  • Third (optional if you want to get rid of some space token), something similar to the following executed at GridKa will be necessary, adapting the values of the corresponding spacereservationid. It is also important to verify the values of vogroup, description in your instance.
   Execute the three SQL commands to get rid of LHCb_RAW.
[root@f01-117-104 ~]# psql spacemanager dcache
psql (11.8)
Type "help" for help.

spacemanager=> update srmspacefile set spacereservationid = 39930230 where spacereservationid = 449420;
UPDATE 29079
spacemanager=> delete from srmspace where id = 449420;
DELETE 1
spacemanager=> update srmspace set vogroup = '/lhcb',  description = 'LHCb-Tape' where id = 39930230;
UPDATE 1

  • Finally, execute:
$ java -jar lhcb-migration.jar [property-file]

If property file is not defined, then `migration.properties` from the local directory will be used.

Brave people can look at the code:

https://github.com/kofemann/lhcb-migration/blob/master/src/main/java/org/dcache/migraion/lhcb/Migration.java


This topic: LHCb > WebHome > LHCbComputing > DCacheNamespaceMigration
Topic revision: r6 - 2020-10-20 - ChristopheHaen
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback