Going from a Classic SE to the DPM

Turning your Classic SE into a DPM is easy : it doesn’t require to move the data in any way. You only need to make the DPM server aware of the files that are present on your Storage Element. In other words, this is only a metadata operation, and no actual file movement is required at all.

How long will it take ?

To give a time estimate, the tests we have performed at CERN took :

  • 4 hours 23 minutes 17 seconds
  • for 236546 files

This gives an average of 14.97 files migrated per second.

Possible scenarios

There are two possibilities :

  • install the DPM servers on the Classic SE, and consider the Classic SE as a pool node as well,
  • install the DPM servers on a different machine, and turn the Classic SE into a DPM pool node.

Preliminary steps

You have to install the DPM servers on a given machine (it can be the Classic SE itself) See the DPM Admin Guide.

If installed on a different machine, the Classic SE will act as a pool node (=disk server) of the DPM.

Important :

Make sure that the VO groups and pool accounts have the same uids/gids on the Classic SE and on the DPM server. Otherwise, the migrated permissions will no be the correct ones.

Permissions

Make sure that the VO groups ids and pool accounts uid/gids correspond on the DPM server and on the Classic SE. Otherwise, the ownership will not be correctly migrated to the DPM Name Server

Get the script

To perform the migration, the IT-GD group provides a migration script. You can find it in the CERN central CVS service (repository lcgware/migration-classicSE-DPM).

You can also download the following tarball: migration-classicSE-DPM.tar.gz (last update: 2005-10-11).

Note that a new version of this script has been rewritten in order to manage problems encountered during the migration (for example when migrating the entries to an already existing DPM server (already having entries): migration-classicSE-DPM2.tar.gz (last update: 2006-03-21).

Configuration

On the classic SE

  • Stop the GridFTP server :

service globus-gridftp stop
chkconfig globus-gridftp off

  • Install the DPM-client package.

  • Set the environment variable DPNS_HOST with the DPNS hostname :

export DPNS_HOST=DPNS_HOSTNAME

  • Put in the /etc/shift.conf the following lines:

RFIOD RTRUST SHORT_DPNS_HOSTNAME LONG_DPNS_HOSTNAME
RFIOD WTRUST SHORT_DPNS_HOSTNAME LONG_DPNS_HOSTNAME
RFIOD XTRUST SHORT_DPNS_HOSTNAME LONG_DPNS_HOSTNAME
RFIOD FTRUST SHORT_DPNS_HOSTNAME LONG_DPNS_HOSTNAME

  • Compile the migration.c file using the Makefile :

make all

On the DPNS server

  • Put in the /etc/shift.conf the following line:

DPNS TRUST SHORT_CLASSIC_SE_HOSTNAME LONG_CLASSIC_SE_HOSTNAME

Migration

Run the following command on the classic SE host:

./migration classicSE_full_hostname classicSE_directory dpm_full_hostname dpm_directory dpm_poolname

where:

  • classicSE_hostname is the hostname (fqdn) of the classic SE.
  • classicSE_directory is the name of the directory where are stored all the files (for example /storage).
  • dpm_hostname is the hostname (fqdn) of the DPM.
  • dpm_directory is the name of the directory where will be stored all the files (for example /dpm/DOMAIN_NAME/home).
  • dpm_poolname is the name of the pool (obtained by using dpm_qryconf) on the DPM.

Post migration steps

If the Classic SE is a separate machine, make sure you turn it into a DPM pool node :

on the Classic SE :

Attention : before doing that, make sure that the entries appear in DPM Name Server as expected

Configure the Classic SE to be a pool node :

  • remove the CASTOR-client RPM
  • install the DPM-client, DPM-rfio-server and DPM-gsiftp-server RPMs
  • configure security (globus, grid-mapfile, gridmapdir, pool accounts
  • create the dpmmgr user/group (with the same uid/gid as on the DPM server)
  • chown root:dpmmgr /etc/grid-security/gridmapdir
  • create /etc/grid-security/dpmmgr
  • chown dpmmgr:dpmmgr /etc/grid-security/dpmmgr
  • cp -p /etc/grid-security/hostcert.pem /etc/grid-security/dpmmgr/dpmcert.pem
  • cp -p /etc/grid-security/hostkey.pem /etc/grid-security/dpmmgr/dpmkey.pem
  • service rfiod start
  • service dpm-gsiftp start

VERY IMPORTANT : Change the ownership of all the Classic SE files/directories :

WARNING : before changing the permissions, make sure that all the files have been properly migrated in the DPNS. Once the permissions changed, you cannot get the old permissions back...

  • chown -R dpmmgr:dpmmgr /YOUR_PARTITION
  • chmod -R 660 /YOUR_PARTITION
  • find /storage -type d -exec chmod 770 {} \; to have the correct permissions on directories

on the DPM server :

Create the pool and add the Classic SE file system to it :

  • export DPM_HOST=YOUR_DPM_SERVER
  • dpm-addpool --poolname POOL_NAME --def_filesize 200M (if the pool doesn't exist yet !)
  • dpm-addfs --poolname POOL_NAME --server CLASSIC_SE_FULL_NAME --fs CLASSIC_SE_FILE_SYSTEM

For more details, refer to the DPM Admin Guide.

Catalog

The entries that exist already in a catalog (RLS or LFC) won't be migrated.

The corresponding entries can still be accessed in the same way as before the migration. For instance :

lcg-cp --vo dteam sfn://ClassiSE_hostname/storage/dteam/generated/2005-03-29/filef70996ba-ba4e-42dc-9bae-03a3d7e7ac31 file:/tmp/test.classic.se.migration.1

Information System

You have to publish the DPM as an SRM in the Information System.

There is no need to publish the Classic SE as such in the Information System.

Troubleshooting

Please send all your questions/comments to hep-service-dpm@cernNOSPAMPLEASE.ch (remove the NONSPAM !) or to yvan.calas@cernNOSPAMPLEASE.ch.

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatgz migration-classicSE-DPM2.tar.gz r1 manage 9.6 K 2006-03-21 - 12:08 YvanCalas migration classic SE
Edit | Attach | Watch | Print version | History: r19 < r18 < r17 < r16 < r15 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r19 - 2007-02-17 - YvanCalas
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback