LCG Web>LCGGridDeployment>LcgDocs>DPMfs (2007-05-18, GergelyDebreczeni)

The DPM file system project

Introduction and foreword

Introduction and foreword

The DPM file system project is my simple attempt to mount DPM servers, not the product of the Data Management team, and is not supported officialy. So complaints, observations, comments should be sent directly to the author instead of sending bugs to savannah.

Furthermore it is still in development phase, there is no stable version. These pages are for myself to keep track of progress and to be turned into some kind of documentation later on.

The DPM fs is implemented using the FUSE kernel module interface. (See links.) The system's file system calls are forwarded to the daemon which communicates with the DPM servers using the rfio and dpns API and sends back the answer to the kernel. The daemon is running as root, but the file system calls are performed in user space, which ensures that the certificate is available. Currently there is three separate module.

the dpmfs daemon
the dpmfs kernel module
the dpmfs NSS module

While the daemon requires the kernel module , the NSS module is not compulsory , it's just an optional add-on to enable the translation of virtual user and group ids into human readable user and group names.

Due to some limitations on the server side it is not possible modify the files on the server. One can read existing ones and create/write new ones only.

The DPM file system modules

Here a short description is given about the module, for further information one can consult with the man pages dpmfs.1 , dpmfs.conf.1, dpmfs-nss.1.

The dpmfs daemon

Package name: DPM-client-dpmfs

The dpmfs daemon is running as root. One can use the

/etc/init.d/dpmfs [ start | stop | status ]

commands to start , stop or see the status of the daemon. On startup the daemon tries to query the top level directory /dpm of the DPNS servers listed in the config file, for this you need to have a proxy at start-up time. It happen in 20 parallel threads, however can take quite a while, please be patient. To be improved in the future.

The DPM server which hasn't responded during startup won't be 'mounted'. If the server becomes available the daemon has to be restarted. Again something to be improved in the future.

The dpmfs kernel module

Package name: DPM-client-dpmfs-kernel-module-KERNELVERSION

Currently the following prebuild kernelmodules are available:

DPM-client-dpmfs-kernel-module-2.6.9-42.0.2.EL.1.cernsmp
DPM-client-dpmfs-kernel-module-2.6.9-42.0.3.EL.1.cernsmp

The kernel modul is that of the slightly modified version of the fuse kernel module version 2.6.5. On daemon's startup it tries to load (modprobe) into the kernel automaticaly.

The dpmfs NSS module

Package name: DPM-client-dpmfs-nss-module

The NSS module is the dpmfs part of the Name Server Switch. After installation you have to include/modify the following lines in your /etc/nsswitch.conf .:

passwd:     files dpmfs
shadow:     files dpmfs
group:      files dpmfs

The behaviour of this module is highly configurable see the dpmfs-nss.1 and the dpmfs.conf.1 man pages for detailed description. Also this part needs significant speed improvement in the future.

Because fetching user information from the server 'on-the-fly' is really slow, the NSS module uses prepared cache files as user database. This cache files are downloaded and extracted into /var/cache/dpmfs on package installation. They contains the UID/GID mappings of users of different servers. Having this information does not involve security risk, since this is publicely available for anybody. However it should be replaced with some better solution in the future.

Log files

Log files are written under /var/log/dpmfs. Their structure is quite self explaining. Don't use the DEBUG loglevel if not necessary, turning it on has severe effect on performance. Use the INFO loglevel.

Configuration

The central config file resides in /etc/dpmfs/dpmfs.conf. The following variables can be set:

 [nss]
# This section has to be _before_ the 'domains' section !

# Wheter or not enabe NSS translation of user uids/gids.
# 1 = yes, 0 = no
nss_enable = 1

# UID shifts
nss_uid_shift_base = 1000000
nss_uid_shift_gap  = 100000

# How to display certificate DNs (Use the full DN or only the CN field instead.)
# 0 = full DN; 1 = CN field only
nss_use_cn = 1

# Maximum length of user names displayed
nss_uname_maxlength = 0


[connection]
# Connection timeout in seconds. This value will be translated to DPNS_CONNTIMEOUT environment variable.
conn_timeout = 3

[cache]
# Set cache lifetime in seconds
cache_lifetime = 1800

# Set cache size in MB.
cache_size = 10


[logging]
# Set the loglevel here
# 0 = SYS ; 1 = ABORT ; 2 = ERROR ; 3 = WARNING ; 4 = INFO ; 5 = DEBUG ; 6 = DEVEL

dpmfs_loglevel = 5

[domains]
# The domains section is to list the DPM servers to be queried.
# The first coloumn is the domain which will appear is /dpm/<domain> in the
# local file system.
# The second and the third coloumn is the name of the DPM and DPNS server.

#     DOMAIN               DPM_HOST                       DPNS_HOST
cc.kek.jp               dg11.cc.kek.jp                  dg11.cc.kek.jp
cern.ch                 lxb1921.cern.ch                 lxb1921.cern.ch

If you happen to have more than one DPM server under the same domain use YAIM's DPNS_BASEDIR variable to set the name space entry point and modify the configuration file for example in the following way:

cern.ch/home               lxb1921.cern.ch             lxb1921.cern.ch
cern.ch/data               lxb1917.cern.ch             lxb1917.cern.ch

As a result lxb1921 will be mounted under /dpm/cern.ch/home while lxb1917 will be mounted under /dpm/cern.ch/data.

Installation

You can wget the rpms from the

http://grid-deployment.web.cern.ch/grid-deployment/dpmfs/repo/rpm/sl4/i386/

web address directly, or set up the repository (recognized by yum and apt).

Repository

The rpms are available via the following yum repository:


#
# The DPMfs yum repositry
#
[main]
[dpmfs]
name=The DPM fs project's repository
baseurl=http://grid-deployment.web.cern.ch/grid-deployment/dpmfs/repo/rpm
enabled=1
protect=1

Install the kernel module suitable for your kernel.

Dependencies

The followings has to be taken into account:

Because of a bug in the rfiod client, the dpmfs daemon works properly only together with a modified version of the libdpm library. This issue will be fixed in the near future, and from that point on you can install dpmfs having the following packages already installed on your computer. ( The packages needed are listed relative to a 'standard' SL4 installation in order of installation.)

vdt_globus_essentials
glite-security-voms-api
glite-security-voms-api-c
lcg-dm-common
DPM-client
compat-openldap
CGSI_gSOAP_2.6
GFAL-client
lcg-CA

And put the following lines into your /etc/ld.so.conf:

/opt/lcg/lib
/opt/globus/lib
/opt/glite/lib

then run ldconfig. If you happen to install dpmfs packages on a glite-UI, then all the dependencies should already be satisfied. A possible source of installing those packages:

/afs/cern.ch/project/etics/repository/externals/globus/4.0.3-VDT-1.6.0/slc4_ia32_gcc346/vdt_globus_essentials-VDT1.6.0x86_rhas_4-1.i386.rpm
/afs/cern.ch/project/etics/repository/org.glite/org.glite.security.voms-api/1.7.16/slc4_ia32_gcc346/glite-security-voms-api-1.7.16-2.slc4.i386.rpm
/afs/cern.ch/project/etics/repository/org.glite/org.glite.security.voms-api-c/1.7.16/slc4_ia32_gcc346/glite-security-voms-api-c-1.7.16-2.slc4.i386.rpm
/afs/cern.ch/project/etics/repository/org.glite/LCG-DM/1.6.4/slc4_ia32_gcc346/lcg-dm-common-1.6.4-3sec.i386.rpm
/afs/cern.ch/project/etics/repository/org.glite/LCG-DM/1.6.4/slc4_ia32_gcc346/DPM-client-1.6.4-3sec.i386.rpm
/afs/cern.ch/project/etics/repository/org.glite/org.glite.data.gfal/1.9.1/slc4_ia32_gcc346/GFAL-client-1.9.1-0.i386.rpm
http://lxb2042.cern.ch/gLite/APT/R3.0-pps/sl4/RPMS.Release3.0/CGSI_gSOAP_2.6-1.1.15-6.i386.rpm
http://linuxsoft.cern.ch/cern/slc4X/i386/yum/os/compat-openldap-2.1.30-6.4E.i386.rpm

Use cases, precautions

In it's present state it's a perfect DPM browser, presenting the information more friendly than the DM command line utilities. However currently it is not recommended to use this tool for large data transfer, it's performance is not (yet) compatitive with the globus utility.
Be careful , if the DPM file system is mounted, then cron jobs and different tools which are supposed to parse and browse the file system can get lost here, and stress the dpm servers. Please make sure that it is not going to happen:
- updatedb Add /dpm into the list of PRUNEFS variable of your /etc/updatedb.conf file, in order to avoid parsing the DPM file system.

Troubleshooting

If something goes wrong you can try the followings:

Exit from all the directories under /dpm

To stop the daemon, try:

       /etc/init.d/dpmfs stop

or kill it directly:

      kill -9 'proc id of dpmfs daemon'

If that doesn't help try to unmount the /dpm directory:
```
      umount -f /dpm
      
```
Try to remove the kernel module:
```
       rmmod -f fuse
      
```

Bugs, Todos, dreams

To be fixed:
- On startup a valid proxy has to be availabe. To be fix
- It's slow. Listing and browsing has the same speed as that of the dpm-*, dpns-* command line utilities, but reading writing of files is from 2x to 4x times slower that globus utility.
To be implemented:
- The actual read/write functionality should be put into the kernel itself.
- DPM server discovery from a BDII.
To test:
- Has to be tested in multi user and multi process environment. Up to now I've test only as a single user.
Dreams:
- Built-in encryption. During read/write if the target/source file full path contains for example an 'encrypted' word then the file is regarded as encrypted and will automaticaly be (de)crypted. Problem is of course the handling of the keys.