The DPM file system project
Introduction and foreword
The DPM file system project is my simple attempt to mount DPM servers, not the product of the Data Management team, and
is not supported officialy. So complaints, observations, comments should be sent directly to the author instead of sending bugs to savannah.
Furthermore
it is still in development phase, there is no stable version. These pages are for myself to keep track of progress and to be turned into some kind of documentation later on.
The DPM fs is implemented using the FUSE kernel module interface. (See links.) The system's file system calls are forwarded to the daemon which communicates with the DPM servers using the
rfio
and
dpns
API and sends back the answer to the kernel. The daemon is running as root, but the file system calls are performed in user space, which ensures that the certificate is available. Currently there is three separate module.
- the dpmfs daemon
- the dpmfs kernel module
- the dpmfs NSS module
While the daemon requires the kernel module , the NSS module is not compulsory , it's just an optional add-on to enable the translation of virtual user and group ids into human readable user and group names.
Due to some limitations on the server side it is not possible modify the files on the server. One can read existing ones and create/write new ones only.
The DPM file system modules
Here a short description is given about the module, for further information one can consult with the man pages
dpmfs.1
,
dpmfs.conf.1
,
dpmfs-nss.1
.
The dpmfs daemon
Package name: DPM-client-dpmfs
The dpmfs daemon is running as root. One can use the
/etc/init.d/dpmfs [ start | stop | status ]
commands to start , stop or see the status of the daemon. On startup the daemon tries to query the top level directory
/dpm
of the DPNS servers listed in the config file, for this you need to have a proxy at start-up time. It happen in 20 parallel threads, however can take quite a while, please be patient. To be improved in the future.
The DPM server which hasn't responded during startup won't be 'mounted'. If the server becomes available the daemon has to be restarted. Again something to be improved in the future.
The dpmfs kernel module
Package name: DPM-client-dpmfs-kernel-module-KERNELVERSION
Currently the following prebuild kernelmodules are available:
-
DPM-client-dpmfs-kernel-module-2.6.9-42.0.2.EL.1.cernsmp
-
DPM-client-dpmfs-kernel-module-2.6.9-42.0.3.EL.1.cernsmp
The kernel modul is that of the slightly modified version of the fuse kernel module version 2.6.5. On daemon's startup it tries to load (
modprobe
) into the kernel automaticaly.
The dpmfs NSS module
Package name: DPM-client-dpmfs-nss-module
The NSS module is the dpmfs part of the Name Server Switch. After installation you have to include/modify the following lines in your
/etc/nsswitch.conf
.:
passwd: files dpmfs
shadow: files dpmfs
group: files dpmfs
The behaviour of this module is highly configurable see the
dpmfs-nss.1 and the
dpmfs.conf.1 man pages for detailed description.
Also this part needs significant speed improvement in the future.
Because fetching user information from the server 'on-the-fly' is really slow, the NSS module uses prepared cache files as user database.
This cache files are downloaded and extracted into
/var/cache/dpmfs
on package installation. They contains the UID/GID mappings of
users of different servers. Having this information does not involve security risk, since this is publicely available for anybody. However it should be replaced with some better solution in the future.
Log files
Log files are written under
/var/log/dpmfs
. Their structure is quite self explaining. Don't use the
DEBUG
loglevel if not necessary, turning it on has severe effect on performance. Use the
INFO
loglevel.
Configuration
The central config file resides in
/etc/dpmfs/dpmfs.conf
. The following variables can be set:
[nss]
# This section has to be _before_ the 'domains' section !
# Wheter or not enabe NSS translation of user uids/gids.
# 1 = yes, 0 = no
nss_enable = 1
# UID shifts
nss_uid_shift_base = 1000000
nss_uid_shift_gap = 100000
# How to display certificate DNs (Use the full DN or only the CN field instead.)
# 0 = full DN; 1 = CN field only
nss_use_cn = 1
# Maximum length of user names displayed
nss_uname_maxlength = 0
[connection]
# Connection timeout in seconds. This value will be translated to DPNS_CONNTIMEOUT environment variable.
conn_timeout = 3
[cache]
# Set cache lifetime in seconds
cache_lifetime = 1800
# Set cache size in MB.
cache_size = 10
[logging]
# Set the loglevel here
# 0 = SYS ; 1 = ABORT ; 2 = ERROR ; 3 = WARNING ; 4 = INFO ; 5 = DEBUG ; 6 = DEVEL
dpmfs_loglevel = 5
[domains]
# The domains section is to list the DPM servers to be queried.
# The first coloumn is the domain which will appear is /dpm/<domain> in the
# local file system.
# The second and the third coloumn is the name of the DPM and DPNS server.
# DOMAIN DPM_HOST DPNS_HOST
cc.kek.jp dg11.cc.kek.jp dg11.cc.kek.jp
cern.ch lxb1921.cern.ch lxb1921.cern.ch
If you happen to have more than one DPM server under the same domain use YAIM's
DPNS_BASEDIR
variable to set the name space
entry point and modify the configuration file for example in the following way:
cern.ch/home lxb1921.cern.ch lxb1921.cern.ch
cern.ch/data lxb1917.cern.ch lxb1917.cern.ch
As a result
lxb1921
will be mounted under
/dpm/cern.ch/home
while
lxb1917
will be mounted under
/dpm/cern.ch/data
.
Installation
You can
wget
the rpms from the
http://grid-deployment.web.cern.ch/grid-deployment/dpmfs/repo/rpm/sl4/i386/
web address directly, or set up the repository (recognized by
yum
and
apt
).
Repository
The rpms are available via the following yum repository:
#
# The DPMfs yum repositry
#
[main]
[dpmfs]
name=The DPM fs project's repository
baseurl=http://grid-deployment.web.cern.ch/grid-deployment/dpmfs/repo/rpm
enabled=1
protect=1
Install the kernel module suitable for your kernel.
Dependencies
The followings has to be taken into account:
Because of a bug in the
rfiod
client, the dpmfs daemon works properly only together with a modified version of the libdpm library. This issue will be fixed in the near future, and from that point on you can install dpmfs having the following packages already installed on your computer. ( The packages needed are listed relative to a 'standard' SL4 installation in order of installation.)
-
vdt_globus_essentials
-
glite-security-voms-api
-
glite-security-voms-api-c
-
lcg-dm-common
-
DPM-client
-
compat-openldap
-
CGSI_gSOAP_2.6
-
GFAL-client
-
lcg-CA
And put the following lines into your
/etc/ld.so.conf
:
/opt/lcg/lib
/opt/globus/lib
/opt/glite/lib
then run
ldconfig
. If you happen to install dpmfs packages on a glite-UI, then all the dependencies should already be satisfied.
A possible source of installing those packages:
/afs/cern.ch/project/etics/repository/externals/globus/4.0.3-VDT-1.6.0/slc4_ia32_gcc346/vdt_globus_essentials-VDT1.6.0x86_rhas_4-1.i386.rpm
/afs/cern.ch/project/etics/repository/org.glite/org.glite.security.voms-api/1.7.16/slc4_ia32_gcc346/glite-security-voms-api-1.7.16-2.slc4.i386.rpm
/afs/cern.ch/project/etics/repository/org.glite/org.glite.security.voms-api-c/1.7.16/slc4_ia32_gcc346/glite-security-voms-api-c-1.7.16-2.slc4.i386.rpm
/afs/cern.ch/project/etics/repository/org.glite/LCG-DM/1.6.4/slc4_ia32_gcc346/lcg-dm-common-1.6.4-3sec.i386.rpm
/afs/cern.ch/project/etics/repository/org.glite/LCG-DM/1.6.4/slc4_ia32_gcc346/DPM-client-1.6.4-3sec.i386.rpm
/afs/cern.ch/project/etics/repository/org.glite/org.glite.data.gfal/1.9.1/slc4_ia32_gcc346/GFAL-client-1.9.1-0.i386.rpm
http://lxb2042.cern.ch/gLite/APT/R3.0-pps/sl4/RPMS.Release3.0/CGSI_gSOAP_2.6-1.1.15-6.i386.rpm
http://linuxsoft.cern.ch/cern/slc4X/i386/yum/os/compat-openldap-2.1.30-6.4E.i386.rpm
Use cases, precautions
- In it's present state it's a perfect DPM browser, presenting the information more friendly than the DM command line utilities. However currently it is not recommended to use this tool for large data transfer, it's performance is not (yet) compatitive with the globus utility.
- Be careful , if the DPM file system is mounted, then cron jobs and different tools which are supposed to parse and browse the file system can get lost here, and stress the dpm servers. Please make sure that it is not going to happen:
- updatedb Add
/dpm
into the list of PRUNEFS
variable of your /etc/updatedb.conf
file, in order to avoid parsing the DPM file system.
Troubleshooting
If something goes wrong you can try the followings:
Bugs, Todos, dreams
- To be fixed:
- On startup a valid proxy has to be availabe. To be fix
- It's slow. Listing and browsing has the same speed as that of the
dpm-*
, dpns-*
command line utilities, but reading writing of files is from 2x to 4x times slower that globus
utility.
- To be implemented:
- The actual read/write functionality should be put into the kernel itself.
- DPM server discovery from a BDII.
- To test:
- Has to be tested in multi user and multi process environment. Up to now I've test only as a single user.
- Dreams:
- Built-in encryption. During read/write if the target/source file full path contains for example an 'encrypted' word then the file is regarded as encrypted and will automaticaly be (de)crypted. Problem is of course the handling of the keys.
Links
Gergely Debreczeni