Manual Configuration

This is really a guide for experts, site admins should follow the puppet installation at Puppet Installation Guide

The guide is valid both for SL6/C6 and SL7/C7 installations and for DPM >= 1.10.

The guide concerns the installation from scratch of a dmlite/dome DPM system, meaning that the legacy stack (including SRM) is not installed.

This guide is for for DPM >= 1.10.

DPM Terminology

  • Head Node The machine with the DomeHead daemon and the database client (and possibly the database itself) and protocol frontends.

  • Disk Servers / Pool Nodes Machines where data is actually stored

  • Client Remote machine talking to the ''Head Node'' and ''Disk Servers''

Repositories

DPM can be installed on SL6 or CentOS7 (or equivalent compatible distributions).

To install a DPM you need to ensure that the EPEL repository is enabled, and that you have configured your certification authorities as described on the EGI IGTF Release Page.

The you can do

yum install dmlite-dpmhead-domeonly
yum install dmlite-dpmdisk-domeonly

The above commands will install the fetch-crl utility which you need to enable. e.g. on CentOS7;

systemctl enable fetch-crl-cron
systemctl start fetch-crl-cron

xrootd

For xrootd, you should also enable the WLCG repository, http://linuxsoft.cern.ch/wlcg/

and then execute

yum install dmlite-dpm-xrootd vomsxrd

on head and disk nodes.

Certificates

You will need a host certificate and private key in order to run DPM. This guide expects them to be found at

/etc/grid-security/hostcert.pem
/etc/grid-security/hostkey.pem

Ensure permissions on hostkey.pem are 600.

SELinux

We recommend running with SELinux disabled.

Firewall Configuration

The following ports should be open, depending on the protocols you intend to use.

Headnode:

  • GRIDFTP: 2811/tcp , 20000-25000/tcp (local and internet)
  • or by setting $GLOBUS_TCP_PORT_RANGE, $GLOBUS_TCP_SOURCE_RANGE
  • LCGDM-DAV: 443/tcp (local and internet)
  • DPM-XROOTD: 1094/tcp (local and internet) ( in case of federations also the port 1095 for cmsd and one port for each federation should be opened locally and to the internet)
  • BDII: 2170/tcp ( local and internel)

Disknode:

  • GRIDFTP: 2811/tcp (local and internet, N.B. in case the DPM is configured with gridftp redirection this port should be firewalled to internet) , 20000-25000/tcp (local and internet)
  • LCGDM-DAV: 443/tcp and 80/tcp (local and internet)
  • DPM-XROOTD: 1095/tcp (local and internet)

Example with firewalld

Here is an example firewalld service assigned to the zone 'public', which is assumed to be the default zone configured in firewalld.

$ cat /etc/firewalld/services/dpmsvc.xml 

<?xml version="1.0" encoding="utf-8"?>
<service>
    <short>dpmsvc</short>
    <description>DPM disk server</description>
    
    <port protocol="tcp" port="80" />
    <port protocol="tcp" port="443" />
    <port protocol="tcp" port="1094" />
    <port protocol="tcp" port="1095" />
    <port protocol="tcp" port="2170" />
    <port protocol="tcp" port="2811" />
    <port protocol="tcp" port="20000-25000" />
    
</service>

and the corresponding zone will make use of the dpmsvc service:

$cat /etc/firewalld/zones/public.xml

<?xml version="1.0" encoding="utf-8"?>
<zone>
  <short>Public</short>
  <description>For use in public areas. You do not trust the other computers on networks to not harm your computer. Only selected incoming connections are accepted.</description>
  <service name="ssh"/>
  <service name="dhcpv6-client"/>
  <service name="dpmsvc"/>
  <port protocol="udp" port="7001"/>
  <port protocol="tcp" port="4241"/>
</zone>

Create the ''dpmmgr'' user

groupadd -g 151 dpmmgr
useradd -c "DPM manager" -g dpmmgr -u 151 -r -m dpmmgr

IPV6 Configuration

If you are installing on a SL7/C7 machine, the DPM services and frontends are going to work out of the box with IPv6

On SL6/C6 instead the configuration file /etc/gai.conf must be created with the following content

label ::1/128 0
label ::/0 1
label 2002::/16 2
label ::/96 3
label ::ffff:0:0/96 4
label fec0::/10 5
label fc00::/7 6
label 2001:0::/32 7
label ::ffff:7f00:0001/128 8

to let gridftp correctly bind to IPv6.

Configuring a Head Node

Security configuration

Certificates

Copy the host certificate/key pair to ''/etc/grid-security/dpmmgr'', owned by ''dpmmgr''

mkdir /etc/grid-security/dpmmgr
cp -a /etc/grid-security/hostcert.pem /etc/grid-security/dpmmgr/dpmcert.pem
cp -a /etc/grid-security/hostkey.pem /etc/grid-security/dpmmgr/dpmkey.pem
chown -R dpmmgr.dpmmgr /etc/grid-security/dpmmgr

N.B. Do not remove the original host certificates

Mapfile

Create the file /etc/lcgdm-mkgridmap.conf, adding entries for the VOs you wish to support

group vomss://lcg-voms2.cern.ch:8443/voms/alice?/alice alice
group vomss://lcg-voms2.cern.ch:8443/voms/atlas?/atlas atlas
group vomss://lcg-voms2.cern.ch:8443/voms/cms?/cms cms
group vomss://lcg-voms2.cern.ch:8443/voms/lhcb?/lhcb lhcb
group vomss://lcg-voms2.cern.ch:8443/voms/ops?/ops ops
group vomss://voms2.cern.ch:8443/voms/alice?/alice alice
group vomss://voms2.cern.ch:8443/voms/atlas?/atlas atlas
group vomss://voms2.cern.ch:8443/voms/cms?/cms cms
group vomss://voms2.cern.ch:8443/voms/lhcb?/lhcb lhcb
group vomss://voms2.cern.ch:8443/voms/ops?/ops ops
group vomss://voms2.hellasgrid.gr:8443/voms/dteam?/dteam dteam
gmf_local /etc/lcgdm-mapfile-local

Create the mapfile cron

55 5,11,17,23 * * * (date; /usr/libexec/edg-mkgridmap/edg-mkgridmap.pl --conf=/etc/lcgdm-mkgridmap.conf --output=/etc/lcgdm-mapfile --safe) >> /var/log/lcgdm-mkgridmap.log 2>&1

Create a local mapfile if it's not present

touch /etc/lcgdm-mapfile-local

Run the cron job manually to generate the /etc/lcgdm-mapfile file.

Set up the Database

MySQL

If running a local db, make sure the MySQL or Mariadb daemon is running

CentOS7:

systemctl start mariadb

SL6:

service mysqld start

Load the database schema

mysql -u root < /usr/share/dmlite/dbscripts/cns_mysql_db.sql
mysql -u root < /usr/share/dmlite/dbscripts/dpm_mysql_db.sql

Setup the DPM database user (replace DPNS_HOST / DPM_HOST/ DPMUSER/ DPMPASS with your config values)

mysql -u root
mysql> use mysql
mysql> GRANT ALL PRIVILEGES ON cns_db.* TO 'DPMUSER'@DPNS_HOST IDENTIFIED BY 'DPMPASS' WITH GRANT OPTION;
mysql> GRANT ALL PRIVILEGES ON cns_db.* TO 'DPMUSER'@localhost IDENTIFIED BY 'DPMPASS' WITH GRANT OPTION;
mysql> GRANT ALL PRIVILEGES ON dpm_db.* TO 'DPMUSER'@DPM_HOST IDENTIFIED BY 'DPMPASS' WITH GRANT OPTION;
mysql> GRANT ALL PRIVILEGES ON dpm_db.* TO 'DPMUSER'@localhost IDENTIFIED BY 'DPMPASS' WITH GRANT OPTION;

Zero /etc/dmlite.conf.d/mysql.conf if present.

cat /dev/null > /etc/dmlite.conf.d/mysql.conf

Follow in instructions for tunining MySQL/Mariadb.

Core setup and configuration

The shared secret

In the following steps, a shared secret is configured in a number of places (herein referenced as <key at least 32 chars>). Ensure that the same key is used in all the configurations. (Exception: /etc/xrootd/dpmxrd-sharedkey.dat is a separate shared secret used only by dmlite-dpm-xrootd).

The config is described in the relevant sections, but for reference, we summarise here where this key must appear:

Head

/etc/domehead.conf
/etc/dmlite.conf.d/domeadapter.conf
/etc/xrootd/xrootd-dpmredir.cfg

Disk

/etc/domedisk.conf
/etc/dmlite.conf.d/domeadapter.conf
/etc/xrootd/xrootd-dpmdisk.cfg 

Dmlite configuration

If necessary, configure /etc/dmlite.conf. The default file shipped with the rpm should work.

Dome configuration

If you are upgrading existing legacy DPM installation to new DOME DPM flavor you have to follow Enabling DOME instructions.

For the configuration, the file /etc/domehead.conf has to be added and configured as follows (fill in the necessary fields)

glb.role: head
glb.debug: 1

glb.auth.urlprefix: /domehead/

glb.task.maxrunningtime: 3600
glb.task.purgetime: 3600

glb.restclient.cli_certificate: /etc/grid-security/dpmmgr/dpmcert.pem
glb.restclient.cli_private_key: /etc/grid-security/dpmmgr/dpmkey.pem
glb.restclient.xrdhttpkey: <key at least 32 chars>

head.dirspacereportdepth: 6
head.put.minfreespace_mb: 1

head.checksum.maxtotal: 1000
head.checksum.maxpernode: 40
head.filepulls.maxtotal: 1000
head.filepulls.maxpernode: 40
head.filepuller.stathook: /usr/share/dmlite/filepull/externalstat_example.sh
head.filepuller.stathooktimeout: 60

# Database
head.db.host: <DBHOST>
head.db.user: <DBUSER>
head.db.password: <DBPASS>
head.db.port: 0
head.db.poolsz: 128

# OIDC authentication (DPM 1.14+)
#head.oidc.allowissuer[]: "/dpm/domain.org/home/cms" "https://cms-auth.web.cern.ch/" cms
#head.oidc.allowissuer[]: "/dpm/domain.org/home/wlcg" "https://wlcg.cloud.cnaf.infn.it/" wlcg
#head.oidc.allowissuer[]: "/dpm/domain.org/home/xdc" "https://iam.extreme-datacloud.eu/" xdc
#head.oidc.allowaudience[]: < The OIDC Client ID for this service >
#head.oidc.allowaudience[]: https://wlcg.cern.ch/jwt/v1/any
#head.oidc.allowaudience[]: https://< your headnode >
##head.oidc.allowaudience[]: https://< your headnode >:non_standard_https_port

# Telemetry (version, host, totalspace, freespace)
#head.informer.mainurl: https://dpmhead-rc.cern.ch/dpminfo

Configure /etc/dmlite.conf.d/domeadapter.conf (fill in the necessary fields)

LoadPlugin plugin_domeadapter_io /usr/lib64/dmlite/plugin_domeadapter.so

LoadPlugin plugin_domeadapter_pools /usr/lib64/dmlite/plugin_domeadapter.so

LoadPlugin plugin_domeadapter_headcatalog /usr/lib64/dmlite/plugin_domeadapter.so


DavixCAPath  /etc/grid-security/certificates
DavixCertPath /etc/grid-security/dpmmgr/dpmcert.pem
DavixPrivateKeyPath /etc/grid-security/dpmmgr/dpmkey.pem

DomeHead http://<your headnode>:1094/domehead

# Token generation
# The token password is any secret string that will be used to create access tokens
# It can be of any length
TokenPassword <key at least 32 chars>
TokenId ip
TokenLife 1000

ThisDomeAdapterDN <DN of head node in / / / format>

Zero /etc/dmlite.conf.d/adapter.conf if present.

cat /dev/null > /etc/dmlite.conf.d/adapter.conf

Info providers and space publishing

The dmlite-shell package comes with utilities for publishing to ldap (BDII) and for creating a summary json file of space usage.

Info provider

If you still run DPM in mixed mode (DOME + legacy to support SRM protocol) then for maximum compatibility it is recommended to rely on old scripts based on dpm-listspaces. Legacy info provider publish BDII data with GLUE1.3 + GLUE2 schema and really old / deprecated tools (e.g. lcg-cp) can rely on data published in GLUE1.3 format for SRM access.

The DOME info provider publish data only in GLUE2 schema and it is invoked by /var/lib/bdii/gip/provider/dome-info-exec. To enable

  • Configure /etc/sysconfig/dpminfo
    • Ensure DPM_INFO_PROVIDER="dome" is set
      • Alternatively, the historical dpm-listspaces can be invoked by setting DPM_INFO_PROVIDER="dpm-listspaces", but only if you run DPM in mixed mode
    • Configure the site name and other parameters
  • Remove the earlier info providers /var/lib/bdii/gip/provider/se-dpm and /var/lib/bdii/gip/provider/service-srm2.2.
Publishing space usage
Publish SRR JSON online automatically with HTTP CGI

Since DPM 1.13.2 storage resource reporting can be published also directly with HTTP CGI interface. WLCG SRR JSON file is generated online without storing data in the intermediate file. This approach has several advantages, because updating storagesummary.json file in the DPM storage can fail for many different reason (no space, no quota, disknode issues, ...). According WLCG storage accounting document SRR data should be provided with at least one protocol and DPM automatically publish these data at http://dpmhead-trunk.cern.ch/static/srr with apache configuration in /etc/httpd/conf.d/zlcgdm-dav.conf

...
# publish WLCG SRR information online (works only with DPM DOME)
ScriptAlias /static/srr "/usr/bin/dpm-storage-summary.cgi"
...

To disable CGI SRR remove these lines from apache configuration or update hiera configuraiton for puppet modules

dmlite::dav::params::enable_srr_cgi: false 

Create & update storagesummary.json file with cron

Use this more complicated and fragile method only if HTTP CGI doesn't fit your requirements.

The script dpm-storage-summary.py will create a file called storagesummary.json which will be used for experiment operations and WLCG storage accounting. It should be run via cron, for example with the following in /etc/cron.hourly/dpm-storage-summary:

#!/bin/bash
/usr/bin/dpm-storage-summary.py --path /dpm/domain.org/home/dteam
/usr/bin/dpm-storage-summary.py --path /dpm/domain.org/home/atlas

which would create a copy of the file under each of the paths shown. It is sufficient to publish this file just once and let all SRR consumer read information from one place.

NOTE: Make sure that the cron file has sufficient permissions to be executed (e.g. 775)

For more reliable publishing it is recommended to use a separate quotatoken for SRR to avoid situation with full VO quotatoken that could prevent updating published data. Create a new directory and associate that directory with quotatoken, but be aware that DPM doesn't allow you to write to quotatoken smaler than associated pool minimum freespace (search for defsize that corresponds to the SRR QT pool in the dmlite-shell -e poolinfo output) and that's why we allocate 4GB (default DOME pool defsize is 3GB) eventhough we are going to store just few kilobytes:

dmlite-shell -e 'mkdir /dpm/domain.org/home/SRR'
dmlite-shell -e 'quotatokenset /dpm/domain.org/home/SRR pool your_existing_pool size 4GB desc SRR groups root'

New path associated with SRR quotatoken can be used same way to publish space occupancy data with /etc/cron.hourly/dpm-storage-summary:

#!/bin/bash
/usr/bin/dpm-storage-summary.py --path /dpm/domain.org/home/SRR --log-level=DEBUG --log-file=/var/log/dpm-srr.log

or you can use puppet to configure similar cron task with following definition

cron {
    'dpm-storage-summary':
        command         => '/usr/bin/dpm-storage-summary.py --path /dpm/domain.org/home/SRR --log-level=DEBUG --log-file=/var/log/dpm-srr.log',
        user            => 'root',
        minute          => '*/15';
}

xrootd

From DPM 1.10 onwards, xrootd has a double role - it acts as a data access frontend but it also provides Dome's HTTP interface.

/etc/sysconfig

SL6:

On SL6, edit /etc/sysconfig/xrootd and

#-------------------------------------------------------------------------------
# Define the instances of xrootd, cmsd and frmd here and specify the option you
# need. For example, use the -d flag to send debug output to the logfile,
# the options responsible for daemonizing, pidfiles and instance naming will
# be appended automatically.
#-------------------------------------------------------------------------------

#-------------------------------------------------------------------------------
# Define the user account name which will be used to start the daemons.
# These may have many unexpected side effects, so be sure you know what you're
# doing before playing with them.
#-------------------------------------------------------------------------------
XROOTD_USER=dpmmgr
XROOTD_GROUP=dpmmgr

#-------------------------------------------------------------------------------
# Define the commandline options for the instances of the daemons.
# The format is:
# DAEMON_NAME_OPTIONS, where:
#   DAEMON - the daemon name, the valid values are: XROOTD, CMSD or FRMD
#   NAME   - the name of the instance, any uppercase alphanumeric string
#            without whitespaces is valid
#-------------------------------------------------------------------------------
XROOTD_REDIR_OPTIONS="-l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-dpmredir.cfg -k fifo"

#-------------------------------------------------------------------------------
# Names of the instances to be started by default, the case doesn't matter,
# the names will be converted to lowercase automatically, use space as a
# separator
#-------------------------------------------------------------------------------
XROOTD_INSTANCES="  redir"
CMSD_INSTANCES=""

export DPM_CONRETRY=0
export DPM_HOST=domehead-trunk.cern.ch
export DPNS_CONRETRY=0
export DPNS_HOST=domehead-trunk.cern.ch
export MALLOC_ARENA_MAX=4
export XRD_MAXREDIRECTCOUNT=1

DAEMON_COREFILE_LIMIT=unlimited

CentOS7:

On CentOS7, the equivalent is achieved by executing sudo systemctl edit xrootd@dpmredir and then

[Unit]
After=network-online.target mariadb.service

[Service]
User=dpmmgr
Group=dpmmgr
RuntimeDirectory=xrootd

Environment=DPM_CONRETRY=0
Environment=DPM_HOST=<headnode>
Environment=DPNS_CONRETRY=0
Environment=DPNS_HOST=<headnode>
Environment=XRD_MAXREDIRECTCOUNT=1
LimitCORE=infinity
Environment=DAEMON_COREFILE_LIMIT=unlimited
# recommended memory allocation library for xrootd, install jemalloc package first
# CentOS7
#Environment=LD_PRELOAD=/usr/lib64/libjemalloc.so.1
# CentOS8
#Environment=LD_PRELOAD=/usr/lib64/libjemalloc.so.2

/etc/xrootd

Edit /etc/xrootd/xrootd-dpmredir.cfg

#ofs.trace all
#xrd.trace all
#cms.trace all
#oss.trace all
#xrootd.trace all
#http.trace all

all.adminpath /var/spool/xrootd
all.pidpath /var/run/xrootd
all.sitename CERN_DPM_TEST
xrd.network nodnr

xrootd.chksum max 100 adler32 md5 crc32

if exec xrootd
xrootd.seclib libXrdSec.so
sec.protocol /usr/lib64 gsi -crl:3 -key:/etc/grid-security/dpmmgr/dpmkey.pem -cert:/etc/grid-security/dpmmgr/dpmcert.pem -md:sha256:sha1 -ca:2 -gmapopt:10 -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so
sec.protocol /usr/lib64 unix
xrootd.export /
ofs.cmslib libXrdDPMFinder.so.3
ofs.osslib +cksio libXrdDPMOss.so.3
ofs.authlib libXrdDPMRedirAcc.so.3
ofs.authorize
ofs.forward all
all.role manager
ofs.ckslib = libXrdDPMCks.so.3
fi

if exec cmsd
    all.role server
fi

############################################
# The following parameters are DPM-specific

dpm.nohv1

if exec xrootd
dpm.xrdserverport 1095
xrd.protocol XrdHttp /usr/lib64/libXrdHttp-4.so
http.exthandler dome /usr/lib64/libdome.so /etc/domehead.conf
http.selfhttps2http yes
http.cert /etc/grid-security/dpmmgr/dpmcert.pem
http.key /etc/grid-security/dpmmgr/dpmkey.pem
http.cadir /etc/grid-security/certificates
http.secretkey  <key at least 32 chars>
http.cipherfilter HIGH

fi
# the following can be used to check for and if necessary add a
# prefix to file names. i.e. to allow access via names like /dteam/the_file
dpm.defaultprefix /dpm/cern.ch/home

dpm.dmconf /etc/dmlite.conf

dpm.mmreqhost localhost

The /etc/xrootd/dpmxrd-sharedkey.dat file should be created to contain the xootd key, which should be the same for all the DPM cluster. Ensure that permissions are 600 and the owner is dpmmgr.

For a key, use a 64 byte string of random ascii characters, no newline. Other keys may be possible, consult the xroot docs for more info.

Directory permissions

chown -R dpmmgr:dpmmgr /var/log/xrootd
chown -R dpmmgr:dpmmgr /var/spool/xrootd/

Configure startup

SL6:

chkconfig xrootd on
service xrootd start

CentOS7:

systemctl enable xrootd@dpmredir
systemctl start xrootd@dpmredir

HTTP

Configure Apache with the event MPM

SL6:

The file /etc/sysconfig/httpd should be configured with:

HTTPD=/usr/sbin/httpd.event

CentOS7:

The file /etc/httpd/conf.modules.d/00-mpm.conf

LoadModule mpm_event_module modules/mod_mpm_event.so

Recommended mpm configuration (mpm_event.conf)

<IfModule mpm_event_module>
    StartServers          4
    ServerLimit          16
    MinSpareThreads       1
    MaxSpareThreads    1200
    ThreadLimit         300
    ThreadsPerChild     300
    MaxClients         1200
<IfVersion >= 2.4>
    MaxRequestWorkers  4800
</IfVersion>
<IfVersion < 2.4>
    MaxClients         4800
</IfVersion>
    MaxRequestsPerChild   100000
</IfModule>

httpd.conf

Configure /etc/httpd/conf/httpd.conf

User dpmmgr
Group dpmmgr

lcgdm-dav (dmlite-apache-httpd) configuration

Following a code restructuration, lcgdm-dav is now distributed as dmlite-apache-httpd.

Edit /etc/httpd/conf.d/zlcgdm-dav.conf.

Set the following (as required)

NSFlags Write RemoteCopy
...
DiskFlags Write RemoteCopy

on SL6 comment out these lines (if present) from the file /etc/httpd/conf/httpd.conf

LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so

On CentOS7 this is done by zeroing the following file

cat /dev/null > /etc/httpd/conf.modules.d/00-dav.conf

On a typical default install,you will have to zero /etc/httpd/conf.d/ssl.conf and /etc/httpd/conf.d/zgridsite.conf. This approach ensures that they are not replaced by subsequent package updates.

cat /dev/null > /etc/httpd/conf.d/ssl.conf
cat /dev/null > /etc/httpd/conf.d/zgridsite.conf

To support WebDAV TPC with tokens you have to add the line NSMacaroonSecret <your_secret_string_longer_then_64_chars> to the already existing section <LocationMatch "^/dpm/.*">. Also check that SSLVerifyClient is set to optional, because in the past this configuration option was set to required which is not compatible with (macaroon) token authentication.

Schedule graceful restarts

A periodic graceful restart of apache has to be scheduled to ensure that it re-reads the CRLs. For example, an /etc/cron.d entry such as the following script;

# graceful http restart
50 */6 * * * root /usr/sbin/apachectl graceful >& /dev/null

Configure startup

SL6:

chkconfig httpd on
service httpd start

CentOS7:

systemctl enable httpd
systemctl start httpd

Note on Nagios and PHP

The php rpm, often pulled in by Nagios probes, introduces a configuration (in /etc/httpd/conf.d/php.conf) which is incompatible with the event MPM used by DPM. The aforementioned config file must be zeroed to allow httpd to start.

Note on OpenID -Connect and WLCG bearer tokens through HTTP

Starting from version 1.14 DPM has built-in support for OpenID -Connect and its bearer tokens, which may or may not comply to the WLCG profile. This summary describes how it works in the case of:

  • a client like DAVIX or curl inserrting the token into a request
  • a browser browsing the logical directory structure or downloading files.

In the case of a command-line client, all the requests that are produced are normally added an Authorization header line, similar to:

> GET /dpm/cern.ch/home/dteam/yuk2 HTTP/1.1
> User-Agent: libdavix/0.7.5 neon/0.0.29
> Keep-Alive: 
> Connection: Keep-Alive
> TE: trailers
> Host: fab-dpm-dev1.cern.ch
> Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

In order to be able to unpack and validate these bearer tokens, Apache (the HTTP frontend of DPM) needs to be added an Apache module. At the time of this writing (Feb 2020) the minimum versions for the packages to install are:

mod_auth_openidc-2.4.0-1.el7.x86_64.rpm
cjose-0.6.1.5-1.el7.x86_64.rpm

Before you start with the DPM OIDC configuration it is necessary to register your DPM (client application) with your OIDC provider (e.g. WLCG AIM). Apache configuration file (normally /etc/httpd/conf.d/zlcgdm-dav.conf) needs to be updated with configuration options for mod_auth_openidc and most essential are following configuration options:

...
...

# The location of the base dmlite configuration file
NSDMLite /etc/dmlite.conf


OIDCResponseType                "code"
OIDCScope                       "openid email profile wlcg.groups"
OIDCProviderMetadataURL         https://wlcg.cloud.cnaf.infn.it/.well-known/openid-configuration
OIDCClientID                    < The OIDC Client ID for this service >
OIDCClientSecret                < The OIDC Client Secret for this service >
OIDCProviderTokenEndpointAuth   client_secret_basic
OIDCCryptoPassphrase            < The OIDC crypto passphrase >
OIDCRedirectURI                 < The Redirect URI >

OIDCOAuthVerifyJwksUri https://wlcg.cloud.cnaf.infn.it/jwk
OIDCOAuthRemoteUserClaim sub

...
...

# Base path for nameserver requests
<LocationMatch "^/dpm/.*">

...
...

  # Use this user for anonymous access
  # It has to be in the mapfile!
  NSAnon nobody:nogroup

  # Check the authorization HTTP header
  <If "%{HTTP:Authorization} =~ /^[Bb][Ee][Aa][Rr][Ee][Rr] dpm-macaroon/">
   AuthType oauth20
   #Require valid-user
  </If>
  <ElseIf "%{HTTP:Authorization} != ''">
   AuthType oauth20
   Require valid-user
  </ElseIf>
  <ElseIf "%{HTTP:User-Agent} =~ /Mozilla|Chrom|MSIE/">
   AuthType openid-connect
   Require valid-user
  </ElseIf>
...
...

The DOME configuration file (normally /etc/domehead.conf) needs to be added some directives:

The directive head.oidc.allowissuer[] is used to specify an array of triples:

  • an absolute logical path
  • an OIDC issuer URL
  • a DPM group name

*Clients coming with a valid OIDC bearer token asking to operate on a certain logical directory path, and authorized by a certain OIDC issuer will be treated as belonging to the given group.*

NOTE: since this is an array, one can add multiple such associations.

The directive head.oidc.allowaudience[] is used to specify an array of OIDC audience strings.

Clients will be authorized only if their audience is among the allowed ones.

Example:

...
head.oidc.allowissuer[]: "/dpm/domain.org/home/cms" "https://cms-auth.web.cern.ch/" cms
head.oidc.allowissuer[]: "/dpm/domain.org/home/wlcg" "https://wlcg.cloud.cnaf.infn.it/" wlcg
head.oidc.allowissuer[]: "/dpm/domain.org/home/xdc" "https://iam.extreme-datacloud.eu/" xdc
head.oidc.allowaudience[]: https://wlcg.cern.ch/jwt/v1/any
head.oidc.allowaudience[]: https://< your headnode >
#head.oidc.allowaudience[]: https://< your headnode >:non_standard_https_port
...

NOTE: a bearer token that follows the WLCG profile can assert that the operation it authorizes is to be performed as member of some groups ( wlcg.groups assertion ). DPM supports this.

NOTE: a bearer token that follows the WLCG profile can provide a scope. DPM supports this, and only operations authorized by the scope will be allowed.

GridFTP

Configure /etc/gridftp.conf as follows:

inetd 0
daemon 1
detach 1
chdir 1
fork 1
single 0

cas 1
secure_ipc 1
ipc_auth_mode host
allow_anonymous 0

log_level ERROR,WARN,INFO
log_single /var/log/dpm-gsiftp/gridftp.log
log_transfer /var/log/dpm-gsiftp/dpm-gsiftp.log
disable_usage_stats 1
usage_stats_target usage-stats.globus.org:4810

# List your disk servers here
remote_nodes disk-server01.domain.org:2811 disk-server02.domain.org:2811
data_node 0
stripe_blocksize 1048576
stripe_layout 2
stripe_blocksize_locked 0
stripe_layout_locked 0

blocksize 262144
sync_writes 0

port 2811

control_preauth_timeout 120
control_idle_timeout 600
ipc_idle_timeout 600
ipc_connect_timeout 600

banner_terse 0
login_msg "Disk Pool Manager (dmlite)"

load_dsi_module dmlite
use_home_dirs 1
debug 0

# trigger non-standard but necessary return of host ip in response to EPSV
epsv_ip 1

The file /etc/sysconfig/globus should be created to contain:

conf=/etc/gridftp.conf
confdir=/etc/gridftp.d
export GLOBUS_THREAD_MODEL="pthread"

Ensure you have the following line in /etc/sysconfig/dpm-gsiftp

OPTIONS="-S -p 2811 -auth-level 0 -dsi dmlite:dome_checksum -disable-usage-stats"

Configure startup

SL6:

chkconfig dpm-gsiftp on
/etc/init.d/dpm-gsiftp start

Disable the globus-gridftp-server

/sbin/chkconfig globus-gridftp-server off

CentOS7:

systemctl enable dpm-gsiftp
systemctl start dpm-gsiftp

Deploying a Disk Node

Security configuration

Copy the host certificate/key pair to ''/etc/grid-security/dpmmgr'', owned by ''dpmmgr''

mkdir /etc/grid-security/dpmmgr
cp -a /etc/grid-security/hostcert.pem /etc/grid-security/dpmmgr/dpmcert.pem
cp -a /etc/grid-security/hostkey.pem /etc/grid-security/dpmmgr/dpmkey.pem
chown -R dpmmgr.dpmmgr /etc/grid-security/dpmmgr

Mountpoints configuration

The storage montpoints should be owned by the user dpmmgr

N.B.It's not adivsed to use as storage mountpoints paths starting with /dpm as this is will create troubles to the WebDav frontend. If this is the case for your installation please contact the support.

Core configuration

dmlite configuration

If necessary, configure /etc/dmlite.conf. The default file shipped with the rpm should work.

Dome configuration

Then the the file /etc/domedisk.conf has to be added and configured as follows - change the relevant fields.

glb.role: disk
glb.debug: 1

glb.auth.urlprefix: /domedisk/

glb.task.maxrunningtime: 3600
glb.task.purgetime: 3600

glb.restclient.cli_certificate: /etc/grid-security/dpmmgr/dpmcert.pem
glb.restclient.cli_private_key: /etc/grid-security/dpmmgr/dpmkey.pem
glb.restclient.xrdhttpkey: <key at least 32 chars>
disk.headnode.domeurl: http://<YOUR HEADNODE FQDN>:1094/domehead
disk.filepuller.pullhook: /usr/share/dmlite/filepull/externalpull_example.sh

Create /etc/dmlite.conf.d/domeadapter.conf.

Update the relevant fields, in particular, use the same TokenPassword that you configured on the head node.

LoadPlugin plugin_domeadapter_diskcatalog /usr/lib64/dmlite/plugin_domeadapter.so

LoadPlugin plugin_domeadapter_io /usr/lib64/dmlite/plugin_domeadapter.so

LoadPlugin plugin_domeadapter_pools /usr/lib64/dmlite/plugin_domeadapter.so


DavixCAPath  /etc/grid-security/certificates
DavixCertPath /etc/grid-security/dpmmgr/dpmcert.pem
DavixPrivateKeyPath /etc/grid-security/dpmmgr/dpmkey.pem

DomeHead http://<your headenode>:1094/domehead
DomeDisk http://<your disknode>:1095/domedisk

# Token generation
TokenPassword <key at least 32 chars>
TokenId ip
TokenLife <tokenlifetime>

ThisDomeAdapterDN <DN of disk node in / / / format>

Remove /etc/dmlite.conf.d/adapter.conf and /etc/dmlite-disk.conf.d/adapter.conf if present.

xrootd

From DPM 1.10 onwards, xrootd has a double role - it acts as a data access frontend but it also provides Dome's HTTP interface.

/etc/sysconfig

The file /etc/sysconfig/xrootd should be modified to contain:

XROOTD_USER=dpmmgr
XROOTD_GROUP=dpmmgr
XROOTD_DISK_OPTIONS="-l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-dpmdisk.cfg -k fifo"
XROOTD_INSTANCES="  disk"
CMSD_INSTANCES=""
DPM_CONRETRY=0
export DPM_CONRETR
DPM_HOST=<DPM_hostname>
export DPM_HOST
DPNS_CONRETRY=0
export DPNS_CONRETRY
DPNS_HOST=<DPNS_hostname>
export DPNS_HOST
MALLOC_ARENA_MAX=4
export MALLOC_ARENA_MAX
XRD_MAXREDIRECTCOUNT=1
export XRD_MAXREDIRECTCOUNT
DAEMON_COREFILE_LIMIT=unlimited

On CentOS7, the equivalent is achieved by executing sudo systemctl edit xrootd@dpmdisk and then

[Service]
User= dpmmgr
Group= dpmmgr
RuntimeDirectory=xrootd
DPM_CONRETRY=0
DPM_HOST=dpmhead-cc7-rc.cern.ch
DPNS_CONRETRY=0
DPNS_HOST=dpmhead-cc7-rc.cern.ch
MALLOC_ARENA_MAX=4
XRD_MAXREDIRECTCOUNT=1

DAEMON_COREFILE_LIMIT=unlimited

/etc/xrootd

The file /etc/xrootd/xrootd-dpmdisk.cfg should contain:

#ofs.trace all
#xrd.trace all
#cms.trace all
#oss.trace all

all.adminpath /var/spool/xrootd
all.pidpath /var/run/xrootd
xrd.network nodnr

xrootd.chksum max 100 adler32 md5 crc32

xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 dest redir fstat info user uct2-int.mwt2.org:9930 dest redir fstat info user atlas-fax-eu-collector.cern.ch:9330
xrootd.async on

if exec xrootd
xrootd.seclib libXrdSec.so
sec.protocol /usr/lib64 gsi -dlgpxy:1 -exppxy:=creds -crl:3 -key:/etc/grid-security/dpmmgr/dpmkey.pem -cert:/etc/grid-security/dpmmgr/dpmcert.pem -md:sha256:sha1 -ca:2 -gmapopt:10 -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:vos=atlas,cms,dteam|grps=/atlas,/cms,/dteam|grpopt=10|dbg
/lib64/libXrdSecgsiVOMS.so
sec.protocol /usr/lib64 unix
#sec.level all compatible
xrootd.export /
xrd.port 1095
xrd.timeout idle 60m
ofs.osslib +cksio libXrdDPMOss.so.3
ofs.authlib libXrdDPMDiskAcc.so.3
ofs.authorize
ofs.persist auto hold 0
ofs.tpc xfr 25 oids fcreds gsi =X509_USER_PROXY pgm /usr/bin/xrdcp --server
all.role server
ofs.ckslib = libXrdDPMCks.so.3
fi

if exec cmsd
all.role server
fi

dpm.nohv1
if exec xrootd
xrd.protocol XrdHttp /usr/lib64/libXrdHttp-4.so
http.exthandler dome /usr/lib64/libdome.so /etc/domedisk.conf
http.selfhttps2http yes
http.cert /etc/grid-security/dpmmgr/dpmcert.pem
http.key /etc/grid-security/dpmmgr/dpmkey.pem
http.cadir /etc/grid-security/certificates
http.secretkey  <key at least 32 chars>
http.cipherfilter HIGH

fi
dpm.dmconf /etc/dmlite.conf

The /etc/xrootd/dpmxrd-sharedkey.dat file should be created to contain the xootd key, which should be the same for all the DPM cluster. Ensure that permissions are 600 and the owner is dpmmgr.

For a key, use a 64 byte string of random ascii characters, no newline. Other keys may be possible, consult the xroot docs for more info.

The config above includes support for the xrootd proxy delegation available in xrootd 4.9.

Directory permissions

chown -R dpmmgr:dpmmgr /var/log/xrootd
chown -R dpmmgr:dpmmgr /var/spool/xrootd/

Configure startup

SL6:

chkconfig xrootd on
service xrootd start

CentOS7:

systemctl enable xrootd@dpmdisk
systemctl start xrootd@dpmdisk

HTTP

Configure Apache with the event MPM

SL6:

The file /etc/sysconfig/httpd should be configured with:

HTTPD=/usr/sbin/httpd.event

CentOS7:

The file /etc/httpd/conf.modules.d/00-mpm.conf

LoadModule mpm_event_module modules/mod_mpm_event.so

Recommended mpm configuration (mpm_event.conf)

<IfModule mpm_event_module>
    StartServers          4
    ServerLimit          16
    MinSpareThreads       1
    MaxSpareThreads    1200
    ThreadLimit         300
    ThreadsPerChild     300
    MaxClients         1200
<IfVersion >= 2.4>
    MaxRequestWorkers  4800
</IfVersion>
<IfVersion < 2.4>
    MaxClients         4800
</IfVersion>
    MaxRequestsPerChild   100000
</IfModule>

httpd.conf

In addition the following configuration changes are needed in /etc/httpd/conf/httpd.conf:

User dpmmgr
Group dpmmgr

SL6:

Comment out these lines if present:

LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so

CentOS7:

On CentOS7 this is done by removing the following file

/etc/httpd/conf.modules.d/00-dav.conf

TODO : KeepAlive configuration

lcgdm-dav (dmlite-apache-httpd) configuration

then both files zgridsite.conf and ssl.conf ( under /etc/httpd/conf.d/) should be emptied, as the configuration will be provided by zlcgdm-dav.conf

Ensure that zlcgdm-dav.conf contains the following outside the Location configuration

#disable gridsite session files generation
GridSiteGridHTTP off
GridSiteAutoPasscode off

Edit /etc/httpd/conf.d/zlcgdm-dav.conf.

Set the following (as required)

NSFlags Write RemoteCopy
...
DiskFlags Write RemoteCopy

N.B Make sure then to change the ownership of the above configuration files to dpmmgr:dpmmgr

Ensure that the directory var/www/proxycache exists with owner/group dpmmgr:dpmmgr

To support WebDAV TPC check that SSLVerifyClient is set to optional, because in the past this configuration option was set to required which is not compatible with (macaroon) token authentication.

Schedule graceful restarts

A periodic graceful restart of apache has to be scheduled to ensure that it re-reads the CRLs.

# graceful http restart
50 */6 * * * /usr/sbin/apachectl graceful >& /dev/null

Configure startup

SL6:

chkconfig httpd on
service httpd start

CentOS7:

systemctl enable httpd
systemctl start httpd

GridFTP

Configure /etc/gridftp.conf as follows:

inetd 0
daemon 1
detach 1
chdir 1
fork 1
single 0

cas 1
secure_ipc 1
ipc_auth_mode host
allow_anonymous 0

log_level ERROR,WARN,INFO
log_single /var/log/dpm-gsiftp/gridftp.log
log_transfer /var/log/dpm-gsiftp/dpm-gsiftp.log
disable_usage_stats 1
usage_stats_target usage-stats.globus.org:4810

data_node 1
stripe_blocksize 1048576
stripe_layout 2
stripe_blocksize_locked 0
stripe_layout_locked 0

blocksize 262144
sync_writes 0

port 2811

control_preauth_timeout 120
control_idle_timeout 600
ipc_idle_timeout 600
ipc_connect_timeout 600

banner_terse 0
login_msg "Disk Pool Manager (dmlite)"

load_dsi_module dmlite
use_home_dirs 1
debug 0

and the file /etc/sysconfig/globus that should be created to contain:

conf=/etc/gridftp.conf
confdir=/etc/gridftp.d
export GLOBUS_THREAD_MODEL="pthread"

Configure startup

SL6:

chkconfig dpm-gsiftp on
/etc/init.d/dpm-gsiftp start

make sure to disable the globus-gridftp-server

/sbin/chkconfig globus-gridftp-server off

!CentOS7

systemctl start  dpm-gsiftp
systemctl enable dpm-gsiftp

Cluster initialisation

For full details on operations such as creating pools, adding filesystems, VO support and quotatoken management, see DpmAdministration.

Edit | Attach | Watch | Print version | History: r74 < r73 < r72 < r71 < r70 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r74 - 2020-11-23 - PetrVokac
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    DPM All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback