DPM Introduction

Overview

The Disk Pool Manager (DPM) is a distributed storage system offering multi-protocol access to a scalable data-store with support for "grid standard" X509/VOMS authentication and authorization.

It is supported by the CERN IT Storage Group and is available through EPEL.

Architecture

Physical architecture of a deployment

DPM maintains its metadata, including the namespace and ACLs, on a head node. Clients contact this node, and are then redirected to one of the many disk servers for data access.

DPMArch.svg

Software stack

A DPM deployment glues some frontends onto some backend resources. This glue is represented by two major subsystems, Dome and dmlite.

dmlite is a library designed to abstract the interactions which a protocol frontend will have with the namespace and the backend resources.

Dome is a service which provides resource management functions through the dmlite API and via native commands.

This picture shows the internal architecture of a DPM head node using DOME. domehead.png

This picture shows the internal architecture of a DPM disk node using DOME. domedisk.png

Frontends

DPM provides standard frontends for the following protocols.

  • HTTP - implemented by Apache
  • xroot - implemented by standard XrootD
  • GridFTP - implemented by the Globus gridftp server

DBs

DPM's metadata persistency is implemented using MySQL. Internally, DOME caches various metadata items in a dedicated write-through cache, which highers the performance.

Backends

DPM can support different backend datastores.

Disk servers

The standard DPM disk server runs a protocol stack on top of its local filesystem, offering access to the data stored there.

Disk cache

DPM has an experimental caching backend, which will retrieve data for the client and manage the storage as a cache.

HDFS

DPM supports HDFS as a backend.

The legacy system

The DPM has gone through a number of generations and this guide describes the final platform for the future, based on DOME, dmlite and standard protocol frontends. Most existing DPM installations are based, at least partially, on a legacy stack which can operated in parallel with the system described here. The legacy stack provides the SRM interface and is described in an appendix to this guide.

Resources

-- OliverKeeble - 2017-06-06

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng domedisk.png r1 manage 156.1 K 2018-12-19 - 15:56 FabrizioFurano  
PNGpng domehead.png r1 manage 128.2 K 2018-12-19 - 15:56 FabrizioFurano  
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2018-12-19 - FabrizioFurano
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    DPM All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback