DPM volatile pools and caching

Since version 1.10, DPM supports the so-called "volatile pools", that can be used to give cache-like behaviour to a DPM pool.

This is what it means:

  • The volatile area is normally accessed by path. Volatile pools are normally assigned to paths through the use of quotatokens.
  • Reads from a volatile area work normally if the file is there
  • If the requested file is not there, DPM will try to fetch it from an external source
  • The FULL FILE is retrieved (no chunks)
  • The client will block while this happens
  • Writes into a volatile pool go into the volatile pool (if there is space)

Marking a pool as “Volatile” triggers the cache-like behaviour for the path that has been assigned to that pool through a quotatoken.

A DPM volatile pool can cache virtually any source of files, remote or not. If there's no space in the pool or in the quota, and a new file has to be pulled, DPM will try to remove some files from the volatile pool, to make space.

How to prepare a DPM setup to retrieve files from external endpoints

DPM needs 2 instrumentation scripts:

  • one implements stat() towards the remote system
  • the other one retrieves the new file from the remote system

The stat script must be configured on the head node, the pull script must be configured on the disk servers. The directive head.filepuller.stathook must be provided to the head node like in the following example:

head.filepuller.stathook: /usr/share/dmlite/filepull/externalstat_example.sh

The directive disk.filepuller.pullhook must be provided to the disk servers corresponding to the volatile pools like in the following example:

disk.filepuller.pullhook: /usr/share/dmlite/filepull/externalpull_example.sh

We provide two simple example scripts that fake an external source, just to show the parameters.

The example scripts can be found in

/usr/share/dmlite/filepull

Who does the integration will have to adapt these two scripts or executables to his own external source. Depending on the external source, it's reasonable to assume that this script will need credentials of some kind to do its work. These credentials cannot be provided by the DPM infrastructure.

NOTE: other more advanced parameters (e.g. max numbers of pulls per server, etc) can influence the behaviour of the file pulling mechanism. For those, we encourage the reader to have a look at the DOME documentation .

Example for a stat() script or executable

Please remind that the stat() script must be as quick and efficient as possible.

#!/bin/sh

# usage: externalstat.sh <lfn>
#
# This is an example script for the DOME file pull hooks
# To return "File not found" to the client, just return nonzero

# This script will make DOME believe that there is an external file
# that has 123456 as its size
# The companion pulling script will have to create such a file when invoked

echo ">>>>> STAT 123456”

Example for a pull script or executable

#!/bin/sh

# usage: externalpull.sh <lfn> <pfn>

# This is an example file puller script, that creates a fake file <pfn>
# by pulling it from nowhere using dd
# If querying an external system, the query should be based on the <lfn>
#

# Let’s claim we’re doing something complex and important. To be dropped in a serious script!
sleep 5

# Pull the file
dd "of=$2" "if=/dev/urandom" bs=123456 count=1

How to configure a volatile pool

Steps:
  • Create a volatile pool using dmlite-shell and add filesystems to it, or use a preexisting one. The command to be used is poolmodify.
    • The pool s_type must be "V"
    • Make sure that the defsize is larger than the biggest file that the pool has to manage

  • Create a Quotatoken pointing to that pool and giving a quota of reasonable size
  • Assign the Quotatoken to your /volatile path
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2018-05-29 - FabrizioFurano
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    DPM All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback