Web access to DPM storage via HTTPS(HTTP)

Introduction

Web access to DPM files is handled by an Apache CGI script running in a virtual host on port 443 (https). This virtual hosts acts as a redirector to redirect requests to the disk server which stores the requested DPM file. Globus/VOMS authentication is handled by mod_gridsite ( http://www.gridsite.org). The current packages support read via GET & write reqeusts via POST + PUT.

Clients have to provide credentials and the CGI script executes credential mapping & access checks in the DPM name server. By default the redirection forwards the client to an Apache virtual host on port 777 (http). Currently VOMS support is not yet enabled in the current packages - only globus grid-map files.

A standard request to browse a DPM directory would be: https://<dpm-head-node>/dpm/cern.ch/home/dteam .... to download a DPM file point to a file with the URL path e.g. https//<dpm-head-node>/dpm/cern.ch/home/dteam/testfile

To ensure authorization by DPM access permissions the Apache server on port 777 runs an authorization plugin which verifies a signature which has been appended by the redirector to the requested URL. The file transport then runs over plain HTTP protocol.

Additionally the client can append to his request as a query option "?protocol=https" to read the file over an encrypted HTTPS connection from an Apache virtual host on port 884.

For convenience another virtual host is running on port 883 on the redirector. Requests made here are automatically redirected to HTTPS transport on port 884 without the need to append the query string option.

• Head-Node
  • Virtual Host Port 443
    • main entrance point for web file access to a DPM 'https://<dpm-headnode>/dpm/cern.ch/filename' with default redirection to HTTP transport
      
  • Virtual Host Port 883
    • main entrance point for web file access to a DPM with forced redirection to HTTPS transport ( slower )
• Disk-Server
  • Virtual Host Port 777
    • tranpsport endpoint for HTTP web file access with redirector authorization
      
  • Virtual Host Port 884
    • tranpsport endpoint for HTTPS web file access with redirector authorization

Source Installation & Setup

Requirements

• Apache Version 2 installed (yum install httpd)
• Apache Devel Package (yum install httpd-devel)
• Openssl installed (yum install openssl)
• Openssl Devel Package (yum install openssl-devel)
• gridsite package installed/mod_gridsite for Apache (yum install gridsite)
• DPM installation in /opt/lcg/ ---++ Source Compilation
• Untar the 3 source tarballs
• Execute in every directory:
  • ./configure --prefix=/opt/lcg/
  • make
  • make install

Configuration

• Preparation for a head node (including a single head/disk-node):
  • Create the directory /opt/lcg/etc/dpm/https/keystore and inside create:
  • A symbolic link named cert.pem target /etc/grid-security/dpmmgr/dpmcert.pem
  • A symbolic link named key.pem target /etc/grid-security/dpmmgr/dpmkey.pem
• Preparation for a disk pool only node:
  • Create the directory /opt/lcg/etc/dpm/https/keystore and inside place:
  • A copy of the cert.pem from the headnode
• Run the configuration script depending if you have a single head/disk-node or a seperate head- and seperate disk-nodes
  • /opt/lcg/etc/dpm/https/conf/dpm-https-conf.sh --type sor
  • /opt/lcg/etc/dpm/https/conf/dpm-https-conf.sh --type head-node
  • /opt/lcg/etc/dpm/https/conf/dpm-https-conf.sh --type disk-node
• Whenever you add or remove disks to a DPM pool, you have to run on the disk node
  • /opt/lcg/etc/dpm/https/conf/dpm-https-conf.sh --pools

Service Startup

• To start the DPM apache service run
  • /etc/init.d/dpm-httpd start
• To stop the DPM apache service run
  • /etc/init.d/dpm-httpd stop
• To check the status run
  • /etc/init.d/dpm-httpd status
• To restart the DPM apache service
  • /etc/init.d/dpm/httpd restart

Log Files

• Apache access & error log files are written under /var/log/dpm-httpd/
  • /var/log/dpm-httpd/access
  • /var/log/dpm-httpd/errors
• The redirection CGI script writes a special log file under /var/log/dpm-httpd/cgilog
  • errors are logged also in syslog
    
• The startup-package configures automatic logration for the 3 logfiles above.
  

Known Problems

dpm-httpd does not start

Starting dpm-httpd: (13)Permission denied: make_sock: could not bind
to  address [::]:884
no listening sockets available, shutting down

Please try to disable SELinux to make it work!

Clients

Web-Browser

• The DPM catalogue can be browsed with any HTTP browser without any special plugins (no JAVA script needed).
  
  
• For browsing via HTTPS & file transfer via HTTP specify in the URL 'https' protocol & the head node of your DPM installation.
  
  
  
• For browsing via HTTPS & file transfer via HTTPS specify in the URL 'https' protocol & the head node of your DPM installation + port 883 !
  
  
• You can navigate the DPM by clicking directory names to enter a directory or 'parent dir' to navigate one level up. Clicking on plain files triggers the download of file depending on its MIME extension and your browser configuration.
  
• To upload a new file use the first form on the head of the page:
  
  
  Enter the new name, click 'post' and use the 'Browse' button in the newly displayed page to select a local file. Start the upload by pressing 'Upload':
  
  
• Be aware that you have only 120s time to select the source file with the 'browse' option! The upload via a browser is implemented using a POST call. After the successful upload the browser will display a 'stat' call on the newly uploaded ">file:<br' src="">
• You can use the same form to create directories, remove files or directories or change the ownership of a file.
• Right of every directory item are forms to modify the displayed item. With 'Post' you can upload a new file to this DPM file. The other buttons 'rm', 'stat', 'chmod' should be straight forward to understand.

Command Line Client

• /opt/lcg/bin/dpm-httpd-cp <src> <dst>
  The copy command uses internally only the 'curl' executable to issue 'GET' and 'PUT' requests. To specify a local file just use the usual path name, to specify a DPM file use the same URL syntax as explained for the web browser client.

  • • read:
      
      
    • write:
      
      
• /opt/lcg/bin/dpm-httpd-cmd [-m <mode>] ls | chmod | stat | rm | mkdir <url>
  The busy box command execute 'ls', 'chmod', 'stat', 'rm' & 'mkdir' DPM commands. The <url> is the same used for web browser access. Successful commands return 0 on the shell, otherwise the error code & message is displayed and the error code returned to the shell.
  
  • • ls:
      
      
    • mkdir
      :
      
    • mkdir with error:
      
      
    • rm:
      
      
    • stat / chmod:
      
      
      

ROOT Interface

• A ROOT TFile plugin based on libcurl is under consideration.
  

-- Main.apeters - 14 Aug 2007