CVMFS Key and Config Distribution Plan

This plan was agreed to in a meeting at CERN on 28 March 2014. Those attending the meeting were:

• Jakob Blomer (CERN)
• Catalin Condurache (RAL)
• Dave Dykstra (FNAL/OSG)
• Gerardo Ganis (CERN)
• Dmitry Ozerov (DESY, via Skype)
• Steve Traylen (CERN)
• Dennis van Dok (NIKHEF, via Skype)

This was a followup to a prior meeting written up at KeyAndConfigDistribution.

This page is mostly Jakob's notes from the meeting, with minor updates for any changes to the plan that were made since. The target cvmfs version for implementing all of this is 2.1.20. The /etc/cvmfs/default.d part was implemented in 2.1.19. As of 16 July 2014 most of it has been integrated into the cvmfs development branch other than the client side of automated stratum 1 ordering. The relevant tickets are CVM-616, CVM-617, CVM-618, CVM-619, CVM-629, and CVM-630.

Synopsis

With more widespread use of cvmfs, the core software should be disentangled configuration-wise from the links to CERN and .cern.ch domain. At the same time, configuration of newly created cvmfs repositories, changes to the public signing keys, and changes in the stratum 0 / stratum 1 relationship should be quickly distributed throughout the grid. That would allow VOs at any scale to distribute their software in the grid.

To this end, cvmfs will be extended to read keys and configuration from another, bootstrap cvmfs repository, the "cvmfs config repository". This meeting decided on the layout and operational model of the (CERN) cvmfs config repository.

In order to avoid the need at sites to manually set the order of stratum 1 servers (and to always check of the latest list of stratum 1 servers in the configuration), the ordering should be automatic. Stratum 1 servers should be extended so that any of them can create an ordered list for the clients. The results will be cached on the site proxies.

Decisions and actions

1. Disentanglement from core cvmfs software and CERN-related configuration
   • CERN specific configuration should be removed from the cvmfs package. The cvmfs-keys package will no longer be a dependency of the cvmfs package. Instead there will packages of the form cvmfs-config-*** (e.g. cvmfs-config-cern). Typically the cvmfs-config-*** packages are mutually exclusive. The cvmfs package alone does not result in a fully functional client.
   • Each future cvmfs-config-*** package "Provides" cvmfs-keys and "Obsoletes" cvmfs-keys.
   • Probably the cvmfs package "Requires" a generic cvmfs-config pseudo-package and each cvmfs-config-*** package also "Provides" cvmfs-config.
2. Distribution of many independent cvmfs domains and repositories in the grid
   • A new repository, for example cvmfs-config-cern.cern.ch, is created that can host public keys and configuration for repositories supposed to run on the grid. These repositories have to be connected in some way to LCG. Similar config repositories can emerge for other large grids such as OSG or EGI.
   • Every cvmfs installation can point to a single cvmfs-config.cern.ch repository by setting

     CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch
     

     (not set by default but by one of the cvmfs-config-*** packages)
   • The cvmfs-config.cern.ch repository can be mounted provided that the cvmfs-config-cern package is installed. The cvmfs-config.cern.ch repository is maintained like other cern.ch repositories.
   • The new repository layout is like the following mock up

     /cvmfs
     `-- cvmfs-config-cern.cern.ch
         `-- etc
             `-- cvmfs
                 |-- config.d
                 |   `-- special.desy.de.conf
                 |-- contact
                 |   |-- cern.ch
                 |   |-- desy.de
                 |   `-- opensciencegrid.org
                 |-- domain.d
                 |   |-- cern.ch.conf
                 |   |-- desy.de.conf
                 |   `-- opensciencegrid.conf
                 `-- keys
                     |-- cern.ch
                     |   |-- key1.pub
                     |   `-- key2.pub
                     |-- desy.de
                     |   |-- key1.pub
                     |   `-- key2.pub
                     `-- opensciencegrid.org
                         |-- key1.pub
                         `-- key2.pub  
     

   • The future order of parsing configuration files by the client is

     /etc/cvmfs/default.conf
     /etc/cvmfs/default.d/*  (e.g. /etc/cvmfs/default.d/10-osg)
     /etc/cvmfs/default.local
     /cvmfs/${CVMFS_CONFIG_REPOSITORY}/etc/cvmfs/domain.d/$domain.conf
     /etc/cvmfs/domain.d/$domain.{conf,local}
     /cvmfs/${CVMFS_CONFIG_REPOSITORY}/etc/cvmfs/config.d/$fqrn.conf
     /etc/cvmfs/config.d/$fqrn.{conf,local}
     

   • When parsing configuration files (sourcing them), cvmfs exports CVMFS_FQRN that can be used by the bash logic in the configuration files. These config files have to be prepared for empty CVMFS_FQRN variable (no syntax error in this case).
   • When parsing configuration files (sourcing them), the current working directory of the shell should be the directory containing the configuration file. That allows to specify the public keys directory as a relative path.
3. Automatic ordering of Stratum 1s
   • Future stratum 1s provide a RESTful service under http://${STRATUM1-HOST}/cvmfs/$fqrn/api/v1.0/…
   • Clients can request

     GET http://${STRATUM1-HOST}/cvmfs/$fqrn/api/v1.0/geo/<proxy address as seen by cvmfs client>/<alphabetical list of stratum 1s separated by comma>
     

     The result is an ordered, comma-separated list of indexes of the input stratum 1s, calculated from GeoIP data. The address of the client is not taken from the proxy address in the URL, that is just for caching of related requests together. Rather, the address is taken from the remote IP address of the TCP socket connecting to the stratum 1.
   • Automatic ordering can be turned on and off through a client configuration parameter. The cvmfs default is off (which might change in the future), the cvmfs-config-cern default will be on.
   • The client should show the effective ordered list through an extended attribute
   • We should document to remove /etc/cvmfs/domain.d/cern.ch.local once this is working well
   • Life time of the response is at least 5 minutes, probably rather 3 days like all other files in cvmfs repositories
   • Extra care in validation of the input path is necessary for keeping the RESTful service secure

-- DaveDykstra - 16 Jul 2014