Berkeley Database Information Index V4

Note: For information on BDIIv5 please see this documentation.

Overview

The Berkeley Database Information Index (BDIIv4) consists of two or more standard LDAP databases that are populated by an update process. Port forwarding is used to enable one database to serve data while the other is refreshing. The update process does the following in an endless loop:

  • If enabled, update configuration file that contains the LDAP URLs from the web.
  • Fork off processes to obtain the LDIF from the URLs.
  • Write the LDIF files in the temporary directory.
  • If enabled, try to obtain an LDIF file that will modify (filter) the data.
  • Modify the data read from the temporary files.
  • Kill the oldest database instance along with any stale clients.
  • Clear the corresponding database.
  • Add the fresh data to a new database.
  • Start the new database.
  • Forward traffic from the BDIIv4 read port to the new database.
  • Sleep for a while.
The update process obtains LDIF from either doing an ldapsearch on LDAP URLs or by running a local script that generates LDIF. The LDIF is then inserted into the LDAP database. Options exist to update the list of LDAP URLs from a web page and to use an LDIF file from a web page to modify the data before inserting it into the database. By default the BDIIv4 is configured to run the Generic Information Provider (GIP).

bdii.png

Configuration

The BDIIv4 requires four configuration files:

  • /opt/bdii/etc/bdii.conf
  • /opt/bdii/etc/bdii-update.conf
  • /opt/bdii/etc/glue-slapd.conf
  • /opt/bdii/etc/default.ldif

bdii.conf

This configuration file is used to configure the BDIIv4 itself. The format is key/value pairs with an '=' sign as the separator. A default configuration file comes with the BDIIv4. This may require editing in order for the BDIIv4 to function as desired. The table belows describes the key/value pairs found in the configuration file.

Key Typical Value Description

BDII_DIR /opt/bdii/ The location and working directory of the bdii
BDII_VAR_DIR /opt/bdii/var The var directory to use for the bdii
BDII_SLAPD_CONF /opt/bdii/etc/glue-slapd.conf The slapd.conf template for use with the bdii
BDII_DEFAULT_LDIF /opt/bdii/etc/default.ldif A default LDIF file to populate the bdii
BDII_PORT_READ 2170 The port which is used for incoming queries
BDII_PORTS_WRITE "2171 2172 2173" The slapd database ports. Each port specified will create a database
BDII_USER edguser The user runing the update process and the slapd databases
BDII_BIND mds-vo-name=local,o=grid The bind point of the database
BDII_SEARCH_FILTER '*' A search filter for use with the ldapsearches
BDII_SEARCH_TIMEOUT 30 Time to wait for LDAP urls to respond
BDII_BREATHE_TIME 60 Time to wait before updating the next database
BDII_AUTO_UPDATE no Whether or not to update the file containing the LDAP URLs
BDII_AUTO_MODIFY no Whether or not to modify the data using an LDIF modify file
BDII_MODIFY_DN no Whether or not to modify the DN of entries so that they all match the BDII_BIND value
BDII_IS_CACHE no Whether or not to reject entries with already match the BDII_BIND value
BDII_UPDATE_URL http://lcg-bdii-conf.cern.ch/bdii-conf/bdii.conf The URL to use to automatically update the LDAP URLs file
BDII_UPDATE_LDIF http://lcg-fcr.cern.ch:8083/fcr-data/exclude.ldif The URL to use to update the LDIF modify file
SLAPD /opt/openldap/libexec/slapd The location of the slapd command
SLAPD_DEBUG_LEVEL 4 The logging level for the slapd
SLAPADD /opt/openldap/sbin/slapadd The location of the slapadd command

bdii-update.conf

This configuration file contains a list of URLS which the BDIIv4 uses to populate itself. The format of the file is :

region-name ldap://host.invalid:2170/mds-vo-name=local,o=grid
local-name file:///somewhere/myscript

The URLs that returns LDIF should be specified along with a name for that information. The name is use as an alias for the URL is useful for troubleshooting. By default this file contains name and URL for the GIP.

Starting and stopping the BDIIv4

The BDIIv4 is started and stopped by a daemon script /etc/rc.d/init.d/bdii

File locations and Descriptions

The following table contains a list of files and locations which may be useful during troubleshooting.

Location Description
/opt/bdii/etc/bdii.conf BDIIv4 configuration file
/opt/bdii/etc/bdii-update.conf List of URLs for the BDIIv4 gather information
/opt/bdii/etc/glue-slapd.conf The slapd.conf template for use with the bdii
/opt/bdii/etc/default.ldif A default LDIF file to populate the bdii
/opt/bdii/sbin/bdii BDIIv4 init.d script
/opt/bdii/sbin/bdii-fwd Port forwarder
/opt/bdii/sbin/bdii-update Main update script
/opt/bdii/sbin/bdii-proxy Creates proxy for the BDIIv4
/opt/bdii/var/bdii.log The BDIIv4 log file
/opt/bdii/var/bdii-update.pid The bdii-update.pid of the main process
/opt/bdii/var/bdii-fwd.log The log file of the bdii-fwd process
/opt/bdii/var/bdii-fwd.conf The port state file used by the bdii-fwd process
/opt/bdii/var/2171 Directory containing slapd data/config for the instance using port 2171
/opt/bdii/var/2171/bdii-slapd.conf The slapd.conf file
/opt/bdii/var/2171/slapd.args The command used to start the slapd
/opt/bdii/var/2171/slapd.pid The c slapd process
/opt/bdii/var/2171/*.dbb The slapd database files
/opt/bdii/var/tmp/stderr.log The stderr log of the slapad and slap add commands
/opt/bdii/var/tmp/*.ldif The resulting response from the LDAP URL
/opt/bdii/var/cache A cache of the resulting response used to fault tolerance

Running Processes

When a BDIIv4 is started, the following processes run:

  • At least one slapd process for each port specified and more depending on the query load.
  • 1 bdii-update process.
  • 1 master bdii-fwd process and 2 (usually short-lived) processes per active client.
  • Periodically, between 1 and 10 ldapsearch processes.
  • Periodically, one slapadd process.

Monitoring the BDIIv4 Instance (BSIM)

The BDIIv4 Status Information Mechanism is intended to allow better monitoring possibilities, spotting of upraising problems and resulting failure prevention. It adds status information about the BDIIv4 instance into the o=infosys root containing metrics like the number of entries added in the last cyle, the time to do so, etc. These Metrics can be queried and used for monitoring systems ( e.g. Nagios).
Quering these metrics is fairly simple:

[root@lxb7508 ]# ldapsearch -LLL -x -t -h grid-bdii-intern.desy.de:2170 -b Hostname=ldap://grid-bdii1.desy.de:2170,o=infosys 
dn: Hostname=ldap://grid-bdii1.desy.de:2170,o=infosys
objectClass: InfoSystemStats
Hostname: ldap://grid-bdii1.desy.de:2170
Entries: 48129
SearchTime: 93
Sources: 1
SourcesTimeout: 0
UpdateTime: 29
SourcesFailed: 0

The following table shows the meaning of these metrics:

Metric Description
Entries All LDAP entries under o=grid.
SearchTime The time [sec] to pull in the information from the GIP / ldapsearches
Sources The number of sources the BDII queries ( in case of a GIP it is always 1 )
SourcesTimeout The number of sources which timed out
UpdateTime The time [sec] the current DB took to be filled up
SourcesFailed The number of sources which failed when querying

Nagios Plugins

The latest nagios plugins can be found in ETICS.

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng bdii.png r1 manage 7.7 K 2008-01-21 - 19:39 LaurenceField  
Edit | Attach | Watch | Print version | History: r22 < r21 < r20 < r19 < r18 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r22 - 2013-09-24 - LaurenceField
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EGEE All webs login

This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright & by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback