Read-only LFC replica check list

This page is aimed to help you setting up a read-only LFC replica server for LHCb.

Check list

Oracle DB

Oracle needs to be available at your site.

Enable Streams

You need to setup Oracle Streams at your site to receive the data coming from CERN.

Check the documents provided by Eva Da Fonte Perez.

LFC machine

Dedicate a machine to the LFC service. See What kind of machine for the minimal requirements.

The LFC server is a read-only replica of the CERN primary LFC server, used by LHCb for failover. Thus, there is no need for redundancy (i.e. several LFC servers with DNS load-balancing).

LFC installation

Pre-requisites

Oracle Client

The LFC installation via YAIM will not work unless oracle-instantclient-basic and oracle-instantclient-sqlplus are installed on the LFC machine.

     $ rpm -qa | grep oracle

NOTE: you don't need oracle-instantclient-jdbc or oracle-instantclient-devel and can safely ignore the corresponding warnings in YAIM.

Oracle version

If your Oracle version is different from 10.2.0.1, you need to change it in /etc/sysconfig/lfcdaemon (after creating it):

     $ cp /etc/sysconfig/lfcdaemon.templ /etc/sysconfig/lfcdaemon

     $ grep oracle /etc/sysconfig/lfcdaemon
     ...
     # - Oracle Home :
     export ORACLE_HOME=/usr/lib/oracle/10.1.0.4/client

tnsnames.ora

A tnsnames.ora file containing the LFC database description should be present under /etc or /home/lfcmgr/.tnsadmin. It can be any other directory provided it is defined in /etc/sysconfig/lfcdaemon uncommented:

     $ more /etc/sysconfig/lfcdaemon
     ...
     # - Directory where tnsnames.ora resides :
     export TNS_ADMIN=/another/directory

ldconfig

The Oracle library path appears in /etc/ld.so.conf and /sbin/ldconfig has been run.

     $ grep oracle /etc/ld.so.conf
     /usr/lib/oracle/10.2.0.1/client/lib

     $ /sbin/ldconfig

read-only LFC

Your LFC should be started as read-only, thus you need to change the /etc/sysconfig/lfcdameon file:

     $ grep READONLY /etc/sysconfig/lfcdaemon
     ...
     RUN_READONLY="yes"

disable virtual ids automatic creation

Your LFC should be started so that the automatic creation of virtual ids is disabled:

In LFC version 1.6.7-1 and 1.6.7-2, this is missing from the /etc/sysconfig/lfcdaemon and /etc/init.d/lfcdaemon files.

• So, replace your /etc/init.d/lfcdaemon file with this one
• And check this sysconfig file, and merge it with your own sysconfig file. Disable the virtual ids automatic creation:

        RUN_DISABLEAUTOVIDS="yes"
     

After starting the LFC, make sure that the -n option appears:

   $ service lfcdaemon start
   $ ps aux | grep lfcdaemon
   lfcmgr   29457  0.0  1.6 235372 4440 ? Ssl  14:51   0:01 /opt/lcg/bin/lfcdaemon -n -t 20 -c /opt/lcg/etc/NSCONFIG -l /var/log/lfc/log

Installation via YAIM

Check the YAIM guide

LFC Admin Guide

To become familiar with LFC service, Tier-1 administrators are encouraged to read the LFC admin guide

These slides might also be helpful

The important part is to start the LFC server as read-only

For help with the LFC server installation, please contact the LFC support team

Open ports

Ports 5010, 8085 and 2170 need to be open to the outside world. See this section of the LFC admin guide.

Test the LFC

• Test that the LHCb data has actually been replicated to your site and is visible in the LFC:

$ lfc-ls /grid/lhcb
SAM
SFT
blouw
cnaf-stress
data
...

• Test that the server has been started in read-only mode:

$ lfc-mkdir /grid/lhcb/hello
cannot create /grid/lhcb/hello: Read-only file system

• Contact Roberto Santinelli to test your freshly installed LFC server. He will be able to test your installation as a member of the LHCb VO.

Troubleshooting

Could not establish context

The user sees:

$ lfc-ls /grid/lhcb
/grid/lhcb: Could not establish context

And the /var/log/lfc/log LFC server log shows:

09/14 09:03:54 23479,0 Cns_serv: Could not establish security context:
_Csec_get_voms_creds: Cannot find certificate of AC issuer for vo lhcb !

The solution is to install lcg-vomscert-4.5.0-1, which creates the /etc/grid-security/vomsdir directory and its content.

Could not get virtual id

The user sees:

$ lfc-ls /
Could not get virtual id: Internal error !
/: No user mapping

And the /var/log/lfc/log LFC server log shows:

09/14 13:45:48 25609,0 Cns_get_usrinfo_by_name: OPEN CURSOR error:
ORA-00980: synonym translation is no longer valid

The problem is that the Cns_groupinfo and Cns_userinfo tables are not replicated to Tier1s. This should be fixed. They cannot be created locally, because the ACLs won't be the same between the different LFC replicas, and there won't be consistency . For more details, ask Roberto...

-- SophieLemaitre - 10 Dec 2007