LFC DB replication tests for LHCb

Database setup

See LFC Streams Replication Tests

LFC

This page is meant to be a work log, nothing more.

15/05/2006 :

1) MV creation now works from @strmtest to @strmtestr1 :

[lxplus068] ~ > sqlplus lcg_lfc_mv_lhcb_ro@strmtestr1
...
SQL> create database link RLS1R1.CERN.CH connect to lcg_lfc_mv_lhcb identified by "1lfc4sc3" using 'rls1r1.cern.ch';

Database link created.

and :

[lxplus068] ~ > sqlplus lcg_lfc_mv_lhcb_ro@strmtestr1
...
SQL> CREATE MATERIALIZED VIEW Cns_file_metadata
BUILD IMMEDIATE
REFRESH START WITH sysdate NEXT sysdate+ 1/24/2
WITH PRIMARY KEY
AS
SELECT * FROM lcg_lfc_mv_lhcb.CNS_FILE_METADATA@rls1r1.cern.ch;

Materialized view created.

2) MV creation script done. Nilo's comments :

• materialized view logs missing -> REFRESH FAST uses materialized view logs then
• constraints not needed, but indexes on constraints yes...
• group the refreshes.

12/05/2006 :

1) Load tool finished. It uses the DIRAC framework to populate the LFC from xxx.dat files.

The script for automatic xxx.dat files creation is :

~slemaitr/public/forMarianne/createDatFiles

Its usage :

./createDatFile --number 40 --dir-number 2 --sub-dir-number 2 --file-number 10 --file /tmp/file.dat --verbose

The files are created are of the form /grid/lhcb/tests/replication_tests_xxx_M/test_dir_xxx/file_xxx.

10/05/2006 :

1) James reinstalled lxb0716, lxb0717 and lxb1540 with Quattor (to be as close as possible to the production setup). Debug LFC RPMs installed on them.

08/05/2006 :

1) LFC RPMs are not compiled in debug mode, that's why the core dump was useless...

2) New RPMs produced. Trying to reproduce the problem.

/afs/cern.ch/user/s/slemaitr/public/forJames/ > ls | sort
lcg-dm-common-1.5.7-2sec.i386.rpm
LFC-client-1.5.7-2sec.i386.rpm
LFC-interfaces-1.5.7-1.i386.rpm
LFC-server-oracle-1.5.7-2sec.i386.rpm

28/04/2006 :

1) lfcdaemon crash reproduced. Core dump enabled this time, but no useful information produced :

 
[root@lxb0716 lfc]# gdb lfcdaemon core.32362
GNU gdb Red Hat Linux (6.3.0.0-1.90rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `/opt/lcg/bin/lfcdaemon -t 20 -c /opt/lcg/etc/NSCONFIG -l /var/log/lfc/log'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /opt/globus/lib/libglobus_gssapi_gsi_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_gssapi_gsi_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libglobus_gss_assist_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_gss_assist_gcc32dbgpthr.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /usr/lib/oracle/10.1.0.3/client/lib/libclntsh.so.10.1...done.
Loaded symbols for /usr/lib/oracle/10.1.0.3/client/lib/libclntsh.so.10.1
Reading symbols from /lib/tls/libpthread.so.0...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /opt/globus/lib/libglobus_gsi_proxy_core_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_gsi_proxy_core_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libglobus_gsi_credential_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_gsi_credential_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libglobus_gsi_callback_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_gsi_callback_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libglobus_oldgaa_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_oldgaa_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libglobus_gsi_cert_utils_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_gsi_cert_utils_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libglobus_gsi_sysconfig_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_gsi_sysconfig_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libglobus_openssl_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_openssl_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libglobus_openssl_error_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_openssl_error_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libglobus_proxy_ssl_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_proxy_ssl_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libglobus_common_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libglobus_common_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libssl_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libssl_gcc32dbgpthr.so.0
Reading symbols from /opt/globus/lib/libcrypto_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/globus/lib/libcrypto_gcc32dbgpthr.so.0
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/oracle/10.1.0.3/client/lib/libnnz10.so...done.
Loaded symbols for /usr/lib/oracle/10.1.0.3/client/lib/libnnz10.so
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /usr/lib/oracle/10.1.0.3/client/lib/libociei.so...done.
Loaded symbols for /usr/lib/oracle/10.1.0.3/client/lib/libociei.so
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /opt/lcg/lib/libCsec_plugin_GSI_thread.so...done.
Loaded symbols for /opt/lcg/lib/libCsec_plugin_GSI_thread.so
Reading symbols from /opt/glite/lib/libvomsc_gcc32dbgpthr.so.0...done.
Loaded symbols for /opt/glite/lib/libvomsc_gcc32dbgpthr.so.0
Reading symbols from /usr/lib/libexpat.so.0...done.
Loaded symbols for /usr/lib/libexpat.so.0
Reading symbols from /usr/lib/libstdc++.so.5...done.
Loaded symbols for /usr/lib/libstdc++.so.5
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
#0  0x00000000 in ?? ()
(gdb) frame 0
#0  0x00000000 in ?? ()

2) Materialized Views creation script ON GOING

A few remarks :

• The primary key is not transfered to the MV. Need to use WITH PRIMARY KEY during MV creation.
• If no primary key in the original table, WITH ROWID has to be specified.
• Foreign keys are not propagated, they should be re-created in the MV (or only the index is creates, as the MV is READ-ONLY. See Nilo's comments).
• Indexes not propagated either. We should re-create them in the MV.
• REFRESH options : FAST (i.e. from MV log), COMPLETE (i.e. recreate all MV), FORCE (i.e. FAST if possible, otherwise COMPLETE)

3) LFC RPMs built against Oracle 10.2.0.1 and installed on lxb0716.cern.ch (temporary 1.5.6 test version, not released yet).

26/04/2006 :

Meeting with Andrei T., Marianne B., Eva D., Dirk D., Barbara M., Jean-Philippe B., James C., Gian Luca, Sophie L.

Actions :

• James : put LFC test servers at CERN in Quattor cluster + enable read-only mode in Quattor. DONE
• Sophie : reproduce crash with core dump. TO BE DONE
• Sophie : build LFC RPMs with Oracle 10.2.0.1 NOT NEEDED: USE CORRESPONDING VERSION OF Instant Client INSTEAD
• Sophie/Marianne/Andrei : create load tool using DIRAC data management utilities.DONE
• Sophie : test with and without last access time update on read-write (Streams + MV).
• Sophie : test using accounts with different number of entries (1 million, 10 millions, 100 millions). -> see latency compared to number of updates with Streams.
• Eva : create the 6 Oracle accounts at CERN DONE + enable Streams. DONE
• Barbara : create the 6 Oracle accounts at CNAF DONE + enable Streams. DONE
• Eva/Barbara : create database link between CERN and CNAF for materialized views creation
• Sophie : script to create materialized view DONE -> send it to Barbara
• Eva/Miguel : take a snapshot of the existing lcg_lfc_lhcb@lcg_lfc data and put it in lcg_lfc_3d_lhcb@strmtest and lcg_lfc_mv_lhcb@strmtest
• Everybody : keep a work log on the Wiki page. DONE
• Sophie : create a tool that checks that the last file (and all the files) inserted in the LFC is present in the read-only.
• Sophie : inform Eva/Barbara every time a test is run.

Before 26/04/2006 :

LFC installation with version 1.5.6.

• lxb0717.cern.ch (lcg_lfc_3d_lhcb@strmtest) : Streams READ-WRITE source LFC
• lxb0716.cern.ch (lcg_lfc_mv_lhcb@strmtest) : MV READ-WRITE source LFC
• lxb1540.cern.ch (lcg_lfc_3d_lhcb_ro@strmtestr1 or lcg_lfc_mv_lhcb_ro@strmtestr1) : Streams/MV target LFC

Very simple tests with Streams and MV showed lfcdaemon crashes. Not clear when and why it happens exactly. It seems to happen when a client was using the LFC at the same time as a replication is triggered.

Dirk's suggestion : it might be due to 10gR1 and 10gR2 interaction -> try to build the RPMs against 10.2.0.1 and see if it happens again. Let's see.

-- SophieLemaitre - 16 May 2006