Difference: Meeting080128 (2 vs. 3)

Revision 32008-01-28 - PhilippeCharpentier

Line: 1 to 1

Bookkeeping meeting, January 28th 2008

Line: 25 to 25

Items for the new system

Three main threads were discussed concerning the new system

Setting up a test system

Elisa reported the efforts to set up the test system in a first instance with the current schema. The main initial problem is to understand how to create the initial tables. There are no instructions, but Elisa and Marianne found an SQL file that can correspond to this. Looking at the code it was not easy to understand why index partitioning is implemented with a predefined segmentation (hardcoded) that seems to correspond only to DC04. This cannot scale when new files/jobs are registered... but to start with we can try this out. It is then necessary to migrate a small fraction of the DB (test DB is smallish), e.g. starting from a simulation production and all its descendants.

Action (Marianne and Elisa): understand how to create a DB schema (first warehouse, then how to re-create views). Get a BK interface work on the test system.

Review of the DB schema

The current warehouse schema is a translation of the former schema, but the flexibility of the (key, value) pairs has been lost. The columns in the params tables are just an "OR" of all the key values that were found at the time of the migration. It makes it quite difficult to add new parameters if necessary. It is therefore proposed to include all columns that are needed for creating the views in the /jobs and /files tables rather than leaving them in a separate table that has as many entries as the mother table.

The provenance parameters should remain in the params table, and we propose to go back to a (key, value) scheme. This would allow an easy display of all parameters in the web page by a generic code.

Action (Elisa): prepare a proposal for the /jobs and /files tables, as well as for a list of key values necessary for converting the existing entries. This proposal should take into account requirements for real data (such as identifying a "job" to a "run" for RAW data).

As suggested by the BKWG, two view schema should be defined separately for real and MC data, and new tables added in order to define in particular "data taking periods" as well as "processing passes". The existing MC views schema should be reviewed in order to address the current problems (limited number of programs in history, multiple versions in the history).

A well specified API should be defined in order to implement BK access through a DIRAC service (see next point). This API must set the baseline for all higher level usage of the BK, either for administrating it (BK manager) or querying it. Elisa already defined part of the API for implementing her prototype python interface.

Action (Elisa): document the prototype API and classes, to be reviewed and augmented in order to accommodate the use cases (to be explicitly listed as well)

Architecture of the BK services

The BK service should be integrated in the DIRAC3 framework. The BK services should be handled in a DIRAC service that will interact with the database (through AMGA). The client interface will use this service as a proxy to the DB.

Action (Zoltan): get documentation on how to set up a DIRAC service and client from DIRAC developers. Implement a dummy server/client, and follow soon with porting Elisa's prototype as a service.

 -- PhilippeCharpentier - 28 Jan 2008
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback