Difference: Meeting071206 (1 vs. 2)

Revision 22007-12-10 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Bookkeeping meeting, December 6th 2007

Line: 6 to 6
  The aim of the meeting was to set the scope of the project, review the current situation and existing developments and define the topics of work. The agenda can be found here
Changed:
<
<

Current bookkeeping system

>
>

Current bookkeeping system (Marianne)

  Marianne presented the current status of the [[http://indico.cern.ch/getFile.py/access?contribId=0&resId=1&materialId=slides&confId=24813][system], which is also in slides from Carmine Cioffi.
Line: 18 to 18
 
  • BkkReceiver: receives XML files from jobs, already using the DISET transport. Currently the implementation for storing the queries is based on the file system. This should be ported to a RequestDB within DIRAC3
Changed:
<
<
  • BkkManager: it checks the consistency of the XML file, in particular if referring to jobs not yet entered in the BK, the XML file is put in hold (in a special directory). This request will be treated after all requests have been treated. The main role (besides checking for inconsistent or missing info) is to sort the entries in the order in which they should be inserted. In the DIRAC3 framework, this should become an agent
>
>
  • BkkManager: uses the NewConfirm servlet which checks the consistency of the XML file, in particular if referring to jobs not yet entered in the BK, the XML file is put in hold (in a special directory). This request will be treated after all requests have been treated. The main role (besides checking for inconsistent or missing info) is to sort the entries in the order in which they should be inserted. In the DIRAC3 framework, this should become an agent BkkManager inserts the data in the warehouse and is responsible for the views creation every night. Is this all in a single script?
 
Changed:
<
<
In construction...
>
>
  • Tomcat servlets: they are responsible for giving access to the browsing web page

  • FileCatalog: XMLRPC service that integrates BK and LFC for getting replica information (used by the BK browser web page and stand-alone client that creates XML catalogs or converts Gaudi options from LFN to PFN).

  • BkkMonitor: verifies the other services are alive, triggers alarms otherwise

Bookkeeping Working Group (Philippe)

Philippe presented the same slides he had prepared for the May '07 Software week that contains the main recommendations of the BKWG. The BKWG was formed during Spring '07 in order to review the current BK from a user's perspective and identify issues related to real data.

The current warehouse schema seems adequate for real data if a DAQ run is assimilated to a job, with several output files. MMinor adaptations might be necessary.

Several new features or concepts for querying the BK for real data need to be introduced (query by date, data taking period, processing pass). Periods and passes can be (re-)defined a posteriori, hence should be expressed in terms of the query criteria. The definitions could be additional tables or any other source of information (implementation detail).

One of the requests from users is to introduce flexibility in the queries compared to the current rigid web interface. One should be able to chain criteria in any order and get the choice of possible other criteria according to the already selected ones. There is a need to have a browser that can be integrated into ganga and also run in stand-alone mode for producing job options. The web browsing capabilities are not considered as a "must". The WG therefore recommends to implement in priority a stand-alone browser (in python and pyQT for the GUI). The web interface could be added if easy to maintain. The web interface should however be used for providing statistics (e.g. for management of productions).

  -- PhilippeCharpentier - 10 Dec 2007

Revision 12007-12-10 - PhilippeCharpentier

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="BKDev"

Bookkeeping meeting, December 6th 2007

Present: R.McNulty, T.Kechadi, Z.Mathe (Dublin), M.Bargiotti, Ph.Charpentier (chair, minutes), B.Koblitz, E.Lanciotti, A.Maier, S.Paterson, R.Santinelli, A.C.Smith (CERN), A.Tsaregorodtsev (Marseille, on the phone)

The aim of the meeting was to set the scope of the project, review the current situation and existing developments and define the topics of work. The agenda can be found here

Current bookkeeping system

Marianne presented the current status of the [[http://indico.cern.ch/getFile.py/access?contribId=0&resId=1&materialId=slides&confId=24813][system], which is also in slides from Carmine Cioffi.

There were discussions on the understanding of the actual content of the views (roottree and JobFileInfo tables). The understanding was that roottree contains for each file an entry with all parameters that queries can be made on. JobFileInfo is a table for each file containing the most usual metadata for that file.

AMGA is used as an access layer to the actual Oracle database. It implements authentication as well as an insulation from the actual backend. The actual BK schema is based on the generic AMGA schema.

Marianne reviewed the services that form the BK system:

  • BkkReceiver: receives XML files from jobs, already using the DISET transport. Currently the implementation for storing the queries is based on the file system. This should be ported to a RequestDB within DIRAC3

  • BkkManager: it checks the consistency of the XML file, in particular if referring to jobs not yet entered in the BK, the XML file is put in hold (in a special directory). This request will be treated after all requests have been treated. The main role (besides checking for inconsistent or missing info) is to sort the entries in the order in which they should be inserted. In the DIRAC3 framework, this should become an agent

In construction...

-- PhilippeCharpentier - 10 Dec 2007

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback