Difference: BKDev (1 vs. 24)

Revision 242011-02-14 - ZoltanMathe1

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 58 to 58
 

Miigration plan of Bookkeeping(2011)

Changed:
<
<
We are using pass indexes to describe a processing chain. The processing chain can have different steps. The processing pass is a group of the pass indexes which are compatible. The Production System are used to define a activity(the processing chain). The data processing can be simulation, reconstruction, reprocessing etc. The users can use the production system the define their requests for data processing. The current processing pass is created and published to the bkk when the production has been created. The processing pass created by the production manager or data quality person. This persons to create a production have to contact with the application persons to know the option files and different attributes. They can make mistake for example: typo, bad option file, forget the file list one stream file. etc. We decided to modify the existing processing passes. We are using steps. Each steps are defined by the application responsible person. They have to provide the option files and the input and output of each steps. The production request people can put together the processing steps.This make simple the production definition. This modification is affected the following systems: Bookkeeping, Production systems and work flow. The main modification has been done on the Bookkeeping system. We are considering the migration plan of bkk system, but also we try to summarize what we need to be done in the other system. You can found the migration plan in the following link Check list.
>
>
We are using pass indexes to describe a processing chain. The processing chain can have different steps. The processing pass is a group of the pass indexes which are compatible. The Production System are used to define a activity(processing chain). The data processing can be simulation, reconstruction, reprocessing etc. The users can use the production system the define their requests for data processing. The current processing pass is created and published to the bkk when the production has been created. The processing pass created by the production manager or data quality person. This persons who create a production have to contact with the application persons to know the option files and different attributes. They can make mistake for example: typo, bad option file, forget the file list one stream file. etc. We decided to modify the existing processing passes. We are using steps. Each steps are defined by the application responsible person. They have to provide the option files and the input and output of each steps. The production request people can put together the processing steps.This make simple the production definition. This modification is affected the following systems: Bookkeeping, Production systems and work flow. The main modification has been done on the Bookkeeping system. We are considering the migration plan of bkk system, but also we try to summarize what we need to be done in the other system. You can found the migration plan in the following link Check list.
 

-- PhilippeCharpentier - 10 Dec 2007

Revision 232011-02-14 - ZoltanMathe1

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 56 to 56
  The new Bookkeeping system is foreseen to be put in production by the beginning of November 2008: updated Check list of all the last pending issues.
Added:
>
>

Miigration plan of Bookkeeping(2011)

We are using pass indexes to describe a processing chain. The processing chain can have different steps. The processing pass is a group of the pass indexes which are compatible. The Production System are used to define a activity(the processing chain). The data processing can be simulation, reconstruction, reprocessing etc. The users can use the production system the define their requests for data processing. The current processing pass is created and published to the bkk when the production has been created. The processing pass created by the production manager or data quality person. This persons to create a production have to contact with the application persons to know the option files and different attributes. They can make mistake for example: typo, bad option file, forget the file list one stream file. etc. We decided to modify the existing processing passes. We are using steps. Each steps are defined by the application responsible person. They have to provide the option files and the input and output of each steps. The production request people can put together the processing steps.This make simple the production definition. This modification is affected the following systems: Bookkeeping, Production systems and work flow. The main modification has been done on the Bookkeeping system. We are considering the migration plan of bkk system, but also we try to summarize what we need to be done in the other system. You can found the migration plan in the following link Check list.

  -- PhilippeCharpentier - 10 Dec 2007

Revision 222009-11-05 - ZoltanMathe1

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 37 to 37
  This section describe Bkk Service development and production setups.
Added:
>
>

Bookkeeping release procedure

This section describe a release procedure of the BKK system.
 

Meetings

Revision 212008-10-17 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 52 to 52
  Go to this page to check the current status of the activity on the BK.
Added:
>
>
The new Bookkeeping system is foreseen to be put in production by the beginning of November 2008: updated Check list of all the last pending issues.
  -- PhilippeCharpentier - 10 Dec 2007 \ No newline at end of file

Revision 202008-10-15 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 15 to 15
 

The Processing Pass

Changed:
<
<
This is a new concept intruduced in the BK. The idea is to group datasets on the basis of the application version and conditions DB tag used to produce them and to produce their ancestors. More information on the Processing Pass.
>
>
This is a new concept intruduced in the BK. The idea is to group datasets on the basis of the application version and conditions DB tag used to produce them and to produce their ancestors.
  How the Processing Pass has been computed for already existing data (DC06 data): offline computation of the Processing Pass.
Added:
>
>
How it is computed for newly inserted data: online computation of the Processing Pass.
 

The Simulation Conditions and Data Taking Conditions tables

The simulation conditions and data taking conditions are new tables for simulated and real data respectively. Data with same simulation or data taking conditions can be merged at processing stage. More details about the simulation conditions and data taking conditions tables.

Revision 192008-10-14 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 17 to 17
  This is a new concept intruduced in the BK. The idea is to group datasets on the basis of the application version and conditions DB tag used to produce them and to produce their ancestors. More information on the Processing Pass.
Added:
>
>
How the Processing Pass has been computed for already existing data (DC06 data): offline computation of the Processing Pass.
 

The Simulation Conditions and Data Taking Conditions tables

The simulation conditions and data taking conditions are new tables for simulated and real data respectively. Data with same simulation or data taking conditions can be merged at processing stage. More details about the simulation conditions and data taking conditions tables.

Revision 182008-09-08 - ZoltanMathe

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 29 to 29
  This section is the Bookkeeping Client documentation.
Added:
>
>

Bookkeeping development and production setups

This section describe Bkk Service development and production setups.

 

Meetings

Revision 172008-07-28 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 38 to 38
 
Changed:
<
<
  • 23 June '08 minutes and general planning for next activities. Go to this page to check the current status of the activity on the BK
>
>
  • 23 June '08 minutes and general planning for next activities.

Current status of the activity on the BK

Go to this page to check the current status of the activity on the BK.

  -- PhilippeCharpentier - 10 Dec 2007 \ No newline at end of file

Revision 162008-07-28 - ZoltanMathe

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 24 to 24
 

Migration of the database to the new schema

After implementing the new database schema in an integration oracle database, we made a migration of the DC06 data. More details on the procedure of the migration and the tool used for it are available in this link: Migration.

Added:
>
>

Bookkeeping Client Documentation

This section is the Bookkeeping Client documentation.

 

Meetings

Revision 152008-07-16 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 17 to 17
  This is a new concept intruduced in the BK. The idea is to group datasets on the basis of the application version and conditions DB tag used to produce them and to produce their ancestors. More information on the Processing Pass.
Added:
>
>

The Simulation Conditions and Data Taking Conditions tables

The simulation conditions and data taking conditions are new tables for simulated and real data respectively. Data with same simulation or data taking conditions can be merged at processing stage. More details about the simulation conditions and data taking conditions tables.

 

Migration of the database to the new schema

After implementing the new database schema in an integration oracle database, we made a migration of the DC06 data. More details on the procedure of the migration and the tool used for it are available in this link: Migration.

Revision 142008-07-02 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 13 to 13
 A new schema for the BK database has been designed, in order to enable the BK to host real data. For more information about the new schema click here.
Added:
>
>

The Processing Pass

 
Added:
>
>
This is a new concept intruduced in the BK. The idea is to group datasets on the basis of the application version and conditions DB tag used to produce them and to produce their ancestors. More information on the Processing Pass.
 

Migration of the database to the new schema

Revision 132008-07-01 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 27 to 27
 
Changed:
<
<
>
>
  • 23 June '08 minutes and general planning for next activities. Go to this page to check the current status of the activity on the BK
  -- PhilippeCharpentier - 10 Dec 2007 \ No newline at end of file

Revision 122008-06-23 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 27 to 27
 
Added:
>
>
  -- PhilippeCharpentier - 10 Dec 2007

Revision 112008-06-20 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 26 to 26
 
Changed:
<
<
>
>
  -- PhilippeCharpentier - 10 Dec 2007 \ No newline at end of file

Revision 102008-06-20 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 26 to 26
 
Added:
>
>
  -- PhilippeCharpentier - 10 Dec 2007 \ No newline at end of file

Revision 92008-06-10 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 25 to 25
 
Added:
>
>
  -- PhilippeCharpentier - 10 Dec 2007 \ No newline at end of file

Revision 82008-06-02 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 24 to 24
 
Added:
>
>
  -- PhilippeCharpentier - 10 Dec 2007

Revision 72008-04-17 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 11 to 11
 

The new schema of the BK database

A new schema for the BK database has been designed, in order to enable the BK to host real data.

Changed:
<
<
The same tables will be used to store metadata relative to real data and simulated data. The main differences with respect to the old schema are the following:
>
>
For more information about the new schema click here.
 
Deleted:
<
<
  • Many obsolete or duplicated attributes have been dropped
  • Some new attributes have been added, relevant for real data
  • The 2 tables relative to files (files and fileParams tables) and the 2 tables relative to jobs ( jobs and jobParams tables) have been merged.
  • Two new tables have been added: DataTakingPeriod and ProcessingPass

A draft of the documentation about this new schema is available in this link:

 

Migration of the database to the new schema

Revision 62008-04-16 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 8 to 8
  As defined in the BKWG, for online data, the "job" entity from which the file originates will be identified to the DAQ "run". All files of that run will be "output files" of that "job/run". There are however some limitations in the current model that need to be addressed urgently in order to be usable for online data. These limitations are described in this twiki page.
Added:
>
>

The new schema of the BK database

A new schema for the BK database has been designed, in order to enable the BK to host real data. The same tables will be used to store metadata relative to real data and simulated data. The main differences with respect to the old schema are the following:

  • Many obsolete or duplicated attributes have been dropped
  • Some new attributes have been added, relevant for real data
  • The 2 tables relative to files (files and fileParams tables) and the 2 tables relative to jobs ( jobs and jobParams tables) have been merged.
  • Two new tables have been added: DataTakingPeriod and ProcessingPass

A draft of the documentation about this new schema is available in this link:

Migration of the database to the new schema

After implementing the new database schema in an integration oracle database, we made a migration of the DC06 data. More details on the procedure of the migration and the tool used for it are available in this link: Migration.

 

Meetings

Revision 52008-03-06 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 13 to 13
 
Added:
>
>
  -- PhilippeCharpentier - 10 Dec 2007

Revision 42008-02-19 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 12 to 12
 
Added:
>
>
  -- PhilippeCharpentier - 10 Dec 2007 \ No newline at end of file

Revision 32008-01-28 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

Line: 11 to 11
 

Meetings

Added:
>
>
  -- PhilippeCharpentier - 10 Dec 2007 \ No newline at end of file

Revision 22007-12-14 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

This page is intended for sharing information amongst developer of the Bookkeeping system. These developments are meant at integrating the Bookkeeping system into the DIRAC framework (as part of DIRAC3) as well as implementing the recommendations of the Bookkeeping Working Group.

Added:
>
>

Schema for Online data

As defined in the BKWG, for online data, the "job" entity from which the file originates will be identified to the DAQ "run". All files of that run will be "output files" of that "job/run". There are however some limitations in the current model that need to be addressed urgently in order to be usable for online data. These limitations are described in this twiki page.

 

Meetings

Revision 12007-12-10 - PhilippeCharpentier

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="LHCbComputing"

Bookkeeping developments

This page is intended for sharing information amongst developer of the Bookkeeping system. These developments are meant at integrating the Bookkeeping system into the DIRAC framework (as part of DIRAC3) as well as implementing the recommendations of the Bookkeeping Working Group.

Meetings

-- PhilippeCharpentier - 10 Dec 2007

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback