Difference: Meeting080623 (1 vs. 36)

Revision 362011-02-15 - ZoltanMathe1

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Revision 352008-09-30 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 280 to 280
  -- ElisaLanciotti - 23 Jun 2008
Changed:
<
<
META FILEATTACHMENT attachment="badProductions.txt" attr="" comment="Comprehensive list of bad productions still in the DB" date="1219074159" name="badProductions.txt" path="badProductions.txt" size="550" stream="badProductions.txt" user="Main.ElisaLanciotti" version="2"
>
>
META FILEATTACHMENT attachment="badProductions.txt" attr="" comment="Comprehensive list of bad productions still in the DB" date="1222787411" name="badProductions.txt" path="badProductions.txt" size="1018" stream="badProductions.txt" user="Main.ElisaLanciotti" version="3"

Revision 342008-09-29 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 272 to 272
  3 - The simulation conditions ID will be added as a new column in the PROCESSING_PASS_ table.
Changed:
<
<
TO be Done
>
>
Done 29 Sept 08
 

More details are given in the processing pass page.

Revision 332008-09-26 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 267 to 267
 
DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31 1<3<5
DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31 + DC06-Stripping_v30 1<3<5<7
Added:
>
>
Done 25 Sept 08

3 - The simulation conditions ID will be added as a new column in the PROCESSING_PASS_ table.

TO be Done

  More details are given in the processing pass page.
Deleted:
<
<
Done 25 Sept 08
  -- ElisaLanciotti - 23 Jun 2008

Revision 322008-09-25 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 249 to 249
 update processing_pass set totalprocpass='DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31' where totalprocpass='DC06-Reco_v30 + DC06-Stripping_v31';
Changed:
<
<
Done (Elisa) 24 Sept 08
>
>
Done 24 Sept 08
  2 - The totalprocpass attribute in the PROCESSING_PASS table will be expressed on the basis of the numeric group ID instead of the group descriptions strings. For example the entry: 'DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31 + DC06-Stripping_v30' will be replaced by: '1<3<5<7'.
Added:
>
>
This table shows the total processing pass old and new format:

Old Format New Format
DC06-Sim 1
DC06-Sim-Reco_v30 2
DC06-Sim-Reco_v31 9
DC06-Sim + DC06-Reco_v30 1<3
DC06-Sim + DC06-Reco_v32 1<6
DC06-Sim + DC06-Recon-L0-v1-lumi2 1<8
DC06-Sim-Reco_v30 + DC06-Stripping_v31 2<5
DC06-Sim-Reco_v31 + DC06-Stripping_v31 9<5
DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31 1<3<5
DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31 + DC06-Stripping_v30 1<3<5<7

More details are given in the processing pass page.

Done 25 Sept 08

 -- ElisaLanciotti - 23 Jun 2008

META FILEATTACHMENT attachment="badProductions.txt" attr="" comment="Comprehensive list of bad productions still in the DB" date="1219074159" name="badProductions.txt" path="badProductions.txt" size="550" stream="badProductions.txt" user="Main.ElisaLanciotti" version="2"

Revision 312008-09-25 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 240 to 240
 

Moreover, after the meeting of Sept 22 some modification has been applied to the PROCESSING_PASS table:

Changed:
<
<
1- the total processing pass includes all the steps, from the simulation. So we have to apply the following changes: for the processing pass DC06-Stripping_v31 + DC06-Stripping_v30 (this affects only productions 200 and 201)

>
>
*1*- the total processing pass includes all the steps, from the simulation. So we have to apply the following changes: for the processing pass DC06-Stripping_v31 + DC06-Stripping_v30 (this affects only productions 200 and 201)

 update processing_pass
Changed:
<
<
set totalprocpass='DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31 + DC06-Stripping_v30' where totalprocpass='DC06-Stripping_v31 + DC06-Stripping_v30';
>
>
set totalprocpass='DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31 + DC06-Stripping_v30' where totalprocpass='DC06-Stripping_v31 + DC06-Stripping_v30';
 

and the proc pass DC06-Reco_v30 + DC06-Stripping_v31 (it affects only prod 2000):


Changed:
<
<
update processing_pass set totalprocpass='DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31' where totalprocpass='DC06-Reco_v30 + DC06-Stripping_v31';
>
>
update processing_pass set totalprocpass='DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31' where totalprocpass='DC06-Reco_v30 + DC06-Stripping_v31';
 

Done (Elisa) 24 Sept 08

Added:
>
>
2 - The totalprocpass attribute in the PROCESSING_PASS table will be expressed on the basis of the numeric group ID instead of the group descriptions strings. For example the entry: 'DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31 + DC06-Stripping_v30' will be replaced by: '1<3<5<7'.
 -- ElisaLanciotti - 23 Jun 2008

META FILEATTACHMENT attachment="badProductions.txt" attr="" comment="Comprehensive list of bad productions still in the DB" date="1219074159" name="badProductions.txt" path="badProductions.txt" size="550" stream="badProductions.txt" user="Main.ElisaLanciotti" version="2"

Revision 302008-09-24 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 235 to 235
  Computation of Processing Pass
Changed:
<
<
It has been decided to restructure the processing_pass table and pass_index table (Mon Sept 1 2008).
>
>
It has been decided to restructure the processing_pass table and pass_index table and to add a PASS_GROUP table (Mon Sept 1 2008).
 
Changed:
<
<
To be done (Elisa)
>
>

Moreover, after the meeting of Sept 22 some modification has been applied to the PROCESSING_PASS table:
1- the total processing pass includes all the steps, from the simulation. So we have to apply the following changes: for the processing pass DC06-Stripping_v31 + DC06-Stripping_v30 (this affects only productions 200 and 201)

update processing_pass
set totalprocpass='DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31 + DC06-Stripping_v30' 
where totalprocpass='DC06-Stripping_v31 + DC06-Stripping_v30';

and the proc pass DC06-Reco_v30 + DC06-Stripping_v31 (it affects only prod 2000):

update processing_pass set totalprocpass='DC06-Sim + DC06-Reco_v30 + DC06-Stripping_v31'
where totalprocpass='DC06-Reco_v30 + DC06-Stripping_v31';

Done (Elisa) 24 Sept 08

  -- ElisaLanciotti - 23 Jun 2008

Revision 292008-09-24 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 83 to 83
 update jobs set Production=200 where (jobs.production=2000 and jobs.programVersion='v30r17' and jobs.programname='Brunel');
This has been fixed on 1 July 08
Added:
>
>
Later it was done also for the AMGA tables in the new schema with the statement:
update dir3 set "user:Production"=201 
where (dir3."user:Production"=2001 and dir3."user:ProgramVersion"='v30r17' and dir3."user:ProgramName"='Brunel');
update dir3 set "user:Production"=200 
where (dir3."user:Production"=2000 and dir3."user:ProgramVersion"='v30r17' and dir3."user:ProgramName"='Brunel');
This has been done on 24 Sept 08
  Populating the Simulation Conditions table for the DC06 data

Revision 282008-09-01 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 228 to 228
  Computation of Processing Pass
Changed:
<
<
The production 1536 has config. version = phys-lumi2 and according to the instructions of Philippe it should be in the group DC06-Sim-Reco_v30 but it has only RDST files, so in my opinion it should be in the group _DC06-Reco_v30_
Wed Aug 13 2008: Philippe confirmed there was a problem with this prod and it can be deleted.
dropped this entry in the prod2pass table and in the passindex table and added this prod to the bad prod list
>
>
It has been decided to restructure the processing_pass table and pass_index table (Mon Sept 1 2008).
 
Changed:
<
<
Prods 37 and 38:

Prod 2146:

>
>
To be done (Elisa)
  -- ElisaLanciotti - 23 Jun 2008

Revision 272008-08-29 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 33 to 33
 Finished at 17:50 Aug 20. A lot of errors due to the problem of multiple input files. Done
4) correct again the production number for prods 201 and 200. This should be done on the Oracle tables, after that Zoltan has copied again them from the AMGA tables.
5) recover again the files and jobs with null prod id that have been deleted. Prods:00001354, 00001501, 00001503. Done
Changed:
<
<
This has to be done. HIGH PRIORITY
>
>
Done by 28 Aug 08
  Removal of bad productions
Line: 54 to 54
  More in general, there a list (see badProductions.txt file in the attachment) that contains all the productions which have to be ignored (for different reasons). I keep this list updated every time that it comes out that a production is bad.
Changed:
<
<
to be done!
>
>
Done. 25 Aug: Provided to Zoltan the list of bad prods
  Post synchronization checks
Line: 115 to 115
  Files with no entry in jobs table

During the migration of the files with null production number, some files have been found with no entry in the jobs table. The files are 29 in total, spread over 5 productions ( 00001323, 00001324, 00001325, 00001326, 00001327). It has been decided to ignore these files, since we don't have the necessary information about them to register them in the new schema, and also because they are only few files.

Changed:
<
<
This is not an issue, just a warning
>
>
This is not an issue, just a warning so I put it in green smile
  Productions with ONLY SIM files

Revision 262008-08-25 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 27 to 27
  Mon Aug 18 2008: the procedure I'll follow will be:
1) identify the new data inserted after Feb 1 2008 (they are about 273000 jobs spread over 103 productions)
Changed:
<
<
No! the list of files has to be retrieved on the basis of the JOBID, not the date, because some old jobs (of 2006) have just been inserted in the last months after Feb 2008.
Ok, we have 2 lists: prodsAfterFeb.txt (all prods with jobid more recent than the jobsid(1 feb) and prodsAfterFeb_byDate.txt (al prods with JOBDATE more recent than Feb 1). Of course, in the first list there are more files, because of some old jobs recently recovered and inserted in the DB. Done
>
>
No! the list of files has to be retrieved on the basis of the JOBID, not the date, because some old jobs (of 2006) have just been inserted in the last months after Feb 2008. Done
  2) delete these productions from the AMGA tables in the new schema. Done
3) copy them again from the production DB to the AMGA tables in the new schema
Finished at 17:50 Aug 20. A lot of errors due to the problem of multiple input files. Done
4) correct again the production number for prods 201 and 200. This should be done on the Oracle tables, after that Zoltan has copied again them from the AMGA tables.
Changed:
<
<
5) recover again the files and jobs with null prod id that have been deleted. Prods:00001354, 00001501, 00001503
>
>
5) recover again the files and jobs with null prod id that have been deleted. Prods:00001354, 00001501, 00001503. Done
  This has to be done. HIGH PRIORITY

Removal of bad productions

Revision 252008-08-20 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Current status of the activity on the BK

Line: 31 to 31
 Ok, we have 2 lists: prodsAfterFeb.txt (all prods with jobid more recent than the jobsid(1 feb) and prodsAfterFeb_byDate.txt (al prods with JOBDATE more recent than Feb 1). Of course, in the first list there are more files, because of some old jobs recently recovered and inserted in the DB. Done
2) delete these productions from the AMGA tables in the new schema. Done
3) copy them again from the production DB to the AMGA tables in the new schema
Changed:
<
<
I started the copy of the prodsAfterFeb_byDate.txt on Aug 19 at 10:25. Finished at 17:58. A lot of errors due to the problem of multiple input files.
Copied the remaining prods on aug 20. Only 1770 gave errors. checking...
>
>
Finished at 17:50 Aug 20. A lot of errors due to the problem of multiple input files. Done
  4) correct again the production number for prods 201 and 200. This should be done on the Oracle tables, after that Zoltan has copied again them from the AMGA tables.
Changed:
<
<
5) recover again the files and jobs with null prod id that have been deleted.
>
>
5) recover again the files and jobs with null prod id that have been deleted. Prods:00001354, 00001501, 00001503
  This has to be done. HIGH PRIORITY

Removal of bad productions

Revision 242008-08-20 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"
Changed:
<
<

Meeting of 23rd June 2008 and planning for next activities

>
>

Current status of the activity on the BK

  Participants: Philippe, Zoltan, Elisa
Line: 31 to 31
 Ok, we have 2 lists: prodsAfterFeb.txt (all prods with jobid more recent than the jobsid(1 feb) and prodsAfterFeb_byDate.txt (al prods with JOBDATE more recent than Feb 1). Of course, in the first list there are more files, because of some old jobs recently recovered and inserted in the DB. Done
2) delete these productions from the AMGA tables in the new schema. Done
3) copy them again from the production DB to the AMGA tables in the new schema
Changed:
<
<
I started the copy of the prodsAfterFeb_byDate.txt on Aug 19 at 10:25. Finished at 17:58. A lot of errors due to the problem of multiple input files.
>
>
I started the copy of the prodsAfterFeb_byDate.txt on Aug 19 at 10:25. Finished at 17:58. A lot of errors due to the problem of multiple input files.
Copied the remaining prods on aug 20. Only 1770 gave errors. checking...
  4) correct again the production number for prods 201 and 200. This should be done on the Oracle tables, after that Zoltan has copied again them from the AMGA tables.
5) recover again the files and jobs with null prod id that have been deleted.
This has to be done. HIGH PRIORITY

Revision 232008-08-20 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Participants: Philippe, Zoltan, Elisa

Deleted:
<
<
List of bad productions
 
Deleted:
<
<
There a list (see badProductions.txt file in the attachment) that contains all the productions which have to be ignored (for different reasons).
  Synchronization of the 2 DB
Line: 31 to 29
  1) identify the new data inserted after Feb 1 2008 (they are about 273000 jobs spread over 103 productions)
No! the list of files has to be retrieved on the basis of the JOBID, not the date, because some old jobs (of 2006) have just been inserted in the last months after Feb 2008.
Ok, we have 2 lists: prodsAfterFeb.txt (all prods with jobid more recent than the jobsid(1 feb) and prodsAfterFeb_byDate.txt (al prods with JOBDATE more recent than Feb 1). Of course, in the first list there are more files, because of some old jobs recently recovered and inserted in the DB. Done
Changed:
<
<
2) delete these productions from the AMGA tables in the new schema
Done
>
>
2) delete these productions from the AMGA tables in the new schema. Done
  3) copy them again from the production DB to the AMGA tables in the new schema
I started the copy of the prodsAfterFeb_byDate.txt on Aug 19 at 10:25. Finished at 17:58. A lot of errors due to the problem of multiple input files.
Changed:
<
<
4) correct again the production number for prods 201 and 200.
>
>
4) correct again the production number for prods 201 and 200. This should be done on the Oracle tables, after that Zoltan has copied again them from the AMGA tables.
  5) recover again the files and jobs with null prod id that have been deleted.
This has to be done. HIGH PRIORITY
Line: 56 to 53
 "00001610"
Added:
>
>
More in general, there a list (see badProductions.txt file in the attachment) that contains all the productions which have to be ignored (for different reasons). I keep this list updated every time that it comes out that a production is bad.
  to be done!

Post synchronization checks

Line: 219 to 218
 1496

Changed:
<
<
There productions are done with a buggy version of brunel. They can savely be removed.
>
>
There productions are done with a buggy version of brunel. They can safely be removed. I have added them to the list in the badProductions.txt file.
 On 5 of Aug 08 it has been decided that: we remove them from the oracle tables in the new schema. For completeness sake, we keep them in the AMGA tables in the new schema. In the computation of the processing pass these prods are skipped.
Deleted:
<
<
Tell Zoltan to remove them from the Oracle tables
  Modifications of the schema

Revision 222008-08-20 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 32 to 32
 No! the list of files has to be retrieved on the basis of the JOBID, not the date, because some old jobs (of 2006) have just been inserted in the last months after Feb 2008.
Ok, we have 2 lists: prodsAfterFeb.txt (all prods with jobid more recent than the jobsid(1 feb) and prodsAfterFeb_byDate.txt (al prods with JOBDATE more recent than Feb 1). Of course, in the first list there are more files, because of some old jobs recently recovered and inserted in the DB. Done
2) delete these productions from the AMGA tables in the new schema
Changed:
<
<
I have deleted the prodsAfterFeb_byDate.txt . I still have to delete the ones which are only in prodsAfterFeb.txt
>
>
Done
  3) copy them again from the production DB to the AMGA tables in the new schema
I started the copy of the prodsAfterFeb_byDate.txt on Aug 19 at 10:25. Finished at 17:58. A lot of errors due to the problem of multiple input files.
4) correct again the production number for prods 201 and 200.

Revision 212008-08-19 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 34 to 34
  2) delete these productions from the AMGA tables in the new schema
I have deleted the prodsAfterFeb_byDate.txt . I still have to delete the ones which are only in prodsAfterFeb.txt
3) copy them again from the production DB to the AMGA tables in the new schema
Changed:
<
<
I started the copy of the prodsAfterFeb_byDate.txt on Aug 19 at 10:25
>
>
I started the copy of the prodsAfterFeb_byDate.txt on Aug 19 at 10:25. Finished at 17:58. A lot of errors due to the problem of multiple input files.
  4) correct again the production number for prods 201 and 200.
5) recover again the files and jobs with null prod id that have been deleted.
This has to be done. HIGH PRIORITY

Revision 202008-08-19 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 28 to 28
 Philippe should provide a list of the productions that are still ongoing. The productions still running will need some special checks in order to unsure that no file/job has been lost during the process of synchronization.

Mon Aug 18 2008: the procedure I'll follow will be:

Changed:
<
<
1) identify the new data inserted after Feb 1 2008 (they are about 273000 jobs spread over 103 productions)
No! the list of files has to be retrieved on the basis of the JOBID, not the date, because some old jobs (of 2006) have just been inserted in the last months after Feb 2008
2) delete these productions from the AMGA tables in the new schema
3) copy them again from the production DB to the AMGA tables in the new schema
4)correct again the production number for prods 201 and 200.
5)recover again the files and jobs with null prod id that have been deleted.
>
>
1) identify the new data inserted after Feb 1 2008 (they are about 273000 jobs spread over 103 productions)
No! the list of files has to be retrieved on the basis of the JOBID, not the date, because some old jobs (of 2006) have just been inserted in the last months after Feb 2008.
Ok, we have 2 lists: prodsAfterFeb.txt (all prods with jobid more recent than the jobsid(1 feb) and prodsAfterFeb_byDate.txt (al prods with JOBDATE more recent than Feb 1). Of course, in the first list there are more files, because of some old jobs recently recovered and inserted in the DB. Done
2) delete these productions from the AMGA tables in the new schema
I have deleted the prodsAfterFeb_byDate.txt . I still have to delete the ones which are only in prodsAfterFeb.txt
3) copy them again from the production DB to the AMGA tables in the new schema
I started the copy of the prodsAfterFeb_byDate.txt on Aug 19 at 10:25
4) correct again the production number for prods 201 and 200.
5) recover again the files and jobs with null prod id that have been deleted.
  This has to be done. HIGH PRIORITY

Removal of bad productions

Revision 192008-08-18 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Participants: Philippe, Zoltan, Elisa

Added:
>
>
List of bad productions
 
Added:
>
>
There a list (see badProductions.txt file in the attachment) that contains all the productions which have to be ignored (for different reasons).
  Synchronization of the 2 DB
Line: 27 to 29
  Mon Aug 18 2008: the procedure I'll follow will be:
1) identify the new data inserted after Feb 1 2008 (they are about 273000 jobs spread over 103 productions)
Added:
>
>
No! the list of files has to be retrieved on the basis of the JOBID, not the date, because some old jobs (of 2006) have just been inserted in the last months after Feb 2008
  2) delete these productions from the AMGA tables in the new schema
3) copy them again from the production DB to the AMGA tables in the new schema
Changed:
<
<
correct again the production number for prods 201 and 200.
>
>
4)correct again the production number for prods 201 and 200.
5)recover again the files and jobs with null prod id that have been deleted.
  This has to be done. HIGH PRIORITY
Added:
>
>
Removal of bad productions

Philippe notices that: this list of event types: [11102003, 11102013, 11102402, 11104103, 11144103, 13102002, 13102013] in phys-lumi2 and phys-lumi5 have a bug and should be removed. Some of them are 1501, 1512, 1514, 1601 (lumi2) and 1523, 1526, 1628 and 1610 (lumi5). But Marianne probably knows the whole list. Marianne said that she will provide the list in the end of August. Here are the productions which correspond to these config versions and event type: "" "00001502" "00001512" "00001514" "00001523" "00001526" "00001528" "00001601" "00001610"

to be done!

  Post synchronization checks

Some consistency checks after the synchronization will be done, to ensure that we get the same results from the 2 (old and new) schemas.

Line: 216 to 238
 Prod 2146:

-- ElisaLanciotti - 23 Jun 2008 \ No newline at end of file

Added:
>
>
META FILEATTACHMENT attachment="badProductions.txt" attr="" comment="Comprehensive list of bad productions still in the DB" date="1219074159" name="badProductions.txt" path="badProductions.txt" size="550" stream="badProductions.txt" user="Main.ElisaLanciotti" version="2"

Revision 182008-08-18 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 25 to 25
  Philippe should provide a list of the productions that are still ongoing. The productions still running will need some special checks in order to unsure that no file/job has been lost during the process of synchronization.
Added:
>
>
Mon Aug 18 2008: the procedure I'll follow will be:
1) identify the new data inserted after Feb 1 2008 (they are about 273000 jobs spread over 103 productions)
2) delete these productions from the AMGA tables in the new schema
3) copy them again from the production DB to the AMGA tables in the new schema
correct again the production number for prods 201 and 200.
  This has to be done. HIGH PRIORITY

Post synchronization checks

Revision 172008-08-15 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 203 to 203
 The production 1536 has config. version = phys-lumi2 and according to the instructions of Philippe it should be in the group DC06-Sim-Reco_v30 but it has only RDST files, so in my opinion it should be in the group _DC06-Reco_v30_
Wed Aug 13 2008: Philippe confirmed there was a problem with this prod and it can be deleted.
Changed:
<
<
drop this entry in the prod2pass table and in the passindex table and add this prod to the bad prod list!
>
>
dropped this entry in the prod2pass table and in the passindex table and added this prod to the bad prod list
  Prods 37 and 38:

Revision 162008-08-13 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 202 to 202
  The production 1536 has config. version = phys-lumi2 and according to the instructions of Philippe it should be in the group DC06-Sim-Reco_v30 but it has only RDST files, so in my opinion it should be in the group _DC06-Reco_v30_
Changed:
<
<
Correct this bug in the buildHistory method!
>
>
Wed Aug 13 2008: Philippe confirmed there was a problem with this prod and it can be deleted.
drop this entry in the prod2pass table and in the passindex table and add this prod to the bad prod list!
  Prods 37 and 38:

Revision 152008-08-13 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 200 to 200
  Computation of Processing Pass
Changed:
<
<
The production 1536 has config. version = phys-lumi2 and according to the instructions of Philippe it should be in the group DC06-Sim-Reco_v30
>
>
The production 1536 has config. version = phys-lumi2 and according to the instructions of Philippe it should be in the group DC06-Sim-Reco_v30
 but it has only RDST files, so in my opinion it should be in the group _DC06-Reco_v30_
Correct this bug in the buildHistory method!
Added:
>
>
Prods 37 and 38:

Prod 2146:

  -- ElisaLanciotti - 23 Jun 2008

Revision 142008-08-13 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 189 to 189
 

There productions are done with a buggy version of brunel. They can savely be removed.

Changed:
<
<
On 5 of Aug 08 it has been decided that: we remove them from the oracle tables in the new schema. For completeness sake, we keep them in the AMGA tables in the new schema.
>
>
On 5 of Aug 08 it has been decided that: we remove them from the oracle tables in the new schema. For completeness sake, we keep them in the AMGA tables in the new schema. In the computation of the processing pass these prods are skipped.
Tell Zoltan to remove them from the Oracle tables
  Modifications of the schema
Line: 197 to 198
 
  • stored procedure to compute and insert the simulation conditions: updated
  • stored procedure to compute the processing pass: updated (Aug 5 2008)
Added:
>
>
Computation of Processing Pass

The production 1536 has config. version = phys-lumi2 and according to the instructions of Philippe it should be in the group DC06-Sim-Reco_v30 but it has only RDST files, so in my opinion it should be in the group _DC06-Reco_v30_
Correct this bug in the buildHistory method!

  -- ElisaLanciotti - 23 Jun 2008

Revision 132008-08-05 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 188 to 188
 1496

Changed:
<
<
This has to be investigated further and possibly fixed (Elisa and Philippe). Until it is not fixed, the table of the processing pass cannot be completed.
>
>
There productions are done with a buggy version of brunel. They can savely be removed.
On 5 of Aug 08 it has been decided that: we remove them from the oracle tables in the new schema. For completeness sake, we keep them in the AMGA tables in the new schema.
  Modifications of the schema

Revision 122008-08-05 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 25 to 25
  Philippe should provide a list of the productions that are still ongoing. The productions still running will need some special checks in order to unsure that no file/job has been lost during the process of synchronization.
Changed:
<
<
This has to be done.
>
>
This has to be done. HIGH PRIORITY
  Post synchronization checks

Some consistency checks after the synchronization will be done, to ensure that we get the same results from the 2 (old and new) schemas.

Changed:
<
<
This has to be done.
>
>
This has to be done, but after the item Synchronization of the 2 DB, see above.
 

BKK client documentation

Line: 81 to 81
  Problem of multiple input files

Data have been migrated to the new schema and here we will have to fix this problem. Philippe should provide some ascii files containing the information to delete some of the entries in the _inputFiles_tables of the problematic productions.

Changed:
<
<
This has to be fixed
>
>
This has to be fixed
  Files with no entry in jobs table
Line: 188 to 188
 1496

Changed:
<
<
This has to be investigated further (Elisa and Philippe).
>
>
This has to be investigated further and possibly fixed (Elisa and Philippe). Until it is not fixed, the table of the processing pass cannot be completed.
  Modifications of the schema

The new schema has been modified. A new table has been created: CONFIGURATIONS with the ConfigName and ConfigVersion. As a consequence, all the stored procedures have to be modified.

Changed:
<
<
  • stored procedure to compute and insert the simulation conditions: they have been updated
  • stored procedure to compute the processing pass: this has to be done (elisa)
>
>
  • stored procedure to compute and insert the simulation conditions: updated
  • stored procedure to compute the processing pass: updated (Aug 5 2008)
 

-- ElisaLanciotti - 23 Jun 2008

Revision 112008-08-04 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 36 to 36
  BKK client documentation
Changed:
<
<
In this twiki a new link has been added with the documentation of the Bkk Client methods developed by Zoltan.
>
>
In this twiki a new link has been added with the documentation of the Bkk Client methods developed by Zoltan.
This has to be done.
  How to deal with the 2000 and 2001 productions
Line: 187 to 188
 1496

Changed:
<
<
This has to be investigated further (Elisa).
>
>
This has to be investigated further (Elisa and Philippe).

Modifications of the schema

The new schema has been modified. A new table has been created: CONFIGURATIONS with the ConfigName and ConfigVersion. As a consequence, all the stored procedures have to be modified.

  • stored procedure to compute and insert the simulation conditions: they have been updated
  • stored procedure to compute the processing pass: this has to be done (elisa)
  -- ElisaLanciotti - 23 Jun 2008 \ No newline at end of file

Revision 102008-07-28 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 14 to 14
  This has been done. The 2 schemas have been populated in parallel since (+ o -) 10 July 08
Deleted:
<
<
  • elisa will migrate the data newly inserted (after Feb 2008) from the old to the new schema.
 
Changed:
<
<
This has to be done

Migration of productions altered after February

>
>
In order to have the migration completed, it is necessary also to migrate the productions altered after February 2008.
  The following productions have been altered after February:
  • 1856 (now finished) all the jobs are in the new schema (1658 jobs and 4974 files) ok
Line: 26 to 23
 
  • 1932
  • 1959
Added:
>
>
Philippe should provide a list of the productions that are still ongoing. The productions still running will need some special checks in order to unsure that no file/job has been lost during the process of synchronization.
  This has to be done.
Added:
>
>
Post synchronization checks

Some consistency checks after the synchronization will be done, to ensure that we get the same results from the 2 (old and new) schemas.

This has to be done.

BKK client documentation

In this twiki a new link has been added with the documentation of the Bkk Client methods developed by Zoltan.

  How to deal with the 2000 and 2001 productions

These 2 productions present a problem for the computation of the processing pass because 2 different versions of the application have been used (Brunelv30 and Brunelv31) within THE SAME production.

Revision 92008-07-28 - ZoltanMathe

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Revision 82008-07-24 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 22 to 22
  The following productions have been altered after February:
  • 1856 (now finished) all the jobs are in the new schema (1658 jobs and 4974 files) ok
Changed:
<
<
  • 1896
>
>
  • 1896 migrated on July 24. (All the jobs and files have been copied, but the logfile is full of errors: multiple input files issue...).
 
  • 1932
  • 1959

Revision 72008-07-24 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 11 to 11
 The procedure is the following:

  • first, we will start to feed in parallel the 2 DB. A script will catch the XML files sent to the volhcb05 and will translate them in the new format (the format to insert data in the new BK). This XML file will be sent to the volhcb07. Then, the BK Agent will be switched on and the data should be inserted into the new schema. During these operations, the BKManager on volhcb05 will always be on.
Added:
>
>
This has been done. The 2 schemas have been populated in parallel since (+ o -) 10 July 08
 
  • elisa will migrate the data newly inserted (after Feb 2008) from the old to the new schema.

This has to be done

Line: 18 to 21
  Migration of productions altered after February

The following productions have been altered after February:

Changed:
<
<
  • 1856 (now finished)
>
>
  • 1856 (now finished) all the jobs are in the new schema (1658 jobs and 4974 files) ok
 
  • 1896
  • 1932
  • 1959
Changed:
<
<
This has to be done. But before we have to start to feed the new schema with the XML files (see previous point)
>
>
This has to be done.
  How to deal with the 2000 and 2001 productions
Line: 56 to 59
 and the table has been populated with 15 rows, defining all the possible simulation conditions of DC06 simulated data.
This has been done
Changed:
<
<
Now the attribute DAQId in the jobs table has to be updated with the corresponding SimId. To be done (Elisa). This has to be done
>
>
Now the attribute DAQId in the jobs table has to be updated with the corresponding SimId.
This has been done on July 24
 For more details see the twiki page for the simulation conditions.
Added:
>
>
Some checks to do:
we have to check that a job has the same simulation conditions of the input jobs.
This has to be done (Elisa).
  Problem of multiple input files

Data have been migrated to the new schema and here we will have to fix this problem. Philippe should provide some ascii files containing the information to delete some of the entries in the _inputFiles_tables of the problematic productions.

Line: 70 to 77
 During the migration of the files with null production number, some files have been found with no entry in the jobs table. The files are 29 in total, spread over 5 productions ( 00001323, 00001324, 00001325, 00001326, 00001327). It has been decided to ignore these files, since we don't have the necessary information about them to register them in the new schema, and also because they are only few files.
This is not an issue, just a warning
Added:
>
>
Productions with ONLY SIM files

During the computation of the processing files, some productions have been found to have only SIM files. This is not foreseen in the normal production workflow. The files of these production do not appear in the InputFiles table, this means that there's no job in the BK which has taken in input those files. Then we have 3 possible cases:
1- The jobs and their output (DIGI) files which have processed these SIM files have been removed from the BK
2- Or they have never been registered in the BK
3- Or these SIM files were not interesting and were not processed.

the productions affected by this problem are 92: 1329, 1330, 1331 1332 1333 1334 1335 1336 1358 1359 1360 1361 1362 1363 1364 1365 1375 1376 1377 1378 1384 1385 1386 1387 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1462 1463 1470 1471 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496

This has to be investigated further (Elisa).

 -- ElisaLanciotti - 23 Jun 2008

Revision 62008-07-17 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Line: 53 to 53
  DETECTORCOND VARCHAR2(256)
LUMINOSITY VARCHAR2(256)
Changed:
<
<
and the table has been populated with 15 rows, defining all the possible simulation conditions of DC06 simulated data. See the attached file to see the content.
>
>
and the table has been populated with 15 rows, defining all the possible simulation conditions of DC06 simulated data.
  This has been done
Changed:
<
<
Now the attribute DAQId in the jobs table has to be updated with the corresponding SimId. To be done (Elisa). This has to be done
>
>
Now the attribute DAQId in the jobs table has to be updated with the corresponding SimId. To be done (Elisa). This has to be done
For more details see the twiki page for the simulation conditions.
  Problem of multiple input files
Line: 70 to 71
  This is not an issue, just a warning

-- ElisaLanciotti - 23 Jun 2008

Deleted:
<
<
META FILEATTACHMENT attachment="simCond.html" attr="" comment="the simulation conditions table data" date="1214826240" name="simCond.html" path="simCond.html" size="5597" stream="simCond.html" user="Main.ElisaLanciotti" version="1"

Revision 52008-07-01 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008 and planning for next activities

Participants: Philippe, Zoltan, Elisa

Deleted:
<
<
Migration of productions altered after February
 
Deleted:
<
<
The following productions have been altered after February:
  • 1856 (now finished)
  • 1896
  • 1932
  • 1959
  Synchronization of the 2 DB
Line: 21 to 15
  This has to be done
Added:
>
>
Migration of productions altered after February

The following productions have been altered after February:

  • 1856 (now finished)
  • 1896
  • 1932
  • 1959

This has to be done. But before we have to start to feed the new schema with the XML files (see previous point)

  How to deal with the 2000 and 2001 productions

These 2 productions present a problem for the computation of the processing pass because 2 different versions of the application have been used (Brunelv30 and Brunelv31) within THE SAME production.

Line: 50 to 54
  LUMINOSITY VARCHAR2(256)

and the table has been populated with 15 rows, defining all the possible simulation conditions of DC06 simulated data. See the attached file to see the content.

Changed:
<
<
Now the attribute DAQId in the jobs table has to be updated with the corresponding SimId. To be done (Elisa).
>
>
This has been done
 
Added:
>
>
Now the attribute DAQId in the jobs table has to be updated with the corresponding SimId. To be done (Elisa). This has to be done
  Problem of multiple input files

Revision 42008-07-01 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"
Changed:
<
<

Meeting of 23rd June 2008

>
>

Meeting of 23rd June 2008 and planning for next activities

  Participants: Philippe, Zoltan, Elisa
Line: 19 to 19
 
  • first, we will start to feed in parallel the 2 DB. A script will catch the XML files sent to the volhcb05 and will translate them in the new format (the format to insert data in the new BK). This XML file will be sent to the volhcb07. Then, the BK Agent will be switched on and the data should be inserted into the new schema. During these operations, the BKManager on volhcb05 will always be on.
  • elisa will migrate the data newly inserted (after Feb 2008) from the old to the new schema.
Added:
>
>
This has to be done
  How to deal with the 2000 and 2001 productions
Line: 30 to 31
 if production==2001 and programVersion==v30 then set production to 201
(action for Elisa).

Added:
>
>
This has been fixed with the following SQL statements (1 july 08):
update jobs set Production=201 where (jobs.production=2001 and jobs.programVersion='v30r17' and jobs.programname='Brunel');
update jobs set Production=200 where (jobs.production=2000 and jobs.programVersion='v30r17' and jobs.programname='Brunel');
This has been fixed on 1 July 08
  Populating the Simulation Conditions table for the DC06 data

Finally, the attributes of the simulation conditions table are:

Line: 48 to 55
  Problem of multiple input files
Changed:
<
<
Data have been migrated to the new schema and here we will have to fix this problem. Philippe should provide some ascii files containing the information to delete some of the entries in the _inputFiles_tables of the problematic productions.
>
>
Data have been migrated to the new schema and here we will have to fix this problem. Philippe should provide some ascii files containing the information to delete some of the entries in the _inputFiles_tables of the problematic productions.
This has to be fixed
  Files with no entry in jobs table
Changed:
<
<
During the migration of the files with null production number, some files have been found with no entry in the jobs table. The files are 29 in total, spread over 5 productions ( 00001323, 00001324, 00001325, 00001326, 00001327). It has been decided to ignore these files, since we don't have the necessary information about them to register them in the new schema, and also because they are only few files.
>
>
During the migration of the files with null production number, some files have been found with no entry in the jobs table. The files are 29 in total, spread over 5 productions ( 00001323, 00001324, 00001325, 00001326, 00001327). It has been decided to ignore these files, since we don't have the necessary information about them to register them in the new schema, and also because they are only few files.
This is not an issue, just a warning
  -- ElisaLanciotti - 23 Jun 2008

Revision 32008-06-30 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008

Line: 53 to 53
  Files with no entry in jobs table
Added:
>
>
During the migration of the files with null production number, some files have been found with no entry in the jobs table. The files are 29 in total, spread over 5 productions ( 00001323, 00001324, 00001325, 00001326, 00001327). It has been decided to ignore these files, since we don't have the necessary information about them to register them in the new schema, and also because they are only few files.
  -- ElisaLanciotti - 23 Jun 2008 \ No newline at end of file
Added:
>
>
META FILEATTACHMENT attachment="simCond.html" attr="" comment="the simulation conditions table data" date="1214826240" name="simCond.html" path="simCond.html" size="5597" stream="simCond.html" user="Main.ElisaLanciotti" version="1"

Revision 22008-06-30 - ElisaLanciotti

Line: 1 to 1
 
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008

Line: 22 to 22
  How to deal with the 2000 and 2001 productions
Added:
>
>
These 2 productions present a problem for the computation of the processing pass because 2 different versions of the application have been used (Brunelv30 and Brunelv31) within THE SAME production.
The following solution has been proposed: to alter the production field in the jobs table for those jobs which have one programVersion (i.e. v30) and set it to a new value (of course taking care of using a production number that will never be used!). The other jobs, those which have the other version (i.e. v31), will be left unchanged.
Proposal for the algorithm:
if production==2000 and programVersion==v30 then set production to 200
if production==2001 and programVersion==v30 then set production to 201
(action for Elisa).
  Populating the Simulation Conditions table for the DC06 data
Changed:
<
<
Finally, the attributes of the simulation conditions table are:
>
>
Finally, the attributes of the simulation conditions table are:
SIMID NUMBER
SIMDESCRIPTION VARCHAR2(256)
BEAMCOND VARCHAR2(256)
BEAMENERGY VARCHAR2(256)
GENERATOR VARCHAR2(256)
MAGNETICFIELD VARCHAR2(256)
DETECTORCOND VARCHAR2(256)
LUMINOSITY VARCHAR2(256)

and the table has been populated with 15 rows, defining all the possible simulation conditions of DC06 simulated data. See the attached file to see the content.
Now the attribute DAQId in the jobs table has to be updated with the corresponding SimId. To be done (Elisa).

 

Problem of multiple input files

Added:
>
>
Data have been migrated to the new schema and here we will have to fix this problem. Philippe should provide some ascii files containing the information to delete some of the entries in the _inputFiles_tables of the problematic productions.
  Files with no entry in jobs table

Revision 12008-06-23 - ElisaLanciotti

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="BKDev"

Meeting of 23rd June 2008

Participants: Philippe, Zoltan, Elisa

Migration of productions altered after February

The following productions have been altered after February:

  • 1856 (now finished)
  • 1896
  • 1932
  • 1959

Synchronization of the 2 DB

The procedure is the following:

  • first, we will start to feed in parallel the 2 DB. A script will catch the XML files sent to the volhcb05 and will translate them in the new format (the format to insert data in the new BK). This XML file will be sent to the volhcb07. Then, the BK Agent will be switched on and the data should be inserted into the new schema. During these operations, the BKManager on volhcb05 will always be on.
  • elisa will migrate the data newly inserted (after Feb 2008) from the old to the new schema.

How to deal with the 2000 and 2001 productions

Populating the Simulation Conditions table for the DC06 data

Finally, the attributes of the simulation conditions table are:

Problem of multiple input files

Files with no entry in jobs table

-- ElisaLanciotti - 23 Jun 2008

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback