WARNING: This web is not used anymore. Please use PDBService.TestbedCernStatus instead!
 
1.March:
  • Source database: rls1r1 (lxshare084d). Document updated and sent.
  • Streams setup started with FNAL.
2.March:
  • Streams configuration finished between CERN and FNAL.
  • Streams setup started with CNAF.
3.March:
  • Instantiation problem between CERN and FNAL: Missing Streams data dictionary. Streams doesn't work.
4.March:
  • Anil proposed to repopulate the Streams data dictionary as solution.
  • CERN and FNAL are now connected and schema capturing is enabled (DML and DDL changes are replicated)
  • Streams configuration finished between CERN and CNAF (DML and DDL changes are replicated)
9. March:
  • Stress tests run up to 25.000 entries. All changes replicated to the replica sites (delay: some seconds)
10.March:
  • Streams setup started with Academia Sinica.
  • Streams configuration finished between CERN and Sinica (DML and DDL changes are replicated)
15.March - 17.March:
  • Stress tests run up to 50.000 entries (one site joined to). All changes replicated to the replica sites (delay is higher replicating to Sinica).
  • > 25.000 entries, LCRs spilled to disk (=> higher delay in replication)
21.March - 24.March:
  • Script to prevent excessive spill of messages(LCRs) from the Streams buffer queue to disk implemented for the testbed environment (in the source database). This script works by stopping the capture process when the number of outstanding messages are above a threshold and then restarting capture when the messages come within the threshold [ see metalink note 259609.1 for detailed info. ]
  • Some stress tests repeated using the script. LCRs are not spilled to disk but delay in replication is increased.
  • Working on Frontier installation: Oracle 10g sw installed and database created.
29.March - 31.March:
  • Trash.Test plan to test streams replication for 3D - Anil Kumar proposal
  • Scripts to test the replication of DDL operations in the testbed environment implemented. Preparing how to automate the tests.
  • Gordon from RAL is preparing to join to testded environment.
  • Working on Frontier installation: accounts.
4.April - 5.April: 6.April:
  • Agents installation on CERN machines for testbed environment done. Important: setting the appropiate value of TZ environmental variable as it's explained in the point before.
  • Streams setup started with Rutherford Appleton Laboratory (RAL).
7.April:
  • Streams configuration finished between CERN and RAL (DML operations are replicated)
8.April:
  • Running tests to test DDL operations replication.
12.April:
  • Propagation disabled from CERN to Sinica due to an archiver problem in Sinica's database: ORA-00257: archiver error. Connect internal only, until freed.
  • The cause is that the archiver process received an error while trying to archive a log. By far the most likely cause of the error is the archive destination device is out of space.
  • Error ORA-10388: parallel query server interrupt (failure) possible cause of the desconnection problem from Sinica. Under investigation.
13.April:
  • DDL tests running.
  • "DROP PUBLIC SYNONYM" operation is not replicated. DDL change ignored by the apply process. However, "DROP SYNONYM" operation is replicated.
14.April:
  • Upgrade of database rls1 at CERN to Oracle 10.1.0.4 successful.
  • Bug found: CONTINUOUS ORA-25228 IN ALERT AFTER UPGRADE CAPTURE TO 10.1.0.4 FROM 10.1.0.2
  • Capture process and queue, and propagation job stopped due to this new bug.
  • Streams replication is still working from the source database rls1r1 to the other databases into the testbed environment.
18.April - 20.April:
  • DDL tests running.
26.April:
  • Frontier installation completed.
29.April:
  • Error in propagation from CERN to SINICA: TNS: connect timeout ocurred. Error repeated several times => propagation disabled.
  • Queue of LCRs with high rate of enqueing messages (no propagation from CERN to SINICA) causes ORA-04031: unable to allocate bytes of shared memory in propagation job from CERN to RAL.
  • Once propagation job to Sinica is enabled again, the error desappears and replication continues working well (no LCRs lost).
  • Replication with Sinica very slow (20 min to be synchronized with all the databases in the testbed environment).
17.May:
  • FNAL: Oracle not available (applying kernel patches) => propagation disabled
  • RAL: TNS no listener => propagation disabled
  • SINICA: database problem => propagation disabled
  • Due to the propagation was disabled in 3 of the 5 sites, lot of LCR's were spilled to disk. Error: unable to allocate bytes of shared memory => stop capture process until all LCR's are propagated.
18.May:
  • FNAL database and RAL listener are up => propagation enabled.
  • Capture process still stopped.
23.May:
  • April CPU 2005 patch 10.1.0.4 applied on rls1 (lxfs5591).
24.May:
  • Sinica database is up => re-enable capture work.
1.June:
  • Upgrade of database rls1r1 (source 3D testbed at CERN) to Oracle 10.1.0.4 successful.
  • Path 4291110 to fix Bug: CONTINUOUS ORA-25228 IN ALERT AFTER UPGRADE CAPTURE TO 10.1.0.4 FROM 10.1.0.2 applied.
  • April CPU 2005 patch 10.1.0.4 applied.
6.June:
  • Sinica database upgraded to 10.1.0.4
7.June:
  • RAL database stopped due to maintenance.
14.June:
  • Script to join to the 3D OEM (prepared by Marta) available (sent to each site).
15.June:
  • FNAL database upgraded to 10.1.0.4
16.June:
  • Sinica is joined to the 3D OEM.
23.June:
  • CNAF is joined to the 3D OEM.
2.July:
  • cooling system problems at CNAF implies to turn off CNAF database.
  • Propagation is disabled from CERN to CNAF.
13.July:
  • CNAF database up and running again.
  • Re-enable propagation from CERN to CNAF. Some minutes to synchronize.
26.July:
  • FNAL database down to apply July CPU patches.
17.August:
  • Sinica's database archiver problem : ORA-00257: archiver error. Connect internal only, until freed. The cause is that the archiver process received an error while trying to archive a log. By far the most likely cause of the error is the archive destination device is out of space.
  • Sinica has disabled the database archivelog mode.
18.August:
  • CNAF supply power problem: they need some days to solve it. Propagation disabled from CERN to CNAF.
  • Preparatoy work for Streams setup started with Karlsuhe.
  • Connection problems: causes unknown. Under investigation.

29.August:

  • Streams configuration finished between CERN and Karlsuhe. Still connection problems: propagation doesn't work properly.

5.September:

  • RAL database upgrade to Oracle version 10.1.0.4 + patches

13.September:

  • CNAF supply power problem solved. Database up and running again. Re-enable propagation from CERN to CNAF.
  • Still connection problems with Karlsuhe.

27.September:

  • Recommendation on streams user privileges from Anil Kumar (FNAL) applied:
    • revoke dba privileges
    • grant connect, resource, select_catalog_role, create public synonym, drop public synonym, exp_full_database and imp_full_database
  • Recommendation on streams administrator privileges not applied because dba privileges for streams administrator is a requirement using 10g version.

28.September:

  • Metalink note 298877.1 10g Streams Recommended Configuration
    • preparatory work to implement some recomendations not applied yet
    • force checkpoint periodically and checkpoint frequency increased for capture process at source database

7.October:

  • COOL accounts setup on CERN databases. Testbed (CERN sites) ready for Streams tests using COOL
  • Preparatory work with BNL to join to the testbed

14.October:

  • ATLASDD accounts setup on CERN databases. Testbed (CERN sites ready for Streams tests using Atlas geometry
  • Description and script for sites to join to the Streams tests using COOL and ATLASDD applications prepared and available in wiki

17.October - 19.October:

  • 3D LCG workshop

25.October:

  • network intervention with Gridka in order to determine the connection problems

27.October:

  • October CPU patches applied on the testbed at CERN

28.October:

  • October CPU patches applied on the 3D OEM at CERN

31.October:

  • Streams configuration finished between CERN and BNL

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2005-11-29 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PSSGroup All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback