This page explains some error scenarios and recovery procedures involving the oracle databases in the pit:

RAC failures:

Power cut:
There is a race condition between the restart of the storage and the restart of the RAC servers. The restart of the storage takes significantly longer than the restart of the servers so usually when the qla24xx module is loaded at boot time, no volumes are being exported by the storage controllers. As a result multipath and rawdevices fail to find any devices and the whole Oracle installation is not found.
  • To establish if this is the case run as root:
     multipath -l 
If no devices are displayed then most probably this is the case of the Oracle failure.
  • FIX: The simplest way is to restart the servers. The alternative is to restart the crs, crsd, cssd and evmd (startup scripts in /etc/init.d with the init. prefix)

Insufficient disk space on the ASM groups:
Usually this
  • To establish if this is the case, use the asmcmd tool to view the available space on the ASM groups:
    • export ORACLE_SID=+ASM as user oracle
    • execute asmcmd
    • type du
  • FIX: Taking into consideration the space available in the current ASM groups, the space consumption is caused by unnecessary archive logs. The best way to remove those is using RMAN:
export ORACLE_SID=<db instance name>
rman target /
report obsolete
delete obsolete
Also files can be removed from ASM using asmcmd. DO NOT REMOVE REDO LOGS. DATABASE FILES CAN BE REMOVED BUT THE DATABASE WILL GO OFFLINE IMMEDIATELY AND MUST BE RESTORED FORM BACKUPS!!!. If you are removing archive logs and there is a database crash before the next backup then information can be lost. To prevent this you should take a backup immediate using RMAN:
export ORACLE_SID=<db instance name>
rman target /
backup database

-- Main.RaduStoica - 17 Mar 2008
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2008-03-17 - RaduStoica
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback