Oracle CPU patch October 2005 installation instructions

• Note the CPU Oct05 patch needs to be applied to the DB Oracle Home and OEM agent home. The following describes the steps for DB patching. OEM agent patching can be done after the DB has been successfully patched.

• Optional: create tar copy of ORACLE_HOME and orainvetory (to force a patch rollback if needed)

• copy Patch and OPatch to server

      mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch.old
      cp -r CPU_Oct05/OPatch $ORACLE_HOME/

10g RAC

• shutdown with srvrctl: db instance, asm instance, listener

      srvctl stop instance -d DBNAME -i INST -o transactional # wait end of transactions before kill sessions
      srvctl stop asm -n `hostname -s`                        # `hostname -s` returns local hostname without domain
      srvctl stop nodeapps -n `hostname -s`  

• check existance of shrept.lst

 
      ls $ORACLE_HOME/network/admin/shrept.lst 

• create g++ link

      sudo ln -s /usr/bin/g++296 /usr/bin/g++

• use new optach to install patch

   cd CPU_Oct05/4567866
   $ORACLE_HOME/OPatch/opatch apply -local
   * Note: might appear error on pre-patch script, just say to continue installation

• Send email to computer.operations@cern.ch warning the reboot of nodes

• Prepare kernel upgrade
  • login as root

      rm /etc/nospma                # enable SPMA to update RPMs
      spma_wrapper.sh               # upgrade RPMs
      ncm-ncd --co grub fmonagent   # update grub and monitoring configuration
      check-this-host               # run all monitoring checks; should (at least) give "OperatingSystem" warning

• reboot the node (shutdown -r now), it should be back in a few minutes
• check errors

      check-this-host                # the OperatingSystem" warning should have disapeared
       
      The following warnings may persist yet for 5-10 minutes and then should disappear:
      crond
      atd
      MSAd
      SureHeartbeat

• Database should be up and available

      crs_stat -t     

• If needed startup the crs components/db instance manually

      srvctl start nodeapps -n `hostname -s`  
      srvctl stop asm -n `hostname -s` 
      srvctl start instance -d DBNAME -i INST

• Proceed to next node of the cluster

• After patch all nodes of cluster, compile objects

      cd $ORACLE_HOME/cpu/CPUOct2005
      sqlplus "/ as sysdba" @catcpu.sql
      cd $ORACLE_HOME/rdbms/admin
      sqlplus "/ as sysdba" @utlrp.sql

• Check all objects are valid

      select OBJECT_NAME from DBA_OBJECTS where status = 'INVALID';

10g single instance database

• Do the OS security upgrade. Send email to computer.operations@cern.ch warning the reboot of nodes

• Prepare kernel upgrade
  • login as root

      rm /etc/nospma                # enable SPMA to update RPMs
      spma_wrapper.sh               # upgrade RPMs
      ncm-ncd --co grub fmonagent   # update grub and monitoring configuration
      check-this-host               # run all monitoring checks; should (at least) give "OperatingSystem" warning

• reboot the node (shutdown -r now), it should be back in a few minutes
• check errors

      check-this-host                # the OperatingSystem" warning should have disapeared
       
      The following warnings may persist yet for 5-10 minutes and then should disappear:
      crond
      atd
      MSAd
      SureHeartbeat

• Check existance of shrept.lst

 
      ls $ORACLE_HOME/network/admin/shrept.lst 

• Create g++ link

      sudo ln -s /usr/bin/g++296 /usr/bin/g++

• Check if the following executables are present in the $PATH: make, ar, ld, and nm.
• Check whether "java" and "jar" executables are present in your Oracle Home. In a standard Oracle installation, "java" is available in "$ORACLE_HOME/jre//bin" directory and "jar" is available in "$ORACLE_HOME/jdk/bin" directory. By default, "opatch" uses the executables from these locations. If not, invoke the "opatch apply" command with the -jdk flag, specifiying the full path to the JDK to be used.

• Shutdown all instances and listeners associated to the ORACLE_HOME being updated:

       lsnrctl stop
       sqlplus "/ as sysdba";
       shutdown immediate;
     

• Use new optach to install patch

   cd CPU_Oct05/4567866
   $ORACLE_HOME/OPatch/opatch apply 
   * Note: some errors might appear - please check the metalink document 333956.1

• Start up the database instance running out of the ORACLE_HOME being patched and run catcpu.sql as follows:

        cd $ORACLE_HOME/cpu/CPUOct2005
        sqlplus "/ as sysdba";
        startup;
        @catcpu.sql;
        exit;
     * Note: some errors might appear - please check the metalink document 333956.1
     

• Compile objects

         cd $ORACLE_HOME/rdbms/admin
         sqlplus "/ as sysdba"
         @utlrp.sql;
      

• Check all objects are valid

      select OBJECT_NAME from DBA_OBJECTS where status = 'INVALID';

9i single instance

• Disable Asynchronous IO

    SQL> alter system set disk_asynch_io=false scope=spfile;
    SQL> shutdown immediate;

      $ cd $ORACLE_HOME/rdbms/lib
      $ make -f ins_rdbms.mk async_off
      $ make -f ins_rdbms.mk ioracle

• Apply Oracle OS-level patch 3006854 and fulfill prerequisits described on metalink in the note 252217.1

       $ sudo mv /usr/bin/gcc  /usr/bin/gcc32
       $ sudo ln -sf /usr/bin/gcc296 /usr/bin/gcc
       $ sudo ln -sf /usr/bin/g++296 /usr/bin/g++
  
       $ cd 3006854
       $ sudo ./rhel3_pre_install.sh
       $ sudo cp /etc/libcwait.so /lib/libcwait.so
       $ sudo chmod u+w /etc/ld.so.preload  
      

In the /etc/ld.so.preload file change /etc/libcwait.so to /lib/libcwait.so

     $ sudo rm /etc/libcwait.so
    

• Upgrade OS as described in 10g single instance database section

• Upgrade OPatch to version 1.0.0.0.53

• Stop Oracle database and the listener and apply the patch

     $ lsnrctl stop
     SQL> shutdown immediate
     $ cd 4567854
     $ $ORACLE_HOME/OPatch/opatch apply -jdk $ORACLE_HOME/jdk
    

• Startup the database, the listener and patch the database

     $ cd $ORACLE_HOME/cpu/CPUOct2005
     $ sqlplus "/ as sysdba"
     SQL> STARTUP
     SQL> @catcpu.sql
     $ lsnrctl start
    

• Compile objects

         cd $ORACLE_HOME/rdbms/admin
         sqlplus "/ as sysdba"
         @utlrp.sql;
      

• Check all objects are valid

      select OBJECT_NAME from DBA_OBJECTS where status = 'INVALID';

• Extra steps added to the CPU maintenance. Change (if needed) the following init parameters:

       control_files=... (multiplex to 2 different file systems + change the parameter value accordingly)
       db_block_checksum = true
       parallel_max_servers = 0 (we discussed this in an earlier meeting)

        Create pfile from spfile:
         create pfile='/tmp/init.ora' from spfile;
        Shutdown the database
        Copy one of the control files from /ORA/dbs03/oradata/SID directory to /ORA/dbs01/oracle/admin/SID/ directory 
        (if they are on the same filesystem, choose a location in a different filesystem):
         cp /ORA/dbs03/oradata/COPVSS10/COPVSS10_control01.ctl  /ORA/dbs01/oracle/admin/COPVSS10/COPVSS10_control02.ctl
        Edit the /tmp/init.ora and set the control_files value to the 2 new control file locations:
         *.control_files='/ORA/dbs03/oradata/COPVSS10/COPVSS10_control01.ctl','/ORA/dbs01/oracle/
                           admin/COPVSS10/COPVSS10_control02.ctl'           
        Startup the database with the init.ora pfile:
         startup pfile='/tmp/init.ora'
        Create spfile from the corrected pfile:
         create spfile from pfile='/tmp/init.ora'
        Restart the database and check the control_files parameter:
         shutdown immediate;
         startup;
         show parameter control_files;
      

• For compass change the soft file limit in /etc/security/limits.conf from 1024 to 2048 (optional, this is to fix an issue seen by Miguel)

• If any filesystem, such as /ORA/dbs03 or other is ext2, change it to ext3

      1. shutdown database
      2. cd /
      3. sudo bash (become root)
      4. mount - (get mounted fs info - filesystems, devices, ext...) 
      5. Check processes running in the filessystem to be changed, kill if something exists:
         sudo /usr/sbin/lsof | grep -i /ORA/dbs0... 
      6. umount /ORA/dbs0...
      7. tune2fs -j /dev/sd... (should be in /sbin directory; /dev/sd... is the device for /ORA/dbs0... which you want to change)
      8. Edit /etc/fstab and correct ext2 to ext3 for /ORA/dbs0... which has been changed 
      9. mount /ORA/dbs0...
      10. You can check the changes doing "mount" command