10g RAC Single node installation

Single node installation for downstreams capture

OS Setup and Prerequisites

• RHEL 4, Oracle user and group, kernel parameters and other OS installation details, as performed by FIO procedures are prerequisites for the following https://twiki.cern.ch/twiki/bin/view/PSSGroup/OSspecification
• upload/refresh to the node the scripts directory with PDB tools (copy them over from another installed node scp -r scripts node_name:$PWD)
• deploy and configure .bashrc (see installed machine)
• sudo echo "session required /lib/security/pam_limits.so" >> /etc/pam.d/login
• create needed directories. Set the ownership properly:

   
        mkdir /ORA/dbs00/oracle
        chown oracle:ci /ORA/dbs00/oracle
        chown oracle:ci /ORA/dbs01/oracle
     

• Create a /etc/oraInst.loc file with the following contents:

inventory_loc=/ORA/dbs01/oracle/oraInventory
inst_group=ci

Setup storage: multipathing with device mapper

• check storage configuration
  • RAID 0+1 configured on the infortrend controller
• Identify and prepare the LUN (1 big LUN) to by used by the DB filesystem
  • check /etc/modprobe.conf against the standard (see installed machine)
  • fdisk -l |grep Disk to list the disks
  • /dev/sdb and /dev/sdc are the same disk visible via 2 paths
  • partition /dev/sdb (1 partition that spans the whole disk) : fdisk /dev/sdb
• Setup multipathing
  • copy /etc/multipath form a deployed system
  • edit the last part of /etc/multipath.conf wit the correct alias
  • generate the wwid with /sbin/scsi_id -g -u -s /block/sdb
  • the multipathed device will be visible in /dev/mapper

• start multipathing (one off)

modprobe dm-multipath
modprobe dm-round-robin
chkconfig multipathd on
multipath

• Useful commands:
  • multipath -> to recreate/refresh dm devices (see /dev/mpath)
  • multipath -l -> to display current config
  • multipath -F -> to flush out all dm aliases (!)
  • multipath -F; chkconfig multipathd off; rmmod qla2400 qla2xx -> remove all config (!)

• Configure Filesystem for Oracle (/ORA/dbs02)

 
mkfs.ext3 -T largefile /dev/sdb1
tune2fs -i 0 /dev/sdb1 (disables automatic fsck)
vi /etc/fstab
 ..add entry for /ORA/dbs02 oracle filesystem (see example on installed system)
 .. (note with device mapper use /dev/mapper/* instead of /dev/mpath/*)
mkdir /ORA/dbs02
mount /ORA/dbs02
mkdir /ORA/dbs02/oracle
chown oracle:ci /ORA/dbs02/oracle
 

• (optional)Write to netops and ask for network aliases to be used in the tnsnames.ora

Oracle binaries Installation

RDBMS binaries installation

Cloning (method 2) is the preferred method to deploy Oracle RDBMS installations, for uniformity and speed.

• Method 1 (without cloning) Use Oracle runInstaller to install RAC on first node only
  • note you should rather use clonig (next paragraph)
  • advanced installation
  • HOME_NAME=OraDb10g_rdbms
  • ORACLE_HOME=/ORA/dbs01/oracle/product/10.2.0/rdbms
  • install Oracle Enterprise Edition (default)
  • choose cluster install on first node only
  • Install only the software (no DB creation at this stage)
  • Apply patchsets and security patches on the installed node

• Method 2, cloning:
  • copy the 'master' tar image from pdb backup (Ex: scp rdbms_10203_singlenode_with5705795.tgz srv2:$PWD)
  • at the destinations
    • tar xfpz rdbms_singlenode_10203_with5705795.tgz
  • on the new nodes perform the clonig operation
    • cd $ORACLE_HOME/clone/bin
    • *perl clone.pl ORACLE_HOME="/ORA/dbs01/oracle/product/10.2.0/rdbms" ORACLE_HOME_NAME="OraDb10g_rdbms"
    • run root.sh o, as instructed by clone.pl

• Run netca
  • listener name: LISTERNER (each node will have a suffix with the node name automatically)
  • choose the correct non-default port
  • after netca, manually edit listener.ora: remove the EXTPROC entry from listener.ora and use node names instead of IPs
  • remove tnsnames.ora (netca creates it only on one node with extproc config that we don't need)

Database creation

Database instance creation

• run dbca to create the DB, post installation steps follow
  • custom database (not from a template)
  • enter DB name (same value for SID prefix)
  • uncheck 'configure for EM flag'
  • input password
  • check 'file system'
  • database files in /ORA/dbs02/oracle/oradata (note dbca will create a subdirs with the db name, etc )
  • recovery files in /ORA/dbs02/oracle/flash_recovery (note dbca will create a subdirs with the db name, etc)
  • use oracle-managed files
  • choose archivelog
  • uncheck all options (dataming, olap,spatial,EM repository and all standard options:jvm, etc)
  • change the following params: db_domain=cern.ch
  • don't tune other parameters yet (leave the defaults) but check block size = 8k, character set = WE8ISO5559P1
  • create database + check 'save' the DB creation scripts
  • NOTE: never click 2 times on the 'java buttons' reaction time can be slow

• fine tune db parameters:
  • sqlsys_DB -> create pfile='/tmp/initdb.txt' from spfile;
  • show parameter spfile
  • shutdown the DB instance
  • edit vi /tmp/initdb.txt (see parameter values below)
  • change the dump directorie filesystem:
    • if needed mkdir /ORA/dbs00/oracle/admin
    • mv /ORA/dbs01/oracle/admin/[DBNAME] /ORA/dbs00/oracle/admin
  • sqlsys_DB -> Ex: create spfile='$ORACLE_HOME/dbs/spfile[DBNAME].ora' from pfile='/tmp/initdb.txt';

change [DBNAME] with the appropriate value

*.compatible='10.2.0.3'
*.db_block_size=8192
*.db_create_file_dest='/ORA/dbs02/oracle/oradata' 
*.db_domain='cern.ch'
# autotuned in 10.2 -> delete the entry from spfile  for 
*.db_file_multiblock_read_count 
*.db_name=
*.db_recovery_file_dest='/ORA/dbs02/oracle/flash_recovery'
*.db_recovery_file_dest_size=2048g
# only if planning to use XDB for ftp *.dispatchers=.'(PROTOCOL=TCP) (SERVICE=[DBNAME]XDB)' 
*.global_names=TRUE
*.job_queue_processes=10
*.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST'
*.log_archive_format='log_%t_%s_%r.arc'
*.log_buffer=10485760 
*.open_cursors=300
*.parallel_max_servers=0  # if streams parallel propagation is needed set this >0
*.pga_aggregate_target=1400m
*.processes=800
*.recyclebin=OFF
*.remote_listener='...listener_alias_here....'
*.remote_login_passwordfile='exclusive'
*.resource_limit=TRUE
*.sga_target=2200m
*.streams_pool_size=256m
*.undo_management='AUTO'
*.undo_retention=3600
*.audit_file_dest='/ORA/dbs00/oracle/admin/[DB_NAME]/adump'
*.core_dump_dest='/ORA/dbs00/oracle/admin/[DB_NAME]/cdump'
*.background_dump_dest='/ORA/dbs00/oracle/admin/[DB_NAME]/bdump'
*.user_dump_dest='/ORA/dbs00/oracle/admin/[DB_NAME]/udump'
*.audit_trail='db'

4 instance-specific parameters, typically set correctly by dbca. 
There is one entry per parameter per instance:
instance_number, local_listener, thread, undo_tablespace

Post Installation

• Check Hugepage memory allocation (if Oracle cannot allocate hugepages it will silently use 'normal' memory)
  • check hugetbs allocation in the last 3 rows of more /proc/meminfo

• Apply catcpu scripts from the latest security patch where relevant. Ex:

       cd $ORACLE_HOME/cpu/CPU...2006
       sqlsys_DB
       SQL> select count(*) from dba_objects where status='INVALID';
       SQL> @catcpu.sql
       SQL> @?/rdbms/admin/utlrp.sql
       SQL> select count(*) from dba_objects where status='INVALID';
       SQL> exit
  

• change redo log size, number and multiplexing as appropriate. Ex: add 5 redo groups per thread, no multiplexing, redo size 1g and drop old redologs (select group#,thread#,bytes/1024/1024 sizeMB from v$log order by 2,1;)

     alter database add logfile thread 1 group 11 size 1g;
     ...
     -drop old redologs
     alter system switch logfile; 
     alter system checkpoint global;
     (alternative: alter system archive log all)
     alter database drop logfile group 1; 
     ...
     

• change undo and temp tbs size as appropriate
• (optional) revoke from public unneeded privs, such as execute on vulnerable packages

       revoke execute on sys.utl_tcp from public;
       revoke execute on sys.utl_http from public;
       revoke execute on sys.utl_smtp from public;
       revoke execute on sys.dbms_export_extension from public;
     

• Edit tnsnames.ora
  • local tnsnames in particular the service_name parameter (add .cern.ch where appropriate)
  • afs version

Other post installation

• Install EM agent in a separate Oracle_Home
• setup RAC services and tns_names aliases
• setup logrotation
• setup account monitoring
• Setup backup (TSM client)
• revoke privileges from public
• add to service and asm monitoring
• see also the post install steps in the PSS wiki 'dba area'

• Created L.C., E.Version 1 - first version, Luca.Canali -AT- cern.ch Sep-2005
• Version 2 - fully updated, revised and tested on 10g R2, Luca Feb-2006
• Version 2.1 - Minor revisions, Luca Apr 2006
• Version 2.2 - Added entry for /etc/rc.local, Luca,Nov 2006
• Version 3.0 - Updated for RHEL4, L.C. Nov 2006
• Version 3.1 - Added device mapper multipath and removed asmlib, L.C. Dec 2006
• Version 3.2 - Changed ssh installation and added fix for CRS Bug 5722352 + rdbms patch 5705795, L.C. Jan 2007