10g RAC on Linux, adding cluster nodes

This is an addendum to the installation procedure in Installation procedure

Complete the "OS Setup and Prerequisites" as in Installation procedure. Note the following extra comments:

• "Configure the private interconnect", edit /etc/hosts on new and old nodes
• " Setup ssh and host equivalence" (.ssh dir), edit on new and old nodes
• other OS operations as in the install guide
• configure multipathing on the new nodes
  • prereq: configure zoning to make the DB shared storage visible to the new nodes
• copy and run the configuration for rawdevices (/etc/sysconfig/devlabel, /etc/init.d/devlabel + chown for eraw devices, see install guide)

Add node(s) to the clusterware

• from an existing cluster node: cd $ORA_CRS_HOME/oui/bin; ./addNode.sh
• select new nodes and click next and then install
• run the scripts on new and old nodes as requested by the pop-up window
  • note: if the node is added after a node removal operation, check the vi $ORA_CRS_HOME/install/rootconfig and update node information where relevant
• $ORA_CRS_HOME/bin/racgons add_config NEW_HOST_NAME:4948
• apply the fix for /etc/init.d/init.cssd as described in the RAC installation procedure

Add node(s) to ASM and the DB

• install rdbms binaries as in Installation procedure using the cloning procedure (see also below for the syntaxmake sure you are using the same Oracle version + patches)
• copy $ORACLE_HOME/dbs from an existing home and do the necessary modifications + create pass filw on the new nodes
• run netca to configure the listener on the new nodes (only)
• check listener.ora (remove exptroc)
• edit tnsnames.ora (add aliases for dbs and listener entries) and distribute to all the cluster nodes
• update the node list on the 'old nodes' with oraInstaller

Update the OLD nodes:
cd $ORACLE_HOME/oui/bin
./runInstaller -silent -updateNodeList CLUSTER_NODES="NODE1,NODE2,NODE3,NODE4" ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME="OraDb10g_rdbms" LOCAL_NODE="NODEXXX"

attach the oracle home on the new nodes (if not already done above)
cd $ORACLE_HOME/clone/bin;
perl clone.pl ORACLE_HOME="/ORA/dbs01/oracle/product/10.2.0/rdbms" ORACLE_HOME_NAME="OraDb10g_rdbms" '-O"CLUSTER_NODES={itrxx,itryy}"' '-O"LOCAL_NODE=itracxx"'

• edit /etc/oratab on the new nodes
• copy the admin directory structure in /ORA/dbs00/oracle/admin/...
• edit ASM spfile
  • instance-dependent parameters: cluster_database_instances,instance_number, local_listener
• add asm instances to CRS and start them

srvctl add asm -n ... -i +ASM3 -o $ORACLE_HOME

• edit DB spfile
  • instance-dependent parameters: instance_number,thread,undo_tablespace,local_listener
  • DB parameters: cluster_database_instances (should be set to the number of nodes, but can be different)
• add the undo tablespaces for the new nodes
• add the redo threads for the new nodes
• add database instances to CRS and add the dependency ASM-DB instance

srvctl add instance -d DBNAME -i INSTNAME -n ...
srvctl modify instance -d {DB_NAME} -i {INSTANCE_NAME} -s  {ASM_INSTANCE_NAME} ...

• startup using srvctl
• modify services to include the new nodes (don't use dbca)

srvctl stop service -d ... -s .... 
srvctl modify service -d ... -s .... 
srvctl start service -d ..

• edit /etc/logrotate.d/ora_cern_listener_udump_bdump_adump_rotate
• on all nodes of the cluster grant the 'sysdba' privilege to the 'pdb_admin' user.
• update /etc/oratab files on all nodes of the cluster
• install or reattach the Oracle_HOME for the EM agent

• Perform a full backup of the DB when finished