Adding nodes to an existing RAC version 10g R1

Purpose

This piece of documentation describes the procedure of adding new nodes to an existing RAC in a way transparent for the database users.

Steps

• Clear all new nodes by removing Oracle software and all Oracle related files. For how to do this look here.

• Decide on what names will have new ASM and DB instances set environment variables that will be used later on. On ALL_NEW_NODES copy over, edit (names of instances) and reload .bashrc file from any of existing nodes of the cluster:

         on each NODE (new and old):
           export OLD_NODE_1=<hostname of the first existing node of the cluster>
           export OLD_NODE_2=<hostname of the second existing node of the cluster>     
           ...
           export NEW_NODE_1=<hostname of the first machine supposed to be added to the cluster>
           export NEW_NODE_2=<hostname of the second machine supposed to be added to the cluster>
           ...
           export ASM_OLD_NAME_1=<name of the asm instance on the first old node>
           export ASM_OLD_NAME_2=<name of the asm instance on the second old node>
           ...
           export ASM_NEW_NAME_1=<name of the asm instance on the first new node>
           export ASM_NEW_NAME_2=<name of the asm instance on the second new node>
           ...
          
         on each NEW_NODE:
           scp oracle@${OLD_NODE_1}:.bashrc ~/
           vi ~/.bashrc
           source ~/.bashrc
       

• Assign private IP addresses to new nodes. Check here to which subnet new IPs should belong:

         on each NEW_NODE:
         sudo vi /etc/sysconfig/network-scripts/ifcfg-eth1    :
           DEVICE=eth1
           BOOTPROTO=static
           IPADDR=192.168.XXX.XXX
           NETMASK=255.255.255.0
           ONBOOT=yes
           TYPE=Ethernet
  
         sudo vi /etc/sysconfig/network-scripts/ifcfg-eth2    :
           DEVICE=eth2
           BOOTPROTO=static
           IPADDR=192.168.XXX.XXX
           NETMASK=255.255.255.0
           ONBOOT=yes
           TYPE=Ethernet
  
         sudo /etc/rc.d/init.d/network restart
       

• On each NODE (new and old) modify /etc/hosts file. The best way to do this is to update the file on one of existing RAC nodes by adding entries describing private, public and virtual interfaces of new nodes and then to distribute this file to all RAC nodes.

• Generate RSA and DSA keys on new nodes (use default location and set empty passphrase). Prepare .ssh/authorized_keys.local and .ssh/authorized_keys files and distribute them to all nodes of the cluster:

         On each NEW_NODE:
           /usr/bin/ssh-keygen -t rsa
           /usr/bin/ssh-keygen -t dsa
        
           vi ~oracle/.ssh/config
             Host *
             ForwardX11 no
             StrictHostKeyChecking=no
  
         On OLD_NODE_1:
           wget https://twiki.cern.ch/twiki/pub/PSSGroup/Installation_verbose/clu-ssh
           chmod +x clu-ssh
           echo "$OLD_NODE_1 $OLD_NODE_2 $NEW_NODE_1 $NEW_NODE_2" > racnodes.db 
           ./clu-ssh racnodes "cat ~/.ssh/id_rsa.pub ~/.ssh/id_dsa.pub" >> ~/.ssh/authorized_keys.local
           cat ~/.ssh/authorized_keys.local >> ~/.ssh/authorized_keys
           scp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.local ${OLD_NODE_2}:~/.ssh
           scp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.local ${NEW_NODE_1}:~/.ssh
           scp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.local ${NEW_NODE_2}:~/.ssh
       

• on OLD_NODE_1 edit the file .ssh/known_hosts. For each entry add a comma separated (no space) list of IPs and host aliases for public and private networks. Example of complete entry: itrac29,itrac29.cern.ch,137.138.35.79,itrac29-v,compr-priv1-1,compr-priv2-1,172.31.5.1,172.31.6.1 ssh-rsa AAAAB3NzaC...

• Deploy modified .ssh/known_hosts file to all cluster nodes using scp and check the configuration:

         scp ~/.ssh/known_hosts ${OLD_NODE_2}:~/.ssh
         scp ~/.ssh/known_hosts ${NEW_NODE_1}:~/.ssh
         scp ~/.ssh/known_hosts ${NEW_NODE_2}:~/.ssh
         ssh PRIVATE_HOSTNAME_OF_ONE_OF_THE_RAC_NODES date
       

• On each NODE (old and new) remove the /etc/issue.net file:

         sudo rm -f /etc/issue.net
       

• on each NEW_NODE configure the hangcheck-timer module and PAM:

         sudo echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modules.conf
         sudo echo "session required /lib/security/pam_limits.so" >> /etc/pam.d/login
       

• Change the FC switche zones (if needed) in a way that new nodes can see te same storage arrays as existing cluster nodes.

• As root on each NEW_NODE copy contents of /etc/sysconfig/devlabel file from any of the old nodes. Reload devlabel. If needed configure ASMLib and rescan disks:

         on each NEW_NODE:
           sudo vi /etc/sysconfig/devlabel
           sudo /sbin/devlabel reload
           sudo /etc/init.d/oracleasm configure
           sudo /etc/init.d/oracleasm scandisks
           sudo /etc/init.d/oracleasm listdisks
       

• From the first node of the cluster start the Clusterware clonning procedure. Follow the Oracle documentation.

         on the OLD_NODE_1:
         cd $ORA_CRS_HOME/oui/bin
         ./addNode.sh
         ...
       

Note 1: Some strange, ignorable errors (during execution of the lsnode.sh script) can pop up at the end.

Note 2: Instead of executing the command mentioned in the point 9 in the Oracle documentation run the following command:

        on OLD_NODE_1:
        cat $ORACLE_HOME/opmn/conf/ons.config
      

        on OLD_NODE_1:
        $ORA_CRS_HOME/bin/racgons add_config ${NEW_NODE_1}:REMOTE_PORT_SELECTED_ABOVE ${NEW_NODE_2:}REMOTE_PORT_SELECTED_ABOVE ...
      

• On the first node of the cluster start the RAC software clonning procedure. Follow the Oracle documentation.

         cd $ORACLE_HOME/oui/bin
         ./addNode.sh
         ...
       

Note 1: In point 9, become root on any of the cluster nodes and run vipca only for new nodes.

       sudo su -
       export DISPLAY=YOUR_PC_NAME:0.0
       cd /ORA/dbs01/oracle/product/10.1.0/rdbms/bin
       ./vipca -nodelist ${NEW_NODE_1},${NEW_NODE_2},...
(note choose to configure the VIP only on eth0)
     

Note 2: In point 10 check with oifcfg tool if cluster interconnect is specified correctrly:

       on OLD_NODE_1:
         oifcfg getif
     

• On one of the new nodes run Network Configuration Assistant (netca) to configure listeners on new nodes.
  • (new nodes) edit listener.ora to remove old listener entries and the extproc config
  • (all nodes) edit tnsnames.ora, in particular the entry LISTENER_{DB_NAME} used by RAC load balancing

• On any of the OLD_NODES connect to the ASM instance and update the spfile.

         on OLD_NODE_1:
         sqlsys_ASM
         for each new node:
         SQL> alter system set instance_number=<number of the instance> scope=spfile sid='<name_of_the_ASM_instance>';
  
         e.g. SQL> alter system set instance_number=3 scope=spfile sid='+ASM3';
       

• On each NEW_NODE create an ASM pfile pointing to the raw device with the spfile:

         on NEW_NODE_1:
           cd $ORACLE_HOME/dbs
           echo "spfile=/dev/raw/raw3" > init${ASM_NEW_NAME_1}.ora
  
         on NEW_NODE_2:
           cd $ORACLE_HOME/dbs
           echo "spfile=/dev/raw/raw3" > init${ASM_NEW_NAME_2}.ora
  
         ...
       

• On each NEW_NODE create password files for ASM instances:

         on NEW_NODE_1:
           cd $ORACLE_HOME/dbs
           orapwd file=./orapw${ASM_NEW_NAME_1} password=xxxxxx entries=20
  
         on NEW_NODE_2:
           cd $ORACLE_HOME/dbs
           orapwd file=./orapw${ASM_NEW_NAME_2} password=xxxxxx entries=20
  
         ...
       

• Create ASM logging directories if they do not exist:

         on each NEW_NODE:
           mkdir -p /ORA/dbs00/oracle/admin/+ASM/adump
           mkdir -p /ORA/dbs00/oracle/admin/+ASM/bdump
           mkdir -p /ORA/dbs00/oracle/admin/+ASM/cdump
           mkdir -p /ORA/dbs00/oracle/admin/+ASM/udump
       

• Register new ASM instances to the Cluster Registry and start them up:

         On any NODE for each NEW_NODE run:
           $ORACLE_HOME/bin/srvctl add asm -n ${NEW_NODE_1} -i ${ASM_NEW_NAME_1} -o ${ORACLE_HOME}
           $ORACLE_HOME/bin/srvctl enable asm -n ${NEW_NODE_1} -i ${ASM_NEW_NAME_1}
           $ORACLE_HOME/bin/srvctl start nodeapps -n ${NEW_NODE_1}
           $ORACLE_HOME/bin/srvctl start asm -n ${NEW_NODE_1}
  
           crsstat.sh       
       

• inspect the spfile, in particular for the asm_diskgroups parameter

        create pfile='/tmp/asmpfile33.txt' from spfile='/dev/raw/raw3';
       

• Connect to each new ASM instance and check whether desired diskgroups have been mounted:

         on each NEW_NODE:
           sqlsys_ASM
           SQL> SELECT * FROM V$ASM_DISKGROUP;
           SQL> exit
       

• From one of the old instances modify the spfile of the DB. Add entries related to instances being added:

         from OLD_NODE_1, for each NEW_INSTANCE run:
         sqlsys_DB
         SQL> alter system set instance_number=<new_DB_instance_number> scope=spfile sid='<new_instance_sid>';
         SQL> alter system set thread=<new_DB_instance_number> scope=spfile sid='<new_instance_sid>';  
       

• From one of the old instances create UNDO tablespaces and redo logfiles for new instances:

         for each NEW_INSTANCE run:
         sqlsys_DB
         SQL> select tablespace_name from dba_tablespaces where tablespace_name like 'UNDO%';
         SQL> create undo tablespace <name_of_the_undo_tablespace> datafile size 1G autoextend on next 100M maxsize 10G  extent management local;
         SQL> alter system set undo_tablespace='<name_of_the_undo_tablespace>' scope=spfile sid='<new_instance_sid>';
  
         SQL> select * from v$log;
         SQL> alter database add logfile thread <new_DB_instance_number> group <group_number> size 500M;
         SQL> alter database add logfile thread <new_DB_instance_number> group <group_number> size 500M;
         SQL> alter database add logfile thread <new_DB_instance_number> group <group_number> size 500M;
         SQL> alter database add logfile thread <new_DB_instance_number> group <group_number> size 500M;
         SQL> alter database add logfile thread <new_DB_instance_number> group <group_number> size 500M;
         SQL> alter database enable public thread <new_DB_instance_number>
       

• For each NEW_DB_INSTANCE copy over the init file from any of the old nodes, create a password file, register the instance in the OCR, create logging directories, and start the instance up:

         on each NEW_NODE:
         scp OLD_NODE_1:$ORACLE_HOME/dbs/init<old_instance_sid>.ora $ORACLE_HOME/dbs/init<new_instance_sid>.ora
         orapwd file='/ORA/dbs01/oracle/product/10.1.0/rdbms/dbs/orapw<new_instance_sid>' password=xxxxx entries=20
         mkdir -p /ORA/dbs00/oracle/admin/DB_NAME/adump
         mkdir -p /ORA/dbs00/oracle/admin/DB_NAME/bdump
         mkdir -p /ORA/dbs00/oracle/admin/DB_NAME/cdump
         mkdir -p /ORA/dbs00/oracle/admin/DB_NAME/udump
  
         srvctl add instance -d <db_name> -i <new_instance_sid> -n <new_node_name>
         srvctl start instance -d <db_name> -i <new_instance_sid>
         sqlsys_DB
         SQL> grant sysdba to pdb_admin;
         SQL> exit
       

• Configure EM agents, register aliases, update services, update public tnsnames.ora files, configure monitoring, implement logrotation

• sample service config

        srvctl config service -d $DB_NAME (displays config)
        srvctl modify service -d $DB_NAME -s lcg_fts -n -i lcgr1 -a "lcgr2,lcgr3,lcgr4"