WARNING: This web is not used anymore. Please use PDBService.Adding_nodes instead!
 

Adding nodes to an existing RAC version 10g R1

Note for 10gR2 changes are needed

Purpose

This piece of documentation describes the procedure of adding new nodes to an existing RAC in a way transparent for the database users.

Steps

  • Clear all new nodes by removing Oracle software and all Oracle related files. For how to do this look here.

  • Decide on what names will have new ASM and DB instances set environment variables that will be used later on. On ALL_NEW_NODES copy over, edit (names of instances) and reload .bashrc file from any of existing nodes of the cluster:
           on each NODE (new and old):
             export OLD_NODE_1=<hostname of the first existing node of the cluster>
             export OLD_NODE_2=<hostname of the second existing node of the cluster>     
             ...
             export NEW_NODE_1=<hostname of the first machine supposed to be added to the cluster>
             export NEW_NODE_2=<hostname of the second machine supposed to be added to the cluster>
             ...
             export ASM_OLD_NAME_1=<name of the asm instance on the first old node>
             export ASM_OLD_NAME_2=<name of the asm instance on the second old node>
             ...
             export ASM_NEW_NAME_1=<name of the asm instance on the first new node>
             export ASM_NEW_NAME_2=<name of the asm instance on the second new node>
             ...
            
           on each NEW_NODE:
             scp oracle@${OLD_NODE_1}:.bashrc ~/
             vi ~/.bashrc
             source ~/.bashrc
         

  • Assign private IP addresses to new nodes. Check here to which subnet new IPs should belong:
           on each NEW_NODE:
           sudo vi /etc/sysconfig/network-scripts/ifcfg-eth1    :
             DEVICE=eth1
             BOOTPROTO=static
             IPADDR=192.168.XXX.XXX
             NETMASK=255.255.255.0
             ONBOOT=yes
             TYPE=Ethernet
    
           sudo vi /etc/sysconfig/network-scripts/ifcfg-eth2    :
             DEVICE=eth2
             BOOTPROTO=static
             IPADDR=192.168.XXX.XXX
             NETMASK=255.255.255.0
             ONBOOT=yes
             TYPE=Ethernet
    
           sudo /etc/rc.d/init.d/network restart
         

  • On each NODE (new and old) modify /etc/hosts file. The best way to do this is to update the file on one of existing RAC nodes by adding entries describing private, public and virtual interfaces of new nodes and then to distribute this file to all RAC nodes.

  • Generate RSA and DSA keys on new nodes (use default location and set empty passphrase). Prepare .ssh/authorized_keys.local and .ssh/authorized_keys files and distribute them to all nodes of the cluster:
           On each NEW_NODE:
             /usr/bin/ssh-keygen -t rsa
             /usr/bin/ssh-keygen -t dsa
          
             vi ~oracle/.ssh/config
               Host *
               ForwardX11 no
               StrictHostKeyChecking=no
    
           On OLD_NODE_1:
             wget https://twiki.cern.ch/twiki/pub/PSSGroup/Installation_verbose/clu-ssh
             chmod +x clu-ssh
             echo "$OLD_NODE_1 $OLD_NODE_2 $NEW_NODE_1 $NEW_NODE_2" > racnodes.db 
             ./clu-ssh racnodes "cat ~/.ssh/id_rsa.pub ~/.ssh/id_dsa.pub" >> ~/.ssh/authorized_keys.local
             cat ~/.ssh/authorized_keys.local >> ~/.ssh/authorized_keys
             scp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.local ${OLD_NODE_2}:~/.ssh
             scp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.local ${NEW_NODE_1}:~/.ssh
             scp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.local ${NEW_NODE_2}:~/.ssh
         

  • on OLD_NODE_1 edit the file .ssh/known_hosts. For each entry add a comma separated (no space) list of IPs and host aliases for public and private networks. Example of complete entry: itrac29,itrac29.cern.ch,137.138.35.79,itrac29-v,compr-priv1-1,compr-priv2-1,172.31.5.1,172.31.6.1 ssh-rsa AAAAB3NzaC...

  • Deploy modified .ssh/known_hosts file to all cluster nodes using scp and check the configuration:
           scp ~/.ssh/known_hosts ${OLD_NODE_2}:~/.ssh
           scp ~/.ssh/known_hosts ${NEW_NODE_1}:~/.ssh
           scp ~/.ssh/known_hosts ${NEW_NODE_2}:~/.ssh
           ssh PRIVATE_HOSTNAME_OF_ONE_OF_THE_RAC_NODES date
         

  • On each NODE (old and new) remove the /etc/issue.net file:
           sudo rm -f /etc/issue.net
         

  • on each NEW_NODE configure the hangcheck-timer module and PAM:
           sudo echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modules.conf
           sudo echo "session required /lib/security/pam_limits.so" >> /etc/pam.d/login
         

  • Change the FC switche zones (if needed) in a way that new nodes can see te same storage arrays as existing cluster nodes.

  • As root on each NEW_NODE copy contents of /etc/sysconfig/devlabel file from any of the old nodes. Reload devlabel. If needed configure ASMLib and rescan disks:
           on each NEW_NODE:
             sudo vi /etc/sysconfig/devlabel
             sudo /sbin/devlabel reload
             sudo /etc/init.d/oracleasm configure
             sudo /etc/init.d/oracleasm scandisks
             sudo /etc/init.d/oracleasm listdisks
         

  • From the first node of the cluster start the Clusterware clonning procedure. Follow the Oracle documentation.
           on the OLD_NODE_1:
           cd $ORA_CRS_HOME/oui/bin
           ./addNode.sh
           ...
         

Note 1: Some strange, ignorable errors (during execution of the lsnode.sh script) can pop up at the end.

Note 2: Instead of executing the command mentioned in the point 9 in the Oracle documentation run the following command:

        on OLD_NODE_1:
        cat $ORACLE_HOME/opmn/conf/ons.config
      
Then use the remote port number found above to run the following command:
        on OLD_NODE_1:
        $ORA_CRS_HOME/bin/racgons add_config ${NEW_NODE_1}:REMOTE_PORT_SELECTED_ABOVE ${NEW_NODE_2:}REMOTE_PORT_SELECTED_ABOVE ...
      

  • On the first node of the cluster start the RAC software clonning procedure. Follow the Oracle documentation.
           cd $ORACLE_HOME/oui/bin
           ./addNode.sh
           ...
         

Note 1: In point 9, become root on any of the cluster nodes and run vipca only for new nodes.

       sudo su -
       export DISPLAY=YOUR_PC_NAME:0.0
       cd /ORA/dbs01/oracle/product/10.1.0/rdbms/bin
       ./vipca -nodelist ${NEW_NODE_1},${NEW_NODE_2},...
(note choose to configure the VIP only on eth0)
     

Note 2: In point 10 check with oifcfg tool if cluster interconnect is specified correctrly:

       on OLD_NODE_1:
         oifcfg getif
     

  • On one of the new nodes run Network Configuration Assistant (netca) to configure listeners on new nodes.
    • (new nodes) edit listener.ora to remove old listener entries and the extproc config
    • (all nodes) edit tnsnames.ora, in particular the entry LISTENER_{DB_NAME} used by RAC load balancing

  • On any of the OLD_NODES connect to the ASM instance and update the spfile.
           on OLD_NODE_1:
           sqlsys_ASM
           for each new node:
           SQL> alter system set instance_number=<number of the instance> scope=spfile sid='<name_of_the_ASM_instance>';
    
           e.g. SQL> alter system set instance_number=3 scope=spfile sid='+ASM3';
         

  • On each NEW_NODE create an ASM pfile pointing to the raw device with the spfile:
           on NEW_NODE_1:
             cd $ORACLE_HOME/dbs
             echo "spfile=/dev/raw/raw3" > init${ASM_NEW_NAME_1}.ora
    
           on NEW_NODE_2:
             cd $ORACLE_HOME/dbs
             echo "spfile=/dev/raw/raw3" > init${ASM_NEW_NAME_2}.ora
    
           ...
         

  • On each NEW_NODE create password files for ASM instances:
           on NEW_NODE_1:
             cd $ORACLE_HOME/dbs
             orapwd file=./orapw${ASM_NEW_NAME_1} password=xxxxxx entries=20
    
           on NEW_NODE_2:
             cd $ORACLE_HOME/dbs
             orapwd file=./orapw${ASM_NEW_NAME_2} password=xxxxxx entries=20
    
           ...
         

  • Create ASM logging directories if they do not exist:
           on each NEW_NODE:
             mkdir -p /ORA/dbs00/oracle/admin/+ASM/adump
             mkdir -p /ORA/dbs00/oracle/admin/+ASM/bdump
             mkdir -p /ORA/dbs00/oracle/admin/+ASM/cdump
             mkdir -p /ORA/dbs00/oracle/admin/+ASM/udump
         

  • Register new ASM instances to the Cluster Registry and start them up:
           On any NODE for each NEW_NODE run:
             $ORACLE_HOME/bin/srvctl add asm -n ${NEW_NODE_1} -i ${ASM_NEW_NAME_1} -o ${ORACLE_HOME}
             $ORACLE_HOME/bin/srvctl enable asm -n ${NEW_NODE_1} -i ${ASM_NEW_NAME_1}
             $ORACLE_HOME/bin/srvctl start nodeapps -n ${NEW_NODE_1}
             $ORACLE_HOME/bin/srvctl start asm -n ${NEW_NODE_1}
    
             crsstat.sh       
         
  • inspect the spfile, in particular for the asm_diskgroups parameter
          create pfile='/tmp/asmpfile33.txt' from spfile='/dev/raw/raw3';
         

  • Connect to each new ASM instance and check whether desired diskgroups have been mounted:
           on each NEW_NODE:
             sqlsys_ASM
             SQL> SELECT * FROM V$ASM_DISKGROUP;
             SQL> exit
         

  • From one of the old instances modify the spfile of the DB. Add entries related to instances being added:
           from OLD_NODE_1, for each NEW_INSTANCE run:
           sqlsys_DB
           SQL> alter system set instance_number=<new_DB_instance_number> scope=spfile sid='<new_instance_sid>';
           SQL> alter system set thread=<new_DB_instance_number> scope=spfile sid='<new_instance_sid>';  
         

  • From one of the old instances create UNDO tablespaces and redo logfiles for new instances:
           for each NEW_INSTANCE run:
           sqlsys_DB
           SQL> select tablespace_name from dba_tablespaces where tablespace_name like 'UNDO%';
           SQL> create undo tablespace <name_of_the_undo_tablespace> datafile size 1G autoextend on next 100M maxsize 10G  extent management local;
           SQL> alter system set undo_tablespace='<name_of_the_undo_tablespace>' scope=spfile sid='<new_instance_sid>';
    
           SQL> select * from v$log;
           SQL> alter database add logfile thread <new_DB_instance_number> group <group_number> size 500M;
           SQL> alter database add logfile thread <new_DB_instance_number> group <group_number> size 500M;
           SQL> alter database add logfile thread <new_DB_instance_number> group <group_number> size 500M;
           SQL> alter database add logfile thread <new_DB_instance_number> group <group_number> size 500M;
           SQL> alter database add logfile thread <new_DB_instance_number> group <group_number> size 500M;
           SQL> alter database enable public thread <new_DB_instance_number>
         

  • For each NEW_DB_INSTANCE copy over the init file from any of the old nodes, create a password file, register the instance in the OCR, create logging directories, and start the instance up:
           on each NEW_NODE:
           scp OLD_NODE_1:$ORACLE_HOME/dbs/init<old_instance_sid>.ora $ORACLE_HOME/dbs/init<new_instance_sid>.ora
           orapwd file='/ORA/dbs01/oracle/product/10.1.0/rdbms/dbs/orapw<new_instance_sid>' password=xxxxx entries=20
           mkdir -p /ORA/dbs00/oracle/admin/DB_NAME/adump
           mkdir -p /ORA/dbs00/oracle/admin/DB_NAME/bdump
           mkdir -p /ORA/dbs00/oracle/admin/DB_NAME/cdump
           mkdir -p /ORA/dbs00/oracle/admin/DB_NAME/udump
    
           srvctl add instance -d <db_name> -i <new_instance_sid> -n <new_node_name>
           srvctl start instance -d <db_name> -i <new_instance_sid>
           sqlsys_DB
           SQL> grant sysdba to pdb_admin;
           SQL> exit
         

  • Configure EM agents, register aliases, update services, update public tnsnames.ora files, configure monitoring, implement logrotation

  • sample service config
          srvctl config service -d $DB_NAME (displays config)
          srvctl modify service -d $DB_NAME -s lcg_fts -n -i lcgr1 -a "lcgr2,lcgr3,lcgr4"
          

Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r11 - 2006-03-10 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PSSGroup All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback