Automatic Recoveries

Automatic test recoveries are indispensable element of any backup system. They do not only help to ensure recoverability but also to detect possible corruptions and other problem that otherwise could have not been easily spotted. Automatic recoveries are especially important in Oracle environment where backups are being taken inline with transactions modifying data and where single backup strategy may consist of many different types of backups.

This Document shows how to configure hardware and software for test recoveries and finally how to schedule and run recoveries themselves.

Configuration of the recovery system

• Dedicate hardware for the test recovery system. Minimum requirements are:
  • at least one mid-range server
  • amount of disk arrays sufficient to fit the biggest database that is going to be recovered. Disk arrays can be configured with RAID5. Flash recovery area is not required
• Configure and install recovery system. Follow installation instructions available here:
  • configure hardware,
  • configure OS, storage, multipathing and oracle account equivalence (in case you have more then 1 mid-range server),
  • install Oracle clusterware (the same version as on production). Name the cluster recor.
  • install Oracle RDBMS software (the same version as on production)
    • use cloning
  • configure listeners using netca,
  • configure ASM instances and tune ASM initialization parameters.
  • create a diskgroup e.g. RECOR_DATADG1
• Configure TSM/TDPO clients on the recovery system nodes:
  • NOTE: TSM/TDPO clients on the test recovery system should be configured configured differently from ordinary RAC nodes. The main difference is that instead of having only one set of dsm.opt and tdpo.opt configuration files there should be one such set per every single database that is supposed to be recovered. For every database that will be recovered there must be also a password file generated. The configuration files should stick to the following naming convention: dsm_DB_NAME.opt, tdpo_DB_NAME.opt.
  • download and untar the archive containing configuration files for major databases.

    wget https://twiki.cern.ch/twiki/pub/PSSGroup/AutomaticRecovery/RECOR_tivoli_config.tgz
    sudo tar xvzPf RECOR_tivoli_config.tgz
          

  • Check if configuration files exist for all databases that will be recovered:

    ls /opt/tivoli/tsm/client/oracle/bin64/
          

  • If there are some databases missing create appropriate configuration files:

    # for each DB_NAME database:
    sudo cp /opt/tivoli/tsm/client/oracle/bin64/dsm.opt /opt/tivoli/tsm/client/oracle/bin64/dsm_DB_NAME.opt
    sudo vi /opt/tivoli/tsm/client/oracle/bin64/dsm_DB_NAME.opt  # set properly name of the TSM server
    sudo cp /opt/tivoli/tsm/client/oracle/bin64/tdpo.opt /opt/tivoli/tsm/client/oracle/bin64/tdpo_DB_NAME.opt
    sudo vi /opt/tivoli/tsm/client/oracle/bin64/tdpo_DB_NAME.opt  # set properly DSMI_ORC_CONFIG and TDPO_NODE variables
    sudo tdpoconf password -TDPO_OPTfile=/opt/tivoli/tsm/client/oracle/bin64/tdpo_DB_NAME.opt
          

• Configure the recovery script:
  • Go to the /backup5/scripts/automatic_recovery directory on the pdb-backup machine and edit the auto_full_recovery.sh script":

    ssh oracle@pdb-backup
    cd /backup5/scripts/automatic_recovery
    vi auto_full_recovery.sh
          

  • Set properly values of the following variables:

    RECOVERY_MACHINE= # should be set to the name of the machine hosting Oracle instance of the recovered database
    RECOVERY_ASM_INSTANCE= # should be set to the name of the ASM instance running on the recovery machine
    RECOVERY_ASM_DISKGROUP= # should be set to the name of the ASM diskgroup that will be used to store recovered data
          

  • In case new databases have been added, make sure that their DBIDs have been added to the dbids.txt file.

Scheduling and running test recoveries

The automatic recovery infrastructure is well integrated with the existing backup infrastructure. Recovery script is supposed to run on the pdb-backup machine and use the same directory structure i.e. /backup5/db_name/... as backup scripts. It is also ready to be run as cron jobs.

The recovery script is named auto_full_recovery.sh and resides in the /backup5/scripts/automatic_recovery directory on the pdb-backup machine. The script accepts 3 arguments:

1. Name of the database (obligatory) e.g. cmsr
2. Recovery point in time (obligatory). This parameter has to be specified in a format understandable to RMAN e.g. "sysdate-1" or "to_date('2007-JUN-24 20:00:00','YYYY-MON-DD HH24:MI:SS')"
3. Recovery parallelism (optional). Default value is 1.

Examples:

/backup5/scripts/automatic_recovery/auto_full_recovery.sh cmsr "sysdate-2" 3
/backup5/scripts/automatic_recovery/auto_full_recovery.sh atlr "to_date('2007-JUN-24 20:00:00','YYYY-MON-DD HH24:MI:SS')" 2
/backup5/scripts/automatic_recovery/auto_full_recovery.sh lcgr "sysdate-1"

The script does the following operations:

• Performs a cleanup of the recovery system.
  • kills database instance with the same name as the one that is going to be created
  • removes instance dump directories
  • removes init and password files with the same names as those that are going to be created
• Creates appropriate dump directories
• Performs health/configuration checks of the recovery system
• Creates a password file
• Restores the spfile
• Restores control files
• Restores and recovers data files
• Produces a log in the /backup5/db_name/logs directory

Useful files

• RECOR_tivoli_config.tgz: TSM/TDPO configuration files for the RECOR system