Operations Support Document for PDB Level 1

*Please make sure you have read the documents linked in

• SupportLevelRoles
• OperationsManual

General system status:

• Check the web page with list of servers and attached storage:
  • Last known URL for that page is https://twiki.cern.ch/twiki/bin/view/PSSGroup/DeployedRACs
  • Reports name of the servers
  • Storages used *by ASM for the data and flash recovery group
• Connect to the OEM via browser to check system activity (server CPU and DB active sessions)
• Linux command line: sar -u 1 100 to measure CPU load
• Linux command line: sar -b 1 100 to measure CPU load
• Linux command line: �ps �ef|grep pmon� -> must output 2 lines, one for ASM and one for the DB instance
• Connect to the DB instance with �sqlsys_DB� (see below for some useful sqlplus commands)

Relevant log files:

• DB alert log
  • Type 'bdump' to go to the directory. The file name is alert*.log
• ASM alert log
  • Type 'bdumpasm' to go to the directory. The file name is alert*.log
• Listener log
  • �cd $ORACLE_HOME/network/log�. The file is listener*.log
• System�s log: /var/log/messages
  • �sudo less /var/log/messages�

Healthcheck of the Oracle clusterware:

• crsstat.sh
• srvctl start service -d dbname can be used to start manually all services for a given db (services which are already up are not touched)
• srvctl start service -d dbname -i instancename -s servicename can be necessary to startup up "stubborn" services

Other clusterware operations:

• srvctl start/stop can be used to start and stop resources managed by the Oracle clusterware. See the syntax for srvctl status above.

Some checks with sqlplus:

• type sqlsys_DB to connect to the db instance with sqlplus and command history
• Check that instances are not close to the maximum number of concurrentprocesses (500 processes per instance):

select inst_id,count(*) from gv$process group by inst_id;

• Check activity on the cluster or drill down on instances close to max processes:

select inst_id,username,osuser,program,last_call_et from gv$session where username is not null;

• check blocking locks:

select * from dba_blockers;

• check for long running ops:

select inst_id,username,time_remaining remaining, elapsed_seconds elapsed from gv$session_longops where time_remaining >0;

• type 'sqlsys_ASM' to connect to the ASM instance with sqlplus and command history
  • List disks and their status: @listdisks
  • List disk groups and their status: select * from v$asm_diskgroup;
  • List ASM rebalancing operations: select * from gv$asm_operation;

Checking the listener on RAC:

• Run ps -elf | grep tns | grep -v grep to find out if the listener process is up.
• Run the crsstat.sh command to find out what the status of the listener according to the clusterware.

Stopping the listener the listener on RAC:

• srvctl stop listener -n XXX (where XXX is name of the node e.g. itrac09)
• If the command above does not work execute: lsrnctl stop listener_XXX (where XXX is name of the node e.g. itrac09)
• If none of the commands above does not work find out listener process pids (ps -ef|grep tnslsnr) and kill it with kill -9

Starting the listener the listener on RAC:

• If the listener is down according to the crsstat.sh command execute: srvctl start listener -n XXX (where XXX is name of the node e.g. itrac09)
• If the listener listener process does not exist and the crsstat.sh command claims that the listener is up: lsrnctl start listener_XXX