Monday, May 6, 2013

PL/SQL package PRICE_RMAN.DBMS_RCVCAT version 11.02.00.02 in RCVCAT database is not current


Problem Description

While starting Oracle RMAN backup whenever we use catalog database as repository it fails with following errors:
$ rman target / catalog catowner@rman_catdb

Recovery Manager: Release 11.2.0.3.0 - Production on Mon May 6 10:06:37 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: METHDEV (DBID=2414256617)
connected to recovery catalog database
PL/SQL package PRICE_RMAN.DBMS_RCVCAT version 11.02.00.02 in RCVCAT database is not current
PL/SQL package PRICE_RMAN.DBMS_RCVMAN version 11.02.00.02 in RCVCAT database is not current

Try to resync catalog, it fails with forllowing errors:

RMAN> resync catalog;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync command on default channel at 05/06/2013 10:07:05
RMAN-10015: error compiling PL/SQL program
RMAN-10014: PL/SQL error 0 on line 1655 column 12: Statement ignored
RMAN-10014: PL/SQL error 306 on line 1655 column 12: wrong number or types of arguments in call to 'CHECKTABLESPACE'

RMAN> exit


Recent Changes

The target database has been recently upgrade to 11.2.0.3 and so RMAN executable is upgraded but catalog database is not aware of this upgrade.

Cause of the Problem
The problem happened due to version mismatch between the RMAN-executable and the RMAN-catalog SCHEMA.

Solution of the Problem
Solution 01:
Upgrade the RMAN catalog SCHEMA. Start the RMAN-executable from the ORACLE_HOME which has been upgraded. There is only a connection to the CATALOG required.
A connection to the TARGET is optional.

For example issue,

$ rman catalog $RMAN_USERID/$RMAN_PASSWD@$RMAN_CONN_STR
RMAN> upgrade catalog;

Solution 02:
Don't use catalog for backup information. So run backup without connecting catalog database.

$ rman target /
$ backup database;


Friday, May 3, 2013

Upgrade to 11.2.0.3 - OCM: ORA-12012 and ORA-29280

Symptoms:

After updated database to 11.2.0.3, getting following error in alert log:

Errors in file /opt/oracle/diag/rdbms/db/trace/db_j001_26027.trc:
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_2"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1

Cause:

OCM is the Oracle Configuration Manager, a tool to proactively monitor your Oracle environment to provide this information to Oracle Software Support.
OCM is trying to write to a local directory which does not exist. Besides that the OCM version delivered with Oracle Database Patch Set 11.2.0.3 is older than the newest available OCM Collector 10.3.7 - the one which has that issue fixed.

Solutions:

you'll either drop OCM completely if you won't use it:

SQL> drop user ORACLE_OCM cascade;

you'll disable the collector jobs:

SQL> exec dbms_scheduler.disable('ORACLE_OCM.MGMT_CONFIG_JOB');
SQL> exec dbms_scheduler.disable('ORACLE_OCM.MGMT_STATS_CONFIG_JOB'); 

Refer to MOS Note: [ID 1453959.1]


Wednesday, May 1, 2013

ora.diskmon offline on 11.2.0.3


As Grid Infrastructure daemon diskmon.bin is used for Exadata fencing, started from 11.2.0.3, resource ora.diskmon will be offline in non-Exadata environment. This is expected behaviour change.


$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       pricedev1-lnx            Started             
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       pricedev1-lnx                                
ora.crf
      1        ONLINE  ONLINE       pricedev1-lnx                                
ora.crsd
      1        ONLINE  ONLINE       pricedev1-lnx                                
ora.cssd
      1        ONLINE  ONLINE       pricedev1-lnx                                
ora.cssdmonitor
      1        ONLINE  ONLINE       pricedev1-lnx                                
ora.ctssd
      1        ONLINE  ONLINE       pricedev1-lnx            OBSERVER            
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.drivers.acfs
      1        ONLINE  ONLINE       pricedev1-lnx                                
ora.evmd
      1        ONLINE  ONLINE       pricedev1-lnx                                
ora.gipcd
      1        ONLINE  ONLINE       pricedev1-lnx                                
ora.gpnpd
      1        ONLINE  ONLINE       pricedev1-lnx                                
ora.mdnsd
      1        ONLINE  ONLINE       pricedev1-lnx  



Modifying AWR Automatic Snapshot Settings


Verify Current setting:

SQL> set linesize 100
SQL> col snap_interval format a20
SQL> col retention format a20
SQL> col topnsql format a20

SQL> select * from dba_hist_wr_control;

      DBID SNAP_INTERVAL        RETENTION            TOPNSQL
---------- -------------------- -------------------- --------------------
2684842560 +00000 01:00:00.0    +00008 00:00:00.0    DEFAULT


Change the interval to 10 minutes and retention to 10 days:

SQL> execute dbms_workload_repository.modify_snapshot_settings( interval => 10,retention => 14400);

Verify change:


SQL> select * from dba_hist_wr_control;

      DBID SNAP_INTERVAL        RETENTION            TOPNSQL
---------- -------------------- -------------------- --------------------
2684842560 +00000 00:10:00.0    +00010 00:00:00.0    DEFAULT

Tuesday, April 30, 2013

High 'direct path read' waits in 11g

Symptoms:

See high 'direct path read' wait when query a large table.

cause:

In 10g, serial table scans for "large" tables used to go through cache (by default) which is not the case anymore.  In 11g, this decision to read via direct path or through cache is based on the size of the table, buffer cache size and various other stats.
Direct path reads are faster than scattered reads and have less impact on other processes because they avoid latches.

Solution:

When using Automatic Shared Memory Management (ASMM) and with buffer cache low limit set at a low end compared to the normal workload requirements and usually after startup, 11g might choose to do serial direct path read scans for large tables that do not fit in the SGA. When ASMM increases the buffer cache due to increased demand, 11g might not again do serial direct path read scans for these same large tables.  If you like to avoid this from happening, you should note the buffer cache and share pool requirements for a normal workload and set the low limits of buffer cache and shared pool in spfile/pfile close to these normal workload values.
db_cache_size
shared_pool_size 


From MOS Note: [ID 793845.1]

Friday, April 26, 2013

DBUA failed in bringing up the database. Either the database is not running from Oracle Home or the correct pfile was not found

Symptoms:

When using dbua to upgrade a database, you might get a pop-up asking for the location of the init.ora, with a message like this:
"Either the database is not running from Oracle Home or the correct pfile was not found" 

Cause:
An earlier upgrade attempt failed, and a dbua file got left "in limbo"

Solution:
cd /u01/app/oracle/cfgtoollogs/dbua/logs

ls
PreUpgradeResults.html  sqls.log  trace.log  Welcome_mydb.txt

rm Welcome_mydb.txt

From MOS Note: [ID 1303325.1]

Wednesday, April 24, 2013

Setup active dataguard using dgmgrl

1. Set apply off
DGMGRL> edit database 'DBAPLC' set state=apply-off;
Succeeded.
DGMGRL> show database 'DBAPLC'

Database - DBAPLC

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-OFF
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    DBAPLC

Database Status:
SUCCESS
2. Open Standby database
SQL> alter database open;

Database altered.
3. Set apply on
DGMGRL> edit database 'DBAPLC' set state=apply-on;
Succeeded.
DGMGRL> show database 'DBAPLC'

Database - DBAPLC

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: ON
  Instance(s):
    DBAPLC

Database Status:
SUCCESS