Problem Description
I switchover from primary to standby successfully but when I swithback to primary I got following error:
DGMGRL> show configuration
Configuration
Name: DBAPRD.intdata.com
Enabled: YES
Protection Mode: MaxPerformance
Databases:
DBALC - Primary database
DBAPRD - Physical standby database
DBADR - Physical standby database
Fast-Start Failover: DISABLED
Current status for "DBAPRD.intdata.com":
SUCCESS
DGMGRL> switchover to 'DBAPRD'
DGMGRL> Performing switchover NOW, please wait...
Error: ORA-16775: target standby database in broker operation has potential data loss
Failed.
Unable to switchover, primary database is still "DBALC"
DGMGRL> show configuration
Configuration
Name: DBAPRD.intdata.com
Enabled: YES
Protection Mode: MaxPerformance
Databases:
DBALC - Primary database
DBAPRD - Physical standby database
DBADBDR - Physical standby database
Fast-Start Failover: DISABLED
Current status for "DBAPRD.intdata.com":
Warning: ORA-16607: one or more databases have failed
Following error was found in alertlog of primary database:
Errors in file /u02/oracle/diag/dbaPRD/diag/rdbms/dbalc/dbaLC/trace/dbaLC_lns1_21171.trc:
ORA-03135: connection lost contact
LGWR: Error 3135 closing archivelog file '(DESCRIPTION=(ADDRESS_LIST = (ADDRESS=(PROTOCOL=tcp)(HOST=racprod1-v)(PORT=1540)))(CONNECT_DATA=(SERVICE_NAME=dbaPRD.intdata.com)(SERVER=DEDICATED)))'
Sat Jun 08 19:20:51 2013
Deleted Oracle managed file +dba_FLASH/dbalc/archivelog/2013_04_14/thread_1_seq_130382.8527.812711313
Waiting for all non-current ORLs to be archived...
Waiting for the ORL for thread 1 sequence 136062 to be archived...
Sat Jun 08 19:20:59 2013
ARC3: Archiving not possible: failed standby destinations
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance dbaLC - Archival Error
ORA-16014: log 2 sequence# 136062 not archived, no available destinations
ORA-00312: online log 2 thread 1: '+dba_DATA/dbalc/onlinelog/group_2.311.794395241'
ORA-00312: online log 2 thread 1: '+dba_FLASH/dbalc/onlinelog/group_2.303.794395255'
Errors in file /u02/oracle/diag/dbaPRD/diag/rdbms/dbalc/dbaLC/trace/dbaLC_arc3_14518.trc:
ORA-16014: log 2 sequence# 136062 not archived, no available destinations
ORA-00312: online log 2 thread 1: '+dba_DATA/dbalc/onlinelog/group_2.311.794395241'
ORA-00312: online log 2 thread 1: '+dba_FLASH/dbalc/onlinelog/group_2.303.794395255'
......
ORA-16416 signalled during: ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN...
Cause:
The target standby database in the broker operation did not have all the redo logs from the primary database.
Solution:
I tried "alter system swith logfile" but it didnt' work.
Restart primary database solve the problem.
Conclusion
Before swithover , run a few log switchover to make sure log can ship to standby.
Saturday, June 8, 2013
switchover failed with ORA-16775: target standby database in broker operation has potential data loss
Understand SCN movement during online user managed backup;
Before online backup, check current scn, system and datafile scn in controlfile and scn in datafile header. As expected checkpoint scn are same in controlfile and datafile headers, and it is behind current scn. SQL> select current_scn, checkpoint_change# from v$database; CURRENT_SCN CHECKPOINT_CHANGE# ----------- ------------------ 325011036 325009912 SQL> select name,checkpoint_change# from v$datafile; NAME CHECKPOINT_CHANGE# -------------------------------------------------- ------------------ +DBA_DATA/dbaprd/datafile/system.270.675355175 325009912 +DBA_DATA/dbaprd/datafile/sysaux.269.675355177 325009912 +DBA_DATA/dbaprd/datafile/undotbs1.266.675355179 325009912 +DBA_DATA/dbaprd/datafile/undotbs2.264.675355187 325009912 +DBA_DATA/dbaprd/datafile/users.263.675355189 325009912 +DBA_DATA/dbaprd/datafile/xml_data.262.675355189 325009912 SQL> select name,checkpoint_change# from v$datafile_header where name like '%users%'; NAME CHECKPOINT_CHANGE# -------------------------------------------------- ------------------ +DBA_DATA/dbaprd/datafile/users.263.675355189 325009912 Start online tablespace backup Oracle did a checkpoint on USERS tablespace and datafile only, and freeze the checkpoint scn on datafile header. SQL> alter tablespace users begin backup; Tablespace altered. SQL> select current_scn, checkpoint_change# from v$database; CURRENT_SCN CHECKPOINT_CHANGE# ----------- ------------------ 325011196 325009912 SQL> select name,checkpoint_change# from v$datafile; NAME CHECKPOINT_CHANGE# -------------------------------------------------- ------------------ +DBA_DATA/dbaprd/datafile/system.270.675355175 325009912 +DBA_DATA/dbaprd/datafile/sysaux.269.675355177 325009912 +DBA_DATA/dbaprd/datafile/undotbs1.266.675355179 325009912 +DBA_DATA/dbaprd/datafile/undotbs2.264.675355187 325009912 +DBA_DATA/dbaprd/datafile/users.263.675355189 325011168 +DBA_DATA/dbaprd/datafile/xml_data.262.675355189 325009912 SQL> select name,checkpoint_change# from v$datafile_header where name like '%users%'; NAME CHECKPOINT_CHANGE# -------------------------------------------------- ------------------ +DBA_DATA/dbaprd/datafile/users.263.675355189 325011168 during the online backup checkpoint scn is freeze on datafile belongs to USERS tablespace SQL> alter system checkpoint; System altered. SQL> select current_scn, checkpoint_change# from v$database; CURRENT_SCN CHECKPOINT_CHANGE# ----------- ------------------ 325011272 325011243 SQL> select name,checkpoint_change# from v$datafile; NAME CHECKPOINT_CHANGE# -------------------------------------------------- ------------------ +DBA_DATA/dbaprd/datafile/system.270.675355175 325011243 +DBA_DATA/dbaprd/datafile/sysaux.269.675355177 325011243 +DBA_DATA/dbaprd/datafile/undotbs1.266.675355179 325011243 +DBA_DATA/dbaprd/datafile/undotbs2.264.675355187 325011243 +DBA_DATA/dbaprd/datafile/users.263.675355189 325011168 +DBA_DATA/dbaprd/datafile/xml_data.262.675355189 325011243 SQL> select name,checkpoint_change# from v$datafile_header where name like '%users%'; NAME CHECKPOINT_CHANGE# -------------------------------------------------- ------------------ +DBA_DATA/dbaprd/datafile/users.263.675355189 325011168 END online tablespace backup; Oracle advanced checkpoint scn on USERS tablespace and datafile only to be same as system checkpoint scn SQL> alter tablespace users end backup; Tablespace altered. SQL> select current_scn, checkpoint_change# from v$database; CURRENT_SCN CHECKPOINT_CHANGE# ----------- ------------------ 325011488 325011243 SQL> select name,checkpoint_change# from v$datafile; NAME CHECKPOINT_CHANGE# -------------------------------------------------- ------------------ +DBA_DATA/dbaprd/datafile/system.270.675355175 325011243 +DBA_DATA/dbaprd/datafile/sysaux.269.675355177 325011243 +DBA_DATA/dbaprd/datafile/undotbs1.266.675355179 325011243 +DBA_DATA/dbaprd/datafile/undotbs2.264.675355187 325011243 +DBA_DATA/dbaprd/datafile/users.263.675355189 325011243 +DBA_DATA/dbaprd/datafile/xml_data.262.675355189 325011243 9 rows selected. SQL> select name,checkpoint_change# from v$datafile_header where name like '%users%'; NAME CHECKPOINT_CHANGE# -------------------------------------------------- ------------------ +DBA_DATA/dbaprd/datafile/users.263.675355189 325011243
Friday, June 7, 2013
Use rman increamental backup to re-sync a physical standby database
Problem description: Got following error in standby alert log: Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE Media Recovery Log +DBA_FLASH/dbalc/archivelog/2013_06_06/thread_1_seq_952.3228.817406977 Errors with log +DBA_FLASH/dbalc/archivelog/2013_06_06/thread_1_seq_952.3228.817406977 MRP0: Background Media Recovery terminated with error 328 Errors in file /u02/oracle/diag/DBAPRD/diag/rdbms/dbalc/DBALC/trace/DBALC_mrp0_2283.trc: ORA-00328: archived log ends at change 324705843, need later change 324707097 ORA-00334: archived log: '+DBA_FLASH/dbalc/archivelog/2013_06_06/thread_1_seq_952.3228.817406977' Managed Standby Recovery not using Real Time Apply Shutting down recovery slaves due to error 328 Recovery interrupted! Errors in file /u02/oracle/diag/DBAPRD/diag/rdbms/dbalc/DBALC/trace/DBALC_mrp0_2283.trc: ORA-00328: archived log ends at change 324705843, need later change 324707097 ORA-00334: archived log: '+DBA_FLASH/dbalc/archivelog/2013_06_06/thread_1_seq_952.3228.817406977' MRP0: Background Media Recovery process shutdown (DBALC) Cause: The issue can be due to many reasons related to archived logs and SCN having wrong meta data. Standby Oracle control file is looking for a specific Archivelog specific SCN which it thinks is required for Recovery is not available in the archived log. Action Plan: Using RMAN Incremental Backups to Roll Forward a Physical Standby Database http://docs.oracle.com/cd/E11882_01/server.112/e25608/rman.htm#CIHIAADC 1 Stop Redo apply and get the SCN on standby database SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; SQL> SELECT CURRENT_SCN FROM V$DATABASE; CURRENT_SCN ----------- 324714843 2 Backup incremental from SCN on primary database RMAN> BACKUP INCREMENTAL FROM SCN 324714843 DATABASE FORMAT '/datapump/ForStandby_%U' tag 'FORSTANDBY'; 3 Copy backup from primary to standby 4 Catalog the backup on standby database RMAN> CATALOG START WITH '/datapump/ForStandby'; 5 Recover the standby database with the cataloged incremental backup RMAN> RECOVER DATABASE NOREDO; you may need to restore standby control first if above command failed. RMAN> STARTUP FORCE NOMOUNT; RMAN> RESTORE STANDBY CONTROLFILE FROM TAG 'FORSTANDBY'; RMAN> ALTER DATABASE MOUNT; RMAN> RECOVER DATABASE NOREDO; 6 Restart Standby database and start redo apply.
Thursday, June 6, 2013
Re-create corrupted ASM disks
Problem description: Following 3 disks can see via OS but not ASM: OCRVOTED01 OCRVOTED02 OCRVOTED03 # multipath -l|grep OCR shared_asm_OCRVOTED01 (1HITACHI_770140123364) dm-24 HITACHI,DF600F shared_asm_OCRVOTED02 (1HITACHI_770140123365) dm-25 HITACHI,DF600F shared_asm_OCRVOTED03 (1HITACHI_770140123366) dm-26 HITACHI,DF600F shared_asm_OCRVOTED04 (1HITACHI_770140123367) dm-27 HITACHI,DF600F shared_asm_OCRVOTED05 (1HITACHI_770140123368) dm-28 HITACHI,DF600F # /etc/init.d/oracleasm listdisks |grep OCR OCRVOTED04 OCRVOTED05 Cause: Disks are corrupted Action Plan: 1. Try to recreate asm disk /etc/init.d/oracleasm createdisk OCRVOTED01 /dev/mapper/shared_asm_OCRVOTED01 Marking disk "OCRVOTED01" as an ASM disk: [FAILED] found following message from /var/log/oracleasm Device "/dev/mapper/shared_asm_OCRVOTED01" is already labeled for ASM disk "" 2. Overwrite the ASM header information on the disk. This can be achieved with the UNIX command dd. The following command will write 100x1024b blocks to the specified raw device: dd if=/dev/zero of=/dev/mapper/shared_asm_OCRVOTED01 bs=1024 count=100 dd if=/dev/zero of=/dev/mapper/shared_asm_OCRVOTED02 bs=1024 count=100 dd if=/dev/zero of=/dev/mapper/shared_asm_OCRVOTED03 bs=1024 count=100 3. Recreate asm disks /etc/init.d/oracleasm createdisk OCRVOTED01 /dev/mapper/shared_asm_OCRVOTED01 /etc/init.d/oracleasm createdisk OCRVOTED02 /dev/mapper/shared_asm_OCRVOTED02 /etc/init.d/oracleasm createdisk OCRVOTED03 /dev/mapper/shared_asm_OCRVOTED03 4. Run oracleasm scandisks on both nodes /etc/init.d/oracleasm scandisks 5. Verify # /etc/init.d/oracleasm listdisks |grep OCR OCRVOTED01 OCRVOTED02 OCRVOTED03 OCRVOTED04 OCRVOTED05
Find mapping of ASM disks to Physical Devices
Query asmdisk to get the major - minor numbers $ /etc/init.d/oracleasm querydisk -d OCRVOTED04 Disk "OCRVOTED04" is a valid ASM disk on device [253, 27] Match the number to physical device $ ls -l /dev|grep 253|grep 27 brw-rw---- 1 root root 253, 27 Jun 2 00:50 dm-27
Tuesday, June 4, 2013
Prerequest check before switchover to physical standby
Verify standby has received all redo from primary
If you are running in Maximum Protection or Maximum Availability mode, running following query on primary database to check if the target of you switchover is synchronized.
SQL> select db_unique_name, protection_mode, synchronization_status, synchronized from v$archive_dest_status;
If Synchronized does not say YES or you are running Maximum Performance mode, running following query on target standby database to get current log sequence that the primary is sending.
SQL> select client_process, process, thread#, sequence#, status from v$managed_standby where client_process = 'LGWR';
CLIENT_P PROCESS THREAD# SEQUENCE# STATUS
-------- --------- ---------- ---------- ------------
LGWR RFS 1 718 IDLE
LGWR RFS 2 11 IDLE
Running following query on primary database to get the current redo sequence#
SQL> select thread#, sequence#, status from v$log where status = 'CURRENT';
THREAD# SEQUENCE# STATUS
---------- ---------- ----------------
1 718 CURRENT
2 11 CURRENT
You can not switchover if standby is not receiving the current redo.
Check the apply is caught up.
Running following query on target standby database, make sure status fo MRP0 is "APPLYING_LOG"
SQL> select inst_id, client_process, process, thread#, sequence#, status from gv$managed_standby where process like 'MRP%';
INST_ID CLIENT_P PROCESS THREAD# SEQUENCE# STATUS
---------- -------- --------- ---------- ---------- ------------
1 N/A MRP0 1 718 APPLYING_LOG
Stop Rman backup and cancel jobs
Subscribe to:
Posts (Atom)