ORA-00328 ORA-00334 MRP0: Background Media Recovery terminated with error 328

環境介紹：前幾天搭建了一套 二節點單實例的 linux+oracle11.2.0.3+dataguard   maximize availability 的環境。

故障現象：今天發現不能同步了，在trace文件alert_orcl.log里發現有報錯信息MRP進程啟不來

MRP0: Background Media Recovery terminated with error 328

ORA-00328: 8386238 , 8972415
ORA-00334: '/opt/oracle/fast_recovery_area/DG_BEI/archivelog/2014_09_28/o1_mf_1_830_b2h9mjmn_.arc'

在報上述錯誤前trace文件alert_orcl.log有報與主庫網絡中斷錯誤

RFS[4]: Assigned to RFS process 20669
RFS[4]: Possible network disconnect with primary database
Sun Sep 28 13:23:48 2014

MRP0: Background Media Recovery terminated with error 328詳細報錯信息如下：

Mon Sep 29 14:32:28 2014
alter database recover managed standby database using current logfile disconnect from session nodelay
Attempt to start background Managed Standby Recovery process (orcl)
Mon Sep 29 14:32:28 2014
MRP0 started with pid=27, OS id=23710
MRP0: Background Managed Standby Recovery process started (orcl)
 started logmerger process
Mon Sep 29 14:32:33 2014
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Clearing online redo logfile 1 /opt/oracle/oradata/orcl/redo01.log
Clearing online log 1 of thread 1 sequence number 835
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 /opt/oracle/oradata/orcl/redo02.log
Clearing online log 2 of thread 1 sequence number 834
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 /opt/oracle/oradata/orcl/redo03.log
Clearing online log 3 of thread 1 sequence number 835
Clearing online redo logfile 3 complete
Media Recovery Log /opt/oracle/fast_recovery_area/DG_BEI/archivelog/2014_09_28/o1_mf_1_830_b2h9mjmn_.arc
Errors with log /opt/oracle/fast_recovery_area/DG_BEI/archivelog/2014_09_28/o1_mf_1_830_b2h9mjmn_.arc
MRP0: Background Media Recovery terminated with error 328
Errors in file /opt/oracle/diag/rdbms/dg_bei/orcl/trace/orcl_pr00_23712.trc:
ORA-00328: 8386238 , 8972415
ORA-00334: : '/opt/oracle/fast_recovery_area/DG_BEI/archivelog/2014_09_28/o1_mf_1_830_b2h9mjmn_.arc'
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Completed: alter database recover managed standby database using current logfile disconnect from session nodelay
MRP0: Background Media Recovery process shutdown (orcl)
Mon Sep 29 14:34:06 2014
RFS[1]: Assigned to RFS process 23726
RFS[1]: Opened log for thread 1 sequence 836 dbid 1356850190 branch 829069458
Archived Log entry 77 added for thread 1 sequence 836 rlc 829069458 ID 0x50df5e0e dest 2:
Mon Sep 29 14:34:08 2014
Primary database is in MAXIMUM AVAILABILITY mode
Changing standby controlfile to RESYNCHRONIZATION level
Standby controlfile consistent with primary
RFS[2]: Assigned to RFS process 23728
RFS[2]: Selected log 4 for thread 1 sequence 838 dbid 1356850190 branch 829069458
Mon Sep 29 14:34:08 2014
RFS[3]: Assigned to RFS process 23730
RFS[3]: Selected log 5 for thread 1 sequence 837 dbid 1356850190 branch 829069458
Mon Sep 29 14:34:08 2014
Archived Log entry 78 added for thread 1 sequence 837 ID 0x50df5e0e dest 1:
Changing standby controlfile to MAXIMUM AVAILABILITY level
RFS[2]: Selected log 5 for thread 1 sequence 839 dbid 1356850190 branch 829069458
Mon Sep 29 14:34:11 2014
Archived Log entry 79 added for thread 1 sequence 838 ID 0x50df5e0e dest 1:

處理過程：

1.備庫上檢查恢復相關的進程，確實少了MRP

select process,status,sequence# from v$managed_standby;

2.在備庫上檢查歸檔日志視圖

sql>select name,sequence#,applied from v$archived_log;

 奇怪的事：trace日志中報錯的歸檔日志顯示被應用了，如紅框中示

3.對比主庫與備庫 2014_09_28歸檔日志，數據和大小都不一樣，備庫比主庫日志數量要多，但比主庫要小，如下圖示：（估計是傳輸日志的時候剛好網絡中斷造成的大小不一致）

4.萬能的google搜索，大概是歸檔日志傳輸出錯，從主庫copy日志到備庫，然后再執行恢復

4.1 以防萬一，先將備庫2014_09_28歸檔日志備份一下，然后刪除備庫2014_09_28歸檔文件夾

4.2 從主庫上scp 復制2014_09_28歸檔文件夾到備庫

4.3 備庫上執行恢復操作

SQL> alter database recover managed standby database using current logfile disconnect;

查看trace日志顯示正常了

5.到備庫上確認歸檔日志 sql>select name,sequence#,applied from v$archived_log;

顯示能正常應用了，不過有一個日志（紅框中示）沒用應用，我猜因為從主庫copy過來的日志文件沒有這個文件（只5個，原備庫有8個），檢查了數據是正常的。故障應該算是解決了。