ORA-600和ORA-7445的错误已经司空见惯了，而Oracle的ORA-700错误还是第一次看到，而看到这个错误，还是在ODA环境中看到的。
数据库环境为ODA，在尝试配置FLASH_CACHE：
SQL> SHOW parameter flash_ NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ db_flash_cache_file string db_flash_cache_size big INTEGER 0 db_flashback_retention_target INTEGER 1440 SQL> ALTER system SET db_flash_cache_size = 60G scope = spfile; System altered. SQL> ALTER system SET db_flash_cache_file= '+REDO' scope = spfile; System altered. |
重新启动数据库后，发现了这个错误：
Tue Aug 14 02:37:13 2012 Encounter problem verifying flash cache +REDO/flash/abc.flash. Disable flash cache AND issue an ORA-700 FOR diagnostics Errors IN file /u01/app/oracle/diag/rdbms/odaenmo/odaenmo1/trace/odaenmo1_gen0_25557.trc (incident=195241): ORA-00700: 软内部错误, 参数: [kcbl2vfyfh_action], [db_flash_cache_file integrity CHECK failed], [], [], [], [], [], [], [], [], [], [] Incident details IN: /u01/app/oracle/diag/rdbms/odaenmo/odaenmo1/incident/incdir_195241/odaenmo1_gen0_25557_i195241.trc Tue Aug 14 02:37:14 2012 Tue Aug 14 02:37:14 2012 |
这是实例1上高级日志的错误，下面是详细的TRACE信息：
Dump file /u01/app/oracle/diag/rdbms/odaenmo/odaenmo1/incident/incdir_195241/odaenmo1_gen0_25557_i195241.trc Oracle DATABASE 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production WITH the Partitioning, REAL Application Clusters, Automatic Storage Management, OLAP, DATA Mining AND REAL Application Testing options ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/dbhome_1 System name: Linux Node name: odaenmo1 Release: 2.6.32-300.11.1.el5uek Version: #1 SMP Wed Mar 7 09:55:50 EST 2012 Machine: x86_64 Instance name: odaenmo1 Redo thread mounted BY this instance: 1 Oracle process NUMBER: 5 Unix process pid: 25557, image: oracle@odaenmo1 (GEN0) *** 2012-08-14 02:37:13.532 *** SESSION ID:(3131.1) 2012-08-14 02:37:13.532 *** CLIENT ID:() 2012-08-14 02:37:13.532 *** SERVICE NAME:(SYS$BACKGROUND) 2012-08-14 02:37:13.532 *** MODULE NAME:() 2012-08-14 02:37:13.532 *** ACTION NAME:() 2012-08-14 02:37:13.532 Dump continued FROM file: /u01/app/oracle/diag/rdbms/odaenmo/odaenmo1/trace/odaenmo1_gen0_25557.trc ORA-00700: 软内部错误, 参数: [kcbl2vfyfh_action], [db_flash_cache_file integrity CHECK failed], [], [], [], [], [], [], [], [], [], [] ========= Dump FOR incident 195241 (ORA 700 [kcbl2vfyfh_action]) ======== ----- Beginning of Customized Incident Dump(s) ----- Flash cache parameter expected: db_unique_name odaenmo, dbid 196211168 inst_id 1, bpid 3, bsz 8192 File header db_unique_name odaenmo, dbid 196211168, inst_id 1, bpid 3, bsz 8192 filetype_str "Oracle RDBMS Flash Cache File" ----- End of Customized Incident Dump(s) ----- *** 2012-08-14 02:37:13.568 dbkedDefDump(): Starting incident DEFAULT dumps (flags=0x2, level=3, mask=0x0) ----- SQL Statement (None) ----- CURRENT SQL information unavailable - no cursor. ----- Call Stack Trace ----- calling CALL entry argument VALUES IN hex location TYPE point (? means dubious VALUE) -------------------- -------- -------------------- ---------------------------- skdstdst()+36 CALL kgdsdst() 000000000 ? 000000000 ? 7FFF491BA3C8 ? 000000001 ? 000000001 ? 000000002 ? ksedst1()+98 CALL skdstdst() 000000000 ? 000000000 ? 7FFF491BA3C8 ? 000000001 ? 000000000 ? 000000002 ? ksedst()+34 CALL ksedst1() 000000000 ? 000000001 ? 7FFF491BA3C8 ? 000000001 ? 000000000 ? 000000002 ? dbkedDefDump()+2741 CALL ksedst() 000000000 ? 000000001 ? 7FFF491BA3C8 ? 000000001 ? 000000000 ? 000000002 ? ksedmp()+36 CALL dbkedDefDump() 000000003 ? 000000002 ? 7FFF491BA3C8 ? 000000001 ? 000000000 ? 000000002 ? ksfdmp()+64 CALL ksedmp() 000000003 ? 000000002 ? 7FFF491BA3C8 ? 000000001 ? 000000000 ? 000000002 ? dbgexPhaseII()+1764 CALL ksfdmp() 000000003 ? 000000002 ? 7FFF491BA3C8 ? 000000001 ? 000000000 ? 000000002 ? dbgexExplicitEndInc CALL dbgexPhaseII() 7F3C50D2E710 ? 7F3C50D398C0 ? ()+750 7FFF491C5F00 ? 000000001 ? 000000000 ? 000000002 ? dbgeEndDDEInvocatio CALL dbgexExplicitEndInc 7F3C50D2E710 ? 7F3C50D398C0 ? nImpl()+767 () 7FFF491C5F00 ? 000000001 ? 000000000 ? 000000002 ? dbgeEndDDEInvocatio CALL dbgeEndDDEInvocatio 7F3C50D2E710 ? 7F3C50D398C0 ? n()+47 nImpl() 7FFF491C5F00 ? 000000001 ? 000000000 ? 000000002 ? kcbl2vfyfh_action() CALL dbgeEndDDEInvocatio 7F3C50D2E710 ? 7F3C50D398C0 ? +3373 n() 7FFF491C5F00 ? 000000001 ? 000000000 ? 000000002 ? ksbabs()+771 CALL kcbl2vfyfh_action() 7FFF491C7738 ? 000000018 ? 7FFF491C5F00 ? 000000001 ? 000000000 ? 000000002 ? ksbrdp()+971 CALL ksbabs() 7FFF491C7738 ? 000000018 ? 7FFF491C5F00 ? 000000001 ? 000000000 ? 000000002 ? opirip()+618 CALL ksbrdp() 7FFF491C7738 ? 000000018 ? 7FFF491C5F00 ? 000000001 ? 000000000 ? 000000002 ? opidrv()+598 CALL opirip() 000000032 ? 000000004 ? 7FFF491C88F8 ? 000000001 ? 000000000 ? 000000002 ? sou2o()+98 CALL opidrv() 000000032 ? 000000004 ? 7FFF491C88F8 ? 000000001 ? 000000000 ? 000000002 ? opimai_real()+261 CALL sou2o() 7FFF491C88D0 ? 000000032 ? 000000004 ? 7FFF491C88F8 ? 000000000 ? 000000002 ? ssthrdmain()+252 CALL opimai_real() 000000000 ? 7FFF491C8AC0 ? 000000004 ? 7FFF491C88F8 ? 000000000 ? 000000002 ? main()+196 CALL ssthrdmain() 000000003 ? 7FFF491C8AC0 ? 000000001 ? 000000000 ? 000000000 ? 000000002 ? __libc_start_main() CALL main() 000000003 ? 7FFF491C8C60 ? +244 000000001 ? 000000000 ? 000000000 ? 000000002 ? _start()+36 CALL __libc_start_main() 000A0B188 ? 000000001 ? 7FFF491C8C58 ? 000000000 ? 000000000 ? 000000003 ? --------------------- Binary Stack Dump --------------------- |
前面提到了ORA-600和ORA-7445这种内部错误很常见，而这种软内部错误的ORA-700错误还是第一次看到。
下面是节点2上的告警：
2012-08-14 02:37:13.192000 +08:00 Encounter problem verifying flash cache +REDO/flash/abc.flash. Disable flash cache AND issue an ORA-700 FOR diagnostics Errors IN file /u01/app/oracle/diag/rdbms/odaenmo/odaenmo2/trace/odaenmo2_gen0_16906.trc (incident=416041): ORA-00700: 软内部错误, 参数: [kcbl2vfyfh_action], [db_flash_cache_file integrity CHECK failed], [], [], [], [], [], [], [], [], [], [] Incident details IN: /u01/app/oracle/diag/rdbms/odaenmo/odaenmo2/incident/incdir_416041/odaenmo2_gen0_16906_i416041.trc L2 cache disabled FOR dbwr 1 L2 cache disabled FOR dbwr 2 L2 cache file closed BY dbwr 0 L2 cache disabled FOR dbwr 0 Dumping diagnostic DATA IN directory=[cdmp_20120814023714], requested BY (instance=2, osid=16906 (GEN0)), summary=[incident=416041]. 2012-08-14 02:37:18.880000 +08:00 Dumping diagnostic DATA IN directory=[cdmp_20120814023714], requested BY (instance=1, osid=25557 (GEN0)), summary=[incident=195241]. 2012-08-14 02:37:55.075000 +08:00 Sweep [inc][416041]: completed Sweep [inc2][416041]: completed |
对应的详细TRACE文件为：
Dump file /u01/app/oracle/diag/rdbms/odaenmo/odaenmo2/incident/incdir_416041/odaenmo2_gen0_16906_i416041.trc Oracle DATABASE 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production WITH the Partitioning, REAL Application Clusters, Automatic Storage Management, OLAP, DATA Mining AND REAL Application Testing options ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/dbhome_1 System name: Linux Node name: odaenmo2 Release: 2.6.32-300.11.1.el5uek Version: #1 SMP Wed Mar 7 09:55:50 EST 2012 Machine: x86_64 Instance name: odaenmo2 Redo thread mounted BY this instance: 2 Oracle process NUMBER: 5 Unix process pid: 16906, image: oracle@odaenmo2 (GEN0) *** 2012-08-14 02:37:13.249 *** SESSION ID:(3131.1) 2012-08-14 02:37:13.249 *** CLIENT ID:() 2012-08-14 02:37:13.249 *** SERVICE NAME:(SYS$BACKGROUND) 2012-08-14 02:37:13.249 *** MODULE NAME:() 2012-08-14 02:37:13.249 *** ACTION NAME:() 2012-08-14 02:37:13.249 Dump continued FROM file: /u01/app/oracle/diag/rdbms/odaenmo/odaenmo2/trace/odaenmo2_gen0_16906.trc ORA-00700: 软内部错误, 参数: [kcbl2vfyfh_action], [db_flash_cache_file integrity CHECK failed], [], [], [], [], [], [], [], [], [], [] ========= Dump FOR incident 416041 (ORA 700 [kcbl2vfyfh_action]) ======== ----- Beginning of Customized Incident Dump(s) ----- Flash cache parameter expected: db_unique_name odaenmo, dbid 196211168 inst_id 2, bpid 3, bsz 8192 File header db_unique_name odaenmo, dbid 196211168, inst_id 1, bpid 3, bsz 8192 filetype_str "Oracle RDBMS Flash Cache File" ----- End of Customized Incident Dump(s) ----- *** 2012-08-14 02:37:13.284 dbkedDefDump(): Starting incident DEFAULT dumps (flags=0x2, level=3, mask=0x0) ----- SQL Statement (None) ----- CURRENT SQL information unavailable - no cursor. ----- Call Stack Trace ----- calling CALL entry argument VALUES IN hex location TYPE point (? means dubious VALUE) -------------------- -------- -------------------- ---------------------------- skdstdst()+36 CALL kgdsdst() 000000000 ? 000000000 ? 7FFF7312DDF8 ? 000000001 ? 000000001 ? 000000002 ? ksedst1()+98 CALL skdstdst() 000000000 ? 000000000 ? 7FFF7312DDF8 ? 000000001 ? 000000000 ? 000000002 ? ksedst()+34 CALL ksedst1() 000000000 ? 000000001 ? 7FFF7312DDF8 ? 000000001 ? 000000000 ? 000000002 ? dbkedDefDump()+2741 CALL ksedst() 000000000 ? 000000001 ? 7FFF7312DDF8 ? 000000001 ? 000000000 ? 000000002 ? ksedmp()+36 CALL dbkedDefDump() 000000003 ? 000000002 ? 7FFF7312DDF8 ? 000000001 ? 000000000 ? 000000002 ? ksfdmp()+64 CALL ksedmp() 000000003 ? 000000002 ? 7FFF7312DDF8 ? 000000001 ? 000000000 ? 000000002 ? dbgexPhaseII()+1764 CALL ksfdmp() 000000003 ? 000000002 ? 7FFF7312DDF8 ? 000000001 ? 000000000 ? 000000002 ? dbgexExplicitEndInc CALL dbgexPhaseII() 7F6850206710 ? 7F68502118C0 ? ()+750 7FFF73139930 ? 000000001 ? 000000000 ? 000000002 ? dbgeEndDDEInvocatio CALL dbgexExplicitEndInc 7F6850206710 ? 7F68502118C0 ? nImpl()+767 () 7FFF73139930 ? 000000001 ? 000000000 ? 000000002 ? dbgeEndDDEInvocatio CALL dbgeEndDDEInvocatio 7F6850206710 ? 7F68502118C0 ? n()+47 nImpl() 7FFF73139930 ? 000000001 ? 000000000 ? 000000002 ? kcbl2vfyfh_action() CALL dbgeEndDDEInvocatio 7F6850206710 ? 7F68502118C0 ? +3373 n() 7FFF73139930 ? 000000001 ? 000000000 ? 000000002 ? ksbabs()+771 CALL kcbl2vfyfh_action() 7FFF7313B168 ? 000000018 ? 7FFF73139930 ? 000000001 ? 000000000 ? 000000002 ? ksbrdp()+971 CALL ksbabs() 7FFF7313B168 ? 000000018 ? 7FFF73139930 ? 000000001 ? 000000000 ? 000000002 ? opirip()+618 CALL ksbrdp() 7FFF7313B168 ? 000000018 ? 7FFF73139930 ? 000000001 ? 000000000 ? 000000002 ? opidrv()+598 CALL opirip() 000000032 ? 000000004 ? 7FFF7313C328 ? 000000001 ? 000000000 ? 000000002 ? sou2o()+98 CALL opidrv() 000000032 ? 000000004 ? 7FFF7313C328 ? 000000001 ? 000000000 ? 000000002 ? opimai_real()+261 CALL sou2o() 7FFF7313C300 ? 000000032 ? 000000004 ? 7FFF7313C328 ? 000000000 ? 000000002 ? ssthrdmain()+252 CALL opimai_real() 000000000 ? 7FFF7313C4F0 ? 000000004 ? 7FFF7313C328 ? 000000000 ? 000000002 ? main()+196 CALL ssthrdmain() 000000003 ? 7FFF7313C4F0 ? 000000001 ? 000000000 ? 000000000 ? 000000002 ? __libc_start_main() CALL main() 000000003 ? 7FFF7313C690 ? +244 000000001 ? 000000000 ? 000000000 ? 000000002 ? _start()+36 CALL __libc_start_main() 000A0B188 ? 000000001 ? 7FFF7313C688 ? 000000000 ? 000000000 ? 000000003 ? --------------------- Binary Stack Dump --------------------- |
从实例2上的详细TRACE文件，可以很轻易的分析出导致这个错误的原因。FLASH CACHE参数期望当前是实例2，但是从FLASH CACHE配置的文件读取，发现文件中记录的是实例1，导致这个错误的原因在于FLASH CACHE配置的文件是各个实例独立的，因此不能配置一个共享的文件给两个实例同时使用。
这个问题还可以通过检查FLASH CACHE的统计信息发现：
SQL> SELECT * FROM v$sysstat WHERE name LIKE '%flash cache eviction%'; STATISTIC# NAME CLASS VALUE STAT_ID ---------- --------------------------------------- ---------- ---------- ---------- 97 flash cache eviction: invalidated 8 1 3647825193 98 flash cache eviction: buffer pinned 8 0 3134415242 99 flash cache eviction: aged OUT 8 0 3307935113 |
将FLASH CACHE的配置修改如下：
SQL> SHOW parameter flash NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ db_flash_cache_file string +REDO/flash/abc.flash db_flash_cache_size big INTEGER 60G db_flashback_retention_target INTEGER 1440 SQL> ALTER system SET db_flash_cache_size = 30g scope = spfile; System altered. SQL> ALTER system SET db_flash_cache_file = '+REDO/flash/odademo1.flash' scope = spfile sid = 'odaenmo1'; System altered. SQL> ALTER system SET db_flash_cache_file = '+REDO/flash/odademo2.flash' scope = spfile sid = 'odaenmo2'; System altered. |
设置正确的FLASH CACHE后，重启数据库，ORA-700的错误不再出现。