ORA-700(kcbl2vfyfh_action)错误

ORA-600和ORA-7445的错误已经司空见惯了,而Oracle的ORA-700错误还是第一次看到,而看到这个错误,还是在ODA环境中看到的。
数据库环境为ODA,在尝试配置FLASH_CACHE:

SQL> SHOW parameter flash_
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_flash_cache_file                  string
db_flash_cache_size big              INTEGER     0
db_flashback_retention_target        INTEGER     1440
SQL> ALTER system SET db_flash_cache_size = 60G scope = spfile;
System altered.
SQL> ALTER system SET db_flash_cache_file= '+REDO' scope = spfile;
System altered.

重新启动数据库后,发现了这个错误:

Tue Aug 14 02:37:13 2012
Encounter problem verifying flash cache +REDO/flash/abc.flash. Disable flash cache AND issue an ORA-700 FOR diagnostics
Errors IN file /u01/app/oracle/diag/rdbms/odaenmo/odaenmo1/trace/odaenmo1_gen0_25557.trc (incident=195241):
ORA-00700: 软内部错误, 参数: [kcbl2vfyfh_action], [db_flash_cache_file integrity CHECK failed], [], [], [], [], [], [], [], [], [], []
Incident details IN: /u01/app/oracle/diag/rdbms/odaenmo/odaenmo1/incident/incdir_195241/odaenmo1_gen0_25557_i195241.trc
Tue Aug 14 02:37:14 2012
Tue Aug 14 02:37:14 2012

这是实例1上高级日志的错误,下面是详细的TRACE信息:

Dump file /u01/app/oracle/diag/rdbms/odaenmo/odaenmo1/incident/incdir_195241/odaenmo1_gen0_25557_i195241.trc
Oracle DATABASE 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
WITH the Partitioning, REAL Application Clusters, Automatic Storage Management, OLAP,
DATA Mining AND REAL Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/dbhome_1
System name:    Linux
Node name:      odaenmo1
Release:        2.6.32-300.11.1.el5uek
Version:        #1 SMP Wed Mar 7 09:55:50 EST 2012
Machine:        x86_64
Instance name: odaenmo1
Redo thread mounted BY this instance: 1
Oracle process NUMBER: 5
Unix process pid: 25557, image: oracle@odaenmo1 (GEN0)
*** 2012-08-14 02:37:13.532
*** SESSION ID:(3131.1) 2012-08-14 02:37:13.532
*** CLIENT ID:() 2012-08-14 02:37:13.532
*** SERVICE NAME:(SYS$BACKGROUND) 2012-08-14 02:37:13.532
*** MODULE NAME:() 2012-08-14 02:37:13.532
*** ACTION NAME:() 2012-08-14 02:37:13.532
Dump continued FROM file: /u01/app/oracle/diag/rdbms/odaenmo/odaenmo1/trace/odaenmo1_gen0_25557.trc
ORA-00700: 软内部错误, 参数: [kcbl2vfyfh_action], [db_flash_cache_file integrity CHECK failed], [], [], [], [], [], [], [], [], [], []
========= Dump FOR incident 195241 (ORA 700 [kcbl2vfyfh_action]) ========
----- Beginning of Customized Incident Dump(s) -----
Flash cache parameter expected: db_unique_name odaenmo, dbid 196211168 inst_id 1, bpid 3, bsz 8192
File header db_unique_name odaenmo, dbid 196211168, inst_id 1, bpid 3, bsz 8192 filetype_str "Oracle RDBMS Flash Cache File"
----- End of Customized Incident Dump(s) -----
*** 2012-08-14 02:37:13.568
dbkedDefDump(): Starting incident DEFAULT dumps (flags=0x2, level=3, mask=0x0)
----- SQL Statement (None) -----
CURRENT SQL information unavailable - no cursor.
----- Call Stack Trace -----
calling              CALL     entry                argument VALUES IN hex      
location             TYPE     point                (? means dubious VALUE)     
-------------------- -------- -------------------- ----------------------------
skdstdst()+36        CALL     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFF491BA3C8 ? 000000001 ?
                                                   000000001 ? 000000002 ?
ksedst1()+98         CALL     skdstdst()           000000000 ? 000000000 ?
                                                   7FFF491BA3C8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksedst()+34          CALL     ksedst1()            000000000 ? 000000001 ?
                                                   7FFF491BA3C8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbkedDefDump()+2741  CALL     ksedst()             000000000 ? 000000001 ?
                                                   7FFF491BA3C8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksedmp()+36          CALL     dbkedDefDump()       000000003 ? 000000002 ?
                                                   7FFF491BA3C8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksfdmp()+64          CALL     ksedmp()             000000003 ? 000000002 ?
                                                   7FFF491BA3C8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbgexPhaseII()+1764  CALL     ksfdmp()             000000003 ? 000000002 ?
                                                   7FFF491BA3C8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbgexExplicitEndInc  CALL     dbgexPhaseII()       7F3C50D2E710 ? 7F3C50D398C0 ?
()+750                                             7FFF491C5F00 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbgeEndDDEInvocatio  CALL     dbgexExplicitEndInc  7F3C50D2E710 ? 7F3C50D398C0 ?
nImpl()+767                   ()                   7FFF491C5F00 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbgeEndDDEInvocatio  CALL     dbgeEndDDEInvocatio  7F3C50D2E710 ? 7F3C50D398C0 ?
n()+47                        nImpl()              7FFF491C5F00 ? 000000001 ?
                                                   000000000 ? 000000002 ?
kcbl2vfyfh_action()  CALL     dbgeEndDDEInvocatio  7F3C50D2E710 ? 7F3C50D398C0 ?
+3373                         n()                  7FFF491C5F00 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksbabs()+771         CALL     kcbl2vfyfh_action()  7FFF491C7738 ? 000000018 ?
                                                   7FFF491C5F00 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksbrdp()+971         CALL     ksbabs()             7FFF491C7738 ? 000000018 ?
                                                   7FFF491C5F00 ? 000000001 ?
                                                   000000000 ? 000000002 ?
opirip()+618         CALL     ksbrdp()             7FFF491C7738 ? 000000018 ?
                                                   7FFF491C5F00 ? 000000001 ?
                                                   000000000 ? 000000002 ?
opidrv()+598         CALL     opirip()             000000032 ? 000000004 ?
                                                   7FFF491C88F8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
sou2o()+98           CALL     opidrv()             000000032 ? 000000004 ?
                                                   7FFF491C88F8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
opimai_real()+261    CALL     sou2o()              7FFF491C88D0 ? 000000032 ?
                                                   000000004 ? 7FFF491C88F8 ?
                                                   000000000 ? 000000002 ?
ssthrdmain()+252     CALL     opimai_real()        000000000 ? 7FFF491C8AC0 ?
                                                   000000004 ? 7FFF491C88F8 ?
                                                   000000000 ? 000000002 ?
main()+196           CALL     ssthrdmain()         000000003 ? 7FFF491C8AC0 ?
                                                   000000001 ? 000000000 ?
                                                   000000000 ? 000000002 ?
__libc_start_main()  CALL     main()               000000003 ? 7FFF491C8C60 ?
+244                                               000000001 ? 000000000 ?
                                                   000000000 ? 000000002 ?
_start()+36          CALL     __libc_start_main()  000A0B188 ? 000000001 ?
                                                   7FFF491C8C58 ? 000000000 ?
                                                   000000000 ? 000000003 ?
--------------------- Binary Stack Dump ---------------------

前面提到了ORA-600和ORA-7445这种内部错误很常见,而这种软内部错误的ORA-700错误还是第一次看到。
下面是节点2上的告警:

2012-08-14 02:37:13.192000 +08:00
Encounter problem verifying flash cache +REDO/flash/abc.flash. Disable flash cache AND issue an ORA-700 FOR diagnostics
Errors IN file /u01/app/oracle/diag/rdbms/odaenmo/odaenmo2/trace/odaenmo2_gen0_16906.trc (incident=416041):
ORA-00700: 软内部错误, 参数: [kcbl2vfyfh_action], [db_flash_cache_file integrity CHECK failed], [], [], [], [], [], [], [], [], [], []
Incident details IN: /u01/app/oracle/diag/rdbms/odaenmo/odaenmo2/incident/incdir_416041/odaenmo2_gen0_16906_i416041.trc
L2 cache disabled FOR dbwr 1
L2 cache disabled FOR dbwr 2
L2 cache file closed BY dbwr 0
L2 cache disabled FOR dbwr 0
Dumping diagnostic DATA IN directory=[cdmp_20120814023714], requested BY (instance=2, osid=16906 (GEN0)), summary=[incident=416041].
2012-08-14 02:37:18.880000 +08:00
Dumping diagnostic DATA IN directory=[cdmp_20120814023714], requested BY (instance=1, osid=25557 (GEN0)), summary=[incident=195241].
2012-08-14 02:37:55.075000 +08:00
Sweep [inc][416041]: completed
Sweep [inc2][416041]: completed

对应的详细TRACE文件为:

Dump file /u01/app/oracle/diag/rdbms/odaenmo/odaenmo2/incident/incdir_416041/odaenmo2_gen0_16906_i416041.trc
Oracle DATABASE 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
WITH the Partitioning, REAL Application Clusters, Automatic Storage Management, OLAP,
DATA Mining AND REAL Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/dbhome_1
System name:    Linux
Node name:      odaenmo2
Release:        2.6.32-300.11.1.el5uek
Version:        #1 SMP Wed Mar 7 09:55:50 EST 2012
Machine:        x86_64
Instance name: odaenmo2
Redo thread mounted BY this instance: 2
Oracle process NUMBER: 5
Unix process pid: 16906, image: oracle@odaenmo2 (GEN0)
*** 2012-08-14 02:37:13.249
*** SESSION ID:(3131.1) 2012-08-14 02:37:13.249
*** CLIENT ID:() 2012-08-14 02:37:13.249
*** SERVICE NAME:(SYS$BACKGROUND) 2012-08-14 02:37:13.249
*** MODULE NAME:() 2012-08-14 02:37:13.249
*** ACTION NAME:() 2012-08-14 02:37:13.249
Dump continued FROM file: /u01/app/oracle/diag/rdbms/odaenmo/odaenmo2/trace/odaenmo2_gen0_16906.trc
ORA-00700: 软内部错误, 参数: [kcbl2vfyfh_action], [db_flash_cache_file integrity CHECK failed], [], [], [], [], [], [], [], [], [], []
========= Dump FOR incident 416041 (ORA 700 [kcbl2vfyfh_action]) ========
----- Beginning of Customized Incident Dump(s) -----
Flash cache parameter expected: db_unique_name odaenmo, dbid 196211168 inst_id 2, bpid 3, bsz 8192
File header db_unique_name odaenmo, dbid 196211168, inst_id 1, bpid 3, bsz 8192 filetype_str "Oracle RDBMS Flash Cache File"
----- End of Customized Incident Dump(s) -----
*** 2012-08-14 02:37:13.284
dbkedDefDump(): Starting incident DEFAULT dumps (flags=0x2, level=3, mask=0x0)
----- SQL Statement (None) -----
CURRENT SQL information unavailable - no cursor.
----- Call Stack Trace -----
calling              CALL     entry                argument VALUES IN hex      
location             TYPE     point                (? means dubious VALUE)     
-------------------- -------- -------------------- ----------------------------
skdstdst()+36        CALL     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFF7312DDF8 ? 000000001 ?
                                                   000000001 ? 000000002 ?
ksedst1()+98         CALL     skdstdst()           000000000 ? 000000000 ?
                                                   7FFF7312DDF8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksedst()+34          CALL     ksedst1()            000000000 ? 000000001 ?
                                                   7FFF7312DDF8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbkedDefDump()+2741  CALL     ksedst()             000000000 ? 000000001 ?
                                                   7FFF7312DDF8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksedmp()+36          CALL     dbkedDefDump()       000000003 ? 000000002 ?
                                                   7FFF7312DDF8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksfdmp()+64          CALL     ksedmp()             000000003 ? 000000002 ?
                                                   7FFF7312DDF8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbgexPhaseII()+1764  CALL     ksfdmp()             000000003 ? 000000002 ?
                                                   7FFF7312DDF8 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbgexExplicitEndInc  CALL     dbgexPhaseII()       7F6850206710 ? 7F68502118C0 ?
()+750                                             7FFF73139930 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbgeEndDDEInvocatio  CALL     dbgexExplicitEndInc  7F6850206710 ? 7F68502118C0 ?
nImpl()+767                   ()                   7FFF73139930 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbgeEndDDEInvocatio  CALL     dbgeEndDDEInvocatio  7F6850206710 ? 7F68502118C0 ?
n()+47                        nImpl()              7FFF73139930 ? 000000001 ?
                                                   000000000 ? 000000002 ?
kcbl2vfyfh_action()  CALL     dbgeEndDDEInvocatio  7F6850206710 ? 7F68502118C0 ?
+3373                         n()                  7FFF73139930 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksbabs()+771         CALL     kcbl2vfyfh_action()  7FFF7313B168 ? 000000018 ?
                                                   7FFF73139930 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksbrdp()+971         CALL     ksbabs()             7FFF7313B168 ? 000000018 ?
                                                   7FFF73139930 ? 000000001 ?
                                                   000000000 ? 000000002 ?
opirip()+618         CALL     ksbrdp()             7FFF7313B168 ? 000000018 ?
                                                   7FFF73139930 ? 000000001 ?
                                                   000000000 ? 000000002 ?
opidrv()+598         CALL     opirip()             000000032 ? 000000004 ?
                                                   7FFF7313C328 ? 000000001 ?
                                                   000000000 ? 000000002 ?
sou2o()+98           CALL     opidrv()             000000032 ? 000000004 ?
                                                   7FFF7313C328 ? 000000001 ?
                                                   000000000 ? 000000002 ?
opimai_real()+261    CALL     sou2o()              7FFF7313C300 ? 000000032 ?
                                                   000000004 ? 7FFF7313C328 ?
                                                   000000000 ? 000000002 ?
ssthrdmain()+252     CALL     opimai_real()        000000000 ? 7FFF7313C4F0 ?
                                                   000000004 ? 7FFF7313C328 ?
                                                   000000000 ? 000000002 ?
main()+196           CALL     ssthrdmain()         000000003 ? 7FFF7313C4F0 ?
                                                   000000001 ? 000000000 ?
                                                   000000000 ? 000000002 ?
__libc_start_main()  CALL     main()               000000003 ? 7FFF7313C690 ?
+244                                               000000001 ? 000000000 ?
                                                   000000000 ? 000000002 ?
_start()+36          CALL     __libc_start_main()  000A0B188 ? 000000001 ?
                                                   7FFF7313C688 ? 000000000 ?
                                                   000000000 ? 000000003 ?
--------------------- Binary Stack Dump ---------------------

从实例2上的详细TRACE文件,可以很轻易的分析出导致这个错误的原因。FLASH CACHE参数期望当前是实例2,但是从FLASH CACHE配置的文件读取,发现文件中记录的是实例1,导致这个错误的原因在于FLASH CACHE配置的文件是各个实例独立的,因此不能配置一个共享的文件给两个实例同时使用。
这个问题还可以通过检查FLASH CACHE的统计信息发现:

SQL> SELECT * FROM v$sysstat WHERE name LIKE '%flash cache eviction%';
STATISTIC# NAME                                         CLASS      VALUE    STAT_ID
---------- --------------------------------------- ---------- ---------- ----------
        97 flash cache eviction: invalidated                8          1 3647825193
        98 flash cache eviction: buffer pinned              8          0 3134415242
        99 flash cache eviction: aged OUT                   8          0 3307935113

将FLASH CACHE的配置修改如下:

SQL> SHOW parameter flash
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_flash_cache_file                  string      +REDO/flash/abc.flash
db_flash_cache_size                  big INTEGER 60G
db_flashback_retention_target        INTEGER     1440
SQL> ALTER system SET db_flash_cache_size = 30g scope = spfile;
System altered.
SQL> ALTER system SET db_flash_cache_file = '+REDO/flash/odademo1.flash' scope = spfile sid = 'odaenmo1';
System altered.
SQL> ALTER system SET db_flash_cache_file = '+REDO/flash/odademo2.flash' scope = spfile sid = 'odaenmo2';
System altered.

设置正确的FLASH CACHE后,重启数据库,ORA-700的错误不再出现。

This entry was posted in ORACLE and tagged , , , , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *