客户的10.2.0.4 RAC for Hp-un环境碰到了这个错误。
错误信息为：
Wed Feb 29 19:42:05 2012 Errors IN file /opt/app/oracle/admin/orcl/udump/orcl1_ora_11261.trc: ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [] ORA-02068: following severe error FROM WEBDB.COM ORA-03113: end-of-file ON communication channel Wed Feb 29 19:42:05 2012 Errors IN file /opt/app/oracle/admin/orcl/udump/orcl1_ora_32036.trc: ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [] ORA-02068: following severe error FROM WEBDB.COM ORA-03113: end-of-file ON communication channel Wed Feb 29 19:42:05 2012 Errors IN file /opt/app/oracle/admin/orcl/udump/orcl1_ora_5935.trc: ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [] ORA-02068: following severe error FROM WEBDB.COM ORA-03113: end-of-file ON communication channel Wed Feb 29 19:42:05 2012 Errors IN file /opt/app/oracle/admin/orcl/udump/orcl1_ora_5026.trc: ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [] ORA-02068: following severe error FROM WEBDB.COM ORA-03113: end-of-file ON communication channel Wed Feb 29 19:42:05 2012 Errors IN file /opt/app/oracle/admin/orcl/udump/orcl1_ora_7620.trc: ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [] ORA-02068: following severe error FROM WEBDB.COM ORA-03113: end-of-file ON communication channel Wed Feb 29 19:42:08 2012 Trace dumping IS performing id=[cdmp_20120229194207] Wed Feb 29 19:42:17 2012 Trace dumping IS performing id=[cdmp_20120229194217] |
这个ORA-600[qersqCloseRem-2]错误非常罕见，在MOS上居然没有任何记载。不过从错误信息进行进一步的分析，这个错误发生在远端数据库的访问异常。
检查进一步的详细信息：
*** 2012-02-29 19:42:05.564 ksedmp: internal OR fatal error ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [] ORA-02068: following severe error FROM WEBDB.COM ORA-03113: end-of-file ON communication channel CURRENT SQL statement FOR this SESSION: SELECT ACCESS_LOG_SEQUENCE.NEXTVAL@WEBDB.COM FROM DUAL ----- PL/SQL Call Stack ----- object line object handle NUMBER name 0x39b5c3720 5 ECOMMERCE.P_USER_AT ----- Call Stack Trace ----- calling CALL entry argument VALUES IN hex location TYPE point (? means dubious VALUE) -------------------- -------- -------------------- ---------------------------- ksedst()+31 CALL ksedst1() 000000000 ? 000000001 ? 7FBFFF4370 ? 7FBFFF43D0 ? 7FBFFF4310 ? 000000000 ? ksedmp()+610 CALL ksedst() 000000000 ? 000000001 ? 7FBFFF4370 ? 7FBFFF43D0 ? 7FBFFF4310 ? 000000000 ? ksfdmp()+21 CALL ksedmp() 000000003 ? 000000001 ? 7FBFFF4370 ? 7FBFFF43D0 ? 7FBFFF4310 ? 000000000 ? . . . 0059DF200 ? 683F6E400000001 ? main()+116 CALL opimai_real() 000000002 ? 7FBFFFF4E0 ? 000000004 ? 7FBFFFF478 ? 0059DF200 ? 683F6E400000001 ? __libc_start_main() CALL main() 000000002 ? 7FBFFFF4E0 ? +219 000000004 ? 7FBFFFF478 ? 0059DF200 ? 683F6E400000001 ? _start()+42 CALL __libc_start_main() 0007139F8 ? 000000002 ? 7FBFFFF628 ? 0052B4BD0 ? 000000000 ? 000000002 ? --------------------- Binary Stack Dump --------------------- |
从详细TRACE分析，在问题发生时刻，正在通过数据库链读取远端序列的值。而此时出现的ORA-3113通信错误，多半与远端数据库状态异常有关。
检查远端数据库的告警日志，果然发现在问题出现时刻，数据库状态异常并最终导致了实例重启：
Wed Feb 29 19:39:29 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:39:30 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed OUT (ORA-3136) . . . Wed Feb 29 19:43:28 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:43:28 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:43:28 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:43:28 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:43:29 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:43:29 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:43:29 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:43:29 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:43:30 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:45:26 2012 PMON failed TO acquire latch, see PMON dump Wed Feb 29 19:46:32 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:46:33 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:46:34 2012 PMON failed TO acquire latch, see PMON dump Wed Feb 29 19:46:40 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:46:43 2012 WARNING: inbound connection timed OUT (ORA-3136) Wed Feb 29 19:46:44 2012 Errors IN file /opt/app/oracle/admin/orcl/bdump/orcl1_asmb_14614.trc: ORA-15064: communication failure WITH ASM instance ORA-03113: end-of-file ON communication channel Wed Feb 29 19:46:44 2012 ASMB: terminating instance due TO error 15064 Wed Feb 29 19:46:44 2012 System state dump IS made FOR LOCAL instance System State dumped TO trace file /opt/app/oracle/admin/orcl/bdump/orcl1_diag_14555.trc Wed Feb 29 19:46:47 2012 Shutting down instance (abort) License high water mark = 1623 Wed Feb 29 19:46:49 2012 Instance TERMINATED BY ASMB, pid = 14614 Wed Feb 29 19:46:52 2012 Instance TERMINATED BY USER, pid = 3684 |
显然远端数据库状态异常是这个ORA-600错误的直接原因。