10.2.0.4 RAC环境的ASM实例出现这个ORA-7445错误。
错误信息为：
Fri DEC 3 19:49:49 2010 Errors IN file /opt/app/oracle/admin/+ASM/bdump/+asm2_lck0_19876.trc: Fri DEC 3 19:50:44 2010 Errors IN file /opt/app/oracle/admin/+ASM/bdump/+asm2_ckpt_19863.trc: ORA-07445: exception encountered: core dump [kgskhighthreshold()+32] [SIGSEGV] [Address NOT mapped TO object] [0x0000001F6] [] [] Fri DEC 3 19:51:00 2010 Trace dumping IS performing id=[cdmp_20101203195100] Fri DEC 3 19:51:27 2010 ospid 19851: network interface WITH IP address 10.10.10.2 IS now running Fri DEC 3 19:51:42 2010 Errors IN file /opt/app/oracle/admin/+ASM/bdump/+asm2_pmon_19843.trc: ORA-00469: CKPT process TERMINATED WITH error Fri DEC 3 19:51:42 2010 Errors IN file /opt/app/oracle/admin/+ASM/bdump/+asm2_lmon_19849.trc: ORA-00469: CKPT process TERMINATED WITH error Fri DEC 3 19:51:43 2010 PMON: terminating instance due TO error 469 |
显然这个ORA-7445的错误最终导致了实例的崩溃，而检查对应的CKPT进程，确发现，导致问题的原因是ORADEBUG命令：
*** 2010-12-03 19:49:49.146 *** SERVICE NAME:() 2010-12-03 19:49:47.380 *** SESSION ID:(36.1) 2010-12-03 19:49:47.380 Received ORADEBUG command 'dump errorstack 1' FROM process Unix process pid: 19845, image: *** 2010-12-03 19:49:49.164 ksedmp: internal OR fatal error ----- Call Stack Trace ----- calling CALL entry argument VALUES IN hex location TYPE point (? means dubious VALUE) -------------------- -------- -------------------- ---------------------------- 惗?ksedst()+31 CALL ksedst1() 000000000 ? 000000001 ? 7FBFFFB0F0 ? 7FBFFFB150 ? 7FBFFFB090 ? 000000000 ? ksedmp()+610 CALL ksedst() 000000000 ? 000000001 ? 7FBFFFB0F0 ? 7FBFFFB150 ? 7FBFFFB090 ? 000000000 ? ksdxfdmp()+1118 CALL ksedmp() 000000001 ? 000000001 ? 7FBFFFB0F0 ? 7FBFFFB150 ? 7FBFFFB090 ? 000000000 ? ksdxcb()+1547 CALL ksdxfdmp() 7FBFFFC240 ? 000000011 ? 000000003 ? 7FBFFFC380 ? 7FBFFFC2E0 ? 000000000 ? sspuser()+111 CALL ksdxcb() 000000002 ? 000000011 ? 000000001 ? 000000001 ? 7FBFFFC2E0 ? 000000000 ? __funlockfile()+64 CALL sspuser() 000000002 ? 000000011 ? 000000001 ? 000000001 ? 7FBFFFC2E0 ? 000000000 ? __libc_poll()+47 signal __funlockfile() 7FBFFFCE20 ? 000000002 ? 000000BB8 ? FFFFFFFFFFFFFFFF ? 000000BB8 ? 000000000 ? --------------------- Binary Stack Dump --------------------- |
本身ORADEBUG就是Oracle未公开的工具，而在不恰当的时候去执行一些命令，导致Oracle出现异常也是非常正常的。
这个故障提醒我们，任何内部的工具都是双刃剑，使用不当就可能会造成严重的后果，因此在产生环境中使用一定要谨慎。