客户11.2 RAC环境出现大量的ORA-7445错误、ORA-4030错误以及ORA-27300、ORA-27301和ORA-27302错误。
错误信息为:
Sat DEC 31 02:33:18 2011 Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_p015_28613.trc: ORA-07445: exception encountered: core dump [_ndoprnt()+4] [SIGSEGV] [ADDR:0xFFFFFFFF7FFE5FE0] [PC:0xFFFFFFFF7ADA92B0] [Address NOT mapped TO object] [] ORA-04030: OUT OF process memory WHEN trying TO allocate 254488 bytes (QERGH hash-agg,kllcqas:kllsltba) Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_p018_28624.trc: ORA-07445: exception encountered: core dump [_ndoprnt()+4] [SIGSEGV] [ADDR:0xFFFFFFFF7FFE5FE0] [PC:0xFFFFFFFF7ADA92B0] [Address NOT mapped TO object] [] ORA-04030: OUT OF process memory WHEN trying TO allocate 254488 bytes (QERGH hash-agg,kllcqas:kllsltba) Exception [TYPE: SIGSEGV, Address NOT mapped TO object] [ADDR:0xFFFFFFFF7FFE5FE0] [PC:0xFFFFFFFF7ADA92B0, _ndoprnt()+4] [flags: 0x0, COUNT: 1]Exception [TYPE : SIGSEGV, Address NOT mapped TO object] [ADDR:0xFFFFFFFF7FFE5EF0] [PC:0xFFFFFFFF7ADA92B0, _ndoprnt()+4] [flags: 0x0, COUNT: 1] Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_p019_28626.trc: ORA-07445: exception encountered: core dump [_ndoprnt()+4] [SIGSEGV] [ADDR:0xFFFFFFFF7FFE5FE0] [PC:0xFFFFFFFF7ADA92B0] [Address NOT mapped TO object] [] ORA-04030: OUT OF process memory WHEN trying TO allocate 254488 bytes (QERGH hash-agg,kllcqas:kllsltba) . . . Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_p017_28617.trc: ORA-07445: exception encountered: core dump [_ndoprnt()+4] [SIGSEGV] [ADDR:0xFFFFFFFF7FFE5EF0] [PC:0xFFFFFFFF7ADA92B0] [Address NOT mapped TO object] [] ORA-04030: OUT OF process memory WHEN trying TO allocate 254488 bytes (QERGH hash-agg,kllcqas:kllsltba) Sat DEC 31 02:33:21 2011 Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_ora_26985.trc (incident=385310): ORA-04030: 在尝试分配 123416 字节 (QERHJ hash-joi,kllcqas:kllsltba) 时进程内存不足 Incident details IN: /app/diag/rdbms/orcl/orcl1/incident/incdir_385310/orcl1_ora_26985_i385310.trc USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. Sat DEC 31 02:33:24 2011 Exception [TYPE: SIGSEGV, Address NOT mapped TO object] [ADDR:0xFFFFFFFF7FFE7088] [PC:0x10780A9C8, sskgds_init()+8] [flags: 0x0, COUNT: 1] Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_p014_28611.trc: ORA-07445: exception encountered: core dump [sskgds_init()+8] [SIGSEGV] [ADDR:0xFFFFFFFF7FFE7088] [PC:0x10780A9C8] [Address NOT mapped TO object] [] ORA-04030: OUT OF process memory WHEN trying TO allocate 254488 bytes (QERGH hash-agg,kllcqas:kllsltba) Sat DEC 31 02:33:24 2011 Exception [TYPE: SIGSEGV, Address NOT mapped TO object] [ADDR:0xFFFFFFFF7FFE7088] [PC:0x10780A9C8, sskgds_init()+8] [flags: 0x0, COUNT: 1] Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_p023_28634.trc: ORA-07445: exception encountered: core dump [sskgds_init()+8] [SIGSEGV] [ADDR:0xFFFFFFFF7FFE7088] [PC:0x10780A9C8] [Address NOT mapped TO object] [] ORA-04030: OUT OF process memory WHEN trying TO allocate 254488 bytes (QERGH hash-agg,kllcqas:kllsltba) Sat DEC 31 02:33:24 2011 Exception [TYPE: SIGSEGV, Address NOT mapped TO object] [ADDR:0xFFFFFFFF7FFE7088] [PC:0x10780A9C8, sskgds_init()+8] [flags: 0x0, COUNT: 1] Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_p020_28628.trc: ORA-07445: exception encountered: core dump [sskgds_init()+8] [SIGSEGV] [ADDR:0xFFFFFFFF7FFE7088] [PC:0x10780A9C8] [Address NOT mapped TO object] [] ORA-04030: OUT OF process memory WHEN trying TO allocate 254488 bytes (QERGH hash-agg,kllcqas:kllsltba) Sat DEC 31 02:33:24 2011 Exception [TYPE: SIGSEGV, Address NOT mapped TO object] [ADDR:0xFFFFFFFF7FFE6F98] [PC:0x10780A9C8, sskgds_init()+8] [flags: 0x0, COUNT: 1] Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_p016_28615.trc: ORA-07445: exception encountered: core dump [sskgds_init()+8] [SIGSEGV] [ADDR:0xFFFFFFFF7FFE6F98] [PC:0x10780A9C8] [Address NOT mapped TO object] [] ORA-04030: OUT OF process memory WHEN trying TO allocate 254488 bytes (QERGH hash-agg,kllcqas:kllsltba) . . . Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_p022_28632.trc: ORA-07445: exception encountered: core dump [sskgds_init()+8] [SIGSEGV] [ADDR:0xFFFFFFFF7FFE6F98] [PC:0x10780A9C8] [Address NOT mapped TO object] [] ORA-04030: OUT OF process memory WHEN trying TO allocate 254488 bytes (QERGH hash-agg,kllcqas:kllsltba) Exception [TYPE: SIGSEGV, Address NOT mapped TO object] [ADDR:0xFFFFFFFF7FFC6CF8] [PC:0x107388144, kgdsdst()+36] [flags: 0x0, COUNT: 1] Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_ora_26985.trc (incident=385311): ORA-07445: 出现异常错误: 核心转储 [kgdsdst()+36] [SIGSEGV] [ADDR:0xFFFFFFFF7FFC6CF8] [PC:0x107388144] [Address NOT mapped TO object] [] ORA-04030: 在尝试分配 123416 字节 (QERHJ hash-joi,kllcqas:kllsltba) 时进程内存不足 Incident details IN: /app/diag/rdbms/orcl/orcl1/incident/incdir_385311/orcl1_ora_26985_i385311.trc USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. Sat DEC 31 02:33:32 2011 Dumping diagnostic DATA IN directory=[cdmp_20111231023332], requested BY (instance=1, osid=26985), summary=[incident=385310]. Sat DEC 31 02:33:37 2011 Process m000 died, see its trace file Sat DEC 31 02:34:35 2011 Process startup failed, error stack: Sat DEC 31 02:34:36 2011 Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_ora_3787.trc (incident=386654): ORA-04030: 在尝试分配 32792 字节 (QERHJ hash-joi,QERHJ Bit vector) 时进程内存不足 Incident details IN: /app/diag/rdbms/orcl/orcl1/incident/incdir_386654/orcl1_ora_3787_i386654.trc Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_psp0_46.trc: ORA-27300: OS system dependent operation:fork failed WITH STATUS: 12 ORA-27301: OS failure message: NOT enough SPACE ORA-27302: failure occurred at: skgpspawn3 Sat DEC 31 02:34:38 2011 Process m000 died, see its trace file USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. Process startup failed, error stack: Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_psp0_46.trc: ORA-27300: OS system dependent operation:fork failed WITH STATUS: 12 ORA-27301: OS failure message: NOT enough SPACE ORA-27302: failure occurred at: skgpspawn3 Sat DEC 31 02:34:43 2011 Process J000 died, see its trace file kkjcre1p: unable TO spawn jobq slave process Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_cjq0_2549.trc: Process startup failed, error stack: Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_psp0_46.trc: ORA-27300: OS system dependent operation:fork failed WITH STATUS: 12 ORA-27301: OS failure message: NOT enough SPACE ORA-27302: failure occurred at: skgpspawn3 Process J000 died, see its trace file kkjcre1p: unable TO spawn jobq slave process Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_cjq0_2549.trc: Sat DEC 31 02:34:47 2011 Exception [TYPE: SIGSEGV, Address NOT mapped TO object] [ADDR:0xFFFFFFFF7FFDE000] [PC:0x1008FD910, ksmpgd_init_ctx()+16] [flags: 0x0, COUNT: 1] ERROR: Unable TO normalize symbol name FOR the following short stack (at offset 210): dbgexProcessError()+164<-dbgePostErrorKGE()+1320<-dbkePostKGE_kgsf()+44<-kgeade()+416<-kgerelv()+144<-kgerev()+56<-kgerec5()+32<-ssexhd()+1448<-__sighndlr()+ 12<-call_user_handler()+992<-ksmpgd_init_ctx()+16<-_$c1A.ksmpgd_scan_pga_heaps_local_detail()+24<-ksm_4030_dump_internal()+208<-ksmarfg()+912<-kghnospc()+429 2<-kghalf()+1416<-qerhjInitializeManagementComponents()+1688<-qerhjFetch()+996<-rwsfcd()+108<-qerhjFetch()+460<-opifch2()+10836<-kpoal8()+3228<-opiodr()+1428 <-ttcpip()+1056<-opitsk()+1528<-opiino()+1000<-opiodr()+1428<-opidrv()+1100<-sou2o()+92<-opimai_real()+304<-ssthrdmain()+320<-main()+308<-_start()+380 Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_ora_3787.trc (incident=386655): ORA-07445: 出现异常错误: 核心转储 [ksmpgd_init_ctx()+16] [SIGSEGV] [ADDR:0xFFFFFFFF7FFDE000] [PC:0x1008FD910] [Address NOT mapped TO object] [] ORA-04030: 在尝试分配 32792 字节 (QERHJ hash-joi,QERHJ Bit vector) 时进程内存不足 Incident details IN: /app/diag/rdbms/orcl/orcl1/incident/incdir_386655/orcl1_ora_3787_i386655.trc USE ADRCI OR Support Workbench TO package the incident. See Note 411.1 at My Oracle Support FOR error AND packaging details. Sat DEC 31 02:34:53 2011 Process startup failed, error stack: Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_psp0_46.trc: ORA-27300: OS system dependent operation:fork failed WITH STATUS: 12 ORA-27301: OS failure message: NOT enough SPACE ORA-27302: failure occurred at: skgpspawn3 Process J000 died, see its trace file kkjcre1p: unable TO spawn jobq slave process Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_cjq0_2549.trc: Process startup failed, error stack: Errors IN file /app/diag/rdbms/orcl/orcl1/trace/orcl1_psp0_46.trc: ORA-27300: OS system dependent operation:fork failed WITH STATUS: 12 ORA-27301: OS failure message: NOT enough SPACE ORA-27302: failure occurred at: skgpspawn3 . . . |
可以看到,整个过程数据库经历了四个7445错误分别为:ORA-7445[_ndoprnt]、ORA-7445[sskgds_init]、ORA-7445[kgdsdst]和ORA-7445[ksmpgd_init_ctx],而这四个ORA-7445显然都是ORA-4030错误导致的。而导致ORA-4030错误的原因,显然是随后操作系统错误ORA-27300、ORA-27301和ORA-27302错误导致的。
查询MOS发现和文档Database Crashes With ORA-04030 ORA-07445 ORA-27300 ORA-27301 ORA-27302 [ID 580552.1]描述的问题非常一致,而导致问题产生的原因和刚才分析的一样,由于系统SWAP空间不足导致了数据库的内存分配报错。
释放SWAP或/tmp下的空间,可以避免错误的出现。