CRS启动报错Failed 3 to bind listening endpoint

Posted on 2012/02/03 by yangtingkun

客户的10.2 RAC for Linux x86-64环境，计划停机后，尝试启动CLUSTER和DB，在一个节点上启动成功，但是另一个节点启动报错。
检查没有成功启动的节点，发现CLUSTER启动在/tmp目录下留下了记录：

[root@smsdbrac1 root]# cd /tmp
[root@smsdbrac1 tmp]# ls -l
total 20
-rw-r--r-- 1 oracle dba 78 Nov 30 00:30 crsctl.14429
-rw-r--r-- 1 oracle dba 78 Nov 30 00:30 crsctl.14509
-rw-r--r-- 1 oracle dba 78 Nov 30 00:30 crsctl.14578
srwxr-xr-x 1 ncli ncli 0 Oct 8 15:22 mapping-ncli
srwxr-xr-x 1 oracle dba 0 May 5 2008 mapping-oracle
srwxr-xr-x 1 root root 0 May 14 2008 mapping-root
drwx------ 2 gdm gdm 4096 Oct 8 15:40 orbit-gdm
drwx------ 2 oracle dba 4096 Oct 10 16:07 orbit-oracle
[root@smsdbrac1 tmp]# more crsctl.14429
Failed 3 TO bind listening endpoint: (ADDRESS=(PROTOCOL=tcp)(HOST=dbprv1))

如果CLUSTER在启动时就报错，说明多半是操作系统或者硬件的配置上存在问题，而从这个信息上看，似乎与网络的配置有关。
由于这次计划停机本身就是要更换交换机，所以首先检查两个服务器是否可以正常连通，以及IP地址的配置是否存在问题。
在排除了这些可能性后，到MOS中查询了一下这个问题。
首先看到的问题和TNS_ADMIN环境变量有关，如果oracle用户设置了这个环境变量，且这个环境变量指定的目录下的sqlnet.ora设置了某些网络相关的参数，那么就会导致这个错误的产生。检查确实发现oracle用户设置了TNS_ADMIN环境变量：

[root@smsdbrac1 tmp]# env|grep TNS
[root@smsdbrac1 tmp]# su - oracle
[oracle@smsdbrac1 oracle]$ env|grep TNS
TNS_ADMIN=/opt/oracle/product/10.2.0/db/network/admin
[oracle@smsdbrac1 oracle]$ more /opt/oracle/product/10.2.0/db/network/admin/ 
afiedt.buf listener.ora samples shrept.lst tnsnames.ora

但是对应目录下并没有sqlnet.ora配置文件，不过处于稳妥，还是在oracle的启动shell中注释了这个环境变量的设置。
尝试启动CLUSTER，问题依旧。
在MOS上第二篇文章描述的问题是由于/etc/hosts文件的属性改变造成的，正常情况下，/etc/hosts的属性应该是(-rw-r–r–)。而检查客户服务器上该文件的配置，发现并不存在这个问题。
不过虽然这个文件本身的属性没有问题，但是检查这个文件的内容时，却发现了问题的端倪：

[root@dbrac1 tmp]# more /etc/hosts
# Do NOT remove the following line, OR various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.200.11 dbrac1.5wan.com.cn dbrac1
192.168.200.13 dbvip1
192.168.0.201 dbprv1.5wan.com.cn dbrac1
192.168.200.12 dbrac2
192.168.200.14 dbvip2
192.168.0.202 dbprv2

很明显，当前节点的private ip对应的主机名出现了错误，这里不但出现了域名，而且最后出现的主机名和public ip对应的主机名出现了重复。显然这是人为修改造成的异常错误。
检查报错信息中主机名信息，可以发现出现错误的就是private ip对应的主机名，显然导致错误的原因是Oracle从/etc/hosts中找不到private ip对应的主机名的配置。
将/etc/hosts中错误的内容修正后，重启CLUSTER，问题解决。

Posted in ORACLE | Tagged , , , | Leave a comment

ORA-600(17281)(1001)错误（二）

Posted on 2012/02/02 by yangtingkun

和第一次碰到这个错误不同，这次是在10.2环境中出现的。
ORA-600(17281)(1001)错误：http://yangtingkun.itpub.net/post/468/514806
告警日志中错误信息为：

Tue Jan 31 15:55:55 2012
Errors IN file /oracle/admin/ORCL/udump/orcl_ora_6225970.trc:
ORA-00600: internal error code, arguments: [17281], [1001], [0x70000059AB9BE38], [], [], [], [], []
ORA-01001: invalid cursor

详细TRACE信息如下：

Dump file /oracle/admin/ORCL/udump/orcl_ora_6225970.trc
Oracle DATABASE 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
WITH the Partitioning, OLAP AND DATA Mining options
ORACLE_HOME = /oracle/orcl/10.2.0
System name:	AIX
Node name:	porcl2
Release:	1
Version:	6
Machine:	00F67AD94C00
Instance name: ORCL
Redo thread mounted BY this instance: 1
Oracle process NUMBER: 503
Unix process pid: 6225970, image: oracleORCL@porcl2
*** SERVICE NAME:(SYS$USERS) 2012-01-31 15:55:55.419
*** SESSION ID:(4136.11880) 2012-01-31 15:55:55.419
*** 2012-01-31 15:55:55.419
ksedmp: internal OR fatal error
ORA-00600: internal error code, arguments: [17281], [1001], [0x70000059AB9BE38], [], [], [], [], []
ORA-01001: invalid cursor
CURRENT SQL statement FOR this SESSION:
DECLARE  cursor NlsParamsCursor IS    SELECT * FROM nls_session_parameters;BEGIN  SELECT Nvl(Lengthb(Chr(65536)), Nvl(Lengthb(Chr(256)), 1))    INTO :CharLength FROM dual;  FOR NlsRecord IN NlsParamsCursor loop    IF NlsRecord.parameter = 'NLS_DATE_LANGUAGE' THEN      :NlsDateLanguage := NlsRecord.value;    elsif NlsRecord.parameter = 'NLS_DATE_FORMAT' THEN      :NlsDateFormat := NlsRecord.value;    elsif NlsRecord.parameter = 'NLS_NUMERIC_CHARACTERS' THEN      :NlsNumericCharacters := NlsRecord.value;    elsif NlsRecord.parameter = 'NLS_TIMESTAMP_FORMAT' THEN      :NlsTimeStampFormat := NlsRecord.value;    elsif NlsRecord.parameter = 'NLS_TIMESTAMP_TZ_FORMAT' THEN      :NlsTimeStampTZFormat := NlsRecord.value;    END IF;  END loop;END;
----- Call Stack Trace -----
calling              CALL     entry                argument VALUES IN hex      
location             TYPE     point                (? means dubious VALUE)     
-------------------- -------- -------------------- ----------------------------
ksedst+001c          bl       ksedst1              000000008 ? 000000000 ?
ksedmp+0290          bl       ksedst               1048E2120 ?
ksfdmp+0018          bl       03F34B6C             
kgeriv+0108          bl       _ptrgl               
kgeasi+0118          bl       kgeriv               11046CD88 ? 1101B37C0 ?
                                                   104A364BC ? 7000000100067F8 ?
                                                   000000000 ?
kgicli+0188          bl       kgeasi               110190630 ? 1104AE040 ?
                                                   438100004381 ? 200000002 ?
                                                   200000002 ? 000000000 ?
                                                   0000003E9 ? 000000002 ?
kgidlt+0398          bl       kgicli               000000800 ? 000000001 ?
kgidel+0018          bl       kgidlt               FFFFFFFFFFF91A8 ? 000000000 ?
                                                   000000001 ? 000000000 ?
                                                   FFFFFFFFFFF9558 ?
perabo+00ac          bl       kgidel               FFFFFFFFFFF90E8 ? 1102247F0 ?
perdcs+0034          bl       perabo               FFFFFFFFFFF9440 ? 110190630 ?
                                                   FFFFFFFFFFF91A0 ?
peidcs+01dc          bl       perdcs               FFFFFFFFFFF92F0 ? 110190630 ?
kkxcls+00a4          bl       peidcs               FFFFFFFFFFF9558 ? 11046A640 ?
kxsClean+0044        bl       kkxcls               1100C99F8 ?
kxsCloseXsc+0444     bl       kxsClean             FFFFFFFFFFF9850 ?
kksCloseCursor+031c  bl       kxsCloseXsc          11046D038 ? 000000000 ?
opicca+00c4          bl       kksCloseCursor       104A503C0 ?
opiclo+0098          bl       opicca               1001376F0 ?
kpoclsa+0050         bl       03F35D58             
opiodr+0adc          bl       _ptrgl               
ttcpip+1020          bl       _ptrgl               
opitsk+10b0          bl       ttcpip               1100C99F8 ? 90000000063A614 ?
                                                   FFFFFFFFFFFB680 ?
                                                   2842288400000001 ?
                                                   9000000007307A8 ? 000000001 ?
                                                   11062EAF8 ? FFFFFFFFFFFB820 ?
opiino+0990          bl       opitsk               000010860 ? 000000000 ?
opiodr+0adc          bl       _ptrgl               
opidrv+0474          bl       opiodr               3C1028E590 ? 4102909B0 ?
                                                   FFFFFFFFFFFF490 ? 0A0082608 ?
sou2o+0090           bl       opidrv               3C02AB579C ? 4A0071248 ?
                                                   FFFFFFFFFFFF490 ?
opimai_real+01bc     bl       01FA4A34             
main+0098            bl       opimai_real          000000000 ? 000000000 ?
__start+0070         bl       main                 000000000 ? 000000000 ?
--------------------- Binary Stack Dump ---------------------

查询MOS发现这是JDBC THIN CLIENT导致的，具体文档参考Intermittent ORA-600 [17281], ORA[1001] With JDBC Thin Client [ID 436914.1]，这个问题与Bug 4359111非常类似，但是这个bug已经在10.2.0.2中被FIXED，而当前数据库的版本是10.2.0.3，而当前的问题影响的版本就包含10.2.0.3，且堆栈中的错误函数可以完全对应到文档给出的错误信息。
这个问题在10.2.0.4以后被修正，11g中同样修正了这个问题。如果短期数据库不会升级，且错误频繁发生，可以考虑替换客户端的ojdbc14.jar文件为10.2.0.4版，因为这个问题的解决是在JDBC客户端，因此客户端的驱动更新后，这个问题就可以被修正，而不需要升级数据库本身的版本。

Posted in BUG | Tagged , , , , , , | Leave a comment

Oracle VM用户手册

Posted on 2012/02/01 by yangtingkun

VM的安装文档解决不了问题，看看VM的用户手册有没有这方面的内容。
简单看了一下文档的目录，感觉这篇文档才是描述如何使用以及配置VM的，其中不但包括虚拟化、Oracle VM以及VM Manager的介绍，还详细描述了如何管理存储、网络、服务器池以及虚拟主机。
不过读完这篇文章能否解决VM创建过程中碰到的问题还未可知，这只能等到看完之后才能知晓。
照例给出在线阅读地址：http://docs.oracle.com/cd/E26996_01/e18549/index.html

Posted in BOOKS | Leave a comment

ORA-600(12333)错误

Posted on 2012/01/31 by yangtingkun

前不久才处理了一个ORA-600[12333]错误，不过那个问题是9i上的。
ORA-600(12333)错误和ORA-600(ttclxx1)错误：http://yangtingkun.itpub.net/post/468/526078
而这个错误是发生在10.2.0.3环境下：

Tue Jan 31 15:31:14 2012
Errors IN file /oracle/admin/ORCL/udump/orcl_ora_40960504.trc:
ORA-00600: internal error code, arguments: [12333], [0], [0], [0], [], [], [], []
详细TRACE文件如下：
Dump file /oracle/admin/ORCL/udump/orcl_ora_40960504.trc
Oracle DATABASE 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
WITH the Partitioning, OLAP AND DATA Mining options
ORACLE_HOME = /oracle/orcl/10.2.0
System name:	AIX
Node name:	porcl2
Release:	1
Version:	6
Machine:	00F67AD94C00
Instance name: ORCL
Redo thread mounted BY this instance: 1
Oracle process NUMBER: 36
Unix process pid: 40960504, image: oracleORCL@porcl2
*** ACTION NAME:() 2012-01-31 15:31:14.454
*** MODULE NAME:(TOAD 10.5.1.3) 2012-01-31 15:31:14.454
*** SERVICE NAME:(SYS$USERS) 2012-01-31 15:31:14.454
*** SESSION ID:(5422.6391) 2012-01-31 15:31:14.454
--------------------
 PROTOCOL VIOLATION 
--------------------
-----Page before-----
1108880E0 00 00 00                                         [...]             
-----Current Page-----
nsbfdbpt 1108880e0
ifccrcp = 1108880e3, ifccrcl = 1108888d5
buffer LENGTH = 2034
1108880E0          00 00 00 00 00 00 00 00 00 00 00 00 00     [.............]
1108880F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
110888100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
110888110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
110888120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
110888130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
110888140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
110888150 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
110888160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
.
.
.
110888890 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1108888A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1108888B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1108888C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1108888D0 00 00 00 00 00                                   [.....]           
hstflg:  0x40002991
hstcflg: 0x00000000
hstpro:  5
hstccs:  (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.10.0.27)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=orcl)(CID=(PROGRAM=C:\Program Files\Quest Software\Toad FOR Oracle 10.5\Toad.exe)(HOST=HOST9)(USER=8000130559))))
--- dump of hsttti ---
1100C9E20 00 04 01 0C 0E 09 0B 0F 04 0A 03 03 01 00 00 0A  [................]
1100C9E30 00 00 00 00 00 00 00 01 01 05 05 01 05 05 05 05  [................]
1100C9E40 06 06 00 00 00 00 00 07 03 03 00 00 00 00 00 00  [................]
1100C9E50 00 00 00 00 00 00 00 00 00 00 13 00 00 00 00 00  [................]
1100C9E60 00 00 00 00 0C 00 00 00 00 00 14 00 00 00 00 00  [................]
1100C9E70 00 00 00 00 00 00 00 00 00 00 00 0D 00 00 11 11  [................]
1100C9E80 04 09 00 00 00 00 05 00 22 00 12 00 13 13 15 15  [........".......]
1100C9E90 17 17 17 17 21 03 00 00 03 13 13 13 00 00 00 00  [....!...........]
1100C9EA0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1100C9EB0 00 00 04 2E 00 00 00 00 0F 0F 0F 1F 20 00 00 00  [............ ...]
1100C9EC0 00 00 00 00 00 00 00 00 00 00 00 00 0F 00 00 00  [................]
1100C9ED0 00 00 00 00 00 00 00 00 2A 00 00 00 00 00 00 00  [........*.......]
1100C9EE0 00 00 00 23 23 23 00 00 00 00 00 00 00 00 00 00  [...###..........]
1100C9EF0 22 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ["...............]
1100C9F00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1100C9F10 00 13 00 00 00 00 00 00 00 00 00 00 00 00 00 35  [...............5]
1100C9F20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1100C9F30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1100C9F40 00 00 03 01 03 03 03 00 00 00 03 03 03 03 03 03  [................]
1100C9F50 03 03 03 03 03 03 03 03 03 03 00 03 03 03 03 03  [................]
1100C9F60 03 03 03 03 00 00 00 03 03 03 00 03 00 03 03 03  [................]
1100C9F70 03 03 03 03 03 03 03 01 00 03 03 00 03 03 00 00  [................]
1100C9F80 00 00 03 03 00 00 00 03 00 00 00 03 00 00 00 00  [................]
1100C9F90 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 03  [................]
1100C9FA0 03 03 03 03 03 03 03 03 00 03 03 03 03 03 03 03  [................]
1100C9FB0 03 03 00 00 03 03 03 03 00 00 00 00 00 03 03 03  [................]
1100C9FC0 03 03 03 03 03 03 03 03 03 03 03 03 00 03 03 03  [................]
1100C9FD0 03 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1100C9FE0 00 00 1C 00 00 00 00 00 00 00 24 25 26 27 28 1E  [..........$%&'(.]
1100C9FF0 00 00 00 2D 00 00 03 03 03 03 03 03 03 03 03 03  [...-............]
1100CA000 03 03 03 2C 00 00 00 00 00 00 00 00 00 00 00 00  [...,............]
1100CA010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1100CA020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  [................]
1100CA030 00 00 00 00 1A 00 00 00 00 00 00 00 00           [.............]   
ttclxlccst:     871
ttclxrccst:     871
ttclxlncst:     2000
ttclxrncst:     2000
ttclxccl2nr:    1
ttclxccn2lr:    1
ttclxncl2nr:    1
ttclxncn2lr:    1
ttclxrccminbpc: 1
ttclxrccmaxbpc: 3
ttclxrncminbpc: 2
ttclxrncmaxbpc: 2
ttclxflags:     0x23
prev funcode: 0x5e
*** 2012-01-31 15:31:14.470
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [12333], [0], [0], [0], [], [], [], []
Current SQL statement for this session:
Select CREATED, LAST_DDL_TIME, OBJECT_ID, STATUS, TIMESTAMP
FROM SYS.ALL_OBJECTS
WHERE object_name = :nm
AND   object_type = :t
AND   owner = :o
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst+001c          bl       ksedst1              900000000312A74 ? 000000000 ?
ksedmp+0290          bl       ksedst               1048E2120 ?
ksfdmp+0018          bl       03F34B6C             
kgeriv+0108          bl       _ptrgl               
kgesiv+0080          bl       kgeriv               000000013 ? 1100D1010 ?
                                                   000000000 ? 00000000A ?
                                                   1100C9B50 ?
ksesic3+0060         bl       kgesiv               104F4E830 ? 000000023 ?
                                                   11025E118 ?
                                                   800000000000F032 ?
                                                   3FFC000000000003 ?
opitsk+0ec0          bl       01FA62EC             
opiino+0990          bl       opitsk               000010860 ? 000000000 ?
opiodr+0adc          bl       _ptrgl               
opidrv+0474          bl       opiodr               3C1028E590 ? 4102909B0 ?
                                                   FFFFFFFFFFFF490 ? 0A0082608 ?
sou2o+0090           bl       opidrv               3C02AB579C ? 4A0071248 ?
                                                   FFFFFFFFFFFF490 ?
opimai_real+01bc     bl       01FA4A34             
main+0098            bl       opimai_real          000000000 ? 000000000 ?
__start+0070         bl       main                 000000000 ? 000000000 ?
--------------------- Binary Stack Dump ---------------------

这个600错误和其他的错误并不相同，上来不是错误信息和错误SQL，而是直接的页面DUMP信息，而且一开始就包含了协议冲突的错误信息。
经分析，问题的描述可以参考文档Bug 8625762 ORA-3137 [12333] due to bind data not read from wire。导致问题的原因在于网络连接中包含的绑定变量信息未被读取。
这个bug影响范围比较广，10.2.0.4、10.2.0.5、11.1.0.6和11.1.0.7都可能存在这个问题，而如果要解决这个问题，需要将版本升级到11.2.0.1或11.1.0.7.3，如果是10.2版本想要通过专门的补丁来修正这个问题，那么至少要先将数据库版本升级到10.2.0.4.3。
除了打补丁和升级之外，还可以尝试通过设置隐含参数”_optim_peek_user_binds”为false来避免这个bug。

Posted in BUG | Tagged , , | Leave a comment

Oracle VM服务器安装和升级手册总结

Posted on 2012/01/30 by yangtingkun

其实文章很短，如果有时间半天就能看完。
感觉Oracle的VM离广泛的应用还有一段距离，在测试过程中，碰到了若干问题都很难解决，一方面错误信息不明确，另一方面网络相关的资料少得可怜，难以进一步分析解决。
其实无论是VM SERVER的安装还是VM MANAGER的安装，步骤都很顺利，没有什么麻烦的，但是将VM SERVER通过图形化工具添加到VM MANAGER中就碰到了很多问题，感觉还不如直接通过命令行方式配置更加直观和方便。
而这篇文档只是在安装结束后就结束了，而关键的注册步骤确没有说明，而且一些错误码也没有给出参考，总的来说无论是产品还是文档VM都还不很成熟。

Posted in BOOKS | Leave a comment

ORA-610内部错误

Posted on 2012/01/29 by yangtingkun

ORA-600内部错误司空见惯了，这个ORA-610的内部错误还是第一次见到。
完整的相关错误信息如下：

Wed Jun 22 10:01:08 2011
Errors IN file /oracle10g/admin/orcl/bdump/orcl_ora_27376.trc:
ORA-00610: Internal error code
Wed Jun 22 10:02:16 2011
WARNING: inbound connection timed OUT (ORA-3136)
Wed Jun 22 10:05:02 2011
Process J000 died, see its trace file
Wed Jun 22 10:05:07 2011
kkjcre1p: unable TO spawn jobq slave process 
Wed Jun 22 10:05:07 2011
Errors IN file /oracle10g/admin/orcl/bdump/orcl_cjq0_12889.trc:
Wed Jun 22 10:07:21 2011
WARNING: inbound connection timed OUT (ORA-3136)
Wed Jun 22 10:20:58 2011
Errors IN file /oracle10g/admin/orcl/udump/orcl_ora_27414.trc:
ORA-27102: OUT OF memory
HPUX-ia64 Error: 12: NOT enough SPACE
Additional information: 108
Additional information: 458752
Wed Jun 22 10:20:59 2011
Errors IN file /oracle10g/admin/orcl/udump/orcl_ora_27414.trc:
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address NOT mapped TO object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: OUT OF memory
HPUX-ia64 Error: 12: NOT enough SPACE
Additional information: 108
Additional information: 458752
Wed Jun 22 10:21:03 2011
Errors IN file /oracle10g/admin/orcl/udump/orcl_ora_27414.trc:
ORA-04030: OUT OF process memory WHEN trying TO allocate 753136 bytes (pga heap,kco buffer)
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address NOT mapped TO object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: OUT OF memory
HPUX-ia64 Error: 12: NOT enough SPACE
Additional information: 108
Additional information: 458752
Wed Jun 22 10:21:03 2011
Errors IN file /oracle10g/admin/orcl/udump/orcl_ora_27414.trc:
ORA-00081: address range [0x6000000000127430, 0x6000000000127434) IS NOT readable
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address NOT mapped TO object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: OUT OF memory
HPUX-ia64 Error: 12: NOT enough SPACE
Additional information: 108
Additional information: 458752
Wed Jun 22 10:21:37 2011
Errors IN file /oracle10g/admin/orcl/udump/orcl_ora_7454.trc:
ORA-27103: internal error
HPUX-ia64 Error: 28: No SPACE LEFT ON device
Additional information: 108
Additional information: 262144
Additional information: -444973056
Wed Jun 22 10:21:39 2011
Errors IN file /oracle10g/admin/orcl/udump/orcl_ora_7454.trc:
ORA-04030: OUT OF process memory WHEN trying TO allocate 753136 bytes (pga heap,kco buffer)
ORA-27103: internal error
HPUX-ia64 Error: 28: No SPACE LEFT ON device
Additional information: 108
Additional information: 262144
Additional information: -444973056

从错误信息上看，在出现ORA-610错误后，很快JOB进程死到，随后系统spawn新的JOB进程时报错，然后就是ORA-27102内部不足的错误，并引发了ORA-7445[kghalf]的错误，以及操作系统上的错误以及ORA-27103以及ORA-4030错误。
仅从现有的错误信息上分析，应该是一个JOB进程运行消耗资源较多的SQL，导致PGA不足，而Oracle尝试从SWAP空间分配内存，但是SWAP空间不足，从而导致了这个问题。
在MOS文档ORA-00610 And/Or “unable to spawn jobq slave process ” And/Or “Process(<>) creation failed” In The Alert Log And/Or TNS-12518/ TNS-12500 In Listener Log [ID 416244.1]中描述了这个问题，导致问题的原因是由于缺少操作系统资源所致，而导致这种内存不足错误的原因多半是由于SWAP空间不足所致。

Posted in ORACLE | Tagged , , , , , , | Leave a comment

ORA-7445(kcsgrsn)错误

Posted on 2012/01/28 by yangtingkun

告警日志中出现这个错误。
错误信息如下：

Thu DEC 23 01:00:02 2010
Starting background process EMN0
EMN0 started WITH pid=290, OS id=21705
Thu DEC 23 01:00:02 2010
Shutting down instance: further logons disabled
Thu DEC 23 01:00:10 2010
Stopping background process QMNC
Thu DEC 23 01:00:11 2010
Stopping background process CJQ0
Thu DEC 23 01:00:13 2010
Stopping background process MMNL
Thu DEC 23 01:00:14 2010
Stopping background process MMON
Thu DEC 23 01:00:15 2010
Shutting down instance (immediate)
License high water mark = 236
Thu DEC 23 01:00:15 2010
Stopping Job queue slave processes
Thu DEC 23 01:00:15 2010
Job queue slave processes stopped
Thu DEC 23 01:05:17 2010
Active CALL FOR process 25698 USER 'oracle' program 'oracleora10@wfrb1'
Active CALL FOR process 8983 USER 'oracle' program 'oracleora10@wfrb1'
Active CALL FOR process 18425 USER 'oracle' program 'oracleora10@wfrb1'
Active CALL FOR process 23144 USER 'oracle' program 'oracleora10@wfrb1'
Active CALL FOR process 28505 USER 'oracle' program 'oracleora10@wfrb1'
Active CALL FOR process 20150 USER 'oracle' program 'oracleora10@wfrb1'
Active CALL FOR process 23164 USER 'oracle' program 'oracleora10@wfrb1'
SHUTDOWN: waiting FOR active calls TO complete.
Thu DEC 23 01:20:10 2010
MMNL absent FOR 1202 secs; Foregrounds taking OVER
Thu DEC 23 02:00:50 2010
SHUTDOWN: Active sessions prevent DATABASE close operation
Thu DEC 23 02:00:50 2010
Starting ORACLE instance (normal)
Thu DEC 23 02:00:53 2010
ALTER DATABASE OPEN
Thu DEC 23 02:00:53 2010
ORA-1531 signalled during: ALTER DATABASE OPEN...
Thu DEC 23 02:15:31 2010
Errors IN file /oracleapp/oracle10g/admin/ora10/udump/ora10_ora_27379.trc:
ORA-01089: immediate shutdown IN progress - no operations are permitted
Thu DEC 23 02:15:32 2010
Errors IN file /oracleapp/oracle10g/admin/ora10/udump/ora10_ora_27379.trc:
ORA-07445: exception encountered: core dump [kcsgrsn()+400] [SIGSEGV] [Invalid permissions FOR mapped object] [0x0000006EC] [] []
ORA-01089: immediate shutdown IN progress - no operations are permitted
Thu DEC 23 02:15:33 2010
Errors IN file /oracleapp/oracle10g/admin/ora10/udump/ora10_ora_27379.trc:
ORA-07445: exception encountered: core dump [kcsgrsn()+400] [SIGSEGV] [Invalid permissions FOR mapped object] [0x0000006EC] [] []
ORA-07445: exception encountered: core dump [kcsgrsn()+400] [SIGSEGV] [Invalid permissions FOR mapped object] [0x0000006EC] [] []
ORA-01089: immediate shutdown IN progress - no operations are permitted

显然这个错误的出现是在SHUTDOWN的过程中，那么多半导致问题的原因会与关闭数据库有关。
查询MOS发现对应的bug信息为Bug 4687581 – Dump (kcsgrsn) during SHUTDOWN [ID 4687581.8]，导致问题的原因和预期的一样，在SHUTDOWN过程中，前台的应用可能会出现ORA-7445[kcsgrsn]的错误。
这个错误没有什么不良影响，完全可以忽略，10.2.0.3及以下版本均可能出现这个问题，Oracle在10.2.0.4和11.1.0.6中解决了这个bug。

Posted in BUG | Tagged , , | 1 Comment

ORA-600(17113)错误

Posted on 2012/01/27 by yangtingkun

又是一个PL/DEVELOPER的bug。
客户数据库是9.2.0.8，告警日志中出现大量下面的错误信息：

Thu DEC 1 08:37:52 2011
Errors IN file /oracle9/app/admin/db/udump/db1_ora_1995124.trc:
ORA-00600: internal error code, arguments: [17113], [0x000000000], [], [], [], [], [], []
Thu DEC 1 08:37:52 2011
Trace dumping IS performing id=[cdmp_20111201083752]
Thu DEC 1 08:38:21 2011
Errors IN file /oracle9/app/admin/db/udump/db1_ora_1995124.trc:
ORA-00600: internal error code, arguments: [17113], [0x000000000], [], [], [], [], [], []
Thu DEC 1 08:39:47 2011
Errors IN file /oracle9/app/admin/db/udump/db1_ora_2982400.trc:
ORA-00600: internal error code, arguments: [17113], [0x000000000], [], [], [], [], [], []
Thu DEC 1 08:41:32 2011
Errors IN file /oracle9/app/admin/db/udump/db1_ora_1995124.trc:
ORA-00600: internal error code, arguments: [17113], [0x000000000], [], [], [], [], [], []
Thu DEC 1 08:42:13 2011
Errors IN file /oracle9/app/admin/db/udump/db1_ora_1995124.trc:
ORA-00600: internal error code, arguments: [17113], [0x000000000], [], [], [], [], [], []
Thu DEC 1 08:43:37 2011
Errors IN file /oracle9/app/admin/db/udump/db1_ora_1995124.trc:
ORA-00600: internal error code, arguments: [17113], [0x000000000], [], [], [], [], [], []

在详细TRACE文件/oracle9/app/admin/db/udump/db1_ora_1995124.trc中，可以看到这是一个DEVELOPER工具发起的会话：

   O/S info: USER: Administrator, term: PC2011100510, ospid: 2676:2564, machine: WORKGROUP\PC2011100510
              program: plsqldev.exe
    application name: PL/SQL Developer, hash VALUE=1190136663
    action name: Main SESSION, hash VALUE=1773317990

而导致错误的SQL语句为：

DECLARE
  t_owner varchar2(30);
  t_name  varchar2(30);
  PROCEDURE check_mview IS
    dummy INTEGER;
  BEGIN
    IF :object_type = 'TABLE' THEN
      SELECT 1 INTO dummy
      FROM sys.all_objects
      WHERE owner = :object_owner
      AND object_name = :object_name
      AND object_type = 'MATERIALIZED VIEW'
      AND rownum = 1;
      :object_type := 'MATERIALIZED VIEW';
    END IF;
  exception
    WHEN others THEN NULL;
  END;
BEGIN
  :sub_object := NULL;
  IF :deep != 0 THEN
    BEGIN
      IF :part2 IS NULL THEN
        SELECT constraint_type, owner, constraint_name
          INTO :object_type, :object_owner, :object_name
          FROM sys.all_constraints c
         WHERE c.constraint_name = :part1 AND c.owner = USER
           AND rownum = 1;
      ELSE
        SELECT constraint_type, owner, constraint_name, :part3
          INTO :object_type, :object_owner, :object_name, :sub_object
          FROM sys.all_constraints c
         WHERE c.constraint_name = :part2 AND c.owner = :part1
           AND rownum = 1;
      END IF;
      IF :object_type = 'P' THEN :object_type := 'PRIMARY KEY'; END IF;
      IF :object_type = 'U' THEN :object_type := 'UNIQUE KEY'; END IF;
      IF :object_type = 'R' THEN :object_type := 'FOREIGN KEY'; END IF;
      IF :object_type = 'C' THEN :object_type := 'CHECK CONSTRAINT'; END IF;        
      RETURN;
    exception
      WHEN no_data_found THEN NULL;
    END;
  END IF;
  :sub_object := :part2;
  IF (:part2 IS NULL) OR (:part1 != USER) THEN
    BEGIN
      SELECT object_type, USER, :part1
      INTO :object_type, :object_owner, :object_name
      FROM sys.all_objects
      WHERE owner = USER
      AND object_name = :part1
      AND object_type IN ('MATERIALIZED VIEW', 'TABLE', 'VIEW', 'SEQUENCE', 'PROCEDURE', 'FUNCTION', 'PACKAGE', 'TYPE', 'TRIGGER', 'SYNONYM')
      AND rownum = 1;
      IF :object_type = 'SYNONYM' THEN
        SELECT s.table_owner, s.table_name
          INTO t_owner, t_name
          FROM sys.all_synonyms s
         WHERE s.synonym_name = :part1
           AND s.owner = USER
           AND rownum = 1;
        SELECT o.object_type, o.owner, o.object_name
          INTO :object_type, :object_owner, :object_name
          FROM sys.all_objects o 
         WHERE o.owner = t_owner
           AND o.object_name = t_name
           AND object_type IN ('MATERIALIZED VIEW', 'TABLE', 'VIEW', 'SEQUENCE', 'PROCEDURE', 'FUNCTION', 'PACKAGE', 'TYPE', 'TRIGGER', 'SYNONYM')
           AND rownum = 1;
      END IF;
      :sub_object := :part2;
      IF :part3 IS NOT NULL THEN
        :sub_object := :sub_object || '.' || :part3;
      END IF;
      check_mview;
      RETURN;
    exception
      WHEN no_data_found THEN NULL;
    END;
  END IF;
  BEGIN
    SELECT s.table_owner, s.table_name
      INTO t_owner, t_name
      FROM sys.all_synonyms s
     WHERE s.synonym_name = :part1
       AND s.owner = 'PUBLIC'
       AND rownum = 1;
    SELECT o.object_type, o.owner, o.object_name
      INTO :object_type, :object_owner, :object_name
      FROM sys.all_objects o 
     WHERE o.owner = t_owner
       AND o.object_name = t_name
       AND object_type IN ('MATERIALIZED VIEW', 'TABLE', 'VIEW', 'SEQUENCE', 'PROCEDURE', 'FUNCTION', 'PACKAGE', 'TYPE', 'TRIGGER', 'SYNONYM')
       AND rownum = 1;
    check_mview;
    RETURN;
  exception
    WHEN no_data_found THEN NULL;
  END;
  :sub_object := :part3;
  BEGIN
    SELECT o.object_type, o.owner, o.object_name
      INTO :object_type, :object_owner, :object_name
      FROM sys.all_objects o
     WHERE o.owner = :part1
       AND o.object_name = :part2
       AND object_type IN ('MATERIALIZED VIEW', 'TABLE', 'VIEW', 'SEQUENCE', 'PROCEDURE', 'FUNCTION', 'PACKAGE', 'TYPE', 'TRIGGER', 'SYNONYM')
       AND rownum = 1;
    check_mview;
    RETURN;
  exception
    WHEN no_data_found THEN NULL;
  END;
  BEGIN
    IF :part2 IS NULL AND :part3 IS NULL
    THEN
      SELECT 'USER', NULL, :part1
      INTO :object_type, :object_owner, :object_name
      FROM sys.all_users u
      WHERE u.username = :part1
      AND rownum = 1;
      RETURN;
    END IF;
  exception
    WHEN no_data_found THEN NULL;
  END;
  BEGIN
    IF :part2 IS NULL AND :part3 IS NULL AND :deep != 0
    THEN
      SELECT 'ROLE', NULL, :part1
      INTO :object_type, :object_owner, :object_name
      FROM sys.session_roles r
      WHERE r.role = :part1
      AND rownum = 1;
      RETURN;
    END IF;
  exception
    WHEN no_data_found THEN NULL;
  END;
  :object_owner := NULL;
  :object_type := NULL;
  :object_name := NULL;
  :sub_object := NULL;
END;

这显然是DEVELOPER在获取数据字典源数据时执行的SQL，关于这个错误的描述可以参考文档：ORA-00600: internal error code, arguments: [17113] using pl/sql developer [ID 396326.1]。
Oracle针对这个问题的解决方案只有一句，把pl/sql developer升级到最新版本。

Posted in BUG | Tagged , , | Leave a comment

ORA-600(504)错误

Posted on 2012/01/26 by yangtingkun

数据库出现ORA-600[504]错误。
错误信息如下：

Tue Oct 5 10:16:18 2010
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_29837.trc:
ORA-00600: internal error code, arguments: [504], [0x38006C260], [160], [7], [shared pool], [4], [0], [0x38006C070]

显然这个问题与共享池有关，查询MOS发现这是9.2上的bug：Bug 2300743 – OERI[504] on “shared pool” latch possible [ID 2300743.8]。导致这个问题的原因是申请共享池的LATCH来释放KGLF HEAP。
这个问题和9.2版本的共享池设置有关，因此除了9.2.0.8版本修正了这个bug外，其余的9.2版本都可能碰到这个问题。除了升级之外，还有一个办法就是设置隐含参数_kghdsidx_count=1，这样可以禁止使用多个共享子池。

Posted in BUG | Tagged , , , | Leave a comment

ORA-600(12333)错误和ORA-600(ttclxx1)错误

Posted on 2012/01/25 by yangtingkun

数据库频繁出现ORA-12333错误，而且随后还出现了ORA-600错误。
错误信息如下：

Mon Aug 10 12:07:41 2009
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_9069.trc:
ORA-00600: internal error code, arguments: [12333], [0], [0], [1], [], [], [], []
Mon Aug 10 12:07:43 2009
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_9069.trc:
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-00600: internal error code, arguments: [12333], [0], [0], [1], [], [], [], []
Mon Aug 10 12:07:46 2009
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_9069.trc:
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-00600: internal error code, arguments: [12333], [0], [0], [1], [], [], [], []
Mon Aug 10 12:07:49 2009
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_9069.trc:
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-00600: internal error code, arguments: [12333], [0], [0], [1], [], [], [], []
Mon Aug 10 15:55:00 2009
Thread 1 advanced TO log SEQUENCE 86
CURRENT log# 2 seq# 86 mem# 0: /opt/oracle/oradata/eomsdb/redo02.log
Tue Aug 11 10:07:13 2009
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_16898.trc:
ORA-00600: internal error code, arguments: [12333], [0], [0], [0], [], [], [], []
Tue Aug 11 11:18:12 2009
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_18734.trc:
ORA-07445: exception encountered: core dump [0000000101A36FBC] [SIGSEGV] [Address NOT mapped TO object] [0x000000000] [] []
Wed Aug 12 09:15:17 2009
Thread 1 advanced TO log SEQUENCE 87
CURRENT log# 3 seq# 87 mem# 0: /opt/oracle/oradata/eomsdb/redo03.log
.
.
.
Tue Aug 18 08:17:00 2009
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_14394.trc:
ORA-00600: internal error code, arguments: [ttclxx1], [256], [768], [], [], [], [], []
Tue Aug 18 08:17:01 2009
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_14394.trc:
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-00600: internal error code, arguments: [ttclxx1], [256], [768], [], [], [], [], []
Tue Aug 18 08:17:03 2009
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_14394.trc:
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-00600: internal error code, arguments: [ttclxx1], [256], [768], [], [], [], [], []
Tue Aug 18 08:17:07 2009
Errors IN file /opt/oracle/admin/eomsdb/udump/eomsdb_ora_14394.trc:
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-07445: exception encountered: core dump [000000010100E8C8] [SIGSEGV] [Address NOT mapped TO object] [0x000000068] [] []
ORA-00600: internal error code, arguments: [ttclxx1], [256], [768], [], [], [], [], []

查询了MOS发现导致ORA-600 12333错误的原因很多，不好确定当前的问题，但是根据随后的ORA-600 ttclxx1找到了导致问题的原因。文档Bug 4577381: ORA-00600: INTERNAL ERROR CODE, ARGUMENTS: [TTCLXX1], [174], [512], [], []描述的就是这个问题，这个问题指向的基本bug为Bug 4551798: [8/8] PEND: ORA-00600 [12333] ON INSERT WITH LOB。
由于Oracle并没有给出这个bug原因的具体描述，只是说明这个bug与LOB的INSERT语句有关，不过Oracle说明在11g中这个bug会被解决。

Posted in BUG | Tagged , , , | Leave a comment