数据库的Oracle 10204 RAC for Windows出现频繁节点重启的问题。
从告警日志看,当前节点的重启一般发生在节点刚启动或关闭时:
Thu May 03 17:22:45 2012 cluster interconnect IPC version:Oracle 9i Winsock2 TCP/IP IPC IPC Vendor 0 proto 0 Version 0.0 PMON started WITH pid=2, OS id=1616 DIAG started WITH pid=3, OS id=120 PSP0 started WITH pid=4, OS id=6104 LMON started WITH pid=5, OS id=3844 LMD0 started WITH pid=6, OS id=6120 LMS0 started WITH pid=7, OS id=3548 LMS1 started WITH pid=8, OS id=5688 LMS2 started WITH pid=9, OS id=3636 LMS3 started WITH pid=10, OS id=3588 MMAN started WITH pid=11, OS id=3168 DBW0 started WITH pid=12, OS id=3208 DBW1 started WITH pid=13, OS id=5784 LGWR started WITH pid=14, OS id=6208 CKPT started WITH pid=15, OS id=3100 SMON started WITH pid=16, OS id=5948 RECO started WITH pid=17, OS id=3748 CJQ0 started WITH pid=18, OS id=7152 MMON started WITH pid=19, OS id=4552 MMNL started WITH pid=20, OS id=6940 Thu May 03 17:22:46 2012 lmon registered WITH NM - instance id 1 (internal mem no 0) Thu May 03 17:22:46 2012 Reconfiguration started (OLD inc 0, NEW inc 8) List OF nodes: 0 1 Global Resource Directory frozen * allocate DOMAIN 0, invalid = TRUE Communication channels reestablished Error: KGXGN aborts the instance (6) Thu May 03 17:22:51 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_lmon_3844.trc: ORA-29702: ??????????? LMON: terminating instance due TO error 29702 Thu May 03 17:22:51 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_pmon_1616.trc: ORA-29702: ??????????? Thu May 03 17:22:51 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_psp0_6104.trc: ORA-29702: ??????????? Thu May 03 17:22:51 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_dbw0_3208.trc: ORA-29702: ??????????? Thu May 03 17:22:51 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_mman_3168.trc: ORA-29702: ??????????? Thu May 03 17:22:51 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_dbw1_5784.trc: ORA-29702: ??????????? Thu May 03 17:22:51 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_ckpt_3100.trc: ORA-29702: ??????????? Thu May 03 17:22:51 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_lgwr_6208.trc: ORA-29702: ??????????? Thu May 03 17:22:52 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_reco_3748.trc: ORA-29702: ??????????? Thu May 03 17:22:52 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_smon_5948.trc: ORA-29702: ??????????? Thu May 03 17:22:52 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_lms1_5688.trc: ORA-29702: ??????????? Thu May 03 17:22:52 2012 Errors IN file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_lms0_3548.trc: ORA-29702: ??????????? Instance TERMINATED BY LMON, pid = 3844 |
而从CSSD日志文件中可以发现下面的信息:
[ CSSD]2012-04-29 16:26:07.953 [7112] >TRACE: clssgmReconfigThread: completed FOR reconfig(13), WITH STATUS(1) 2012-04-30 09:07:04.718: [ OCROSD]utgdv:11:could NOT READ reg VALUE ocrmirrorconfig_loc os error= 操作系统找不到已输入的环境选项。 2012-04-30 09:07:04.718: [ OCROSD]utgdv:11:could NOT READ reg VALUE ocrmirrorconfig_loc os error= 操作系统找不到已输入的环境选项。 [ CSSD]2012-04-30 09:07:04.765 >USER: Copyright 2012, Oracle version 10.2.0.4.0 [ CSSD]2012-04-30 09:07:04.765 >USER: CSS daemon log FOR node crct-oadb, NUMBER 1, IN cluster crs [ CSSD]2012-04-30 09:07:04.765 [3780] >TRACE: clssscmain: local-ONLY SET TO FALSE [ clsdmt]Listening TO (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=61180)) [ CSSD]2012-04-30 09:07:04.781 [3780] >TRACE: clssnmReadNodeInfo: added node 1 (crct-oadb) TO cluster [ CSSD]2012-04-30 09:07:04.781 [3780] >TRACE: clssnmReadNodeInfo: added node 2 (crct-oapt) TO cluster [ CSSD]2012-04-30 09:07:04.828 [3724] >TRACE: clssnm_skgxninit: Compatible vendor clusterware NOT IN USE [ CSSD]2012-04-30 09:07:04.828 [3724] >TRACE: clssnm_skgxnmon: skgxn init failed [ CSSD]2012-04-30 09:07:04.843 [3780] >TRACE: clssnmNMInitialize: misscount SET TO (60) [ CSSD]2012-04-30 09:07:04.843 [3780] >TRACE: clssnmNMInitialize: Network heartbeat thresholds are: impending reconfig 30000 ms, reconfig START (misscount) 60000 ms [ CSSD]2012-04-30 09:07:04.843 [3780] >TRACE: clssnmDiskStateChange: state FROM 1 TO 2 disk (0/\\.\votedsk1) [ CSSD]2012-04-30 09:07:04.843 [3112] >TRACE: clssnmvDPT: spawned FOR disk 0 (\\.\votedsk1) [ CSSD]2012-04-30 09:07:06.843 [3112] >TRACE: clssnmDiskStateChange: state FROM 2 TO 4 disk (0/\\.\votedsk1) [ CSSD]2012-04-30 09:07:06.843 [4492] >TRACE: clssnmvKillBlockThread: spawned FOR disk 0 (\\.\votedsk1) initial sleep INTERVAL (1000)ms |
根据这些信息查询,发现属于10.2.0.4上的bug:10gR2/11gR1: Instances Abort With ORA-29702 When The Server is rebooted or shut down [ID 752399.1]。这个bug影响10.2.0.1到10.2.0.4以及11.1.0.6和11.1.0.7版本。
Oracle给出的解决方案是修改操作系统启动时调用的K96 link替换为K19 link。不过当前版本是Windows环境,显然这种解决方法并不适用。恐怕除了升级版本外,没有什么太好的其他解决方法。
将产品环境部署在Windows环境下的系统确实少见,而在Windows上部署RAC的就更是凤毛麟角了,而大多数这样部署的不只是对于Oracle不了解,连Windows和Linux的稳定性的差别都不是很清楚,出现各种问题的几率自然要大得多了。