某业务生产系统中，三节点的rac数据库中3号节点因故障停机后，进行crs的重启。重启完成后，发现数据库的监听未起来，启动的过程中并提示crs-5005错误。
一、问题过程
查看监听，发现监听no service
ywdb03@/oracle/grid/crs>lsnrctl statLSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production on 11-APR-2014 18:52:56Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
Start Date 11-APR-2014 18:52:50
Uptime 0 days 0 hr. 0 min. 6 sec
Trace Level off
Security ON: Local OS Authentication
SNMP ON
Listener Parameter File /oracle/grid/product/network/admin/listener.ora
Listener Log File /home/grid/diag/tnslsnr/ywdb03/listener/alert/log.xml
Listening Endpoints Summary...(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
The listener supports no services
The command completed successfully
通过srvctl去启动监听，发现一直提示CRS-5005错误
ywdb03@/oracle/grid/crs>srvctl start listener -n ywdb03
PRCR-1013 : Failed to start resource ora.LISTENER.lsnr
PRCR-1064 : Failed to start resource ora.LISTENER.lsnr on node ywdb03
CRS-5005: IP Address: 192.168.10.6 is already in use in the network
CRS-2674: Start of 'ora.ywdb03.vip' on 'ywdb03' failed二、解决过程
我们查看数据库另外另个节点，看是否有端口占用这个地址，结果发现在2号节点中发现问题。
ywdb02@/tmp/ibmsupt#ifconfig -a
en8: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>inet 192.168.10.2 netmask 0xffffff80 broadcast 192.168.10.127inet 192.168.10.7 netmask 0xffffff80 broadcast 192.168.10.127inet 192.168.10.6 netmask 0xffffff80 broadcast 192.168.10.127inet 192.168.10.5 netmask 0xffffff80 broadcast 192.168.10.127tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en9: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>inet 172.16.1.2 netmask 0xffffff00 broadcast 172.16.1.255tcp_sendspace 65536 tcp_recvspace 65536 rfc1323 1
en0: flags=5e080822,c0<BROADCAST,NOTRAILERS,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
en1: flags=5e080822,c0<BROADCAST,NOTRAILERS,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
en4: flags=5e080822,c0<BROADCAST,NOTRAILERS,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
en5: flags=5e080822,c0<BROADCAST,NOTRAILERS,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255inet6 ::1%1/0tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1删除该端口占用地址。
ywdb02@/tmp/ibmsupt#ifconfig en8 delete 192.168.10.6
ywdb02@/tmp/ibmsupt#ifconfig -a
en8: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>inet 192.168.10.2 netmask 0xffffff80 broadcast 192.168.10.127inet 192.168.10.7 netmask 0xffffff80 broadcast 192.168.10.127inet 192.168.10.5 netmask 0xffffff80 broadcast 192.168.10.127tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en9: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>inet 172.16.1.2 netmask 0xffffff00 broadcast 172.16.1.255tcp_sendspace 65536 tcp_recvspace 65536 rfc1323 1
en0: flags=5e080822,c0<BROADCAST,NOTRAILERS,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
en1: flags=5e080822,c0<BROADCAST,NOTRAILERS,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
en4: flags=5e080822,c0<BROADCAST,NOTRAILERS,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
en5: flags=5e080822,c0<BROADCAST,NOTRAILERS,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255inet6 ::1%1/0tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 13号节点重启监听服务，服务正常起来
ywdb03@/oracle/grid/crs>srvctl stop listener -n ywdb03
ywdb03@/oracle/grid/crs>srvctl start listener -n ywdb03三，问题说明
我们也可以通过重启所有的节点的监听服务来释放占用，最终恢复3号节点的监听服务（但由于当时我们另两个数据库节点正在支撑业务服务，所以不能通过该方式进行解决）。