Gangmax Blog

Fix Dubbo RegistryService "Failed to connect" error

The error happens when starting Tomcat which contains a Dubbo service provider web application and the corresponding consumer web application. The exception stack trace is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
2017-03-28 10:57:12 [localhost-startStop-1] ERROR c.a.d.r.transport.AbstractClient -  [DUBBO] Failed to start NettyClient /172.17.0.1 connect to the server /0.0.0.0:9090 (check == false, ignore and retry later!), cause: client(url: dubbo://0.0.0.0:9090/com.alibaba.dubbo.registry.RegistryService?application=dmse-service-consumer&callbacks=10000&check=false&codec=dubbo&connect.timeout=10000&dubbo=2.5.3&heartbeat=60000&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&logger=slf4j&methods=lookup,unsubscribe,subscribe,unregister,register&organization=JD&owner=HuangGang&pid=32068&reconnect=false&send.reconnect=true&sticky=true&subscribe.1.callback=true&timeout=10000&timestamp=1490669831474&unsubscribe.1.callback=false) failed to connect to server /0.0.0.0:9090, error message is:Connection refused, dubbo version: 2.5.3, current host: 172.17.0.1
com.alibaba.dubbo.remoting.RemotingException: client(url: dubbo://0.0.0.0:9090/com.alibaba.dubbo.registry.RegistryService?application=dmse-service-consumer&callbacks=10000&check=false&codec=dubbo&connect.timeout=10000&dubbo=2.5.3&heartbeat=60000&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&logger=slf4j&methods=lookup,unsubscribe,subscribe,unregister,register&organization=JD&owner=HuangGang&pid=32068&reconnect=false&send.reconnect=true&sticky=true&subscribe.1.callback=true&timeout=10000&timestamp=1490669831474&unsubscribe.1.callback=false) failed to connect to server /0.0.0.0:9090, error message is:Connection refused
        at com.alibaba.dubbo.remoting.transport.netty.NettyClient.doConnect(NettyClient.java:124) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.remoting.transport.AbstractClient.connect(AbstractClient.java:280) ~[dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.remoting.transport.AbstractClient.<init>(AbstractClient.java:103) ~[dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.remoting.transport.netty.NettyClient.<init>(NettyClient.java:61) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.remoting.transport.netty.NettyTransporter.connect(NettyTransporter.java:37) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.remoting.Transporter$Adpative.connect(Transporter$Adpative.java) [na:2.5.3]
        at com.alibaba.dubbo.remoting.Transporters.connect(Transporters.java:67) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchanger.connect(HeaderExchanger.java:37) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.remoting.exchange.Exchangers.connect(Exchangers.java:102) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.dubbo.LazyConnectExchangeClient.initClient(LazyConnectExchangeClient.java:79) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.dubbo.LazyConnectExchangeClient.request(LazyConnectExchangeClient.java:104) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.request(ReferenceCountExchangeClient.java:81) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:96) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:48) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:74) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:227) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.common.bytecode.proxy0.subscribe(proxy0.java) [na:2.5.3]
        at com.alibaba.dubbo.registry.dubbo.DubboRegistry.doSubscribe(DubboRegistry.java:138) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.registry.support.FailbackRegistry.subscribe(FailbackRegistry.java:189) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.registry.integration.RegistryDirectory.subscribe(RegistryDirectory.java:133) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.registry.dubbo.DubboRegistryFactory.createRegistry(DubboRegistryFactory.java:80) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.registry.support.AbstractRegistryFactory.getRegistry(AbstractRegistryFactory.java:94) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.registry.RegistryFactory$Adpative.getRegistry(RegistryFactory$Adpative.java) [na:2.5.3]
        at com.alibaba.dubbo.registry.integration.RegistryProtocol.refer(RegistryProtocol.java:240) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper.refer(ProtocolFilterWrapper.java:60) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.ProtocolListenerWrapper.refer(ProtocolListenerWrapper.java:63) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.Protocol$Adpative.refer(Protocol$Adpative.java) [na:2.5.3]
        at com.alibaba.dubbo.config.ReferenceConfig.createProxy(ReferenceConfig.java:397) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.config.ReferenceConfig.init(ReferenceConfig.java:300) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.config.ReferenceConfig.get(ReferenceConfig.java:138) [dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.config.spring.ReferenceBean.getObject(ReferenceBean.java:65) [dubbo-2.5.3.jar:2.5.3]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:275) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:585) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) [spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866) [spring-context-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542) [spring-context-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:668) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:634) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:682) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:553) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:494) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:138) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at javax.servlet.GenericServlet.init(GenericServlet.java:158) [servlet-api.jar:3.1.FR]
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1183) [catalina.jar:8.5.11]
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1099) [catalina.jar:8.5.11]
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:989) [catalina.jar:8.5.11]
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4913) [catalina.jar:8.5.11]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5223) [catalina.jar:8.5.11]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [catalina.jar:8.5.11]
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752) [catalina.jar:8.5.11]
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728) [catalina.jar:8.5.11]
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734) [catalina.jar:8.5.11]
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:952) [catalina.jar:8.5.11]
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1823) [catalina.jar:8.5.11]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_121]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_121]
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384) ~[netty-3.2.5.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354) ~[netty-3.2.5.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276) ~[netty-3.2.5.Final.jar:na]
        ... 3 common frames omitted
2017-03-28 10:57:12 [localhost-startStop-1] ERROR c.a.d.registry.dubbo.DubboRegistry -  [DUBBO] Failed to subscribe consumer://172.17.0.1/com.alibaba.dubbo.registry.RegistryService?application=dmse-service-consumer&callbacks=10000&check=false&connect.timeout=10000&dubbo=2.5.3&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&logger=slf4j&methods=lookup,unsubscribe,subscribe,unregister,register&organization=JD&owner=HuangGang&pid=32068&reconnect=false&sticky=true&subscribe.1.callback=true&timeout=10000&timestamp=1490669831474&unsubscribe.1.callback=false, waiting for retry, cause: Failed to invoke the method subscribe in the service com.alibaba.dubbo.registry.RegistryService. Tried 3 times of the providers [0.0.0.0:9090] (1/1) from the registry 0.0.0.0:9090 on the consumer 172.17.0.1 using the dubbo version 2.5.3. Last error is: Failed to invoke remote method: subscribe, provider: dubbo://0.0.0.0:9090/com.alibaba.dubbo.registry.RegistryService?application=dmse-service-consumer&callbacks=10000&check=false&connect.timeout=10000&dubbo=2.5.3&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&logger=slf4j&methods=lookup,unsubscribe,subscribe,unregister,register&organization=JD&owner=HuangGang&pid=32068&reconnect=false&sticky=true&subscribe.1.callback=true&timeout=10000&timestamp=1490669831474&unsubscribe.1.callback=false, cause: client(url: dubbo://0.0.0.0:9090/com.alibaba.dubbo.registry.RegistryService?application=dmse-service-consumer&callbacks=10000&check=false&codec=dubbo&connect.timeout=10000&dubbo=2.5.3&heartbeat=60000&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&logger=slf4j&methods=lookup,unsubscribe,subscribe,unregister,register&organization=JD&owner=HuangGang&pid=32068&reconnect=false&send.reconnect=true&sticky=true&subscribe.1.callback=true&timeout=10000&timestamp=1490669831474&unsubscribe.1.callback=false) failed to connect to server /0.0.0.0:9090, error message is:Connection refused, dubbo version: 2.5.3, current host: 172.17.0.1

It shows the problem happens when initializing the consumer web application. Here is the Dubbo Spring configuration file content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:util="http://www.springframework.org/schema/util"
       xmlns:aop="http://www.springframework.org/schema/aop"
       xmlns:dubbo="http://code.alibabatech.com/schema/dubbo"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
                           http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
                           http://www.springframework.org/schema/context
                           http://www.springframework.org/schema/context/spring-context-3.2.xsd
                           http://www.springframework.org/schema/util
                           http://www.springframework.org/schema/util/spring-util-3.2.xsd
                           http://www.springframework.org/schema/aop
                           http://www.springframework.org/schema/aop/spring-aop.xsd
                           http://code.alibabatech.com/schema/dubbo
                           http://code.alibabatech.com/schema/dubbo/dubbo.xsd"
       default-autowire="byName"
       default-lazy-init="false">

    <!-- service provider name -->
    <dubbo:application name="dmse-service-consumer" owner="HuangGang" organization="JD" />
    <!-- zookeeper registry server address -->
    <dubbo:registry address="zookeeper://127.0.0.1:2181" />

    <!-- Enable monitoring feature -->
    <dubbo:monitor protocol="registry"/>

    <!-- Close starting-up check for service dependency's availability -->
    <dubbo:consumer check="false" />
    <dubbo:registry check="false" />

    <dubbo:reference id="sparkService" interface="com.jd.dmse.contract.SparkService" version="1.0.0" />
</beans>

Here is how I think:

  1. The reported error is about “RegistryService” in the consumer web application. So it must be related to the “Registry” configuration.

  2. There are three items in the configuration related to registry. The first 2 items seems not problematic.

  3. For the 3rd item(<dubbo:registry check=”false” />), the original purpose of it is to disable the “RegistryService” check when doing the initialization. But, after checking the official document:

说明：如果有多个不同的注册中心，可以声明多个<dubbo:registry>标签，……

  1. So, actually the 3rd item declares a new registry server although no server information is given, instead of disabling the initialization check for the first item which declares the “zookeeper://127.0.0.1:2181” registry server. From the log outout by default the “0.0.0.0:9090” server info is used if no server info is given.

So the fix is easy, remove the “<dubbo:registry check=”false” />” configuration item.

Test and it works as expectation.

Comments