Kubernetes: "Node failed to start!"

1,338 views
Skip to first unread message

miloss....@gmail.com

unread,
Aug 3, 2018, 4:34:18 AM8/3/18
to Hazelcast
Hello.
We are successfully using Hazelcast outside Docker with tcp-ip enabled (members definition) and also in Swarm using discovery.
However we are migrating to Kubernetes and when deployed to Kubernetes (https://github.com/hazelcast/hazelcast-kubernetes) following exception occures on "HazelcastInstance hazelcastInstance = HazelcastInstanceFactory.newHazelcastInstance(hazelcastConfig);"

Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
        ... 186 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
        ... 199 more
Caused by: java.lang.IllegalStateException: Node failed to start!
        at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:136)
        at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:195)
        at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:174)
        at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:124)
....

Versions:
<version.hazelcast-kubernetes>1.1.0</version.hazelcast-kubernetes>
<version.hazelcast>3.9.2</version.hazelcast>

I set logging to 'FINEST' but didn't find anything more specific.

XML configuration:
<hazelcast xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://www.hazelcast.com/schema/config
           http://www.hazelcast.com/schema/config/hazelcast-config-3.9.xsd">

    <properties>
            <property name="hazelcast.logging.type">slf4j</property>
            <property name="hazelcast.discovery.enabled">true</property>
            <property name="hazelcast.health.monitoring.level">OFF</property>
    </properties>

    <network>
    <join>
      <!-- deactivate normal discovery -->
        <multicast enabled="false"/>

        <!-- activate the Kubernetes plugin -->
        <discovery-strategies>
        <discovery-strategy enabled="true"
                            class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">

          <properties>
            <!-- configure discovery service API lookup -->
              <property name="service-name">MUCH-HAZELCAST-SERVICE</property>
              <property name="service-label-name">much-hazelcast-cluster01</property>
              <property name="service-label-value">true</property>
              <property name="namespace">MUCH-HAZELCAST-NAMESPACE</property>
          </properties>
        </discovery-strategy>
      </discovery-strategies>
    </join>
  </network>
</hazelcast>

Thank you in advance for your response.

Bet regards,

Milos.


Guglielmo Nigri

unread,
Aug 3, 2018, 7:40:24 AM8/3/18
to Hazelcast
Hi Milos,

Well, I noticed that while you show xml configuration (which looks correct at first glance) you are passing a `hazelcastConfig` variable to the factory.
How do you create the config object? Does it contain the same config as the xml?

By the stack trace it looks like you are using Spring (with autowire apparently), would you share that part of your code / Spring configuration?

Can you also share more of the log? Especially what gets logged before this exception (when `node.start()` is called internally and fails).

Best regards,
 
Guglielmo

miloss....@gmail.com

unread,
Aug 3, 2018, 9:35:17 AM8/3/18
to Hazelcast
Thank you for fast response.

XML configuration I posted is from file "hazelcast-kubernetes.xml" and in log I can see message "Configuring Hazelcast from 'hazelcast-kubernetes.xml'" so I believe the right configuration is loaded.
Loading of configuration:
@Bean
@Profile("kubernetes")
public Config configKubernetes(){
    return new ClasspathXmlConfig("hazelcast-kubernetes.xml");
}

Hazelcast is later created like this (simplified code): 
@Bean
public HazelcastInstance hazelcastInstance(Config hazelcastConfig){
//...configuration of caches etc. ...
HazelcastInstance hazelcastInstance = HazelcastInstanceFactory.newHazelcastInstance(hazelcastConfig); //here exception occures
return hazelcastInstance;
}


I'm using Hazelcast also for creating cache manager like this (in the same @Configuration class):
@Bean
public HazelcastCacheManager cacheManager(HazelcastInstance hazelcastInstance) {
    return new HazelcastCacheManager(hazelcastInstance);
}

In dependences I have "hazelcast-all" to be able to simply integrate Hazelcast with spring and also "hazelcast-kubernetes" for discovery in Kubernetes:
<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast-all</artifactId>
</dependency>
<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast-kubernetes</artifactId>
    <exclusions>
        <exclusion>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast</artifactId>
        </exclusion>
    </exclusions>
</dependency>

More log:
Multiple beans are not created because hazelcast instance is not created and it causes that cache manager also isn't created
...
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'timeCacheService' defined in class path resource [time.xml]: Cannot resolve reference to bean 'cacheManager' while setting bean property 'cacheManger'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cacheManager' defined in net.myapp.much.backend.config.HazelcastConfig: Unsatisfied dependency expressed through method 'cacheManager' parameter 0; nestedexception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in net.myapp.much.backend.config.HazelcastConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1531)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1276)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
        ... 166 more
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cacheManager' defined in net.myapp.much.backend.config.HazelcastConfig: Unsatisfied dependency expressed through method 'cacheManager' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in net.myapp.much.backend.config.HazelcastConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:467)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
        ... 176 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in net.myapp.much.backend.config.HazelcastConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
        ... 186 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
        ... 199 more
Caused by: java.lang.IllegalStateException: Node failed to start!
        at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:140)
        at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:196)
        at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:175)
        at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:125)
        at net.myapp.much.backend.config.HazelcastConfig.hazelcastInstance(HazelcastConfig.java:96)

Hope it helps. And thank you.
Milos.

Guglielmo Nigri

unread,
Aug 24, 2018, 11:03:21 AM8/24/18
to Hazelcast
Hi Milos,

My apologies, I was away and did not follow up on this.

What's the current status, did you succeed in running on Kubernetes?

Guglielmo

miloss....@gmail.com

unread,
Aug 30, 2018, 5:01:41 AM8/30/18
to Hazelcast
Hi.
I already fixed this problem.
Thank you.

Milos.

jonas....@aspect3.ch

unread,
Dec 18, 2018, 7:52:54 AM12/18/18
to Hazelcast
Hi, Miloš,

That would be nice to share how you solved your problem. What was the problem ?

Cheers,

Jonas

Shikhar Srivastava

unread,
Apr 5, 2022, 5:36:15 AM4/5/22
to Hazelcast

Hi,  Miloš and jonas,

I am facing the exact same issue right now. Could any of you tell me on how to resolve this? It would be a great help!

Thanks already,

Shikhar

pratik parshetti

unread,
Dec 15, 2022, 4:55:05 AM12/15/22
to Hazelcast
Can you please share the solution. I'm facing same issue.
Thanks

Reply all
Reply to author
Forward
0 new messages