My Application kills Hazelcast

[Ulrich Mayring]

I've successfully configured Hazelcast and verified that it is working within Payara 172 full with a test application. So in general Hazelcast is working fine, but my production app causes it to shut down on me. Here's from the logfile:

[2017-07-26T01:56:58.845+0200] [Payara 4.1] [INFO] [] [com.hazelcast.core.LifecycleService] [tid: _ThreadID=158 _ThreadName=admin-thread-pool::admin-listener(4)] [timeMillis: 1501027018845] [levelValue: 800] [[
  [10.99.0.1]:5701 [dev] [3.8] [10.99.0.1]:5701 is SHUTTING_DOWN]]

[2017-07-26T01:56:58.848+0200] [Payara 4.1] [INFO] [] [com.hazelcast.internal.partition.impl.MigrationManager] [tid: _ThreadID=158 _ThreadName=admin-thread-pool::admin-listener(4)] [timeMillis: 1501027018848] [levelValue: 800] [[
  [10.99.0.1]:5701 [dev] [3.8] Shutdown request of [10.99.0.1]:5701 is handled]]

[2017-07-26T01:56:58.821+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=158 _ThreadName=admin-thread-pool::admin-listener(4)] [timeMillis: 1501027018821] [levelValue: 1000] [[
  No valid EE environment for injection of my.package.HazelcastProvider]]

The my.package.HazelcastProvider looks like this (shortened to show only the relevant bits):

@ApplicationScoped
public class HazelcastProvider {

    private HazelcastInstance instance = null;

    @Produces
    @Singleton
    @Hazelcast
    public HazelcastInstance create() throws NamingException {
        if (instance == null) {
            Context ctx = new InitialContext();
            instance = (HazelcastInstance) ctx.lookup("payara/Hazelcast");
        }
        return instance;
    }

    public void close(@Disposes @Hazelcast HazelcastInstance instance) {
        instance.shutdown();
    }

Then I have defined my @Hazelcast annotation like so:

@Qualifier
@Retention(RUNTIME)
@Target({TYPE, METHOD, FIELD, PARAMETER})
public @interface Hazelcast {

}

And finally the culprit class that uses the injection:

@Singleton
public class Maintenance {

    @Inject
    @Hazelcast
    HazelcastInstance hazelcast;

    @Schedule(hour = "*", minute = "*/1", persistent=false)
    public void maintain() {
        Logger.getAnonymousLogger().info("Hazelcast " + hazelcast.getName() + " ready: " + hazelcast.getLifecycleService().isRunning());
    }

My test application did not have the fancy annotation and Provider class, it just did a manual JNDI lookup for "payara/Hazelcast" and that worked.

What happens here with my production app is that it takes down Hazelcast whenever it's deployed. The deployment itself works though and after restarting Payara my application can use Hazelcast just fine. But the next deployment kills Hazelcast again.

Now, I don't doubt that my code can be wrong in some way and that it perhaps tries to exploit a non-existing feature here. But I think that in any event it shouldn't be able to kill Hazelcast entirely. This could cause havoc for running applications, which have absolutely nothing to do with the possibly defect app. Maybe this lack of sandboxing is a bug?

Ulrich

[Steve Millidge]

A couple of things here;

1) You can already inject the Hazelcast instance into your code by using. I just noticed this is not documented anywhere.;

@Inject
HazelcastInstance instance;

2) In your code your producer is killing the Hazelcast instance on undeployment via the code below. This shuts down the Hazelcast Instance when the bean is destroyed which will happen at undeployment as it is a Singleton bean but may occur at other times;

    public void close(@Disposes @Hazelcast HazelcastInstance instance) {
        instance.shutdown();
    }


Steve


[Ulrich Mayring]

Hi Steve,

many thanks for your help, the injection works indeed out of the box and my provider was the culprit. I thought this dispose method would be called when Payara shuts down, not on every undeploy. Which, to me, begs the question whether an app should be allowed to shut down Hazelcast, when other apps are still using it?

Ulrich