NO_HANDLERS: (NO_HANDLERS,-1) No handlers for address .....

[Vinay Samudre]

Hi All,

I saw couple of threads mentioning NO_HANDLERS error but can't see resolution, we are running 12 nodes vert.x(3.3.3) cluster in linux. 

Every now and then we are getting NO_HANDLERS errors for random calls, even though the node is running and some calls are working.

After restarting the node which has handler it all seems to be working. we are running all nodes in embedded mode, can anyone please shed some light on why this could be happening?

My be we are missing come config at vert.x level, or operating server level. We used to get this error with vert.x 2 but not as often as we are getting it now.

Thanks,

Vinay

Following method is used to start embedded vert.x:

public void start(String host, int port, String vertxDomain) {

ClusterManager mgr = new HazelcastClusterManager();

VertxOptions vertxOptions = new VertxOptions();

vertxOptions.setClustered(true);

String thisHostName = getDefaultAddress();

vertxOptions.setClusterHost(thisHostName);

vertxOptions.setClusterPort(port);

vertxOptions.setWorkerPoolSize(100);

vertxOptions.setMetricsOptions(

new DropwizardMetricsOptions().setJmxEnabled(true).setJmxDomain(host + "-" + vertxDomain + "-" + port)

.addMonitoredEventBusHandler(new Match().setValue(".*").setType(MatchType.REGEX)));

System.out.println("Starting vert.x for " + vertxDomain);

Vertx.clusteredVertx(vertxOptions, res -> {

if (res.succeeded()) {

vx = res.result();

vertxstarted.set(true);

System.out.println("Vertx initialised for : " + vertxDomain + vx);

} else {

System.out.println("Vertx initialisation failed: for " + vertxDomain + res.cause());

vertxstarted.set(false);

}

});

synchronized (vertxstarted) {

try {

while (false == vertxstarted.get()) {

System.out.println("waiting for vert.x to start ....." + vertxDomain);

Thread.sleep(1000);

}

} catch (Exception e) {

System.out.println("ERROR while waiting for vert.x to start ....." + vertxDomain);

e.printStackTrace(System.out);

}

}

try {

Thread.sleep(2000);

} catch (InterruptedException e) {

// TODO Auto-generated catch block

e.printStackTrace();

}

System.out.println("vert.x started for " + vertxDomain);

// registerExitHandler(pm);

}

and this is to deployment:

public void deployVerticle(boolean worker, boolean multiThreaded, int instances, JsonObject config, String main) {

AtomicBoolean verticleDeployed = new AtomicBoolean(false);

System.out.print("deploying " + main + "....");

DeploymentOptions options = new DeploymentOptions();

if (multiThreaded) {

options.setMultiThreaded(multiThreaded);

}

if (worker) {

options.setWorker(worker);

}

options.setInstances(instances);

if (!config.isEmpty()) {

options.setConfig(config);

}

System.out.println("Start deploying verticle: " + main);

Consumer<Vertx> runner = vertx -> {

try {

vertx.deployVerticle(main, options, res -> {

if (res.succeeded()) {

String verticleDeploymentId = res.result();

verticleDeployed.set(true);

System.out.println(main + " Verticle deployed successfully");

} else {

System.out.println(main + " Verticle deployed failed");

verticleDeployed.set(false);

}

});

} catch (Throwable t) {

System.out.println(main + " Verticle deployment failed: " + t);

t.printStackTrace();

}

};

runner.accept(vx);

synchronized (verticleDeployed) {

try {

while (false == verticleDeployed.get()) {

System.out.println(main + " waiting for verticle deployment to finish");

Thread.sleep(1000);

}

} catch (Exception e) {

System.out.println(main + "ERROR while waiting for verticle deployment to start .....");

e.printStackTrace(System.out);

}

}

try {

Thread.sleep(10000);

} catch (InterruptedException e) {

// TODO Auto-generated catch block

e.printStackTrace();

}

System.out.println("End deploying verticle: " + main);

}

[Asher Tarnopolski]

can be related to this: https://github.com/eclipse/vert.x/pull/1594

[Thomas SEGISMONT]

Does anything happen in production prior to losing event bus subscription? There are known issues about the cluster manager when multiple nodes are killed (or crash) at the same time. But if the cluster was working and nodes haven't been stopped, no data should be lost.

2017-02-06 10:19 GMT+01:00 Asher Tarnopolski <ata...@gmail.com>:

can be related to this: https://github.com/eclipse/vert.x/pull/1594

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/cef3a82d-5ab5-46fa-9f47-a36f7893ae04%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Vinay Samudre]

For various reasons like software enhancement releases, we are gracefully stopping, deploying and restarting services. All the nodes join correctly but for some reason some of the handlers in a node give NO_HANDLERS.

Does anyone know when this issue will be fixed? 

On Monday, February 6, 2017 at 8:50:23 PM UTC+11, Thomas Segismont wrote:

Does anything happen in production prior to losing event bus subscription? There are known issues about the cluster manager when multiple nodes are killed (or crash) at the same time. But if the cluster was working and nodes haven't been stopped, no data should be lost.

2017-02-06 10:19 GMT+01:00 Asher Tarnopolski <ata...@gmail.com>:

can be related to this: https://github.com/eclipse/vert.x/pull/1594

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

[Thomas SEGISMONT]

OK so it happens when you restart nodes, and I guess you restart a lot of them at the same time. One way to avoid loosing clustered data (event bus subscriptions and haInfo) is to increase the number of backups for the corresponding multimap (__vertx.subs) and map (__vertx.haInfo). Beware that versions up to 3.3.3 have a bug in the default cluster setup file (https://github.com/vert-x3/vertx-hazelcast/commit/4b6e6f815a221306ae93c500b5e62d8c1452397a)

We are working for a fix in 3.4

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f57cba16-a472-4114-90e1-1763e72c2501%40googlegroups.com.

[Vinay Samudre]

Thanks Thomas.

We are using 3.3.3, I can't find "__vertx.haInfo" also "__vertx.subs" is declared as <map name="__vertx.subs">

Can you please point me to correct cluster.xml version for vert.x 3.3.3?

cheers,

Vinay

On Tuesday, February 7, 2017 at 7:32:55 PM UTC+11, Thomas Segismont wrote:

OK so it happens when you restart nodes, and I guess you restart a lot of them at the same time. One way to avoid loosing clustered data (event bus subscriptions and haInfo) is to increase the number of backups for the corresponding multimap (__vertx.subs) and map (__vertx.haInfo). Beware that versions up to 3.3.3 have a bug in the default cluster setup file (https://github.com/vert-x3/vertx-hazelcast/commit/4b6e6f815a221306ae93c500b5e62d8c1452397a)

We are working for a fix in 3.4

2017-02-07 0:17 GMT+01:00 Vinay Samudre <vinay.s...@gmail.com>:

For various reasons like software enhancement releases, we are gracefully stopping, deploying and restarting services. All the nodes join correctly but for some reason some of the handlers in a node give NO_HANDLERS.

Does anyone know when this issue will be fixed? 

On Monday, February 6, 2017 at 8:50:23 PM UTC+11, Thomas Segismont wrote:

Does anything happen in production prior to losing event bus subscription? There are known issues about the cluster manager when multiple nodes are killed (or crash) at the same time. But if the cluster was working and nodes haven't been stopped, no data should be lost.

2017-02-06 10:19 GMT+01:00 Asher Tarnopolski <ata...@gmail.com>:

can be related to this: https://github.com/eclipse/vert.x/pull/1594

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/cef3a82d-5ab5-46fa-9f47-a36f7893ae04%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

[Thomas SEGISMONT]

Have a look at the commit on Github (link in my previous message). It contains a fixed version of the cluster config.

And yes, subscriptions are stored in a multimap so it should be configured as such. 

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/37d875cb-1484-4b2f-b1cb-704173559452%40googlegroups.com.

[Vinay Samudre]

Hi Thomas,

cluster config in https://github.com/vert-x3/vertx-hazelcast/commit/4b6e6f815a221306ae93c500b5e62d8c1452397a will work in vert.x 3.3.3?

[Thomas SEGISMONT]

Yes. Make sure to set the max number of backups (6).

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/d98ede75-a2d8-4167-909c-858c33ee35cf%40googlegroups.com.

[Vinay Samudre]

Thank you Thomas.

I have made this change in one of sandpits, tried restart few times and it all seems to be ok :-)

Will deploy this in dev, test and uat and then in production. Will report later.