error message:"WARNING: No pong from server 10.60.11.133:25556 - will consider it dead" vert.x 2 CR1

[Vinay Samudre]

Hi vert.x community,

I previously tried embedding vertx 2.0 snapshot 2 and it all worked with single verticle in embedded mode(please not that i had to embed because we have huge backend service running with things like embedded mule and what not.....) I never worked with vert.x before but it just worked, its really great platform!!!

Now with vert.x-2.0-CR1 I have created a module which includes another resource module, deployed, and accessed it from command line as follows

1. vertx runmod au.edu.vu~simple-module~v1 -cluster -cluster-port 25555

2. run another vertx as embedded, and passed message to the above module and it works :-) [ i had to do this because this client is a portal application(different team) that will call backed vertx app]. This one runs on port 25556.

3. but after few seconds (about half a minute) I see following message in both console

No pong from server 10.60.11.133:25555 - will consider it dead, timerID: 2539

holder org.vertx.java.core.eventbus.impl.DefaultEventBus$ConnectionHolder@1879c

I tried this with beta2 and I saw above message every few seconds only in vertx runmod command console and not in client console but,

 both client and server kept on working.....but with CR2 i see message in both consoles and it stops responding to event bus messages.....

This could be my ignorance, can someone please enlighten me?

Thanks,

Vinay

[Tim Fox]

On 17/06/13 05:38, Vinay Samudre wrote:
> Hi vert.x community,
>
> I previously tried embedding vertx 2.0 snapshot 2 and it all worked with
> single verticle in embedded mode(please not that i had to embed because we
> have huge backend service running with things like embedded mule and what
> not.....) I never worked with vert.x before but it just worked, its really
> great platform!!!
>
>
> Now with vert.x-2.0-CR1 I have created a module which includes another
> resource module, deployed, and accessed it from command line as follows
>
> 1. vertx runmod au.edu.vu~simple-module~v1 -cluster -cluster-port 25555

You don't need to specify cluster-port in Vert.x 2.0 - it will
automatically assign a free port

> 2. run another vertx as embedded, and passed message to the above module
> and it works :-) [ i had to do this because this client is a portal
> application(different team) that will call backed vertx app]. This one runs
> on port 25556.
> 3. but after few seconds (about half a minute) I see following message in
> both console
>
> No pong from server 10.60.11.133:25555 - will consider it dead, timerID:
> 2539
> holder
> org.vertx.java.core.eventbus.impl.DefaultEventBus$ConnectionHolder@1879c

I would guess you are blocking an event loop thread, but without seeing
some code that reproduces the problem it's impossible to tell.

[Vinay Samudre]

Hi Tim, Thanks for reply.

Here is my module

$echo $VERTX_MODS

/home/vinay/mygroovy/vertx-modules/module-build

$pwd

/home/vinay/mygroovy/vertx-modules/module-build/au.edu.vu~simple-module~v1

$ls -l

drwxr-xr-x 2 vinay vinay   4096 Jun 12 14:44 au

drwxr-xr-x 3 vinay vinay 20480 Jun 13 11:15 lib

-rw-r--r--    1 vinay vinay     100 Jun 14 11:04 mod.json

$cat mod.json

{

   "main": "au.SimpleVerticle",

   "includes": "au.edu.vu~backend~v1",

   "auto-redeploy": true

}

$cat au/SimpleVerticle.java (this class is in src/main/java/au directory which gets copied in au directory)

package au;

import java.util.Map;

import java.util.Date;

import org.vertx.java.core.Handler;

import org.vertx.java.platform.Verticle;

import org.vertx.java.core.eventbus.EventBus;

import org.vertx.java.core.eventbus.Message;

import org.vertx.java.core.json.JsonObject;

import au.edu.vu.its.as.rcs.RCSIdmServer;

import au.edu.vu.its.as.adminidm.IdmServerConsole;

import au.edu.vu.its.as.util.*;

import org.apache.commons.beanutils.BeanUtils;

import org.apache.commons.lang.exception.ExceptionUtils;

import java.util.LinkedHashMap;

import java.util.concurrent.CountDownLatch;

import java.util.concurrent.TimeUnit;

public class SimpleVerticle extends Verticle {

   public Handler<Message<JsonObject>> getEchoHandler() {

      Handler<Message<JsonObject>> handler =

         new Handler<Message<JsonObject>>() {

            public void handle(Message<JsonObject> message) {

               String name = message.body().getString("name");

               message.body().putString("returnValue", "Hello " + name);

               System.out.println("reply=" + message.body());

               message.reply(message.body());

            }

        };

      return handler;

   }

   public void start() {

      vertx.eventBus().registerHandler("simple.echo",

                                       getEchoHandler());

      System.out.println("SimpleVerticle started......");

   }

}

Also, when i redeploy my module I get following exception.....

Module has changed - redeploying module from directory /home/vinay/mygroovy/vertx-modules/module-build/au.edu.vu~simple-module~v1

Unhandled exception

java.lang.IllegalStateException: Not a worker thread

        at org.vertx.java.platform.impl.DefaultPlatformManager.checkWorkerContext(DefaultPlatformManager.java:517)

        at org.vertx.java.platform.impl.DefaultPlatformManager.doDeployMod(DefaultPlatformManager.java:634)

        at org.vertx.java.platform.impl.DefaultPlatformManager.doRedeploy(DefaultPlatformManager.java:310)

        at org.vertx.java.platform.impl.DefaultPlatformManager.access$1000(DefaultPlatformManager.java:52)

        at org.vertx.java.platform.impl.DefaultPlatformManager$7$1.handle(DefaultPlatformManager.java:257)

        at org.vertx.java.platform.impl.DefaultPlatformManager$7$1.handle(DefaultPlatformManager.java:254)

        at org.vertx.java.core.impl.CountingCompletionHandler.callHandler(CountingCompletionHandler.java:73)

        at org.vertx.java.core.impl.CountingCompletionHandler.checkDone(CountingCompletionHandler.java:91)

cheers,

Vinay

[Tim Fox]

I don't believe you're using CR2.

The line numbers of your stack trace don't line up and I believe the
"not a worker..." issue was solved quite some time ago.

[Vinay Samudre]

Hi Tim,

I do not have access to the source code now, but will get back to you tomorrow.

I believe line numbers are from beta2, which was available from vertx.io download link some time back.

CR2? or do u mean vertx CR1? i mean available from https://bintray.com/vertx/downloads/distribution/2.0.0-CR1? thats what I tried, and thats the one gives

me "No pong from server 10.60.11.133:25556 - will consider it dead" and after that it stops working.

By the way i searched "CR2" and didn't find any thing.

cheers,

Vinay

[Tim Fox]

Yes CR1, sorry.

beta2 is old

[Vinay Samudre]

Hi Tim,

vertx version: vert.x-2.0.0-CR1

I can confirm that with simple module run with runmod command and accessed with embedded vertx client works for a while and then after about 30 to 50 seconds gives me "No pong from server 10.60.11.133:25555 - will consider it dead, timerID:......." error message in both server and client console.

I have attached the module for your reference.....module reloading *is* working, that was my mistake, i was including vertx jars in included module.

Now i have removed the include and still having this issue..........please help

cheers,

Vinay

[Vinay Samudre]

Hi Tim,

Here is my client code, I needed to do this because our internal client wants synchronous access to vertx methods. Am doing something wrong?, is it ok to call await on latch declared in main method from which I started the client??

class Test {

  PlatformManager pm;

   public static void main(String[] args) throws IOException {

      Test t = new Test();

      t.createPM(25556, "esbdev");

      while(true) {

        System.out.println("press enter....");

        int inChar = System.in.read();

        t.getEcho();

      }

   }

   private void createPM(int port, String host) {

      pm =  PlatformLocator.factory.createPlatformManager(port, host);

      registerExitHandler(pm);

   }

   private static void registerExitHandler(final PlatformManager mgr) {

     mgr.registerExitHandler(new VoidHandler() {

       public void handle() {

         mgr.stop();

       }

     });

 public JsonObject execute(final Map map) {

    String method = (String) map.get("method");

    final Map ret = new LinkedHashMap();

    final CountDownLatch methodExecutionLatch = new CountDownLatch(1);

    Handler<Message<JsonObject>> getReturnHandler = new Handler<Message<JsonObject>>() {

       public void handle(Message<JsonObject> message) {

         System.out.println("from server:" + message.body());

         Exception me = (Exception)message.body().getField("exception");

         Exception re = (Exception)message.body().getField("rootException");

         map.put("returnValue", message.body().getField("returnValue"));

         methodExecutionLatch.countDown();

         System.out.println(" ***********  return handler executed **************");

       }

    };

    JsonObject arg = new JsonObject(map);

    pm.vertx().eventBus().send(method, arg, getReturnHandler);

    try {

       System.out.println("wait start");

       methodExecutionLatch.await((int)map.get("timeout"), TimeUnit.SECONDS);

       System.out.println("wait end");

       if ( methodExecutionLatch.getCount() > 0 ) {

           map.put("error", "timedout ....waited for " + map.get("timeout") + "seconds");

           methodExecutionLatch.countDown();

        }

     } catch (Exception e) {

        System.out.println(" ***********  Latch await interrupted ***************");

        map.put("error", "timeout await interrupted.");

        map.put("exception", e);

     }

     return new JsonObject(map);

  }

  public void getEcho() {

     Map m = new HashMap();

     m.put("name", "XXXXXXXXXXX");

     m.put("method", "simple.echo");

     m.put("timeout", 2);

     m.put("returnClass", String.class);

     JsonObject jo = execute(m);

     System.out.println("returned:" + jo);

     System.out.println("error:" + jo.getString("error"));

     System.out.println("returned:" + jo.getField("returnValue"));

  }

}

cheers,

Vinay

[Tim Fox]

Pro tip: use gist.github.com for code. It's a PITA having to copy and
paste out of emails, reformat everything and add all the imports back in...

[Vinay Samudre]

Thanks for the tip Tim,

https://gist.github.com/anonymous/5803146

here is gist for client.

cheers,

Vinay

[Tim Fox]

I've had a look at this and it appears that the second pong that is
written on one side of the connection isn't being read out the other side.

I suspect this is a Netty issue.

Can you add a github issue?

Unfortunately Norman is away on holiday for the next week or so.

[Tim Fox]

Ok, it appears that the problem occurs because the same buffer is written more than once.

I can fix it locally, but there's a bigger issue here about how we allow Vert.x buffers to be written more than once to the socket.

[Tim Fox]

https://github.com/vert-x/vert.x/issues/648

On Tuesday, 18 June 2013 09:19:07 UTC+1, Tim Fox wrote:

[Vinay Samudre]

Hi Tim,

Thanks for looking into this, I will keep an eye on this thread and issue, and once resolved I will give it another try.

Thanks again.

cheers,

Vinay

[Tim Fox]

I put a workaround in yesterday so it should work now

[Tim Fox]

BTW, the way you are embedding is not right here.

It appears you want to embed core directly, not embed the platform.

PlatformManager should be used when you want to embed the *platform* not
when embedding core.

For embedding core, you should use VertxFactory to get your Vertx instance.

Please see the vert.x 2.0 docs on embedding for more info.

[Vinay Samudre]

Thanks Tim, 

Will check the docs and use VertxFactory for client.

Thanks again,

Vinay

[actorGeek]

Hi Tim:

I posted an issue I ran into last couple of days ..... here is the original post https://groups.google.com/forum/#!topic/vertx/e_R6xoJb7v0

It is the same issue being discussed in this thread ..... No PONG from Server .....

I ran into this error just by increasing the worker verticle response size from 3K to 300K .....

This post thread seems to indicate that the EB is being flooded and suggesting to introduce flow control .... but the problem is my client is apache bench and all I did to start seeing this error was increase the size of the response from the worker verticle. I don't believe I am blocking any event loop because I was working fine with 60K concurrent connections (from apache bench) when the worker verticle was responding with a 3K response.

Any ideas on what may be causing this or how to troubleshoot this further ?

Thanks

[Clayton Piscopo]

Hi,

We recently encountered a very similar issue to the one mentioned in this thread when using version 2.0.2-final. By any chance was this issue resolved, please?