Networking problems with Vert.x 3.4.1 on AWS EC2 c4.xlarge instance

[Arik Bron]

Hi,

We run into a problem with 1 of our AWS EC2 c4.xlarge instances. It looks like when the total incoming traffic reaches about 1GB/5min, the instance is unable to process all EventBus messages, as well as is unable to support external communication via WebClient. According to various internet resources the VM itself should be able to process up to 1Gbit/s which is about 37.5GB/5min.

Here is the exception that we see in Unix journal file:

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: Unhandled exception

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: java.lang.NullPointerException

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at io.vertx.core.net.impl.NetClientImpl$1.handleMsgReceived(NetClientImpl.java:197)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at io.vertx.core.net.impl.VertxNetHandler.lambda$channelRead$0(VertxNetHandler.java:64)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:335)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at io.vertx.core.impl.TaskQueue.lambda$new$0(TaskQueue.java:60)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at java.lang.Thread.run(Thread.java:745)

We would appreciate all the help any of you guys can provide in either understanding the specifics of the VM / Vert.x configuration or guidance on EventBus utilization that can help us to resolve this.

[Arik Bron]

Also, we're using hazelcast 3.6.3.

[Arik Bron]

And one more input - all messages are received by the VM via EventBus: there is a vertical that listens to published messages and sends them to a corresponding handler instances.

[Arik Bron]

Additional update: following AWS Support specialist recommendation we enabled Enhanced Networking on all involved VMs. Still, Vertx throws the mentioned above NullPointerException without any visible reason.

As long as the Vert.x development community is unable to provide any suggestions on why this happens we have to consider 3.4.1 not suitable for truly large-scale applications.

[Thomas SEGISMONT]

This issue has been fixed in 3.4.2 : https://github.com/eclipse/vert.x/issues/1906

You should either upgrade to 3.4.2 or a later version.

2018-04-16 2:32 GMT+02:00 Arik Bron <arkad...@gmail.com>:

Additional update: following AWS Support specialist recommendation we enabled Enhanced Networking on all involved VMs. Still, Vertx throws the mentioned above NullPointerException without any visible reason.

As long as the Vert.x development community is unable to provide any suggestions on why this happens we have to consider 3.4.1 not suitable for truly large-scale applications.

OK. It's nice to get your opinion but it doesn't help finding solutions though.

 

On Sunday, April 15, 2018 at 12:56:53 AM UTC+3, Arik Bron wrote:

And one more input - all messages are received by the VM via EventBus: there is a vertical that listens to published messages and sends them to a corresponding handler instances.


On Saturday, April 14, 2018 at 8:03:22 PM UTC+3, Arik Bron wrote:

Also, we're using hazelcast 3.6.3.

On Saturday, April 14, 2018 at 7:33:21 PM UTC+3, Arik Bron wrote:

Hi,

We run into a problem with 1 of our AWS EC2 c4.xlarge instances. It looks like when the total incoming traffic reaches about 1GB/5min, the instance is unable to process all EventBus messages, as well as is unable to support external communication via WebClient. According to various internet resources the VM itself should be able to process up to 1Gbit/s which is about 37.5GB/5min.

Here is the exception that we see in Unix journal file:

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: Unhandled exception

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: java.lang.NullPointerException

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at io.vertx.core.net.impl.NetClientImpl$1.handleMsgReceived(NetClientImpl.java:197)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at io.vertx.core.net.impl.VertxNetHandler.lambda$channelRead$0(VertxNetHandler.java:64)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:335)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at io.vertx.core.impl.TaskQueue.lambda$new$0(TaskQueue.java:60)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

Apr 14 03:31:19 ip-XX-XX-XX-XX.us-east-1.compute.internal run_vertx[9193]: at java.lang.Thread.run(Thread.java:745)

We would appreciate all the help any of you guys can provide in either understanding the specifics of the VM / Vert.x configuration or guidance on EventBus utilization that can help us to resolve this.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f516fe35-d8a9-4528-b576-13b265470717%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Arik Bron]

Thank you Thomas!

1) We're trying to move to Vert.x 3.5.1. Will post an update.

2) The opinion is important to those who try developing real-life large scale applications. I mentioned only one specific version - 3.4.1. I will update again after moving to 3.5.1.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

[Arik Bron]

Hi,

Thomas - thank you again for the suggestion. We moved to Vet.x 3.5.1 - the problem is resolved now!

[Thomas SEGISMONT]

Thanks for the update

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/918cee24-29dc-4745-b821-80e529d81911%40googlegroups.com.