Monitoring network traffic from Hazelcast

[Chris Boldon]

Is there a metric emitted (or an easy way of monitoring) the network traffic between Hazelcast nodes? A glance through the doco and Google search didn't yield any results.

[Lucas Beeler]

Hi Chris,

One way to do it is to use a standard JMX client to monitor the various network I/O MBeans on the cluster nodes. Note that this won’t give you cluster-wide statistics, just per-node statistics, but if you’re looking for packets in/out per unit time and bytes in/out per unit time those should be available as part of the standard JMX network suite.

Take care,

Lucas

On Fri, Apr 10, 2020 at 1:23 PM Chris Boldon <cbo...@gmail.com> wrote:

Is there a metric emitted (or an easy way of monitoring) the network traffic between Hazelcast nodes? A glance through the doco and Google search didn't yield any results.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CA%2BR9vP3v8Duj3_zxqUUxWyMp18AfS5EgQ9Gmgu3ZHmtrmrXAmQ%40mail.gmail.com.

--

Lucas BEELER Senior Solutions Architect       hazelcast®     2 W 5th Ave, Ste 300 | San Mateo, CA 94402 | USA +1 (650) 521-5453 | hazelcast.com

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

[Emre Aydın]

Hi Chris,

If you enable diagnostics logging, you will see the following two which measures all traffic a Hazelcast member observes:

13-04-2020 10:39:45 1586763585386 Metric[tcp.bytesReceived=1540194]
13-04-2020 10:39:45 1586763585386 Metric[tcp.bytesSend=791453]

Note that this is not traffic only between Hazelcast nodes. It includes any traffic that goes through the member (including clients, WAN replication, REST API etc.).

You can get network traffic based on type of the communication if you enable advanced networking and configure separate endpoints for member to member communication and other types of communication. It was implemented via this PR in version 3.12.3 - https://github.com/hazelcast/hazelcast/pull/15541. These stats are not exposed as part of the public API of Management Center yet, but if you're interested, we can let you know once it is.

Regards,

Emre

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAPT_Z_NsStSL8Q5_hmcLsRiR9io%2Btzqi4jd9YKRvyAZ-qu0V3g%40mail.gmail.com.

--

Emre Aydın Management Center Team Lead

hazelcast®     2 W 5th Ave, Ste 300 | San Mateo, CA 94402 | USA +1 (650) 521-5453 | hazelcast.com

[Ozan Kılıç]

Diagnostics also provides cumulative metrics of each connection like below. Enabled via -Dhazelcast.diagnostics.metric.level=info

You can visualize using Peter's metric-plot. 

23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].in.bytesRead=616534935]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].in.completedMigrations=73]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].in.idleTimeMs=221]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].in.normalFramesRead=772300]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].in.ownerId=0]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].in.priorityFramesRead=11]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].in.processCount=777982]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].in.startedMigrations=73]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].out.bytesWritten=1213358590]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].out.completedMigrations=2]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].out.idleTimeMs=221]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].out.normalFramesWritten=772300]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].out.ownerId=0]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].out.priorityFramesWritten=3]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].out.priorityWriteQueueSize=0]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].out.processCount=772411]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].out.startedMigrations=2]
23-09-2019 20:24:16 1569270256947 Metric[tcp.connection[/10.20.71.4:22815->/10.104.107.103:1415].out.writeQueueSize=0]

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAAg5FnpF3KrpfKaPohWdPjUPP%3DDyyRYUj8itStzS9MJDPW733Q%40mail.gmail.com.

--

Ozan KILIC Solutions Architect - Support       hazelcast®