agent to cmr timing out

[Viswa]

Hi,

We are trying inspectIT in our environment. 

Its pretty easy to setup the agent and CMR. (version 1.6.9.83)

We are facing a weird problem. When we start the agent it connect to CMR but after 2 minutes its starts timing out. we dont see it connects again.

For POC, we are running agent and CMR on the same VM.

Agent Logs:

2016-09-29 11:22:14,741: 26826  [           main] INFO  spectit.agent.java.SpringAgent - Using agent version 1.6.0.0.

2016-09-29 11:23:55,385: 127470 [:1475162592749]] WARN  analyzer.impl.ByteCodeAnalyzer - Error occurred instrumenting the byte code of class com.ibm.as400.access.AS400JDBCStatement. Sending the class structure to the CMR resulted in a time-out.

2016-09-29 11:24:10,766: 142851 [ervice-thread-2] INFO  .analyzer.impl.AnalyzeCallable - Type could not be sent to the CMR due to the ServerUnavailableException.

CMR Logs:

2016-09-29 11:18:11,380: 20579  [           main] INFO      rocks.inspectit.server.CMR - CMR started in 20243.087109 ms

2016-09-29 11:22:13,989: 263188 [         Server] INFO  all.minlog.MinlogToSLF4JLogger - kryonet: Connection 1 connected: /127.0.0.1

2016-09-29 11:22:14,067: 263266 [ervice-thread-0] INFO  er.service.RegistrationService - Trying to register Agent 'InspectIT_TC'

2016-09-29 11:22:14,073: 263272 [ervice-thread-0] INFO  er.service.RegistrationService - Successfully registered the Agent 'InspectIT_TC' with id 1, version 1.6.0.0 and following network interfaces:

2016-09-29 11:22:14,073: 263272 [ervice-thread-0] INFO  er.service.RegistrationService - |- IPv4: 10.100.40.239

2016-09-29 11:22:14,073: 263272 [ervice-thread-0] INFO  er.service.RegistrationService - |- IPv4: 127.0.0.1

2016-09-29 11:23:55,281: 364480 [ervice-thread-0] INFO  r.util.AgentStatusDataProvider - Platform 1 timed out.

2016-09-29 11:23:59,628: 368827 [         Server] INFO  all.minlog.MinlogToSLF4JLogger - kryonet: Connection 1 disconnected.

For CMR we are using the JRE provided in the respository.

For agent we are using 1.7

java version "1.7.0_111"

Java(TM) SE Runtime Environment (build 1.7.0_111-b13)

Java HotSpot(TM) 64-Bit Server VM (build 24.111-b13, mixed mode)

But I dont see any port connectivity issues. Is there any known issue around this?

Regards,

Viswanath

[Patrice Bouillet]

Hi Viswanath,

do I get you right, that the version of the CMR is 1.6.9 and the agent is running with 1.7 ? If so, this won't work. Please download version 1.7 for all of the components (1.7.3 is latest):

https://github.com/inspectIT/inspectIT/releases/tag/1.7.3.86

If this still doesn't work in your setup, let me know and we'll take a closer look into all of this!

Cheers,

Patrice

[viswanath sriram]

Hi Patrice,

Thanks for getting back on this.

The agent and CMR both are from Inspect 1.6.9 .

1.7 is about the java version we are using for agent.

Initially we tried with Inspect 1.7.3 as well and seeing the same issue. Then in documentation we see 1.6.9 is most stable version. So we downgraded the version from 1.7.3 to 1.6.9 but facing the same errors.

Regards,

Viswanath

[Patrice Bouillet]

Hi Viswanath,

understood. Does this exception you see in the logs always happen with the same class (com.ibm.as400.access.AS400JDBCStatement) or is it different? Could you check if in the agent logs folder and exception log file there is more information you can paste here?

Thanks

Patrice

[viswanath sriram]

Hi Patrice,

This is what i see in exception log. Some times I see a different class as well. But most of the times i see this class.

2016-09-29 11:03:54,219: 122023 [:1475161381741]] WARN  analyzer.impl.ByteCodeAnalyzer - Error occurred instrumenting the byte code of class com.ibm.as400.access.AS400JDBCStatement. Sending the class structure to the CMR resulted in a time-out.

2016-09-29 11:23:55,385: 127470 [:1475162592749]] WARN  analyzer.impl.ByteCodeAnalyzer - Error occurred instrumenting the byte code of class com.ibm.as400.access.AS400JDBCStatement. Sending the class structure to the CMR resulted in a time-out.

~

[viswanath sriram]

I just changed the version to IBM java.

java version "1.7.0"

Java(TM) SE Runtime Environment (build pxa6470sr6-20131015_01(SR6))

IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20131013_170512 (JIT enabled, AOT enabled)

And this started working fine with out any disconnectivity from the agent But i dont see getting CPU metrics now.

[Ivan Senic]

Hi viswanath,

Thanks for the error report you provided. In general the timeout while trying to instrument the class is not a killer error, the class simply will not be instrumented. However, seams like you are hitting some kind of connection problem, which make your agent disconnect from the CMR and thus you can not send any data more. Changing the JVM version should not have any impact here, it should be independent of the version.

What would really help us in understanding what is going on is to send us the complete log folder you have on the agent ([AGENT_DIR]/logs) and the CMR ([CMR_DIR]/logs). Also I would like to know if you are connecting one agent or more at the same time?

Thanks in advance,

Ivan

--
You received this message because you are subscribed to the Google Groups "inspectIT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inspectit+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/inspectit/1e7b994c-15c3-4438-a0dd-af701b339fcb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Ivan Senić
senic...@gmail.com
+381 65 5365272

[Ivan Senic]

Hi viswanath,

I also saw one strange thing in your logs:

2016-09-29 11:22:14,741: 26826  [           main] INFO  spectit.agent.java.SpringAgent - Using agent version 1.6.0.0.

The version 1.6.0.0 does not exists, it's only used when you build the product locally outside the official build process. Are you sure you did get the version from the official GitHub repository? Seams confusing to me..

Greets,

Ivan 

[viswanath sriram]

Hi Ivan,

I will rollback the change to replicate the same scenario and send you the logs today.

To debug this scenario, I have got the latest code and change the ANALYZE_TIMEOUT_MILLIS in ByteCodeAnalyzer.java to 10secs(just for testing) and deployed. I think i have sent the logs on top of that to you and thats why you are seeing 

2016-09-29 11:22:14,741: 26826  [           main] INFO  spectit.agent.java.SpringAgent - Using agent version 1.6.0.0.

But the issue exists when i used actual jar taken from https://github.com/inspectIT/inspectIT/releases/tag/1.6.9.83

Not all the times but few times I see below exception in logs.

[ervice-thread-2] INFO  .analyzer.impl.AnalyzeCallable - Type could not be sent to the CMR due to the ServerUnavailableException.

But I see CMR process will be running (local host) and don't see any weird thing in the logs.

Regards,

Viswanath

To unsubscribe from this group and stop receiving emails from it, send an email to inspectit+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/inspectit/1e7b994c-15c3-4438-a0dd-af701b339fcb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[viswanath sriram]

Let me add also one point..when i say disconnected.. from the UI, I see the agent status as 'No keep alive signal for <<X time>>' but not disconnected status.

So the data is missing from that <<X time>>

Regards,

Viswanath

[Ivan Senic]

Hi Viswanath,

We invested lately a lot on fixing the connection issues in inspectIT. If you have time, I encourage you to have a look on our new preview version (https://github.com/inspectIT/inspectIT/releases/tag/1.7.5.88) and check if this fixes the problems you encountered.

Cheers,

Ivan