How to monitor multiple java processes

[infor...@gmail.com]

Hello,

I have a java application running in wildfly which I can monitor only to some extent. It contains the following lines of code

Runtime runtime = Runtime.getRuntime();
process = runtime.exec(command); // String[]

which starts a new process in a new jvm.

Unfortunately, I am not able to monitor anything in that process. Thus my question is, how to adapt/configure FusionReactor or what kind of information do I need to additionally pass over to the new process, so that it is also monitored.

I appreciate any input!

thank you!

[charlie arehart]

Well, I don't think you're going to be able to get FR to monitor the process launched by that Runtime exec() method, because what it's launching is not a Java process, which is all FR can monitor.

Let me elaborate, both to help others following along, and to make sure we're on the same page (and feel free to correct me where you may think I'm wrong, or am not understanding your need).

As you may know, Runtime exec method is for running a process in the native OS (like launching an exe in Windows, which has its own warts, as discussed in a classic and still-valid article from 2000, https://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html).

Anyway, the key point in our discussion today is that such a process (launched via the exec method, from within your wildfly or any Java app) will then run on its own in the OS, and is not therefore itself a Java process.

As FR can only monitor Java processes (and those in which its javagent has been indicated in the jvm startup args), FR can have no insight into the process that was launched. It can, of course, watch the request DOING the launch, which may be useful especially if you do a waitfor or other sort of process to await or process the results of the process called.

Any thoughts?

/charlie

[infor...@gmail.com]

Hi Charlie,

thank you for your feedback and for your explanation! Yes, we are on the same page :-)

Based on it, I would like to ask some questions and plz note that I am totally new to FR:

a) would it be possible to start up a second FR (i.e. FR2) - manually or programmatically - and have this one monitor the launched java process? The javaagent may be provided as an additional argument to the jvm startup arguments.

b) is it possible to export some kind of report from FR2 once/before the launched process terminates?

c) if so, is it also possible to integrate that report back into the original FR instance?

thx!

[michael...@intergral.com]

Hi,

a) would it be possible to start up a second FR (i.e. FR2) - manually or programmatically - and have this one monitor the launched java process? The javaagent may be provided as an additional argument to the jvm startup arguments.

While in theory, it would be possible to start a new FusionReactor process. I would not recommend this as it will take around 1 second to initialize FusionReactor. To guarantee your request will be tracked by FusionReactor you will have to wait until it is initialized.

This will mean that for every exec call you make you will add around 1 second to your request runtime, significantly slowing down your application.

b) is it possible to export some kind of report from FR2 once/before the launched process terminates?

The data for this request would be contained within your FusionReactor log files and could be accessed here.

c) if so, is it also possible to integrate that report back into the original FR instance?

It is not possible to integrate this request back into the user interface of the main FusionReactor.

An alternative way to do this would be to use the FusionReactor cloud so that data from both the main FusionReactor and the exec FusionReactor process were visible within the same user interface.

For this to work you would need to guarantee the process was active long enough to send data to the Cloud.

In short, while it would be possible to do as you suggest above I do not recommend this as there are many caveats to this approach.

If you believe you have an issue occurring in this exec process it is possible to install the Java agent and run FusionReactor in this manner whilst you diagnose the issue, potentially utilizing the debugger to find the issue but for long-term monitoring this solution does not seem efficient.

Michael Flewitt

FusionReactor Support Team

[infor...@gmail.com]

Hi Michael,

a) would it be possible to start up a second FR (i.e. FR2) - manually or programmatically - and have this one monitor the launched java process? The javaagent may be provided as an additional argument to the jvm startup arguments.

While in theory, it would be possible to start a new FusionReactor process. I would not recommend this as it will take around 1 second to initialize FusionReactor. To guarantee your request will be tracked by FusionReactor you will have to wait until it is initialized.

well, the execution of such processes last between 1 and 7 hours. Thus, I wouldn't mind paying the additional 1 second.

Is there a simple way to detect that FR has been fully initialized?

 

b) is it possible to export some kind of report from FR2 once/before the launched process terminates?

The data for this request would be contained within your FusionReactor log files and could be accessed here.

c) if so, is it also possible to integrate that report back into the original FR instance?

It is not possible to integrate this request back into the user interface of the main FusionReactor.

An alternative way to do this would be to use the FusionReactor cloud so that data from both the main FusionReactor and the exec FusionReactor process were visible within the same user interface.

For this to work you would need to guarantee the process was active long enough to send data to the Cloud.

which process - the main or the exec?

 

If you believe you have an issue occurring in this exec process it is possible to install the Java agent and run FusionReactor in this manner whilst you diagnose the issue, potentially utilizing the debugger to find the issue but for long-term monitoring this solution does not seem efficient.

As I mentioned, those processes run for several hours. What alternative would you suggest instead - if any available?

Thank you!

[michael...@intergral.com]

Hi,

As the process lasts for 1 to 7 hours running another FusionReactor process for this will be a viable option.

My previous statements were based on the idea that you were using the exec command to run a simple request then terminating the process.

To detect FusionReactor is running you can simply view the logs of the Java process and you will see a block like this:

INFO Fusionreactor: --------------------------------------------------------------------------------------

INFO Fusionreactor: FusionReactor - Copyright (C) Intergral GmbH. All Rights Reserved

INFO Fusionreactor: Revision:  7.1.0 fusionreactor.7495.73060

INFO Fusionreactor: Date    :  Fri, 5 Jan 2018 09:33:00 GMT

INFO Fusionreactor: Java    :  1.8.0_121 [25.121-b13] Oracle Corporation (64 bit)

INFO Fusionreactor: OS      :  Linux [3.19.0-32-generic] amd64

INFO Fusionreactor: --------------------------------------------------------------------------------------

Passing data from the exec FusionReactor process to the main (Wildfly) FusionReactor is not possible. However, using the Cloud is a viable option.

Instead of viewing the application data in the user interface in the FusionReactor instance itself you will view your data in our Cloud service. All your FusionReactor instances will send data here and you will see an aggregate view of all applications/processes rather than per JRE process.

One thing to ensure is that for each exec process you have a dedicated fusionreactor.jar file as the fusionreactor.jar file can only be used by one JRE at a time and if a second exec process attempts to access a jar file that is in use FusionReactor will not run.

The FusionReactor Cloud is a separate service that we offer, to access the Cloud you will need to contact our sales team at sa...@fusion-reactor.com

Michael Flewitt

FusionReactor Support Team

[charlie arehart]

I feel like this conversation is heading down a new track. Informatiok had said in his first reply today that he and I were “on the same page”, with regard to my initial response to him: that in his using runtime.exec() to launch a process, that would be launching a new Windows process (assuming your Wildfly is running on Windows), and not a Java one.

So how are either of you now feeling that such a Windows process, launched this way, could be monitored by FR? :-) It’s a not a Java process. Michael, are you seeing/thinking/saying something different now, with your replies?

And informatiok, now that you have specified that this “other process” you’re launching will be running for possibly several hours, were you REALLY wanting to monitoring (from within FR) that non-Java process (which FR can’t do, as it only monitors requests run within a java process), or were you really perhaps wanting just to be able to monitor when that process was DONE?

In that case, I go back to a point I made originally: when you launch the process (with the runtime.exec), you have the option to putting in a wait, which makes this new process synchronous (to the launching Wildfly request), such that FR (in monitoring that Wildfly request) can indeed tell you if/when that launched request is “done” (in that it can tell you when the launching one, waiting for it, is running or done).

Does that get you to your desired solution? Or were you really intending that this wildfly request would use the runtime.exec to launch this Windows (or Linux) process asynchronously (“fire and forget”), such that once launched, it’s now running as a non-java OS process outside of Wildfly? If the latter, then I’m not seeing how either of you are expecting FR to be able to monitor that non-java OS process itself.

Not trying to complicate matters or criticize. Just trying to understand how we seem to have come to such divergent opinions on this matter.

/charlie

 

From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com] On Behalf Of michael...@intergral.com
Sent: Monday, January 8, 2018 08:49 AM
To: FusionReactor <fusion...@googlegroups.com>
Subject: Re: [fusionreactor] How to monitor multiple java processes

 

Hi,

 

As the process lasts for 1 to 7 hours running another FusionReactor process for this will be a viable option.

 

<snip>

 

[Mikey Flewitt]

Hi Charlie,

From point A of the initial response "would it be possible to start up a second FR (i.e. FR2) - manually or programmatically - and have this one monitor the launched java process? The javaagent may be provided as an additional argument to the jvm startup arguments.". I am assuming the exec process being launched is indeed running Java and that JVM arguments can be provided to this process as suggested.

I agree that if the exec command is not running Java and/or the JVM arguments cannot be provided it is not possible to monitor this with FusionReactor in any way.

I hope this clears up the confusion.

Michael Flewitt

FusionReactor Support Team

[charlie arehart]

OK, and yep. So informatiok, please do clarify for us: is the os process you are launching from Wildfly via runtime.exec itself a java thing? :-)

Again, I had said in my first response that FR could not monitor that new process if it was not java, to which you said we were “on the same page”. But then you went on to ask more questions, which Michael kindly answered. If you answer the one above, we should be able to come to resolution on this, and let us know if you feel things are now clear.

/charlie

 

From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com] On Behalf Of Mikey Flewitt
Sent: Tuesday, January 9, 2018 07:29 AM
To: FusionReactor <fusion...@googlegroups.com>
Subject: Re: [fusionreactor] How to monitor multiple java processes

 

Hi Charlie,

 

From point A of the initial response "would it be possible to start up a second FR (i.e. FR2) - manually or programmatically - and have this one monitor the launched java process? The javaagent may be provided as an additional argument to the jvm startup arguments.". I am assuming the exec process being launched is indeed running Java and that JVM arguments can be provided to this process as suggested.

 

I agree that if the exec command is not running Java and/or the JVM arguments cannot be provided it is not possible to monitor this with FusionReactor in any way.

 

I hope this clears up the confusion.

 

Michael Flewitt

[infor...@gmail.com]

Hi Charlie and Michael,

first, thank you for all the support!

Maybe I misunderstood Charlie's explanation in the first place when he stated "..what it's launching is not a Java process...". So, I assumed exec launches an OS-related process (in my case its linux) which in turn takes over the coordination to launch my java process.... Anyhow, to be clear, command[0] equals the java path in "runtime.exec(command);" followed by a sequence of other arguments (like cp) to get a java process run. I am sorry for the caused confusion. 

So, to sum up all the good ideas/input, the following should work in theory

1) wildfly-java-process (which is monitored by main-FR) calls runtime.exec(command)

2) new FR instance (exec-FR) needs to be started

3) exec-java-process waits until exec-FR is running

4) exec-java-process is executed and exec-FR collects data / logs

5) when exec-java-process has finished, exec-FR may be killed too

6) exec-FR log files need to be uploaded to cloud so that they get aggregated with main-FR log

I am not sure about step 6. How does the "cloud" know how to relate (an independently created) exec-FR logs with main-FR log? In my case up to 30 exec-java-process may run at the same time.

Thank you!

[charlie arehart]

Thanks, but for all that there seems still a potential question. Can you please be very specific: is the OS (Linux) process you are launching (via runtime.exec, in that command argument) itself A JAVA PROCESS? Don’t think that because it’s launched FROM a Java app that it IS one. The question is very simply: IS it one. IS the thing you’re launching literally some other Java process?

If so, it may help to hear also what kind of thing it is, to help go forward, including to the other questions/observations you made.

Thanks.

/charlie

 

From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com] On Behalf Of infor...@gmail.com
Sent: Tuesday, January 9, 2018 05:12 PM
To: FusionReactor <fusion...@googlegroups.com>
Subject: Re: [fusionreactor] How to monitor multiple java processes

 

Hi Charlie and Michael,

--
You received this message because you are subscribed to the Google Groups "FusionReactor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fusionreacto...@googlegroups.com.
To post to this group, send email to fusion...@googlegroups.com.
Visit this group at https://groups.google.com/group/fusionreactor.
For more options, visit https://groups.google.com/d/optout.

[michael...@intergral.com]

Hi,

So to confirm is the command you are running something like "java -cp example.jar -javaagent... -X256m"? In this case you will be running a Java process within a linux process.

If so then using FusionReactor is as simple as adding your javaagent, more information on this can be found here: https://docs.fusion-reactor.com/display/FR71/FusionReactor+Java+Agent

Your Java process will not be fully initialized until FusionReactor is running.

When the process terminates FusionReator will do the same.

The cloud collects it's data as each individual FusionReactor instance sends data packs every minute and this data is aggregated by our cloud service. No log files will be required to analyze this data as it will all be available in the cloud user interface.

Rather than the data being instance specific, you will be able to track data on an application level.

Michael Flewitt

FusionReactor Support Team

[charlie arehart]

Ah, I see now that your last note said, “, command[0] equals the java path in "runtime.exec(command);" followed by a sequence of other arguments (like cp) to get a java process run”. I just wasn’t putting together the references there to the command being “the java path” and the “arguments” being “cp…to get a java process to run”.

My bad, so again all that Michael added will help you, and his next that I see he just wrote.

/charlie

[infor...@gmail.com]

Hi,

yes, it is something like "java -cp example.jar -javaagent... -X256m"

Thank you both for the support - I think I know now what to do!

br