Query related to metrics collected using Cryostat

[Shreya Biradar]

Hi team,
I'm a beginner exploring Cryostat and JFR.
I'm curious to understand the difference between the metrics collected using JDK Flight Recorder (JFR) and Prometheus specific to Java runtime and why should one prefer Cryostat( JFR for cloud and containers) over Prometheus?

[Andrew Azores]

Hi Shreya,

Sure, it's a very understandable question. There are commonalities
between the two.

I would first highlight that JFR is specific to HotSpot (OpenJDK) and
can gather a lot of very detailed events and metrics about the JVM
internals, and JFR's format can also include additional information
around events that you might not normally get from a broader monitoring
solution like Prometheus. For example, JFR events can include detailed
stack traces, thread states, lock contention, etc., and the JFR files
encode this very efficiently. You may find JFR is better suited for deep
dives into performance profiling than Prometheus because of these reasons.

Second, Cryostat is able to connect to JVMs using the JMX protocol,
which is also a built-in feature in OpenJDK. All you need to do is set
some JVM startup flags to turn on JMX and Cryostat can connect,
dynamically start/stop JFR, and collect data for you. No added
dependencies, no code changes, no rebuild. For Prometheus you would need
to add the Prometheus client dependencies to your application, add code
to turn on the default Prometheus collectors, and rebuild/redeploy the
application.

Hope this helps.

--
Andrew Azores
Principal Software Engineer, Java Monitoring
Red Hat Canada

[Shreya Biradar]

Thank you for sharing the insights Andrew.

[Shreya Biradar]

Hi Andrew, 

I have few follow up questions regarding Cryostat - 
1.What is the potential impact on the existing performance of a Java application by integrating Cryostat. Specifically, I would like to know if integrating Cryostat with the current application might have adverse effects on its performance.  As Cryostat is an extension of JFR, which has low overhead of about 1-2% in production environments, does the overhead remain the same?

2. The API exposed by Cryostat provides with  various options to manage events, event templates, record and download recordings etc. Are there specific endpoints

 which can provide users with all the metrics captured by JVM events using JFR, similar to other monitoring tools like Prometheus?

Thanks in advance!
Shreya.

[Andrew Azores]

Hi Shreya,

On 2023-06-01 07:09, Shreya Biradar wrote:
> Hi Andrew,
>
> I have few follow up questions regarding Cryostat -

Happy to hear it :-)

> 1.What is the potential impact on the existing performance of a Java
> application by integrating Cryostat. Specifically, I would like to know
> if integrating Cryostat with the current application might have adverse
> effects on its performance.  As Cryostat is an extension of JFR,
> which has low overhead of about 1-2% in production environments, does
> the overhead remain the same?

Cryostat is not really an "extension" of JFR - it is a remote client
that collects JFR, or if you use the Cryostat Agent then that is a local
client that pushes JFR. The performance overhead on your application
will be only the performance overhead of JFR itself, a small amount of
memory to load the Agent if you use it, plus the network bandwidth
required to transfer the JFR data out of the application and into
Cryostat's storage.

>
> 2. The API exposed by Cryostat provides with  various options to manage
> events, event templates, record and download recordings etc. Are there
> specific endpoints
>  which can provide users with all the metrics captured by JVM events
> using JFR, similar to other monitoring tools like Prometheus?
>

There are a few options, but none exactly equivalent to what Prometheus
does. Cryostat's design is much more focused on exposing JFR files for
profiling, rather than exposing metrics for monitoring.

The first option is to simply download the JFR binary recording file and
process that further. There are various tools for this that come with
OpenJDK, such as the `jfr` command you can execute in your shell, but
you can also write Java code to process JFR files.

The second option is to use Cryostat's jfr-datasource component. This
can load a JFR file, parse it, and expose it as Grafana-compatible JSON
timeseries metrics. A typical Cryostat deployment already includes one
jfr-datasource instance, but it will be hidden away inside the Pod and
not accessible from the outside world. You could create a custom
deployment that exposes it, or you could run a jfr-datasource instance
separately and put JFR files into it that you downloaded from the
Cryostat API yourself, then scrape the timeseries data from it for
monitoring.

The third option would be to use Cryostat's Automated Analysis Reports.
This does not give you all of the JFR event data, but rather a distilled
view of the data after it has been run through JMC's analysis rules
engine that highlights potential issues. Cryostat can provide you this
data in JSON format. You might find it interesting to periodically
request this JSON data from the API and then use that for monitoring.

> Thanks in advance!
> Shreya.