collect server and prometheus

[wtp...@gmail.com]

Hi,

We use javamelody to monitor a number of pods and use the collect server.   Under the collect server, can we use prometheus format of reporting so that we can generate alerts in grafana ?

Thank you !

[evernat]

Hello,

Yes you can use prometheus format from a collect server.
Given an URL in the collect server like http://mycollectserver:8080/?application=my_app then you call the URL http://mycollectserver:8080/?application=my_app&format=prometheus from prometheus.

So, I think that the scrape config in prometheus would be like the following to scrape my_app data from mycollectserver:

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'my_app'

    scheme: http
    # metrics_path without /monitoring in the collect server URL
    metrics_path: '/'

    params:
      application: ['my_app']
      includeLastValue: ['false']
      format: ['prometheus']

    # basic_auth if the parameter authorized-users is used in the collect server
    # basic_auth:
    #   username: my_username
    #   password: my_password

    scrape_interval: 60s

    static_configs:
      - targets: ['mycollectserver:8080']
        labels:
          env: 'unit'

A scrape interval of 60s in prometheus seems enough to me because the default collect frequency is 60s in the collect server. That is the default value of the resolution-seconds parameter in collect server is 60.

For reference on the optional collect server:
https://github.com/javamelody/javamelody/wiki/UserGuideAdvanced#optional-centralization-server-setup

bye,

Emeric

[wtp...@gmail.com]

Thanks.

I used the following command:

https://***Server***/javamelody?application=auth-service&format=prometheus

An error was displayed:

Prometheus from collector server is not supported for several nodes in one application - configure Prometheus to scrape nodes directly or declare several applications in the collector server.

Is there something need to set up ?

[evernat]

Ah ok. The problem for the prometheus format is that your app is declared in the collect server to monitor several instances as only one app. That is one app added in the collect server, with several URL targets.

In that case, the collect server doesn't know how to export that in the prometheus format: consider the javamelody_system_tmp_space_free_bytes metric: the collect server can't export this metric for one instance and then at the same time export the same metric with the same name for another instance in the same app. The prometheus exporter in the collect server don't know how to export metrics several times, neither to "aggregate" those metrics between several instances which may not make much sense sometimes.

Sorry for this restriction.

So, if you can for those instances and if it is worth the configuration, I suggest that you add each instance as a separate app in the collect server (and not several instances as a single app). Then you can declare in prometheus a scrape config for each separated app in the collect server. I think that is what you want to have alerts in grafana on the metrics per instance. (If the number of instances varies frequently, I don't know if you can do all that automatically and I would be happy to hear it.)

And if you want, you can also add an "aggregated app" in the collect server from the separately added apps, to replace the.app you have at the moment.

bye,

Emeric