Please review MP0002 - Metrics/Telemetry

[Heiko Rupp]

Hey,

I have updated the Metrics/Telemetry proposal at 

https://github.com/eclipse/microprofile-evolution-process/pull/26, which supersedes the old PR12 (which I have closed).

Please review and give feedback.

There is an example implementation of the code at https://github.com/pilhuhn/mp-metrics available, that you

can check out and play with.

In the proposal are still open questions, marked with a 'Q:'

I have now "manually" implemented the units. @Werner: how would that look like with pulling in UoM?

 Thanks

      Heiko

[Werner Keil]

Heiko,

Thanks for the update. Will have a closer look. Scott also just gives a brief overview to the JCP EC.

The enum approach in

https://github.com/pilhuhn/mp-metrics/blob/master/src/main/java/de/bsd/mp_metrics/MpMUnit.java

is close to the Enum implementation

https://github.com/unitsofmeasurement/uom-impl-enum

The biggest difference would be having an enum per supported quantity. I also use that approach for a current client where service health is also applied. 

An alternative would be how Parfait does it using uom-se: https://github.com/performancecopilot/parfait/tree/master/parfait-core

Parfait also works on JSON support very similar to the way it's needed here, but I think most of it is part of an ongoing GSOC work we are mentoring. Maybe next year GSOC could also be something to consider for certain MP features/subprojects btw. ?;-)

Regards,

Werner

[Heiko Rupp]

There is an example implementation of the code at https://github.com/pilhuhn/mp-metrics available, that you

can check out and play with.

I now have also implemented a Prometheus text-style endpoint. The downside is that it does not support units. Also the inline metadata makes requests for metric data much heavier. 

[Felix Wong]

Hello Heiko and the community,

This is Felix Wong from IBM WebSphere development team.  I will join you guys to work on the Microprofile Metrics proposal.  I will take a look of the proposal and the sample implementation. 

Thanks!
Felix

[Heiko Rupp]

Hey Felix,

Am Montag, 29. Mai 2017 18:06:12 UTC+2 schrieb Felix Wong:

Hello Heiko and the community,

This is Felix Wong from IBM WebSphere development team.  I will join you guys to work on the Microprofile Metrics proposal.  I will take a look of the proposal and the sample implementation. 

Great!

 

[Arthur De Magalhaes]

Hey,

Are there any concerns that using host:port/metrics will collide with other microservices that were already using the /metrics context root?

Thanks,

Arthur

[Felix Wong]

Hi Heiko,

I have a few questions on the proposal.

1. Did you have a list of metrics collection tools (e.g. collectd) in mind that will work with the proposed http format?
2. It mentioned that the metadata shouldn't be changed for the life of the container.  However, tags are included in the metadata.  Does this mean tags can be set once and cannot be changed for the life of the container?
3. Are required metrics mandatory for all microprofile implementations?
4. The required metrics are not listed in the proposal.  Do you have a list of required metrics in mind?
5. What is the purpose of the configuration yaml file?
6. In the configuration file, do we have to list the required metrics?

 

[Heiko Rupp]

Am Dienstag, 30. Mai 2017 23:07:00 UTC+2 schrieb Arthur De Magalhaes:

Hey,

Are there any concerns that using host:port/metrics will collide with other microservices that were already using the /metrics context root?

Yes, this is possible. I think the root path ( /metrics ) does not matter that much in the sense that one can most often easily tune it in the gathering agents.

Do you think something like  /_metrics as default may more sense?

[Heiko Rupp]

Hey Felix,

thanks for your feedback.

Am Mittwoch, 31. Mai 2017 18:15:19 UTC+2 schrieb Felix Wong:

Hi Heiko,

I have a few questions on the proposal.

1. Did you have a list of metrics collection tools (e.g. collectd) in mind that will work with the proposed http format?

No, not really. I am well aware that this is proprietary, but easy to parse. As you saw, I also implemented exposing via Prometheus(P8s) text format, but this has the disadvantage that the P8s format does not cater for units yet 

 

2. It mentioned that the metadata shouldn't be changed for the life of the container.  However, tags are included in the metadata.  Does this mean tags can be set once and cannot be changed for the life of the container?

Yes. What use case to change the tags do you have in mind? When making the tags 'immutable' allows to only grab the metadata once at container startup (or even when a new deployment is deployed)

 

3. Are required metrics mandatory for all microprofile implementations?

Yes. The intent here is to have at least a minimal common set, that tools can always rely on

 

4. The required metrics are not listed in the proposal.  Do you have a list of required metrics in mind?

Not really. I was hoping that the group would come up with a set of  5-10 required metrics out of the experience they have.

 

5. What is the purpose of the configuration yaml file?

I guess it is an implementation detail. I included it in order to show how it could look like. Should probably not be listed in a spec.

 

6. In the configuration file, do we have to list the required metrics?

My implementation gets them from that configuration yaml file along with the vendor-specific ones. I guess that is left to the implementor as in point 5. 

 Cheers 

   Heiko

 

[Arthur De Magalhaes]

I was thinking something like /mp/metrics or /microprofile/metrics.   Probably there will be more REST APIs that MP will want to expose in the future, so it would be nice to have a common API prefix that: (1) helps avoid collision with existing services, and (2) helps group together the set of APIs that are exposed by MP.  

Thoughts?

[Werner Keil]

Why not keep it simple like https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html?

Although Spring usually does not miss an opportunity to present and market itself, there is no /spring/health or /springboot/health there either. So why not try to make also possible migration to MicroProfile solutions easier by sticking to an established pattern.

What Heiko did for Swarm earlier https://github.com/wildfly-swarm/wildfly-swarm-examples/tree/master/jaxrs/health also exposes the /health endpoint on a given context root.

So unless the already started Health module was to establish a prefix as a standard, I would not do this for metrics alone.

Cheers,

Werner

[Felix Wong]

Hey Heiko,

Thanks for your reply! 

DropWizard seems to be a popular framework for metrics.  Does it make sense to have the MP Metrics API closely resemble DropWizard's API so that DropWizard users can migrate easily to MP.

Regards,
Felix

[Heiko Rupp]

Am Donnerstag, 1. Juni 2017 17:34:47 UTC+2 schrieb Arthur De Magalhaes:

I was thinking something like /mp/metrics or /microprofile/metrics.   Probably there will be more REST APIs that MP will want to expose in the future, so it would be nice to have a common API prefix that: (1) helps avoid collision with existing services, and (2) helps group together the set of APIs that are exposed by MP.  

While I think that a prefix /mp or /microprofile is attractive and would prevent clashes with other usages of /metrics, I also think that /metrics is (like /health) sort of a default for telemetry agents. 

I guess I would prefer to use /metrics as a default, but with the possibility to change this. 

[Heiko Rupp]

Felix,

> DropWizard seems to be a popular framework for metrics.  Does it make sense to have the MP Metrics API closely resemble DropWizard's API so that DropWizard users can migrate easily to MP.

Do you mean the REST-api of DW-Metric's MetricServlet? Or the JavaApi to register and report all the data.

If the latter, I think this is an implementation detail of a MP implementation how to obtain the data.

For the REST-Api I fail to find documentation of the data format that is exposed. Can you send me a pointer to it please?

Thanks

  Heiko

[Felix Wong]

AFAIK, DropWizard has a MetricsServlet and it simply prints the metrics out in JSON. 

BTW, What do you think of having a weekly call to discuss the metrics spec?

Regards,
Felix

[Heiko Rupp]

Hey Felix,

sorry I missed that message.

Am Freitag, 16. Juni 2017 16:21:48 UTC+2 schrieb Felix Wong:

AFAIK, DropWizard has a MetricsServlet and it simply prints the metrics out in JSON. 

Yes, but in what format? :-)

 

And yes, some more regular meetings to set this up is a good idea.

  Heiko 

[Emily Jiang]

I suggest you guys to setup a hangout and put it on the MP calendar so that it is open to everyone who is interested. You can take a look at the Config and Fault Tolerance calendar entries.

Emily

[Kevin Sutter]

Due to the pending change in our "evolution process", I'm just checking in to see the state of the current Health Metrics proposal.  Has the team agreed that it's in a state for approval?  Or, are there some outstanding issues?  The proposal doesn't have to be the "final answer" with all of the t's crossed and i's dotted.  Those details can be worked out while defining the spec, api, and tck.

Thoughts?
Kevin

[donbo...@gmail.com]

Kevin, I believe that approval step was already done in another thread (see 

"Metrics/Telemetry proposal and MP 1.1"). In case folks are wondering who I am, Felix Wong and I work together (we're both from WebSphere Liberty). Regards, Don

[Kevin Sutter]

Thanks, Don, for pointing out that previous conversation:
https://groups.google.com/forum/#!topic/microprofile/axZPSJ9TIFI

It looks like Ken has already prompted Heiko on the IP validation issue in his PR:
https://github.com/eclipse/microprofile-evolution-process/pull/26

So, if we can get that cleaned up, then I think we're good for merging the proposal into the repo and get started on the real work -- spec, api, tck.

Thanks!
Kevin

[Felix Wong]

Here is the sample output from DropWizard's MetricsServlet for a simple application.

{

• "version": "3.0.0",
• "gauges": { },
• "counters": {
  • "counter1": {
    • "count": 4
    }
  },
• "histograms": { },
• "meters": { },
• "timers": {
  • "timer1": {
    • "count": 4,
    • "max": 1.1128E-4,
    • "mean": 5.648429229979686E-5,
    • "min": 2.8266E-5,
    • "p50": 3.6244E-5,
    • "p75": 5.3224000000000004E-5,
    • "p95": 1.1128E-4,
    • "p98": 1.1128E-4,
    • "p99": 1.1128E-4,
    • "p999": 1.1128E-4,
    • "stddev": 3.200802012241128E-5,
    • "m15_rate": 0.5977839392019586,
    • "m1_rate": 0.5680177658517294,
    • "m5_rate": 0.5933885815286469,
    • "mean_rate": 0.28077315267369113,
    • "duration_units": "seconds",
    • "rate_units": "calls/second"
    }
  }

[Felix Wong]

Regarding the proposed Hangout, please choose a time slot that works for you from the following link.  All are welcome.

https://beta.doodle.com/poll/imep6ezh66wb9msg

Regards,
Felix

[Heiko Rupp]

One thing we should do in the future is also to address the respective other specs and give them room for their own data to expose next to /base /vendor and /application

[donbo...@gmail.com]

Heiko, I think having metrics for other specs in MP would be good.  JAX-RS in particular would likely have lots of possible metrics.  Would you see that as being another branch next to /base /vendor and /application?  See #2 below as it would have similar considerations.

A couple of thoughts on the list of metrics that I thought would be good to discuss...

1.  I don't see, in the current doc under review, a way to specify metrics for objects with multiple instances.  For example, what would the name of a metric look like for a servlet response time (knowing that you might have 100 servlets and you want to track metrics separately for each of them).  The same question could be asked for things like thread pools, connection pools, EJB hit counts, etc.

2.  We might want to prescribe a set of metrics for servers providing JEE features.  That may not need to be in the first official spec, but something that could perhaps be added in the future.  We could potentially 'borrow' the set of metric names defined in the JSR77 spec.  Maybe a /jee set?

[Heiko Rupp]

From the Doodle, it looks like Tue 27th at 3pm CEST is the best time. Unfortunately I don't have write access to the calendar.

Emily - I think you have, could you please add this? 

I think we should also to weekly hangouts until the 1st version of the spec is out (?)

Thanks Emily, Felix and Don

   Heiko

[Heiko Rupp]

Am Mittwoch, 21. Juni 2017 12:55:11 UTC+2 schrieb donbo...@gmail.com:

Heiko, I think having metrics for other specs in MP would be good.  JAX-RS in particular would likely have lots of possible metrics.  Would you see that as being another branch next to /base /vendor and /application?  See #2 below as it would have similar considerations.

Yes, next to /base etc. Could also go below /base, but I think next to it is better.

 

A couple of thoughts on the list of metrics that I thought would be good to discuss...

1.  I don't see, in the current doc under review, a way to specify metrics for objects with multiple instances.  For example, what would the name of a metric look like for a servlet response time (knowing that you might have 100 servlets and you want to track metrics separately for each of them).  The same question could be asked for things like thread pools, connection pools, EJB hit counts, etc.

This is a good question. I have been thinking about it in the past and put it aside: I see (but may be mistaken) servers that follow Microprofile as servers that do one job and which are then wired together in a distributed set of Microservices.

Of course that does not prevent a server from having many servlets or many EJBs, but would that really be what people are looking for?

 

2.  We might want to prescribe a set of metrics for servers providing JEE features.  That may not need to be in the first official spec, but something that could perhaps be added in the future.  We could potentially 'borrow' the set of metric names defined in the JSR77 spec.  Maybe a /jee set?

Potentially - why not :) I think we should try to also get feedback from community after the 1st version is out.

[Emily Jiang]

Heiko,

yes, I can add the calendar invitation for you. Do you have a hangout url so that I can also put in?
Emily

[Felix Wong]

Emily,

Can you create a new Hangout URL?  We can reuse this one in future metrics calls.  Thanks!

Felix

[Heiko Rupp]

Emily,

my bad, yes: https://bluejeans.com/2042160481

[Emily Jiang]

Done. I was going to create a google hangout for you as suggested by Felix. However, bluejens is even better. I have put an entry in the calendar, repeat weekly till the end of this year. You can cancel this once it is no longer needed.

The calendar and its ics file can be found from MicroProfile wiki. Just import the ics file to your calendar. The calendar contains all MP meetings. You can choose which ones to attend.

Emily

[Heiko Rupp]

Thank you!

[donbo...@gmail.com]

Hi Heiko,

On your comment...

    This is a good question. I have been thinking about it in the past and put it aside: I see (but may be mistaken) servers that follow Microprofile as servers that do one job and which are then wired together in a distributed set of Microservices.

    Of course that does not prevent a server from having many servlets or many EJBs, but would that really be what people are looking for?

I can see a few reasons to support metrics for things that have multiple instances:

1.  applications people write may have use for that

2.  we may want to have different microservice endpoints each have their own set of metrics.  eg. counts and response time tracking for each endpoint.

3.  many different things can be the runtime for a microservice -- including traditional app servers that have multiple connection pools, multiple servlets, etc.  They will want to use the metrics API for all metric instrumentation, so will need the ability to expose metrics for things that aren't singletons.

[Felix Wong]

Just a reminder that we will have our first Hangout call on Tuesday 6/27 1pm GMT or 9am ET.  All are welcome.

[Heiko Rupp]

I have started a minutes document at https://docs.google.com/document/d/1qXlENF6zP5lVMlFpKRKHkmEHTHfDGU-91QoOTXpesm0/edit?usp=sharing

[Kevin Sutter]

The request to create the microprofile-metrics github repo:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=518854

Normally, this is completed within 24 hours.

Thanks, Kevin

[Heiko Rupp]

And it is live and I have added the original proposal as first pull-request (after converting to AsciiDoc)

https://github.com/eclipse/microprofile-metrics/pull/1

Who has commit rights?

Thanks Kevin for organising this.

  Heiko

[Felix Wong]

Raymond Lam has created a PR to update the base metrics

https://github.com/eclipse/microprofile-metrics/pull/2

[Felix Wong]

We will meet twice a week in the first 3 weeks in August.  The additional meeting date/time are as follows:-

Aug 3rd, 2pm GMT/10am ET

Aug 10th, 2pm GMT/10am ET

Aug 17th, 2pm GMT/10am ET