Labels on Prometheus metrics when serving multiple models with Tensorflow Serving

[Adrian Gay]

Hi

I posted this question on SO:

https://stackoverflow.com/questions/64466957/labels-on-prometheus-metrics-when-serving-multiple-models-with-tensorflow-servin

If what I am asking is not currently possible, is this something that I could possibly make a contribution on, ie. adding labels to the existing Prometheus metrics?

Thanks

Adrian

[Adrian Gay]

Answering my own question... following the trail in TF Serving and TF Core code in Github:
https://github.com/tensorflow/tensorflow/blob/2bbba9a08410460f5bdd3ff7c23b65bf0c9fb447/tensorflow/core/lib/monitoring/collected_metrics.h
https://github.com/tensorflow/tensorflow/blob/7df9d5ab2a8928c82615fbb634efd10a8230acf9/tensorflow/core/framework/metrics.cc
https://github.com/tensorflow/tensorflow/blob/0e718f2b0a9de489f135b9228f2db11c00a857d5/tensorflow/core/distributed_runtime/graph_mgr.cc
I see that it’s possible to add label properties to a ```MetricDescriptor```, and I see where ```UpdateGraphExecTime``` updates the histogram data, but this is not 'model-aware' and it seems non-trivial to find how, say, model name and version can be retrieved in order to label the metric. Somewhere further up the call chain would need to provide a reference downwards to this point? If anyone wants to help work on this as a feature, please get in touch, as my C++ skills are 20+ years old!

Thanks

Adrian