Why does the federate endpoint return all metrics as untyped?

[心若寒江雪]

curl http://127.0.0.1:9090/federate?match[]={job!=""}

output:

# TYPE scrape_duration_seconds untyped scrape_duration_seconds{instance="172.31.24.196:18080",job="rs"} 0.000687693 1671095235507 # TYPE scrape_samples_post_metric_relabeling untyped scrape_samples_post_metric_relabeling{instance="172.31.24.196:18080",job="rs"} 0 1671095235507 # TYPE scrape_samples_scraped untyped scrape_samples_scraped{instance="172.31.24.196:18080",job="rs"} 0 1671095235507 # TYPE scrape_series_added untyped scrape_series_added{instance="172.31.24.196:18080",job="rs"} 0 1671095235507 # TYPE up untyped up{instance="172.31.24.196:18080",job="rs"} 0 1671095235507

How does upper Prometheus in the Prometheus federated cluster know what kind of Metrics they are getting from the federate interface of lower Prometheus?

[心若寒江雪]

prometheus federate api

# TYPE hermes_req_proc_time untyped
hermes_req_proc_time{statsdtype="timers",gs_app="someapp",job="aggregator_10s",instance="172.24.0.7:9080",gs_dc="us-east-1",gs_svcuser="svcuser2",gs_statistics="lower",gs_service="hermes",gs_hostname="someapp-01",gs_env="dev",gs_instanceid="8ef31bcd",id="prom-1"} 156 1473292292860
# TYPE netdev untyped
netdev{dev="eth0",gs_svcuser="svcuser2",gs_env="dev",gs_instanceid="aabd3924",type="byteout",instance="container-metrics:9092",gs_dc="us-east-1",job="promclient_triton_5s",id="prom-1"} 2.198199005e+09 1473292299525
# TYPE hermes_req_proc_time untyped
hermes_req_proc_time{gs_dc="us-east-1",job="aggregator_10s",gs_statistics="sum",gs_service="hermes",gs_instanceid="8ef31bcd",gs_svcuser="svcuser2",statsdtype="timers",gs_env="dev",instance="172.24.0.7:9080",gs_app="someapp",gs_hostname="someapp-01",id="prom-1"} 6778 1473292292860

[Bjoern Rabenstein]

[Apologies for late answer.]

On 15.12.22 01:41, 心若寒江雪 wrote:
>
> How does upper Prometheus in the Prometheus federated cluster know what
> kind of Metrics they are getting from the federate interface of lower
> Prometheus?

Sadly, Prometheus doesn't really persist the metric type. It does some
caching of this kind of metadata (availably via a separate API, see
https://prometheus.io/docs/prometheus/latest/querying/api/#querying-target-metadata
), but that's only for the most recent scrapes and separate from the
TSDB.

Federation data is served from the TSDB, and at that point, Prometheus
is completely ignorant of the original type and has to mark everything
as "untyped".

I would love to see this changed (and it was certainly the plan from
the very beginning), but there is no concrete roadmap how to get there
yet. Maybe a Prometheus 3.x feature...

Note that remote-write has at least some support for the metric
type. If you could utilize remote-write for your use case, you might
have more success with the metric type.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in