Prometheus file format

[Moe]

Hey,

I am using JMXCollector to fetch metrics and writing it into a file using the TextFormat and the format looks like the sample below.

# HELP kafka_server_socket_server_metrics_processor_queue_size_p99  kafka.server:name=null,type=socket-server-metrics,attribute=processor-queue-size-p99
# TYPE kafka_server_socket_server_metrics_processor_queue_size_p99 untyped
kafka_server_socket_server_metrics_processor_queue_size_p99{listener="PLAINTEXT",} 0.0

I have two questions. 

1. How do I load this file into Prometheus or grafana to visualise the metrics?

2. The metrics do not have the time stamp in them at the end. How do I get the timestamp?

Thanks in advance!

[Brian Candler]

1. Either drop the file into a directory for the node_exporter textfile collector; or put the file into a webserver directory, and scrape it from Prometheus

2. The timestamp is the time that the metric was scraped by Prometheus

Note that Prometheus isn't a general-purpose database, and although you can ingest historical data via "backfilling", this isn't suitable for continuous ingestion. There is also the remote-write protocol, but that's not something that's easy to integrate with for regular users.

A normal exporter works on-demand: that is, when it receives a HTTP request it collects data as of that instant in time, and returns it in the HTTP response. Rather than fight against Prometheus, I think it's better to swim with the flow.

[Moe]

Thanks Brian that was really helpful,

2. The use case I want this for doesn't need continuous ingestion. In that case is there a way for me to add time stamp to MetricFamilySample? 

[Stuart Clark]

On 07/08/2023 21:00, Moe wrote:
> Thanks Brian that was really helpful,
>
> 2. The use case I want this for doesn't need continuous ingestion. In
> that case is there a way for me to add time stamp to MetricFamilySample?

That isn't how Prometheus works. It will scrape that metric every 10
seconds to 2 minutes. If you need to know when that set of metrics was
created a common pattern is to include a metric who's value is the
timestamp.

--
Stuart Clark

[Matt Doughty]

I just wanted to chime in to say that you almost never need to know
exactly when a metric is updated.
Before you start exporting additional timestamp metrics, you should
really ask yourself whether you actually need them. What problem do
they actually solve?

--Matt

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d401af4d-8583-d2d6-dbbe-89853000e421%40Jahingo.com.



--
--Matt

[Moe]

You're right I don't need the exact time when the metric is fetched. I only need it to differentiate between iterations within the batch job. Then is creating a separate metric the best way to go?

[Matt Doughty]

So you are trying to get discreet metrics for every run of the batch
job? That sounds like an unbounded cardinality problem as you would
end up with a timeseries for every run of the batch job.
Am I misunderstanding or is this accurate?

--Matt

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bf01a491-2b12-4f10-b7db-fbf865647496n%40googlegroups.com.



--
--Matt

[Stuart Clark]

On 08/08/2023 20:31, Matt Doughty wrote:
> So you are trying to get discreet metrics for every run of the batch
> job? That sounds like an unbounded cardinality problem as you would
> end up with a timeseries for every run of the batch job.
> Am I misunderstanding or is this accurate?
>

>> You're right I don't need the exact time when the metric is fetched. I only need it to differentiate between iterations within the batch job. Then is creating a separate metric the best way to go?
>>

If that is the case then Prometheus isn't the right tool. Having
distinctly detectable groups of data for a particular job run indicates
you are talking about events which are quite different to metrics. For
events you'd want to be looking at tools such as Elasticsearch, Loki or
a standard SQL database.

Events and metrics can (and often are) used in parallel. For example
Prometheus would tell you that the average job runtime is 5 minutes over
the past 3 hours, but you'd then use the events system to find the exact
durations for each run (or the number of events processed, or the error
message returned, etc.).

--
Stuart Clark