dynamic variables

[Anton Kiryushkin]

Hello.

I have a task to calculate the nginx average response time for a dynamic amount of servers. Is there a way to dynamically define variables based on the data from a log to calculate the average for each server? Maybe there is a way to define a variable as an argument to mtail programs?

[Jamie Wilkinson]

I recommend using Prometheus or another collection tool that can perform that job. 

If you have mtail extracting the metrics from the ngonx logs on each ngonx server then your Prometheus instance can collect from all of them as they dynamically change. Prometheus will attach labels to match each target.

Then you can perform the average computation once in Prometheus for all the targets at once.

See

https://google.github.io/mtail/Deploying.html

and https://google.github.io/mtail/Interoperability.html

On Wed, 3 July 2024, 07:32 Anton Kiryushkin, <sw...@fotofor.biz> wrote:

Hello.

I have a task to calculate the nginx average response time for a dynamic amount of servers. Is there a way to dynamically define variables based on the data from a log to calculate the average for each server? Maybe there is a way to define a variable as an argument to mtail programs?

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/fd54048b-1935-4953-82e7-e538dc517551n%40googlegroups.com.

[Jamie Wilkinson]

I find that surprising, but if that were the case I'd investigate a sharded collection setup with a tree like structure of Prometheuses.

On Wed, 3 July 2024, 17:41 Anton Kiryushkin, <sw...@fotofor.biz> wrote:

Unfortunately, there are too many rows in this case which would dramatically slow down Prometheus.

Best regards,

Anton Kiryushkin

[Jamie Wilkinson]

I'm sorry then, I don't understand what you're asking.

Can you give some specific examples of what you've tried and what the expected output is?

On Wed, 3 July 2024, 20:09 Anton Kiryushkin, <sw...@fotofor.biz> wrote:

It doesn’t matter how many Prometheus instances I run. This is just wasting of the resources.

Mtail can calculate average value itself and save a lot of time and resources.

Best regards,

Anton Kiryushkin

[Jamie Wilkinson]

How many nginx servers?

Are you running one central log collection with one mtail or many mtails, one per nginx?

Over what time period are you computing the average response time? Is this a cumulative average or a rolling average?

Do you need average time per server or are there groups of servers in aggregate?

What collection mechanism are you currently using?

On Wed, 3 July 2024, 21:41 Anton Kiryushkin, <sw...@fotofor.biz> wrote:

Okay. I'll try again.

I have servers with nginx, and I want to know the average response time for certain types of requests based on their logs. There are about 10 million of these requests per day from each server. I want to know the average response time for each server.

My original question was whether it is possible to calculate this average time for each server without declaring hidden variables for each server. And the answer is obviously no.

Ingesting 10 million lines into Prometheus or VictoriaMetrics seems very wasteful because querying for one server might still work in a reasonable time, but querying for a hundred servers likely won't.

--

Best regards,

Anton Kiryushkin