Push metrics to M3DB

Pavan B

unread,

Jun 10, 2020, 8:07:34 PM6/10/20

to M3

Hello,

We are looking to push data from the metric collectors to M3DB. We want to avoid the multi-hop path of collector to prometheus-push-gateway to prometheus to m3 coordinator to m3db.

I checked out uber-go/tally, is seems to have a M3DB client, but uses UDP based client. Is this the right go client for M3DB? Are there setting on the client/M3DB side to make this work?

On a separate note, should we use M3Aggregator cluster to keep the clients isolated from the M3DB node cluster changes?

Appreciate any input on this.

Thanks

-Pavan

Srikrishna Paparaju

unread,

Jun 10, 2020, 10:33:17 PM6/10/20

to Pavan B, M3

On Thu, Jun 11, 2020 at 5:37 AM Pavan B <bas...@gmail.com> wrote:

Hello,

We are looking to push data from the metric collectors to M3DB. We want to avoid the multi-hop path of collector to prometheus-push-gateway to prometheus to m3 coordinator to m3db.

I checked out uber-go/tally, is seems to have a M3DB client, but uses UDP based client. Is this the right go client for M3DB? Are there setting on the client/M3DB side to make this work?

https://github.com/m3db/prometheus_client_golang

On a separate note, should we use M3Aggregator cluster to keep the clients isolated from the M3DB node cluster changes?

M3Aggregator does the downsampling. What kind of isolation are you referring to ?

Appreciate any input on this.

Thanks
-Pavan

--
You received this message because you are subscribed to the Google Groups "M3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to m3db+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/m3db/3c1ab018-0dfd-4084-8ac0-29c6b8f36cdeo%40googlegroups.com.

sayf eddine Hammemi

unread,

Jun 11, 2020, 1:58:01 AM6/11/20

to Srikrishna Paparaju, Pavan B, M3

I guess you want to push directly into the coordinator, but you need to do that in the Prometheus format.

To view this discussion on the web visit https://groups.google.com/d/msgid/m3db/CAPDqkKF5anggp78dakHmg%2BP6JA0X2PH3XbcM-bpyao8oGn1GMA%40mail.gmail.com.

Rob Skillington

unread,

Jun 19, 2020, 1:23:58 PM6/19/20

to sayf eddine Hammemi, Srikrishna Paparaju, Pavan B, M3

Hey Pavan,

Your best option is to push to the coordinators using the Prometheus Remote Write endpoint:

https://docs.m3db.io/coordinator/api/remote/

Here is a Go client:

https://github.com/m3dbx/prometheus_remote_client_golang

And a Java client:
https://github.com/m3dbx/prometheus_remote_client_java

The tally library with push was mainly used at Uber, there is no end-to-end integration with open source M3. It could be plausible with community help to support this, but it just hasn't received a lot of attention. Here's the work in progress PR that has been put on pause with respect to priorities:
https://github.com/m3db/m3/pull/1241

Apologies for the delay getting back to you, I've been in a reduced capacity on paternity leave(!).

Best,

Rob

To view this discussion on the web visit https://groups.google.com/d/msgid/m3db/CALnV8WgDFMV8PdXXbpjmf1pWzXSoLnqKLd6YQURv_T%2B39EkD7A%40mail.gmail.com.

Pavan Basetty

unread,

Jun 19, 2020, 5:34:29 PM6/19/20

to Rob Skillington, sayf eddine Hammemi, Srikrishna Paparaju, M3

Hi Rob,

Thanks for your reply. Hope you are enjoying parenthood :-)

Thanks for confirming that Prometheus Remote Write API is the right solution. I was able to use the Go client you referred to write metrics to M3.

It would be good if this code could be enhanced to cache and batch metrics to reduce the number of calls to M3DB. Cache could also be used to manage Counters (i.e always incrementing), as opposed to having the applications manage the cumulative values. I understand cache brings in the possibility of losing metrics.