Pushing KairosDB Collected Metrics to an External Endpoint

[Eric Xue]

Hello,

I've been reading over the documentation and also the GitHub repo, but seem to only be able to find plugins and methods that allow for sending data INTO KairosDB, not out to a different endpoint or tool. i.e. metrics4j, Prometheus adapter, Graphite Protocol, etc. 

I'm looking to POST all of the collected metrics from KairosDB to an external HTTP endpoint in a JSON payload format: https://kairosdb.github.io/docs/build/html/Statistics.html#metrics-reported

Is there a plugin or method out there that would make this easy? 

Thank you!

[Greg Matza]

Eric,

--Full disclosure that I work on ScyllaDB.--

If you use Scylla as your KairosDB storage, you may be able to do this. You might have to write a simple service that leverages Scylla JSON support and posts the data to the appropriate endpoint. Or you might be able to use an existing integration with Scylla CDC (Kafka, most likely).

Description of Scylla CDC:

https://www.scylladb.com/2020/07/23/using-change-data-capture-cdc-in-scylla/

Description of Scylla's support of JSON

https://docs.scylladb.com/getting-started/json/

https://www.scylladb.com/2019/01/10/json-support-in-scylla/

 

If this looks interesting, let me know. I might be able to find some folks at Scylla who would be interested in advising you.

Greg

--
You received this message because you are subscribed to the Google Groups "KairosDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kairosdb-grou...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kairosdb-group/8365daa4-41d6-4e5f-ba89-b595a86aa58en%40googlegroups.com.

[Brian Hawkins]

Are you wanting to forward data that has been sent to kairos or are you just wanting to send out karios internal metrics?  I'm guessing the latter.  A plugin would have to be written but it wouldn't be hard.  in the next release I'm planning on integrating my metrics4j library that will allow you to send those metrics elsewhere with ease.

By the way I'm really close on finishing the current release, just have to document a couple more things and it is done.

Brian

[Eric Xue]

Actually the former, the data and metrics from the DB itself versus just the app's internal metrics

[Brian Hawkins]

What you are after is https://github.com/kairosdb/kairos-remote.  This plugin was written so local kairos instances in data centers could forward the metrics they got on to a central collection point.  The plugin is simple and you should be able to modify it to your specific need.  The plugin listens on the event bus to all incoming metrics, it then batches them up on disk and forwards them on.

Brian