high volume inserts into VictoriaMetrics

[Cary Whitney]

Hi,

Trying to insert a high volume data source into VictoriaMetrics.  I've currently implemented the InfluxDB method but since it is http, I'm getting sockets in TIME_WAIT.  I'm bulk loading 10K records at a time and have 6 to 10 TIME_WAIT connections.  This may not sound like a lot but this is only from a test system and the main system I expect to see about 300+ connection TIME_WAIT connections for each of the 4 collection methods.

Thus I am wondering if there is an effective data streaming method to send data to Victoria?  An API or something?  Not all data sources can be Prometheus.

Also any hints would be appreciated.

Thanks

Cary

[Aliaksandr Valialkin]

Hi Cary,

There are the following approaches for inserting high volumes of data into VictoriaMetrics over a small number of TCP connections:

- To send chunked data in InfluxDB format over HTTP. VictoriaMetrics accepts arbitrary number of http chunks in a single request.

- To send batches in InfluxDB format over HTTP keep-alive connections. VictoriaMetrics accepts arbitrary number of http requests over a single connection.

- To stream all the data in Graphite or OpenTSDB formats over a small number of TCP connections. See https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/Single-server-VictoriaMetrics#how-to-send-data-from-graphite-compatible-agents-such-as-statsd .

--
You received this message because you are subscribed to the Google Groups "VictoriaMetrics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to victoriametri...@googlegroups.com.
To post to this group, send email to victori...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/victoriametrics/7d8f41e8-e5ae-4bb4-94f8-794238952c91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best Regards,

Aliaksandr

[Cary Whitney]

Thank you very much.

Keep-alive did the trick.  Actually from the docs you pointed me to, it is the default and I found that the library/helper package was the issue.  It was specifically open/closing on each connection.  

Thanks

Cary

On Saturday, May 18, 2019 at 2:25:35 PM UTC-7, Aliaksandr Valialkin wrote:

Hi Cary,

There are the following approaches for inserting high volumes of data into VictoriaMetrics over a small number of TCP connections:

- To send chunked data in InfluxDB format over HTTP. VictoriaMetrics accepts arbitrary number of http chunks in a single request.

- To send batches in InfluxDB format over HTTP keep-alive connections. VictoriaMetrics accepts arbitrary number of http requests over a single connection.

- To stream all the data in Graphite or OpenTSDB formats over a small number of TCP connections. See https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/Single-server-VictoriaMetrics#how-to-send-data-from-graphite-compatible-agents-such-as-statsd .

On Fri, May 17, 2019 at 8:53 PM Cary Whitney <clwh...@lbl.gov> wrote:

Hi,

Trying to insert a high volume data source into VictoriaMetrics.  I've currently implemented the InfluxDB method but since it is http, I'm getting sockets in TIME_WAIT.  I'm bulk loading 10K records at a time and have 6 to 10 TIME_WAIT connections.  This may not sound like a lot but this is only from a test system and the main system I expect to see about 300+ connection TIME_WAIT connections for each of the 4 collection methods.

Thus I am wondering if there is an effective data streaming method to send data to Victoria?  An API or something?  Not all data sources can be Prometheus.

Also any hints would be appreciated.

Thanks

Cary

--
You received this message because you are subscribed to the Google Groups "VictoriaMetrics" group.

To unsubscribe from this group and stop receiving emails from it, send an email to victori...@googlegroups.com.

To post to this group, send email to victori...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/victoriametrics/7d8f41e8-e5ae-4bb4-94f8-794238952c91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best Regards,

Aliaksandr

[Aliaksandr Valialkin]

Hi Cary,

I'd like to notify you about changed Influx line protocol mappings starting from v1.18.0 - now measurement goes to metric name prefix instead of "measurement" label, so metric name becomes

{measurement}.{field_name}

for each field in the ingested influx line.

This should improve usability by simplifying time series filters in PromQL queries. For example,  usage_user{measurement="cpu"} now becomes cpu.usage_user .

To unsubscribe from this group and stop receiving emails from it, send an email to victoriametri...@googlegroups.com.

To post to this group, send email to victori...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/victoriametrics/3859be2b-181e-4b85-8624-96921f748de9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Best Regards,

Aliaksandr