Support for cold writes & backfilling

[Loic C]

Hello,

M3DB has very promising storage efficiency and performances. I would love to try it.

The issue with M3DB is that we cannot backfill archive data ... Which is a general use case for us (and many others I guess).

The documentation still states it is not possible: https://m3db.github.io/m3/m3db/

“The project has aimed to avoid compactions when at all possible, currently the only compactions M3DB performs are in-memory for the mutable compressed time series window (default configured at 2 hours). As such out of order writes are limited to the size of a single compressed time series window. Consequently backfilling large amounts of data is not currently possible.”

But I found those issues, but they are not very instructive

• https://github.com/m3db/m3/issues?q=is%3Aissue+cold+write+label%3A%22C%3A+Out+of+Order+Writes%22
• https://github.com/m3db/m3/issues/1799
• https://github.com/m3db/m3/issues/1978
• https://github.com/m3db/m3/issues/1809
• https://github.com/m3db/proposal/issues/5

I wonder what is the current support for writing or rewriting data fro the past in M3DB.

Would you please explain what is the current status

Thanks

[Rob Skillington]

Hey Loic,

Apologies for a significantly delayed response. Right now it's not currently possible just using the standard measures to write data into the DB due to our lack of support for compactions. 

We'll be releasing a command line utility in the near future to do this in a semi-online manner (requiring a rolling restart after executing the tool). 

While we understand this is a significant limitation, we would like to progress the project to a point where there is a hyper-optimized manner for appending data in historical blocks of time series data so that this can be done efficiently without impacting the performance of the TSDB which if it was bolted on right now would cause such a side effect.

Thanks for the query and apologies again for the delay. 

Feel free to join us on Slack at https://bitly.com/m3slack to get a much faster response, I'll keep a closer eye on the mailing list too.

Best,

Rob

--
You received this message because you are subscribed to the Google Groups "M3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to m3db+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/m3db/a316736c-2b93-44f0-bc3f-be3cab63069e%40googlegroups.com.