Writing to a boost gzip filter stream

[emyrj...@gmail.com]

Hi,

I have an application that is currently using gzipped tab seperated fields file. The fields are a mix of strings and ints (it's the lstat info for every file in a filesystem). To store the full path in this file I base64 encode it so that the TSV format doesn't break when paths contain unprintable characters. I load this into a clickhouse database for reporting. Having to do base64 decoding in clickhouse is a pain and slows down the queries.

I want to switch my lstap collector to output capnp messages to avoid having to use base64 encoding. I currently use the boost gzip filter streams to do on the fly compression for my tsv lines. I'd like to do the same with capnp messages but it seems the only way to write to a file is using file descriptors. How can I write messages to a boost::io::filtering_ostream ?

Many thanks,

Emyr

[Kenton Varda]

Hi Emyr,

You can use capnp::writeMessage() to write to any kind of stream.

https://github.com/capnproto/capnproto/blob/7e2f70f354e2ca3d9de73ddaf974408ddbd98866/c++/src/capnp/serialize.h#L162

This function writes to a `kj::OutputStream`, which is an abstract interface that you can implement any way you want. It only has a couple methods:

https://github.com/capnproto/capnproto/blob/7e2f70f354e2ca3d9de73ddaf974408ddbd98866/c++/src/kj/io.h#L77

(capnp::writeMessage() always calls the second version of write(), which takes an array of byte arrays, to write the entire message all at once.)

In fact, there's even already an implementation of kj::OutputStream that gzip-compresses the output:

https://github.com/capnproto/capnproto/blob/7e2f70f354e2ca3d9de73ddaf974408ddbd98866/c++/src/kj/compat/gzip.h#L69

So you don't even need to use boost for that.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/7b25d572-414e-481a-a9e8-587b6ed522fco%40googlegroups.com.