Persistence as a service with Protobuf/GRPC and Chronicle Queue under the hood

[Vachagan Balayan]

I'm trying to create a event store based on Chronicle queue (each event stream is one chronicle queue), and expose the interface with GRPC.

Before this my read/write logic with chronicle was with byte[] (i know thats not good but it was temporary solution)

(pardon me for kotlin code, if necessary i can write java version)

fun readFrameAsByteArray(tailer: ExcerptTailer): ByteArray {
    // read next message size
    val result = ByteArray(tailer.readInt())
    // read message content
    tailer.read(result)
    tailer.finish()
    return result
}

fun writeByteArrayFrame(appender: ExcerptAppender, data: ByteArray) : Long{
    // current index we write to
    val index = appender.index()
    // calculate total message size
    val msgSize = 4 + data.size
    appender.startExcerpt(msgSize.toLong())
    // write length of the message
    appender.writeInt(data.size)
    // write message content
    appender.write(data)
    appender.finish()

    return index
}

As you can see i'm writing message length then actual data to the queue... so that when reading i know how many bytes do i need to read.

QUESTION 1. IndexedChronicle knows where exactly any message starts, 

taking that into account in order to read the message N i read from position(N) to position(N+1)...
so can i avoid putting message lengths yet be able to read messages one by one?

One of the issues i knew for a while is that i create useless byte[] objects, now i'm trying to avoid that as GRPC messages carry ByteString (which is like a ByteBuffer)
so my interface of interaction with chronicle essentially looks like this now

fun readFrameAsByteString(tailer: ExcerptTailer): ByteString {
    // todo
}

fun writeByteStringFrame(appender: ExcerptAppender, data: ByteString) : Long{
    // todo
}

If answer to my question 1 is negative then I need to keep this [ length | data ] message format,
thus i need to skip 4 butes (for length) write the data, then go back and write those 4 bytes now that i know how long my message was.

I remember there was some solution for this so that you don't create unnecessary byte[] if i'm not mistaken Peter had this in his blog but i cant find it :(

Please help me out on this one... i'll avoid lot of unnecessary garbage and inefficiency if i just implement properly these read/write functions.
Thanks in advance.

[Vachagan Balayan]

ByteArray in kotlin is same as primitive byte[] in java.

The question 2 is how do i skip 4 bytes, write N bytes, then go and write those skipped bytes?

[Peter Lawrey]

I would use Chronicle Queue v4. Indexed Chronicle would pad some messages which why the length might be required.
Queue v4 doesn't do this. It also supports reading to/from a byte [] with a length so you can reuse a byte [] and avoid garbage.
It also supports deserialization of reused mutable objects without the need for generated code so you don't create any garbage end to end.

Regards, Peter.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vachagan Balayan]

Hi Peter, I'm upgrading to V4 but it seems that interfaces had changed, and i'm getting lot of compilation errors.

Is there any document describing changes compared to 3.x version?

[Peter Lawrey]

As you are writing raw data, most likely you want to use.

appender.writeBytes(bytes);

or you can use any combination of Bytes methods eg.

appender.writeBytes(b -> b.write(bytesArray));