JVM crash with Chronicle Queue

[Anders Peterson]

We've had several JVM crashes lately related to our usage of Chronicle Queue.

# Problematic frame:

# J 15285 C2 net.openhft.chronicle.wire.BinaryWire.readText(ILjava/lang/Appendable;)Ljava/lang/Appendable; (367 bytes) @ 0x00007f691f4d1f93 [0x00007f691f4d14e0+0xab3]

# Problematic frame:

# J 18319 C2 net.openhft.chronicle.bytes.BytesInternal.parseUtf8_SB1(Lnet/openhft/chronicle/bytes/Bytes;Ljava/lang/StringBuilder;I)V (146 bytes) @ 0x00007f4bbf0fe1a3 [0x00007f4bbf0fe040+0x163]

<artifactId>chronicle-bom</artifactId>

<version>1.13.12</version>

We create JSON-strings and store those in Chronicle Queue. One object/message is sometimes quite large resulting in huge JSON-strings. (Most of the time they're not huge.)

appender.writeDocument(writer -> writer.getValueOut().text(tmpValue));

tailer.readDocument(reader -> reader.getValueIn().textTo(tmpStringBuilder));

In our code (temporarily for debugging purposes) we parse the JSON-strings back to Objects before submitting them to Chronicle - this always works. 

Apart from the JVM crashes we've seen several cases where the returned strings (reading back from Chronicle Queue) are not valid/usable JSON-strings. They appear to be truncated and/or missing parts in the middle.

How large "things" can you safely write to a queue?

Looking at the source code for SingleChronicleQueue and AbstractChronicleQueueBuilder I see there are attributes "blockSize" and "bufferCapacity" but I see no comments or documentation on what these actually mean (and I'm not able to deduce this from reading the code).

What's the correct way to configure the queues to be able handle larger messages/strings?

Thanks,

/Anders

[Peter Lawrey]

By default, you should be able to store strings of up to 16 MB without a problem. Larger strings are possible but we don't have tests for such large strings.
Would it be possible to create a unit test which demostrates your problem?

Peter.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anders Peterson]

Thanks for answering,

Is that 16MB as in the “overlapSize”?

My colleague mentioned we had gotten a 32MB message the other day. That’s just one example. We need to be able to handle that order of magnitude with a healthy margin.

We have changed the “blockSize” to 512M to see if that makes a difference. Is that something that could make a difference?


I can try to construct a test case...


/Anders

> To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/2ErOQXd9uFg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.

[Peter Lawrey]

The overlap size is the largest you can safely write without bounds checking. We can change the write text to check bounds periodically to lift this restriction. I suggest you reconsider writing such large entries as you can't sustain messages that large and not overload your system.
Would you consider using compression?

Peter.

> To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/2ErOQXd9uFg/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to java-chronicle+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

[Anders Peterson]

It’s only a few freak messages that are that big. I don’t think periodic checkis will help. Does Chronicle Queue have a compression feature?

Right now I hope that increasing the blockSize (overlapSize) solves the problem. What downsides should we expect if we increase the blockSize by a factor 4 or 8?


/Anders

> > To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/2ErOQXd9uFg/unsubscribe.

> > To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups "Chronicle" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/2ErOQXd9uFg/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.

[Peter Lawrey]

Chronicle Wire which queue is based on supports compression. We could add it to queue for entries like this.
Chronicle Wire Enterprise supports delta compression for the serialization of objects and this reduces the size by around 10x without the overhead of compression. It's also up to 3x faster as it writes less data.

> > To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/2ErOQXd9uFg/unsubscribe.

> > To unsubscribe from this group and all its topics, send an email to java-chronicle+unsubscribe@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups "Chronicle" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/2ErOQXd9uFg/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to java-chronicle+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

[Anders Peterson]

https://github.com/OpenHFT/Chronicle-Queue/pull/282

That pull request contains test cases that reproduce both "not getting back the same string" and "crashing the JVM".

/Anders

[Rob Austin]

I’ll take a look

rob

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

[Rob Austin]

we have raised issue https://github.com/OpenHFT/Chronicle-Queue/issues/283 and merged your pull request

Rob

On 8 Aug 2016, at 13:22, 'Anders Peterson' via Chronicle <java-ch...@googlegroups.com> wrote:

[Anders Peterson]

I increased the blockSize from 64MB (the default value) to 512MB (x8 larger). Since then we've had no problems.

The longest string we've seen so far is 24,830,885 characters long. That's a 48MB Java String instance, but only 24MB in the queue file, right? Which should I compare to the overlapSize?

I think it's ok that there is a (configurable) limit to what can be safely written to a queue, but if that limit is passed you should get an exception or something.

/Anders



On Saturday, August 6, 2016 at 1:05:02 PM UTC+2, Anders Peterson wrote:

[Peter Lawrey]

For now this is a work around. a 24 million character string should use 24 MB if there is no multi-byte characters. This means you need an overlap of 32 MB or a block size of 128 MB.

We have an open issue to avoid needing to change this setting to write larger strings.

--

You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

[Anders Peterson]

Ok, thanks, /Anders

> To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/2ErOQXd9uFg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.