UTFDataFormatRuntimeException while reading map

[Mickael Marrache]

Hi,

I sometimes get the following error:

net.openhft.chronicle.bytes.UTFDataFormatRuntimeException: malformed input: partial character at end

        at net.openhft.chronicle.bytes.BytesInternal.parseUtf82(BytesInternal.java:406) ~[chronicle-bytes-1.5.1.jar:?]

        at net.openhft.chronicle.bytes.BytesInternal.parseUtf8_SB1(BytesInternal.java:339) ~[chronicle-bytes-1.5.1.jar:?]

        at net.openhft.chronicle.bytes.BytesInternal.parseUtf8(BytesInternal.java:97) ~[chronicle-bytes-1.5.1.jar:?]

        at net.openhft.chronicle.bytes.StreamingDataInput.parseUtf8(StreamingDataInput.java:265) ~[chronicle-bytes-1.5.1.jar:?]

        at net.openhft.chronicle.wire.BinaryWire.getStringBuilder(BinaryWire.java:421) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.BinaryWire.readText(BinaryWire.java:860) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.text(BinaryWire.java:1623) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.object(BinaryWire.java:2590) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.object0(BinaryWire.java:2507) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.object(BinaryWire.java:2457) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.ValueIn.object(ValueIn.java:357) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.MarshallableIn.readMap(MarshallableIn.java:109) ~[chronicle-wire-1.5.2.jar:?]

        at com.mycompany.service.MessageWriterImpl.run(MessageWriterImpl.java:89)

Any idea what is the cause of the issue?

Thanks,

Mickael

[Mickael Marrache]

Sometimes, I see this exception instead:

java.lang.UnsupportedOperationException: code=0xC5

        at net.openhft.chronicle.wire.BinaryWire.readText(BinaryWire.java:862) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.text(BinaryWire.java:1623) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.object(BinaryWire.java:2590) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.object0(BinaryWire.java:2507) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.object(BinaryWire.java:2457) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.ValueIn.object(ValueIn.java:357) ~[chronicle-wire-1.5.2.jar:?]

        at net.openhft.chronicle.wire.MarshallableIn.readMap(MarshallableIn.java:109) ~[chronicle-wire-1.5.2.jar:?]

        at com.mycompany.service.MessageWriterImpl.run(MessageWriterImpl.java:89) ~

It is related to the second thread where I mentioned the UnsupportedOperationException.

I have the input that caused the issue but I can't publish it. However, I know that one of the values in my Map is a string that contains special characters. However, my string is encoded using UTF-8 therefore it should be fine.

[Peter Lawrey]

This usually means the reader is trying to read text where there is binary data. To figure out why I would need to reproduce it. Could you create a test for me?

Regards, Peter.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mickael Marrache]

The exception thrown is not always the same.

You can use the following tests as an example:

@Test
 public void test() throws Exception {
     final ChronicleQueue queue = ChronicleQueueBuilder
       .single("your-path")
       .rollCycle(RollCycles.MINUTELY).build();
 
     //Write
     ExcerptAppender appender = queue.createAppender();
     Map<String, Object> writtenMap = new HashMap<String, Object>();
     writtenMap.put("key", "İŞ");
     appender.writeMap(writtenMap);
     appender.close();
 
     //Read
     ExcerptTailer tailer = queue.createTailer();
     Map<String, Object> readMap = tailer.readMap();
     tailer.close();
 }

Using this test I got the following error while reading:

java.nio.BufferUnderflowException

at net.openhft.chronicle.bytes.AppendableUtil.parse8bit_SB1(AppendableUtil.java:187)

at net.openhft.chronicle.wire.BinaryWire.readField(BinaryWire.java:376)

at net.openhft.chronicle.wire.BinaryWire.readField(BinaryWire.java:317)

at net.openhft.chronicle.wire.BinaryWire.readEventName(BinaryWire.java:290)

at net.openhft.chronicle.wire.MarshallableIn.readMap(MarshallableIn.java:109)

at com.mycompany.MyTest.test(MyTest.java:33)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)

at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)

at org.junit.runners.ParentRunner.run(ParentRunner.java:363)

at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)

at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

[Peter Lawrey]

Please don't call appender.close () as this can corrupt the data. I have removed it from the API as it's availability was accidental.

[Mickael Marrache]

Ok. I removed the close statement. Same issue remains. (and I know you didn't mean the close statement was the cause, but at least I can tell you that the data corruption occurring here is not related to the close statement :) )

[Peter Lawrey]

I have added this test and fixed a couple of things.  It checks every possible valid code point up to 65535

@RunWith(value = Parameterized.class)

public class MapWireTest {

    private final WireType wireType;

    private final Map m;

    public MapWireTest(WireType wireType, Map m) {

        this.wireType = wireType;

        this.m = m;

    }

    @Parameterized.Parameters

    public static Collection<Object[]> combinations() {

        List<Object[]> list = new ArrayList<>();

        WireType[] wireTypes = {WireType.TEXT, WireType.BINARY};

        for (WireType wt : wireTypes) {

            for (int i = 0; i < Character.MAX_VALUE; i += 128) {

                Map<Integer, String> map = new LinkedHashMap<>();

                for (int ch = i; ch < i + 128; ch++) {

                    if (Character.isValidCodePoint(ch)) {

                        final String s = Character.toString((char) ch);

                        map.put(i, s);

                    }

                }

                Object[] test = {wt, map};

                list.add(test);

            }

        }

        return list;

    }

    @Test

    public void writeMap() {

        Bytes bytes = Bytes.elasticByteBuffer();

        Wire wire = wireType.apply(bytes);

        wire.getValueOut()

                .marshallable(m);

//        System.out.println(wire);

        Map m2 = wire.getValueIn()

                .marshallableAsMap(Object.class, Object.class);

        assertEquals(m, m2);

    }

}

[Mickael Marrache]

Thamks.

Does that mean the issue is fixed? If yes, when do you plan to release?

[Peter Lawrey]

Hi,
    It means an issue was fixed. I feel the test coverage needs to be improved. We will have a release early this week.

Regards, Peter.