Question on use of Bytes

[Andrew Oswald]

Greetings,

I'm at a bit of a loss as to the proper usage of Chronicle Bytes, specifically its use of bytesForRead.  First I'll describe my usecase:

I've got a collection of objects that, amongst other things, encapsulate: a byte[], the data offset in said byte[], and the respective data length.  I'd like to pass the data through v4 Chronicle-Queue and was thinking I'd use Bytes.allocateElasticDirect() to get a nice contiguous buffer to store them in the interim.  I'd add data to the buffer by way of Bytes.write(byte[] bytes, int offset, int length), which is an absolutely perfect match to what I'm presented with.  Along the way, taking the respective slice of data and adding its pointer reference to the data to a collection of an AbstractMarshallable.

The thought for assigning the respective byte[] was to utilize bytesForRead which would provide me with a BytesStore for use in serialization.  This is analogous to the nio.ByteBuffer usage of slice(), but this is where I'm confused.  Do I need to explicitly assign the readPosition and readLimit to the Bytes that are given back to me through use of bytesForRead?  I would have minimally thought its read position would have been automatically set...  Please see below code as an example:

public class HelloWorld {

    

    private static final byte[] hwBytes = "Hello, World!".getBytes();

    

    public static void main(String[] args) {

        

        NativeBytes<Void> nativeBytes = Bytes.allocateElasticDirect();

        

        nativeBytes.write(hwBytes, 0, hwBytes.length);

        long writePosition = nativeBytes.writePosition();

        // is it normal to have to set the readLimit?

        Bytes<Void> bytesForRead1 = nativeBytes.bytesForRead().readLimit(writePosition);

        

        nativeBytes.write("Farewell, good-bye!");

        
        // is it normal to have to set the readPosition?

        Bytes<Void> bytesForRead2 = nativeBytes.bytesForRead().readPosition(writePosition);

        

        System.out.println("nativeBytes: " + nativeBytes.toDebugString());

        System.out.println("bytesForRead1: " + bytesForRead1.toString() + " " + bytesForRead1.toDebugString());

        System.out.println("bytesForRead2: " + bytesForRead2.toString() + " " + bytesForRead2.toDebugString());

    }

}

Note: I realize if there were more than two entries, per this approach, I'd need to set both readPosition as well as readLimit to the ones in the "middle".

Also: it seems VanillaBytes' bytesForRead implementation's call to isClear is always true and it seems that if it weren't I'd get read position and limit automatically assigned on my slice.

So, am I doing something incorrectly?  Is there a better way to accomplish the same objective?

thanks!

[Rob Austin]

Is this the sort of thing that you want to do, its easier to use wire and document, rather than work with the read/write positions/limits.

 

import net.openhft.chronicle.queue.ExcerptAppender;
import net.openhft.chronicle.queue.ExcerptTailer;
import net.openhft.chronicle.queue.impl.RollingChronicleQueue;
import net.openhft.chronicle.wire.DocumentContext;
import net.openhft.chronicle.wire.Wires;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.util.concurrent.TimeoutException;

/**
 * @author Rob Austin.
 */
public class Example {

    public static void main(String[] args)   throws TimeoutException, IOException {
        try (final RollingChronicleQueue queue = SingleChronicleQueueBuilder.binary(getTmpDir()).build()) {

            final ExcerptAppender appender = queue.createAppender();

            try (DocumentContext dc = appender.writingDocument()) {
                dc.wire().write("my-bytes").bytes("Farewell, good-bye!".getBytes());
            }


            final ExcerptTailer tailer = queue.createTailer();

            try (DocumentContext dc = tailer.readingDocument()) {

                // you can remove this line in you want <----- ********
                System.out.println(Wires.fromSizePrefixedBlobs(dc));

                byte[] bytes = dc.wire().read(() -> "my-bytes").bytes();

                System.out.println("tailer=" + new String(bytes));
            }

            // also take a look at
            System.out.println("this is whats in the queue file=" + queue.dump());

        }
    }


    // *************************************************************************
    //
    // *************************************************************************

    public static File getTmpDir() throws IOException {
        return Files.createTempDirectory("chronicle" + "-").toFile();
    }
}

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rob Austin]

what type of objects are you trying to send are they simple DTO’s if so I suggest you make them implement net.openhft.chronicle.wire.Marshallable or extend AbstractMarshallable

then use the wire.typedMarshable() or wire.marshable() method

otherwise you can use the object() method in wire. like this

import net.openhft.chronicle.queue.ExcerptAppender;
import net.openhft.chronicle.queue.ExcerptTailer;
import net.openhft.chronicle.queue.impl.RollingChronicleQueue;
import net.openhft.chronicle.wire.DocumentContext;
import net.openhft.chronicle.wire.Wires;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;

import java.util.HashMap;
import java.util.Map;

import java.util.concurrent.TimeoutException;

/**
 * @author Rob Austin.
 */
public class Example {

    public static void main(String[] args) throws TimeoutException, IOException {

        Map anExampleObject = new HashMap<>();
        anExampleObject.put("some", "data");

        try (final RollingChronicleQueue queue = SingleChronicleQueueBuilder.binary(getTmpDir()).build()) {

            final ExcerptAppender appender = queue.createAppender();

            try (DocumentContext dc = appender.writingDocument()) {

                dc.wire().write("my-map").object(anExampleObject);

            }


            final ExcerptTailer tailer = queue.createTailer();

            try (DocumentContext dc = tailer.readingDocument()) {

                // you can remove this line in you want <----- ********
                System.out.println(Wires.fromSizePrefixedBlobs(dc));

                Map result =   dc.wire().read(() -> "my-map").object(Map.class);

                System.out.println("tailer result=" + result.get("some"));

            }

            // also take a look at
            System.out.println("this is whats in the queue file=" + queue.dump());

        }
    }


    // *************************************************************************
    //
    // *************************************************************************

    public static File getTmpDir() throws IOException {
        return Files.createTempDirectory("chronicle" + "-").toFile();
    }
}

but unless you have to send bytes, I recommend you just pass the objects to chronicle wire and let it deserialise them for you. If you want more control make the objects net.openhft.chronicle.wire.Marshallable.

Rob 

 ,

On 17 Jun 2016, at 16:35, Andrew Oswald <andrew...@gmail.com> wrote:

[Rob Austin]

Also you may find this BLOG useful as an introduction into using some of the features of chronicle queue https://vanilla-java.github.io, although its main focus is on chronicle queue with micro services

[Andrew Oswald]

Is this the sort of thing that you want to do

No, as I attempted to convey in my original post, I've got objects which contain byte[], offset int to data in said byte[], as well as data length int.  This given format is in perfect alignment to the Bytes.write API; I'm simply trying to remove any superfluous bytes from the source byte[] and get the data into a CQ conducive format for use through methodWriter and methodReader semantics.

thanks.

[Andrew Oswald]

or extend AbstractMarshallable

Yes, AbstractMarshallable, as mentioned in my original post.

thanks. 

[Andrew Oswald]

Also you may find this BLOG useful

Yes, I've read much of it, thanks.

Which brings me back to my original question: is explicit assignment of readPosition and readLimit correct or is that an indication that I'm doing something incorrectly?

thanks.

[Rob Austin]

Do I need to explicitly assign the readPosition and readLimit to the Bytes that are given back to me through use of bytesForRead

no

 I would have minimally thought its read position would have been automatically set...  

Bytes is a low level interface and its simpler to use some of the methods I sent you to write bytes to a chronicle queue, or even better the objects directly,

but bytes has a readPosition(), readLimit() 

writePosition() writeLimit()

as you read from the bytes you can read up to the readLimit(), if you change the writePosition() you are in effect also changing the readLimit()

The thought for assigning the respective byte[] was to utilise bytesForRead which would provide me with a BytesStore for use in serialization.  

 In short, I don’t recommend you take the approach that you are suggesting. Its simpler to let chronicle do the serialisation for you.

[Andrew Oswald]

Ok, so what do I need to do to have my Bytes slices have the proper indexes assigned?  In other words, I don't want to consume more heap, but need pointers to the would-be off-heap slices.  It works as I've provided in my example (explicitly assigning those values), but it just doesn't seem correct to not have the library do that automatically.  That make sense?

Forget about all the CQ stuff; my mistake for providing that as context for what I'm trying to do.  Given an off-heap Bytes, when iteratively writing to it, how can I subsequently get a "slice" with the proper indexes assigned?

thanks again.

[Rob Austin]

Ok - if you are not using chronicle queue. And just want an example using bytes I can send you an example - give me a couple of hours as I'm in the middle of something else at the moment. 

Rob

Sent from my iPhone

[Peter Lawrey]

There was a bug in SubBytes which bytesForRead was using where the readLimit was set to the readPosition instead of the writeLimit.  I have fixed this and this test passes.

@Test
public void testBytesForRead() {
    byte[] byteArr = new byte[128];
    for (int i = 0; i < byteArr.length; i++)
        byteArr[i] = (byte) i;
    Bytes bytes = Bytes.wrapForRead(byteArr);
    bytes.readSkip(8);
    Bytes bytes2 = bytes.bytesForRead();
    assertEquals(128 - 8, bytes2.readRemaining());
    assertEquals(8, bytes2.readPosition());
    assertEquals(8, bytes2.readByte(bytes2.start()));
    assertEquals(9, bytes2.readByte(bytes2.start() + 1));
    assertEquals(9, bytes.readByte(9));
    bytes2.writeByte(bytes2.start() + 1, 99);
    assertEquals(99, bytes.readByte(99));
}

[Andrew Oswald]

Thanks for helping out with this, guys, very much appreciated.

So it seems the missing piece for me was invoking readSkip on my nativeBytes, which then has the effect of VanillaBytes.bytesForRead()'s call to isClear() to return false and thusly return a SubBytes.

What's the general idea for calling readSkip?

thanks again!

[Rob Austin]

Regarding

What's the general idea for calling readSkip?

Are you asking, when would you use this. ?

Sent from my iPhone

[Andrew Oswald]

Yes.  I assume it's more than just a coordination mechanism to subsequently call bytesForRead and there's good reason it's needed as a prerequisite for getting back SubBytes.

Mostly just out of curiosity.

[Andrew Oswald]

Oh, in the case of first byte[] write into a Bytes and invoking bytesForRead, is there a best practice to follow in order to not get back a VanillaBytes by way of having isClear return false?  Something like always provide a byte that won't be used and set the readSkip to 1?

thanks.

[Rob Austin]

I'm not sure about needing it as a prerequisite for subBytes. 

But the usual patten is to write the length followed by the bytes. Then when reading - you read the Len and then sometimes you want to discard/ignore the next chunk of data, so would readSkip(<the Len just read>)

Sent from my iPhone

--

[Andrew Oswald]

Without altering the underlying Bytes' readPosition, isClear() would always return true (and return a VanillaBytes w/out setting read position and limit):

From VanillaBytes:

    @NotNull

    @Override

    public Bytes<Underlying> bytesForRead() throws IllegalStateException {

        return isClear()

                ? new VanillaBytes<>(bytesStore, writePosition(), bytesStore.writeLimit())

                : new SubBytes<>(bytesStore, readPosition(), readLimit());

    }

From Bytes:

    /**

     * @return is the readPosition at the start and the writeLimit at the end.

     */

    default boolean isClear() {

        return start() == readPosition() && writeLimit() == capacity();

    }

This still seems a bit strange to me, but if you guys are saying that's how it works, a-ok by me.  Maybe I'll post another code sample after kicking the tires a little more.

[Rob Austin]

I could be wrong about this ( I'd have to check ), but my understanding is that you should consider your Bytes to be your slice. ( Bytes is not thread safe, it has a read and write positions )

It's the BytesStore that hold the underline bytes. ( but this is thread safe and has no such read and write indexes ), for indexes you should use Bytes. 

Sent from my iPhone

[Andrew Oswald]

So after invoking Bytes.allocateElasticDirect(), I put in a dummy byte and move the readPosition by way of readSkip(1):

public class HelloWorld {

private static final byte[] hwBytes = "Hello, World!".getBytes();

private static final byte[] ihBytes = "I'm here!".getBytes();

private static final byte[] gbBytes = "Farewell, good-bye!".getBytes();

public static void main(String[] args) {

NativeBytes<Void> nativeBytes = Bytes.allocateElasticDirect();

// following two lines seem odd, but so be it?...

nativeBytes.writeByte((byte)0);

nativeBytes.readSkip(1);

nativeBytes.write(hwBytes, 0, hwBytes.length);

Bytes<Void> bytesForRead1 = nativeBytes.bytesForRead().readLimit(nativeBytes.writePosition());

nativeBytes.readSkip(hwBytes.length);

nativeBytes.write(ihBytes);

Bytes<Void> bytesForRead2 = nativeBytes.bytesForRead().readLimit(nativeBytes.writePosition());

nativeBytes.readSkip(ihBytes.length);

nativeBytes.write(gbBytes);

Bytes<Void> bytesForRead3 = nativeBytes.bytesForRead().readLimit(nativeBytes.writePosition());

System.out.println("nativeBytes: " + nativeBytes.toDebugString());

System.out.println("bytesForRead1: " + bytesForRead1.toString() + " " + bytesForRead1.toDebugString() + " " + bytesForRead1.readRemaining());

System.out.println("bytesForRead2: " + bytesForRead2.toString() + " " + bytesForRead2.toDebugString() + " " + bytesForRead2.readRemaining());;

System.out.println("bytesForRead3: " + bytesForRead3.toString() + " " + bytesForRead3.toDebugString() + " " + bytesForRead3.readRemaining());

}

}

Output:

nativeBytes: [pos: 23, rlim: 42, wlim: 8EiB, cap: 8EiB ] ٠Hello, World!I'm here!‖Farewell, good-bye!‡٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠٠

bytesForRead1: Hello, World! [pos: 1, rlim: 14, wlim: 14, cap: 14 ] ‖Hello, World!‡ 13

bytesForRead2: I'm here! [pos: 14, rlim: 23, wlim: 23, cap: 23 ] ‖I'm here!‡ 9

bytesForRead3: Farewell, good-bye! [pos: 23, rlim: 42, wlim: 42, cap: 42 ] ‖Farewell, good-bye!‡ 19

Yes, plan is to only ever have a single Thread ever manipulate a Bytes.  :)

thanks.

[Peter Lawrey]

ReadSkip moves the read position to where you want the start to be. You could just use readPosition(x)  the readRemaining is the same after you do this.