Regarding busy loop for getting updates of Chronicle queue

[Vachagan Balayan]

I'm just starting with chronicle queue, so i might be missing something.

As i know if we have a queue updated from application 1 then the application 2 needs to have some busy loop calling .nextIndex() until it gets true.

What about using WatchService and instead of busy looping use events just like in NIO?

I haven't yet tried this out, and i'm not sure if watchService will pick up changes made to RandomAccessFile but if anyone tried it, or know the reason not to use this please let me know. 

Otherwise i'll try it and let you know under this thread.

[Peter Lawrey]

I haven't tried using the WatchService for this, but it is a good suggestion. 

Even if it's a bit slow, it might be a good option for some situations.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vachagan Balayan]

Thanks Peter, i'm now trying out the queue itself, after that i'll implement single threaded event loop listening for update events using WatchService, although i'm not sure how do i benchmark it against busy loop.

Few questions regarding the queue if i may. 

(P.S. I don't want to bother you much with my questions so if there is a blog post or an article somewhere to get me going with queue please refer it, 

i'm not sure i found something better than the github readme)...

1. Given an ExcerptTailer created from indexed chronicle.

Almost all read* methods have documentation stating "This method will block if no input is available", which raised a question if the current thread will block who is going to notify? is there a busy loop in the background (which i assume there should not be one)... 
I've tried it and it seems that its not blocking but rather returning some default value or throwing java.nio.BufferUnderflowException... 
Is the documentation outdated?

[Peter Lawrey]

I suggest you do a ping/pong test.  Write from "A" process to a queue and have an another "B" process read it. "B" process writes to a second queue which the first "A" reads.  This is the RTT (Round Trip Time) You can add the timings to a histogram e.g. net.openhft.chronicle.core.util.Histogram and at the end print out the distribution.

--

[Peter Lawrey]

This is a copy-paste error.  None of the operations are blocking, nor where they ever. 

In the case of read() it can return -1 if there is no more data, however most operations will throw a BufferUnderflowException as you suggest.

In Chronicle Bytes we have a readIncompleteLong() which will the remaining data as a long (to a maximum of 8 bytes)  This is used for hash code calculations.

You might like to look at the chronicle-queue module which will be the next version Queue v4.

Regards,

   Peter.

[Vachagan Balayan]

Peter currently i'm using this https://github.com/OpenHFT/Chronicle-Queue isnt this the module you refered in your reply? (i simply dont know about any other chronicle-queue project).

Also in order to create realistic comparison between busy loop and watch service (event loop) approaches, i'd like to know if it is common to run the busy loop in your worker thread? I've read somewhere that one must run a separate thread (assigned to a dedicated core), so i suppose that when a worker thread wants to wait for an update it literally wait()'s on some object which is then notified from the busy loop runing in a different thread than the worker right? or it works differently?

[Peter Lawrey]

There is a module under this repo called chronicle-queue which has the v4 code.

The assumption is that the busy waiting thread is not only the monitor but the consumer as well. Ie there is only one thread involved. It is also the one to write any output to say another thread. Ie message in to message out there only needs to be one thread end to end.

On 5 Dec 2015 6:35 am, "Vachagan Balayan" <vachagan...@gmail.com> wrote:

Peter currently i'm using this https://github.com/OpenHFT/Chronicle-Queue isnt this the module you refered in your reply? (i simply dont know about any other chronicle-queue project).

Also in order to create realistic comparison between busy loop and watch service (event loop) approaches, i'd like to know if it is common to run the busy loop in your worker thread? I've read somewhere that one must run a separate thread (assigned to a dedicated core), so i suppose that when a worker thread wants to wait for an update it literally wait()'s on some object which is then notified from the busy loop runing in a different thread than the worker right? or it works differently?

--

[Andrew Oswald]

Greetings, folks, hope all is well.

I'm at a bit of a loss as to just how a v4 SingleChonicleQueueBuilder's resultant ChronicleQueue's ExcerptTailer listens for messages.  The older version of the API had nextIndex() returning a boolean, but what's the approach for the new v4 ExcerptTailer, in essence, what replaced nextIndex()?

thanks!!

-andy

[Peter Lawrey]

The general approach is to attempt to read a message and the method returns false if there is no message.

If you are using the try-with-resource readingDocument you have to call isPresent()

Here is an example of where you provide a lambda to read the bytes of your message which will return true if a message is present.

default boolean readBytes(@NotNull ReadBytesMarshallable reader) {
    try (DocumentContext dc = readingDocument()) {
        if (!dc.isPresent())
            return false;
        reader.readMarshallable(dc.wire().bytes());
    }
    return true;
}

Note: there simplest approach is to use a higher level API which uses a method reader

https://vanilla-java.github.io/2016/03/24/Microservices-in-the-Chronicle-world-Part-2.html

In this approach you create an interface which describes all the message you would like to write, and a component which implements this interface. When you call the method writer, it turns each method call into a message, when you tell the method reader to readOne() it read a method call and invokes it on your component.

I recommend you read the whole series on Microservices and Lambda Architecture here.

http://vanilla-java.github.io/tag/Microservices/

Reading in reverse order of listing (i.e. the order I wrote them)

Regards,

   Peter.

[Andrew Oswald]

Very cool stuff!  And in the case of the method reader invoking component methods, is the opportunity there for optimizing the number of objects that get created?

Thanks so much for the reply and for making this open-source!

[Peter Lawrey]

You can do it without creating object at all.

Note: whenever a method is called with a DTO you actually get the same object passed each time so if you want to retain data it must be copied.

On the upside, the same object is reused.

Peter.