Re: Usage for a durable staging area, plus Recovery on crash

[Rob Shepherd]

private static int getJournalledInt(Location loc, Journal journal) throws IllegalStateException, IOException {

        byte[] bytes = journal.read(loc, ReadType.ASYNC);

        int year = ByteBuffer.wrap(bytes).getInt();

        return year;

}

[Julien Eluard]

Hi Rob,

I encountered the same scenario a couple days ago. I believe this is a bug: see the report here https://github.com/sbtourist/Journal.IO/issues/23.

I also pushed a pull request that fixed the issue for me: https://github.com/sbtourist/Journal.IO/pull/24

You might want to take a look to see if it solves it for you too?

Julien

Le jeudi 24 janvier 2013 15:34:53 UTC-3, Rob Shepherd a écrit :

Dear Group, 

I'm just looking at Journal.IO which seems like a suitable fit for what I need - nevertheless, please can somebody confirm the following.

1. Can I use it as the intermediate durable for multiple reader & writer threads.  

E.g. I have a web request thread which performs some transactions. Metadata about those transaction is recorded. As the DB transaction are committed, the meta-events are synced to a journal.io (Writer) - a thread-pool of readers picks items out of the journal, does some stuff, then deletes the item in the journal.

If I wanted to treat it like a FIFO, surely journal.write() is the best way to append. 

1b. But what is the right way to "peek" at the front? is it journal.redo().interator().next()?

1c. then what is the best approach to subsequently "pop" off the end?

2. If the application crashes, then upon reboot, can I purge the journal of items before going into the main mode of execution?

2b. is journal.redo() the right thing to do?

Further reading.

As an experiment to all of the above, I created a multi-threaded application which uses an AtomicInteger to insert "years" into the journal. 

Readers access the journal and delete entries that are not leap years.

When I restart the application it uses redo() to take a look at the remaining items (these should all be leap years, plus unserviced items at the end, right?)

I expected this to work however it fails:

for(Iterator<Location> it = journal.redo().iterator(); it.hasNext(); )
{
        Location loc = it.next();
        if( ! (loc == null ))
        {
                System.out.println(loc.getPointer());
                int leapYear = getJournalledInt(loc, journal);
                System.out.println(leapYear + " - " + isLeapYear(leapYear));
                it.remove();
        }
}

This breaks with:

7

4 - true

Exception in thread "main" java.lang.IllegalStateException

at journal.io.api.Journal.sync(Journal.java:270)

at journal.io.api.DataFileAccessor.updateLocation(DataFileAccessor.java:60)

at journal.io.api.Journal.delete(Journal.java:337)

at journal.io.api.Journal$Redo$1.remove(Journal.java:1070)

at Test.main(Test.java:47)

Caused by: java.lang.NullPointerException

at journal.io.api.Journal$WriteFuture.get(Journal.java:1009)

at journal.io.api.Journal$WriteFuture.get(Journal.java:988)

at journal.io.api.Journal.sync(Journal.java:265)

... 4 more

what am I doing wrong?

Many thanks for input, suggestions and advice.

Rob

[Rob Shepherd]

Thanks Julien,

I'll apply the patch tomorrow and see if it works.

Thanks, Rob

[Rob Shepherd]

[Sergio Bossa]

Hi Rob!

> 1. Can I use it as the intermediate durable for multiple reader & writer
> threads.

Yes.

> If I wanted to treat it like a FIFO, surely journal.write() is the best way
> to append.
> 1b. But what is the right way to "peek" at the front? is it
> journal.redo().interator().next()?

Yes, redoing is the way to traverse journal entries from the beginning.

> 1c. then what is the best approach to subsequently "pop" off the end?

Just mark as deleted the journal location you want to "pop".

> 2. If the application crashes, then upon reboot, can I purge the journal of
> items before going into the main mode of execution?
> 2b. is journal.redo() the right thing to do?

Not sure I understand here: could you elaborate more?

> Exception in thread "main" java.lang.IllegalStateException
> at journal.io.api.Journal.sync(Journal.java:270)
> at journal.io.api.DataFileAccessor.updateLocation(DataFileAccessor.java:60)
> at journal.io.api.Journal.delete(Journal.java:337)
> at journal.io.api.Journal$Redo$1.remove(Journal.java:1070)
> at Test.main(Test.java:47)
> Caused by: java.lang.NullPointerException
> at journal.io.api.Journal$WriteFuture.get(Journal.java:1009)
> at journal.io.api.Journal$WriteFuture.get(Journal.java:988)
> at journal.io.api.Journal.sync(Journal.java:265)
> ... 4 more

As already answered, this is a bug which is going to be solved soon :)

Feel free to get back with any more questions.
Cheers,

Sergio B.

--
Sergio Bossa
http://www.linkedin.com/in/sergiob

[Sergio Bossa]

Thanks for the pull request, Julien: going to have a look :)

[Sergio Bossa]

Rob, Julien, the issue is now fixed on master. Do let me know if it
works for you guys :)

[Rob Shepherd]

On Friday, 25 January 2013 00:47:31 UTC, Sergio B. wrote:

Rob, Julien, the issue is now fixed on master. Do let me know if it
works for you guys :)

Yes this fixes that issue thanks.

Thanks especially for the quick fix to both you Sergio and Julien.

Rob