Testing 1.3 delete performance

[Rob Shepherd]

There was some discussion about the performance of deleting journal entries which led to issue #30 https://github.com/sbtourist/Journal.IO/issues/30

and subsequently Sergio made a commit https://github.com/sbtourist/Journal.IO/commit/ead66d0 to improve the performance.

I just wanted to share my feedback of this improvement.

My scenario is to use journal IO as a buffer between threads in an application server for handling audit log entries asynchronous to web activity.

Web-threads produce packed audit codes/references and then save to the journal and then signal the new journal location to a worker thread pool.

The worker pool uses a slave DB to transform references to human text for pretty audit log entries and then saves to the audit archive. The worker then deletes the audit entry from the journal.

Using journal.io allows the web thread to finish quickly, knowing that the basic audit stamp is safe on disk for something to expand and archive later on.

I made a small test harness to evaluate the performance when I noticed that deleting journal entries was a fairly slow activity.

There was a significant lag between production of journal entries and the subsequent read+delete of each entry when a number of threads performed each activity.

Test Harness: https://gist.github.com/robshep/4669110/raw/4a50073b609a14c05c4bef08578438b22626a86e/JIOTest.java

_before_

> rev: b73ccf8

> N records: 50000

> writers: 1

> readers: 1

> records size 104 bytes

> physical sync: true

Time to complete production: 3076s

Time to complete consumption: 3652s

Consumption/Production Lag: 576s

_after_

> rev: ead66d0

> N records: 50000

> writers: 1

> readers: 1

> records size 104 bytes

> physical sync: true

Time to complete production: 2970s

Time to complete consumption: 2970s

Consumption/Production Lag: 0s

So, the main benefit for my example here - which mirrors my requirements - is that the code for this commit has decreased the overhead for deleting so much that there is now no lag.

Many thanks for this effort Sergio. We are very grateful.

[Sergio Bossa]

Great to hear, Rob, and thanks for the writeup!

> --
> You received this message because you are subscribed to the Google Groups
> "Journal.IO" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to journalio+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Sergio Bossa
http://www.linkedin.com/in/sergiob