Random access file in process visibility

[Eitan]

I want to use a file in a single process with single writer or multi
readers mutual exclusion, for that purpose I use number of
RandomAccessFile instances over the same file which are open in read
mode and are used concurrently. I also have a single RandomAccessFile
open in "rw" mode which updates the file (not when it is being
actively read).
I do not close the Random Access File handles at any point.

I wanted to know if the data which is being written to the file with
the RandomAccessFile in rw mode will always be visible to the readers
after the the write is complete?
(I am not using rws mode for performance issues, only for these
readers which are in the same process).

Thanks
Eitan

[Christian Catchpole]

Can you even get a second lock on a file that way? Sounds dangerous.
I would just create a some kind of service that each Thread can access
that will synchronise disk access down to one RandomAccessFile.

[Eitan]

I am trying to avoid locking the access to file for multiple readers
can be multi concurrent readers and it will be a bottleneck.
I've created a multi threaded test for it and it seems to work fine,
but I am not still not confident about it. I was hoping to find some
documentation about it which specifies the behavior.

On Dec 20, 11:40 pm, Christian Catchpole <christ...@catchpole.net>
wrote:

[Ben Schulz]

My guess is that Java IO takes advantage of any OS optimizations there
are, on another OS there may be a per-file-pointer cache or something.
That would obviously invalidate your assumptions. I suggest you look
into using asynchronous I/O which will get rid of any multiple-reader-
bottlenecks. This should also rule out third parties tampering with
the file (since you already hold the write lock).

With kind regards
Ben

[Eitan]

Looking at FileChannel it has the following part in its documentation:

The view of a file provided by an instance of this class is guaranteed
to be consistent with other views of the same file provided by other
instances in the same program. The view provided by an instance of
this
class may or may not, however, be consistent with the views seen by
other
concurrently-running programs due to caching performed by the
underlying
operating system and delays induced by network-filesystem protocols.
This
is true regardless of the language in which these other programs are
written, and whether they are running on the same machine or on some
other
machine. The exact nature of any such inconsistencies are system-
dependent
and are therefore unspecified.

This looks exactly like the guarantee I need, now I've looked into
RandomAccessFile.getChannel() implementation and it holds a single
member of FileChannel which is initialized on demand and the next
calls will return the same instance.

So the only way I see to create multiple FileChannel instances over
the same file is to open multiple RandomAccessFile instances over that
file and use getChannel() on each of these instances. Since
FileChannel does provide this guarantee I tend to believe it relies
upon a guarantee that RandomAccessFile provide, otherwise it would
need some mechanism to identify that different FileChannel instances
over different RandomAccessFile instances are actually over the same
file in order to provide the specified guarantee.

What do you think?

[Christian Catchpole]

Is not the simplest solution to allow each thread to get a reference
to the *single* RandomAccessFile instance and simply synchronize on it
while reading (not forgetting to reset the offset into the file).
It's not going to be super performant, but how is it ever going to be
with multiple threads thrashing on one file? If anything it might
reduce thrashing as you are controlling the points of file
contention. What is the ratio of time spent reading the file to time
spent operating on the data?

If all these threads will in-fact thrash on the file it might only be
solved with:

1. Better scheduling than "free for all" threads.
2. a large OS level disk cache
3. an in memory disk cache
4. some kind of abstract layer over the file which does caching and
returns you "model objects" rather than being seen as a byte stream.

[Eitan]

What about what I said above, using multiple FileChannel/
RandomAccessFile to read from the same file (not writing, there's a
single writer multi reader mutual exclusion protection over these
FileChannel/RandomAccessFile access), all I am worried about is the
visibility that updates the writer will do will be visible to the
readers once done, FileChannel guarantee this and I suspect it must
rely on this kind of guarantee from the RandomAccessFile that creates
it though it is not documented, But according to the simple analysis I
of the RandomAccessFile getChannel method I did, it seems that it does
need to guarantee it in order for FileChannel to guarantee it.

Since my implementation here is just an "unlimited" single list of
objects that starts in the memory and once there's no room continue in
the disk, I went for this solution and not using an over kill embedded
data base. I only need to support appending objects at the end of the
list, removing from the start and iterating over the list in read only
mode. (Either from the start to the end or from the end to the start).

Regarding the caching you suggested, I created caching mechanism and
there are a few layers of abstraction over this file, but, I am
talking here about the lowest level of the abstraction which access
the file and that might occur concurrently and I want to remove the
contention there as well. (for instance 10 threads currently iterating
over the list all at a different location) besides the whole issue
here is to move the data from the memory to the disk because I am out
of memory space, so caching the 10g file is not an option, I only
cache the end and the start of the list since they are accesses most
frequently.
The time spending ratio here is not the problem, there can be tens of
threads concurrently iterating over the list and I want them to be
able to iterate concurrently and not wait for each other. According to
amdal's law even if the time spent on reading is just 10%, I can only
accelerate this process by at most 10 so 100 threads iterating will
work no faster than 10.

Basically if I get the visibility guarantee then my multi
RandomAccessFile in read mode over the file is good enough solution,
let the disk and the OS do the caching of accessing the same file
sections and keep my program more concurrent.

Thanks

On Dec 24, 11:45 pm, Christian Catchpole <christ...@catchpole.net>

[Viktor Klang]

Why a file mate?
Why not Redis, Cassandra, MongoDB, SQLite whatnot?

--

You received this message because you are subscribed to the Google Groups "The Java Posse" group.
To post to this group, send email to java...@googlegroups.com.
To unsubscribe from this group, send email to javaposse+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Blog: klangism.blogspot.com
Twttr: twitter.com/viktorklang
Code: github.com/viktorklang

[Eitan]

Because it's a simple feature that I need and I don't want an overkill
solution that I do not control its thread count, its memory
consumption and cpu usage.
It is just a simple component inside my highly performance server that
needs to keep a single list of redo log in the disk if it takes to
much memory which will be very light and won't impact the server
performance. This occurs when the backup server it replicates to is
temporarily down and the redo log starts to accumulate. There are many
consideration which I can't really explain, but I do not wish to use
any embedded or remote third party overkill product for this very
simplified capabilities, like I've described above, unlimited single
list of elements that can only be added at the end of the list,
removed from the first and iterated over the list in read only mode.

Any other generic product addresses a much more complex scenario and
probably will perform worse because of that.

I already implemented it and it works fine, but I am not sure about
the multi reader over a file regarding the visibility issue and I was
hoping that someone knows what kind of guarantee is there for the
visibility of multiple RandomAccessFile inside the same process over
the same file. (only one in rw mode and the rest in r mode)

Thanks

> > javaposse+...@googlegroups.com<javaposse%2Bunsu...@googlegroups.com>

[Eitan]

Because it's a simple feature that I need and I don't want an overkill
solution that I do not control its thread count, its memory
consumption and cpu usage.
It is just a simple component inside my highly performance server that
needs to keep a single list of redo log in the disk if it takes to
much memory which will be very light and won't impact the server
performance. This occurs when the backup server it replicates to is
temporarily down and the redo log starts to accumulate. There are many
consideration which I can't really explain, but I do not wish to use
any embedded or remote third party overkill product for this very
simplified capabilities, like I've described above, unlimited single
list of elements that can only be added at the end of the list,
removed from the first and iterated over the list in read only mode.

Any other generic product addresses a much more complex scenario and
probably will perform worse because of that.

I already implemented it and it works fine, but I am not sure about
the multi reader over a file regarding the visibility issue and I was
hoping that someone knows what kind of guarantee is there for the
visibility of multiple RandomAccessFile inside the same process over
the same file. (only one in rw mode and the rest in r mode)

Thanks

On Dec 25, 12:41 pm, Viktor Klang <viktor.kl...@gmail.com> wrote:

> > javaposse+...@googlegroups.com<javaposse%2Bunsu...@googlegroups.com>

[Christian Catchpole]

Well, as you are finding, this "simple" approach is causing you
problems. The only thing I can add is to just be cautious about
relying on side-effects or presumptions about these multiple readers.
Because they could change at any time. I've never considered such an
approach because I have always thought of a File Stream / Reader and
the actual file as a 1:1 relationship. Anything else (except for 100%
read only) sounds dodgy. You mention you sometimes going into append
mode. But this would invalidate all the other read locks.

I agree that caches are not magic - they only work when on average,
access tends to cause clumps of common data.

If you were to pull apart the guts of database i wonder if you would
find multiple file readers. My guess is no.

> > > javaposse+...@googlegroups.com<javaposse%2Bunsubscribe@googlegroups .com>

[Eitan]

So do you think that the FileChannel documentation is misleading?

"The view of a file provided by an instance of this class is
guaranteed
to be consistent with other views of the same file provided by other
instances in the same program"

This is taken from the javadoc of FileChannel.
Thanks

On Dec 26, 3:09 am, Christian Catchpole <christ...@catchpole.net>
wrote:

> ...
>
> read more »

[Christian Catchpole]

Not misleading - I was just talking about the standard file objects,
not these channels. So this might be your answer then.

> ...
>
> read more »