Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Fav. Memory Stream Impl.

60 views
Skip to first unread message

Jan Burse

unread,
Nov 25, 2011, 12:39:23 PM11/25/11
to
Dear All,

Would be interested in a good memory stream
implementation. Requirement will be multiple
readers / writers and positioning.

Just found this code:

http://www.mikepaul.ca/index.php?option=com_content&view=article&id=3:net-style-memorystream-for-javaj2me&catid=4:mobile-dev&Itemid=4
.Net style MemoryStream for Java/J2ME!
Thursday, 27 May 2010 22:52 | Written by Michael Paul

Although it has read / write and positioning,
I don't think it is thread safe. Any other
implementations around.

Maybe some operations that provide InputStream /
OutputStream factory methods.

But

markspace

unread,
Nov 25, 2011, 1:34:46 PM11/25/11
to
On 11/25/2011 9:39 AM, Jan Burse wrote:
> Dear All,
>
> Would be interested in a good memory stream implementation.

What is a "memory stream?"

Java already has bounded and unbounded queues, in thread safe and
non-thread safe version. For special pruposes Java also has things like
memory buffers for logging.

What's the actual use model for this thing?

(Esp. for those of us who don't know anything about C#/.Net)

Jan Burse

unread,
Nov 25, 2011, 2:59:37 PM11/25/11
to
markspace schrieb:
> For special pruposes Java also has things like memory buffers for logging.

Which classes? Part of JDK or some utility?

markspace

unread,
Nov 25, 2011, 3:16:04 PM11/25/11
to
It wouldn't kill you to look through the Java API yourself, you know.
Specifically for the logger, since that's what you seem to be interested in:

<http://docs.oracle.com/javase/7/docs/api/java/util/logging/MemoryHandler.html>


Jan Burse

unread,
Nov 25, 2011, 3:23:29 PM11/25/11
to
markspace schrieb:
No the memory stream need not be cyclic. The Java/J2ME
code reference I gave has maximally 256*chunks of size
16384 each. And there is something with a cicular buffer
in it.

But I guess the .NET memory streams are not cyclic. I am
looking for something like the .NET memory streams, so
that I can get rid of temporary file names in my
application.

Bye

Jan Burse

unread,
Nov 25, 2011, 3:23:50 PM11/25/11
to
Jan Burse schrieb:
>
> But I guess the .NET memory streams are not cyclic. I am
> looking for something like the .NET memory streams, so
> that I can get rid of temporary file names in my
> application.

http://msdn.microsoft.com/en-us/library/system.io.memorystream.aspx

markspace

unread,
Nov 25, 2011, 4:00:26 PM11/25/11
to

Jan Burse

unread,
Nov 25, 2011, 4:32:21 PM11/25/11
to
markspace schrieb:
The memory stream should be sharable.
Thats why I wrote:

> Requirement will be multiple readers / writers

And also seekable.
Thats why I wrote.

> and positioning.

To my knowledge ByteArrayOutputStream and ByteArrayInputStream
do not satisfy these requirements.

Bye

Jan Burse

unread,
Nov 25, 2011, 4:36:06 PM11/25/11
to
Jan Burse schrieb:
> The memory stream should be sharable.

Well to some extend ByteArrayOutputStream and
ByteArrayInputStream are shareble. They are
at least thread safe, I find:

public synchronized void write(int b) {

So the same output object can be used by multiple
threads. And the same input object can be used
by multiple threads.

But I would like to be able to do the same as with
a temporary file. Some threads open it for read,
and some threads open it for write.

Bye

Arne Vajhøj

unread,
Nov 25, 2011, 5:00:57 PM11/25/11
to
Use java.nio.ByteBuffer and handle concurrency in your code.

Arne


Robert Klemme

unread,
Nov 25, 2011, 5:22:26 PM11/25/11
to
Why do you want to do that with binary data? What is your use case?

Kind regards

robert


--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

Roedy Green

unread,
Nov 26, 2011, 3:26:25 AM11/26/11
to
On Fri, 25 Nov 2011 18:39:23 +0100, Jan Burse <janb...@fastmail.fm>
wrote, quoted or indirectly quoted someone who said :

>Would be interested in a good memory stream
>implementation. Requirement will be multiple
>readers / writers and positioning.

What do you use it for? What does it do for you?
Is it just an ram-resident InputStream?
--
Roedy Green Canadian Mind Products
http://mindprod.com
Premature optimization is the root of all evil.
~ Donald Ervin Knuth 1938-01-10
Unfortunately, some have misread this quotation as optimisation is
in itself evil, or even that is it wicked to consider speed when
choosing an algorithm. I counter, it is less work to do it right
the first time. Knuth was referring to clarity-destroying
micro-optimisation, now authomatically handled by optimising
static compilers and HotSpot runtimes.

Jan Burse

unread,
Nov 26, 2011, 6:05:27 AM11/26/11
to
Roedy Green schrieb:
> On Fri, 25 Nov 2011 18:39:23 +0100, Jan Burse<janb...@fastmail.fm>
> wrote, quoted or indirectly quoted someone who said :
>
>> Would be interested in a good memory stream
>> implementation. Requirement will be multiple
>> readers / writers and positioning.
>
> What do you use it for? What does it do for you?
> Is it just an ram-resident InputStream?

From my post 25.11.2011 21:23:

Jan Burse

unread,
Nov 26, 2011, 6:10:00 AM11/26/11
to
Arne Vajhøj schrieb:
> Use java.nio.ByteBuffer and handle concurrency in your code.

ByteBuffer does not extend automatically. If I do
put and reach the size of the byte buffer, I will
get an exception BufferOverflowException.

That is at least how I interpret the javadoc.

I am not looking for a bounded memory stream, it
should be unbounded, like (temporary) files are.
You just write and write and write, thats it.

Size control of the memory stream can be done
via close truncate for example.

The reference to the memory stream should be
still valid when an input stream or output stream
sitting on it does a close. With the same memory
stream reference I should be able to create new
input or output streams over it, that will work
on the previously written content.

Bye

Lew

unread,
Nov 26, 2011, 2:22:44 PM11/26/11
to
GREAT! Now we have to do homework just to understand your question! That's a *real* inducement to help you.

How about you answer the question here, hm?

What about these "memory streams" do you want to recapitulate? What is the purpose in your project?

--
Lew

Jan Burse

unread,
Nov 26, 2011, 2:44:37 PM11/26/11
to
Lew schrieb:
> GREAT! Now we have to do homework just to
> understand your question! That's a*real*
> inducement to help you.

My grandmother did understand it in a blink.

Bye

Lew

unread,
Nov 26, 2011, 8:15:28 PM11/26/11
to
You're just piling on the inducements, ain't'cha, Dale Carnegie?

--
Lew

Robert Klemme

unread,
Nov 27, 2011, 3:43:32 PM11/27/11
to
http://docs.oracle.com/javase/6/docs/api/java/io/PipedInputStream.html
http://docs.oracle.com/javase/6/docs/api/java/io/PipedOutputStream.html

Still, why do you want to do this? What problem are you trying to
solve? I'm not convinced that it is a good idea - especially when using
unbounded buffers.

Cheers

robert

Jan Burse

unread,
Nov 27, 2011, 6:13:00 PM11/27/11
to
Robert Klemme schrieb:
>
> http://docs.oracle.com/javase/6/docs/api/java/io/PipedInputStream.html
> http://docs.oracle.com/javase/6/docs/api/java/io/PipedOutputStream.html
>
> Still, why do you want to do this? What problem are you trying to
> solve? I'm not convinced that it is a good idea - especially when using
> unbounded buffers.

A pipe is not a memory stream. In a pipe the reader
does read where the writer does write. But I don't
have this requirement.

The only requirement I have, is that I can use
the memory streams in place of >>TEMPORARY FILES<<.
Whereas >>TEMPORARY FILES<< have a name and usually
exist on some media.

The memory streams should not exist on some media,
except in the memory. But they should share with
>>TEMPORARY FILES<< multiple readers / writers and
positioninng. Since I can use a >>TEMPORARY FILES<<
name and open a RandomAccessFile with it.

So basically in my application I would like to have
some class, call it MemoryStream. And It would like
to be able to instantiate it. And this instance serves
than as a kind of a file name.

So this instance I should be able to use where I normally
would use the >>TEMPORARY FILES<< name. The only places
that come to my mind are:
- creating an input stream from a file name, should
then work from a memory stream instance.
- creating an output stream from a file name, should
then work from a memory stream instance.
- creating a random acccess file from a file name,
should then work from a memory stream instance.

What the memory stream instance will have in common
with a >>TEMPORARY FILES<< name, is that it will go
away when the application closes. So they are very
similar to >>TEMPORARY FILES<< with the option delete
on close.

Also I would like to be able to use the same memory stream
to open input streams, output streams, random access files,
as many as I want. This is also provided by >>TEMPORARY
FiLES<<, once a >>TEMPORARY FILES<< name is established, it
can be used for all kinds of stuff.

Here is a little example:
A large clip board: This could be realized by a temporary
file. When you make the copy operation, you simply write
the copied content into the temporary file. When you make
the past operation you reopen the temporary file and
read the content that should be pasted from it.

The above could be realized by even passing the >>TEMPORARY
FILES<< name around in the system clipboard, so that the copy
paste works across applications. Provided that the applications
understand the content type.

But memory streams cannot passed around like this. Here is
the difference to >>TEMPORARY FILES<<. In their simple
implementation they are confined to the application that
created them.

Of course we could extend them and provide some RMI extension
of them, or what ever, so that the a memory stream references
becomes mobile. Or we could implement them in special shared
memory of the operating system, so that a direct shared access
from multiple applications is possible.

This mobility and shared access across multiple application,
which is a property of >>TEMPORARY FILES<<, is not required
for the memory streams I am looking for. So I guess if my
grandmother could program as she can knit pullovers for me,
she might do a memory stream class for me.

But unfortunately she knows only COBOL and I am looking for
a Java library. But she assured me that something like this
must exist on the web, and that I don't need to go into lengths
to code it by myself. So she told me:

It's not difficult, but why reinvent the wheel?

Bye

Robert Klemme

unread,
Nov 28, 2011, 2:14:39 AM11/28/11
to
On 11/28/2011 12:13 AM, Jan Burse wrote:
> Robert Klemme schrieb:
There is a lot of "could" and "should" in there but no description of a
real problem you are trying to solve. Basically if you want to use
other libraries which depend on java.io to open files you have little
chance to smuggle a "memory stream" in there without going through the
effort to modify byte code of classes. So your "memory stream" could be
made to work easily with your own code only. But then, you'd probably
be far better off using a data structure with specific type and not
sequences of bytes. If you need to store unstructured text, you can use
a StringBuffer. If you need to store other kinds of data you can use
any of the classes from java.util.

Kind regards

robert


PS: I hope you are aware that writing to files does not necessarily
create disk IO. For short lived temporary files chances are that you
delete the file before it got written to disk. And then there are in
memory file systems available which you can mount and use like any other
file system and which hold all the data in memory only.

Jan Burse

unread,
Nov 28, 2011, 3:10:09 AM11/28/11
to
Robert Klemme schrieb:
> Basically if you want to use other libraries which depend on java.io to
> open files you have little chance to smuggle a "memory stream" in there
> without going through the effort to modify byte code of classes.

No, I don't think that the above claim is right. Look see what
is written for the class InputStream:

This abstract class is the superclass of
all classes representing an input
stream of bytes.
http://docs.oracle.com/javase/1.4.2/docs/api/java/io/InputStream.html

So with proper OO I could implement:

class MemoryStream {

InputStream createInput();

}

class MemoryInput extens InputStream {
...
}

The factory method could return a MemoryInput instance
from a MemoryStream instance. Similarly I could
provide OutputStrem createOutput etc..

The MemoryStream would be not the first stream that
has been created outside of java.lang via proper OO.
Think for example of the request and respons streams
of a web server. They are also made like this.

Bye

Robert Klemme

unread,
Nov 28, 2011, 5:47:54 AM11/28/11
to
And how do you make library code use your memory stream factory
class? Remember, there will by typically a line like this somewhere

FileInputStream fileIn = new FileInputStream(fileName);

There is no factory. There is just an invocation of the constructor.

From what you write I get the impression that you are hooked into your
idea of using in memory piles of bytes as replacement for temporary
files because it looks like a good idea (and simple to do) on first
sight. But I haven't seen anything so far which would convince me
that it's really a good idea altogether.

Kind regards

robert

Jan Burse

unread,
Nov 28, 2011, 8:58:43 AM11/28/11
to
Robert Klemme schrieb:
> And how do you make library code use your memory stream factory
> class? Remember, there will by typically a line like this somewhere
>
> FileInputStream fileIn = new FileInputStream(fileName);

I will replace these constructors. The input streams /
output streams are factored via methods from the
memory stream instance, and not passed as an argument
to other constructors.

> From what you write I get the impression that you are hooked into your
> idea of using in memory piles of bytes as replacement for temporary
> files because it looks like a good idea (and simple to do) on first sight.

Well it might be a matter of taste what is a
good idea and what not.

In the present case I am expecting more performance,
since these memory streams will be used in high
frequency for small contents of about 10-50 bytes.

Maybe a part of Google Protocol Buffers could do:
http://code.google.com/intl/de-DE/apis/protocolbuffers/docs/reference/java/index.html

They have newInput() and newOutput(). Have to check.

Bye

Robert Klemme

unread,
Nov 28, 2011, 1:52:40 PM11/28/11
to
On 28.11.2011 14:58, Jan Burse wrote:
> Robert Klemme schrieb:
>> And how do you make library code use your memory stream factory
>> class? Remember, there will by typically a line like this somewhere
>>
>> FileInputStream fileIn = new FileInputStream(fileName);
>
> I will replace these constructors. The input streams /
> output streams are factored via methods from the
> memory stream instance, and not passed as an argument
> to other constructors.

And I said you will have to use bytecode manipulation for this in case
of other libraries (i.e. ones which you do not control or have access to
sources to).

>> From what you write I get the impression that you are hooked into your
>> idea of using in memory piles of bytes as replacement for temporary
>> files because it looks like a good idea (and simple to do) on first
>> sight.
>
> Well it might be a matter of taste what is a
> good idea and what not.

In a way, yes. I'd rather say it's a matter of goals which one wants to
achieve. There are means which work well for some ends and not so well
for others. If the goal is clear solutions can often be pretty easy
aligned according to "good" and "not good".

> In the present case I am expecting more performance,
> since these memory streams will be used in high
> frequency for small contents of about 10-50 bytes.

*What* content? Why do you not answer this question? You didn't even
state that you do not want to answer the question (which I could
understand if you do not want to leak out ideas you are working on).
But coming here and asking while at the same time ignoring those who are
willing to help to find a good solution is not a strategy to success.

Regards

Jan Burse

unread,
Nov 28, 2011, 6:22:48 PM11/28/11
to
Robert Klemme schrieb:
> *What* content? Why do you not answer this question? You didn't even
> state that you do not want to answer the question (which I could
> understand if you do not want to leak out ideas you are working on). But
> coming here and asking while at the same time ignoring those who are
> willing to help to find a good solution is not a strategy to success.

Strange approach to make help dependent on disclosure
of application details. Please note this is a public
forum, so when somebody posts a solution here everybody
profits from this, not only eventually me.

Bye

Jan Burse

unread,
Nov 28, 2011, 6:39:44 PM11/28/11
to
Jan Burse schrieb:
Last but not least, the solution poster might also
profit by getting some feedback etc..

Bye

Gene Wirchenko

unread,
Nov 28, 2011, 8:03:37 PM11/28/11
to
On Tue, 29 Nov 2011 00:22:48 +0100, Jan Burse <janb...@fastmail.fm>
wrote:

>Robert Klemme schrieb:
>> *What* content? Why do you not answer this question? You didn't even
>> state that you do not want to answer the question (which I could
>> understand if you do not want to leak out ideas you are working on). But
>> coming here and asking while at the same time ignoring those who are
>> willing to help to find a good solution is not a strategy to success.
>
>Strange approach to make help dependent on disclosure
>of application details. Please note this is a public

Leaving out critical details can lead to suggestions that are not
so good, or even bad.

>forum, so when somebody posts a solution here everybody
>profits from this, not only eventually me.

That is the idea, yes.

Sincerely,

Gene Wirchenko

Mayeul

unread,
Nov 29, 2011, 6:25:20 AM11/29/11
to
On 28/11/2011 19:52, Robert Klemme wrote:
> *What* content? Why do you not answer this question? You didn't even
> state that you do not want to answer the question (which I could
> understand if you do not want to leak out ideas you are working on). But
> coming here and asking while at the same time ignoring those who are
> willing to help to find a good solution is not a strategy to success.

Long story short, *what* in the world needs some kind of in-memory file
virtualization, that could not be done better with the ultra-simple
concept of ArrayList? Why is there such a need for bytes, what are bytes
any use of?

In the Java World it is often the case that you do not want to work with
bytes, you merely write and read them on the boundaries of your program
to the external world. You'd usually work with structured data, and
therefore not keep bytes lying in-memory.
Still, it rarely makes anything more complex than it needs to be.
Everytime one feels it does, one should just question one's approach to
the task.
I spoke of ArrayList just as an example. Whatever the case is, someone
can probably point out a satisfactory way to address it. It might have
nothing to do with file virtualization, though.

--
Mayeul

Jan Burse

unread,
Nov 29, 2011, 8:37:48 AM11/29/11
to
Mayeul schrieb:
> I spoke of ArrayList just as an example. Whatever the case is, someone
> can probably point out a satisfactory way to address it. It might have
> nothing to do with file virtualization, though.

Well idea is not to change the original application which uses
already some byte serialization in a fundamental way. But only
to replace the temporary files by memory streams.

Of course one could opt for a total rewrite of the original
application, but then there is no guarantee that a non-byte
serialization gives a better design.

Also we cannot deduce from the requirement of an byte input /
output stream that the application deals with binary data. It
could be also that on top of the byte input / output stream
character reader / writer are created.

So that basically the application deals with character streams,
and maybe with something like documents or some such. It might
be that ArrayLists come into play when working with documents,
but only peripherically, for example JTextPane & Co.. there is
an array of elements involved which represent paragraphs.

But when you strip all the styles from JTextPane & Co you
get to the characters of the paragraphs. And XML / HTML shows
you for example how you can serialize characters and styles
into character streams.

Now again one could opt for working with a DOM instead of working
with streams. But in XML both options are viable and not alien
to Java. Sometimes stream based approaches are even more efficient
than DOM based approaches.

So be it as it is, the requiremets stands at it is, there is
the request for memory streams, despite possible alternative
designs. Just assume that among the variant designs, we currently
focus on memory streams and not something else.

Bye

Lew

unread,
Nov 29, 2011, 10:37:06 AM11/29/11
to
Jan Burse wrote:
> So be it as it is, the requiremets stands at it is, there is
> the request for memory streams, despite possible alternative
> designs. Just assume that among the variant designs, we currently
> focus on memory streams and not something else.

Doesn't seem like this group has what you're looking for. The thread's gone on for a while with no movement on your rather narrow and close-to-the-vest requirements. I suggest expanding your search beyond Usenet.

GIYF. And it's faster, too.

--
Lew

Martin Gregorie

unread,
Nov 29, 2011, 3:26:19 PM11/29/11
to
On Tue, 29 Nov 2011 14:37:48 +0100, Jan Burse wrote:

> Mayeul schrieb:
>> I spoke of ArrayList just as an example. Whatever the case is, someone
>> can probably point out a satisfactory way to address it. It might have
>> nothing to do with file virtualization, though.
>
> Well idea is not to change the original application which uses already
> some byte serialization in a fundamental way. But only to replace the
> temporary files by memory streams.
>
As you say the individual file sizes are a few tens of bytes, have you
looked at using ByteInputStream and ByteOutputStream?

If you have considered them, why are they unsuitable?



--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |

Jan Burse

unread,
Nov 29, 2011, 3:34:54 PM11/29/11
to
Martin Gregorie schrieb:
> As you say the individual file sizes are a few tens of bytes, have you
> looked at using ByteInputStream and ByteOutputStream?
>
> If you have considered them, why are they unsuitable?

They could play a part in the game. But two problems are
unsolved: sharing and positioning.

The memory stream should work like a >>TEMPORARY FILE<<,
than can be opened, and thus shared by multiple readers
and writes. And the readers and writers should be able
to perform a seek() in any direction if desired.

Bye

Martin Gregorie

unread,
Nov 29, 2011, 4:05:51 PM11/29/11
to
On Tue, 29 Nov 2011 21:34:54 +0100, Jan Burse wrote:

> Martin Gregorie schrieb:
>> As you say the individual file sizes are a few tens of bytes, have you
>> looked at using ByteInputStream and ByteOutputStream?
>>
>> If you have considered them, why are they unsuitable?
>
> They could play a part in the game. But two problems are unsolved:
> sharing and positioning.
>
My bad: I meant ByteArrayInputStream and ByteArrayOutputStream

The names are a little confusing: ByteArrayInputStream reads *from* a
byte array. The ByteArrayOutputStream writes *into* a byte array.

I've used them to move byte arrays to and from CLOB columns in databases,
where they were straight-forward to use with CLOB sizes ranging from a
few bytes to a few megabytes, though I never had more than one or two
instances in use at a time.

Jan Burse

unread,
Nov 29, 2011, 4:59:06 PM11/29/11
to
Martin Gregorie schrieb:
I don't see a constructor where I could hand over a
buffer content from the memory stream. So that multiple
reader and/or writers can share the same buffer content.

Remember requirement is something along:

class MemoryStream {

InputStream createInput();
OutputStream createOutput();
...

}

class MemoryInput extens InputStream {
...
}

class MemoryOutput extens OutputStream {
...
}

markspace

unread,
Nov 29, 2011, 5:06:30 PM11/29/11
to
On 11/29/2011 1:59 PM, Jan Burse wrote:

> I don't see a constructor where I could hand over a
> buffer content from the memory stream. So that multiple
> reader and/or writers can share the sa


OK, I'll throw myself on this grenade.

<http://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayInputStream.html#ByteArrayInputStream%28byte[]%29>

Martin Gregorie

unread,
Nov 29, 2011, 8:08:21 PM11/29/11
to
On Tue, 29 Nov 2011 22:59:06 +0100, Jan Burse wrote:

> Martin Gregorie schrieb:
>> On Tue, 29 Nov 2011 21:34:54 +0100, Jan Burse wrote:
>>
>>> Martin Gregorie schrieb:
>>>> As you say the individual file sizes are a few tens of bytes, have
>>>> you looked at using ByteInputStream and ByteOutputStream?
>>>>
>>>> If you have considered them, why are they unsuitable?
>>>
>>> They could play a part in the game. But two problems are unsolved:
>>> sharing and positioning.
>>>
>> My bad: I meant ByteArrayInputStream and ByteArrayOutputStream
>>
>> The names are a little confusing: ByteArrayInputStream reads *from* a
>> byte array. The ByteArrayOutputStream writes *into* a byte array.
>>
>> I've used them to move byte arrays to and from CLOB columns in
>> databases,
>> where they were straight-forward to use with CLOB sizes ranging from a
>> few bytes to a few megabytes, though I never had more than one or two
>> instances in use at a time.
>>
>>
> I don't see a constructor where I could hand over a buffer content from
> the memory stream. So that multiple reader and/or writers can share the
> same buffer content.
>
The idea is that you write to the ByteArrayOutputStream, close it, grab
the byte array using toByteArray() and store that until it is needed.
Then you construct a ByteArrayInputStream with the byte array as the
constructor argument. This does, of course, assume that you never want to
write and read an instance simultaneously.

I'd think implementing non-simultaneous access should be easy enough to
bury inside a <MemoryStream> class.

Beyond that I'm guessing since you've said little or nothing about
whether you need simultaneous read and write access to a <MemoryStream>
instance, whether an instance gets read more than once and how many
instances can exist at the same time.

John B. Matthews

unread,
Nov 29, 2011, 9:23:05 PM11/29/11
to
This related example uses an in-memory byte stream to transport
character encoded data:

<http://groups.google.com/group/comp.lang.java.programmer/msg/4f6eab3230444531>

--
John B. Matthews
trashgod at gmail dot com
<http://sites.google.com/site/drjohnbmatthews>

Jan Burse

unread,
Nov 30, 2011, 8:03:45 AM11/30/11
to
Martin Gregorie schrieb:
> Beyond that I'm guessing since you've said little or nothing about
> whether you need simultaneous read and write access to a<MemoryStream>
> instance, whether an instance gets read more than once and how many
> instances can exist at the same time.

I only said multiple readers and writers. But idea
is concurrently and simultaneous. Idea is not
sequentially one after the other.

What will transpire between these multiple readers
and writers will be the size of the memory stream,
and eventually a delete status. And of course the
content.

No special limit on the number of memory stream instances,
and no special limit on the number of input / output /
random access objects created from one memory stream.

Bye

Jan Burse

unread,
Jan 27, 2012, 6:11:13 PM1/27/12
to
Roedy Green schrieb:
> On Fri, 25 Nov 2011 18:39:23 +0100, Jan Burse<janb...@fastmail.fm>
> wrote, quoted or indirectly quoted someone who said :
>
>> Would be interested in a good memory stream
>> implementation. Requirement will be multiple
>> readers / writers and positioning.
>
> What do you use it for? What does it do for you?
> Is it just an ram-resident InputStream?

Dear All,

I found an interesting implementation. On linux
level there is ashmem for Android. And then on Java
level there is an implementation of a class called
MemoryFile. The memory file acts as a factory for
input/output streams:


http://grepcode.com/file/repository.grepcode.com/java/ext/com.google.android/android/2.2_r1.1/android/os/MemoryFile.java#MemoryFile

public InputStream getInputStream() {
return new MemoryInputStream();
}

public OutputStream getOutputStream() {
return new MemoryOutputStream();
}

The above two streams account for sequential access,
and via mark/release some pseudo random access.
But there is also an api for random access. Namely
the following two beasts:

public int readBytes(byte[] buffer, int srcOffset,
int destOffset, int count);

public void writeBytes(byte[] buffer, int srcOffset,
int destOffset, int count)

Compare to a normal read/write there is an additional
parameter for the offset inside the memory file. But
the implementation is not really light weight. Namely
the memory file can shared accross processes, which
I do not need:

http://elinux.org/Android_Kernel_Features#ashmem

But at least it gives an idea how to bootstrap a
memory file implementation. Namely the input/output
streams are implemented based on the read/write with
the extra argument. Makes for examples mark/release
quite simple....

Cool!

Bye


Arne Vajhøj

unread,
Feb 6, 2012, 8:59:08 PM2/6/12
to
On 11/26/2011 6:10 AM, Jan Burse wrote:
> Arne Vajhøj schrieb:
>> Use java.nio.ByteBuffer and handle concurrency in your code.
>
> ByteBuffer does not extend automatically. If I do
> put and reach the size of the byte buffer, I will
> get an exception BufferOverflowException.
>
> That is at least how I interpret the javadoc.
>
> I am not looking for a bounded memory stream, it
> should be unbounded, like (temporary) files are.
> You just write and write and write, thats it.
>
> Size control of the memory stream can be done
> via close truncate for example.
>
> The reference to the memory stream should be
> still valid when an input stream or output stream
> sitting on it does a close. With the same memory
> stream reference I should be able to create new
> input or output streams over it, that will work
> on the previously written content.

Given that memory/available address space is
bounded then what you can store in memory will
be bounded no matter what.

You will just need to code to handle it.

Arne



Arne Vajhøj

unread,
Feb 6, 2012, 9:06:55 PM2/6/12
to
It happens all the time that people ask for help
with implementation of solution XYZ while there is another
solution ABC that is a much better solution
for the problem to be solved.

Picking the right solution is the difficult part and
implementing it is the easy part (if it is a good
solution).

The interest in helping implement XYZ when poster
will not reveal what problem is being solved is not
particular attractive, because the chance that XYZ
is optimal is very small.

Arne


Jan Burse

unread,
Feb 7, 2012, 3:59:33 AM2/7/12
to
Arne Vajhøj schrieb:
Well I can assure you, the memory
streams are not used to wipe your ass.

If you post more of this narrow minded
stuff, I might get injured from laughing...

Bye

Daniel Pitts

unread,
Feb 7, 2012, 12:59:03 PM2/7/12
to
Only a fool calls his teacher foolish. There is nothing narrow minding
about Arne's post. There is however close-mindedness in your own reply.

Engineers look over the most solutions, and choose the ones that work
"best" (where "best" is situational). Often times, not having to
implement potentially tricky algorithms is a good start. As well as not
having to refit existing API's to support a different mode of operation.

If, as an engineer, you decide XYZ is the right solution, but need help
finding how to implement XYZ, it is often a good idea to explain how and
why you came to your conclusion. Either you'll get second opinions on
other approaches you might not have considered, or you'll get agreement
and instantaneous help on XYZ.

Arne Vajhøj

unread,
Feb 7, 2012, 9:02:09 PM2/7/12
to
> Well I can assure you, the memory
> streams are not used to wipe your ass.
>
> If you post more of this narrow minded
> stuff, I might get injured from laughing...

I think it requires an unusual understanding of "narrow"
to consider looking at other possible solutions to be
narrow minded.

Arne

Jan Burse

unread,
Feb 8, 2012, 3:55:23 AM2/8/12
to
Arne Vajhøj schrieb:
> I think it requires an unusual understanding of "narrow"
> to consider looking at other possible solutions to be
> narrow minded.
>
> Arne
>

Well with your attitude you would not be
fit to implement against a specfication.

Which means you would not be fit to work in
a large team where contract based
components are built.

Because of your relentless stupid asking
you would not be able to write one single
code of line.

My specs were and are pretty clear...

John B. Matthews

unread,
Feb 8, 2012, 12:18:09 PM2/8/12
to
In article <jgtd9s$lta$1...@news.albasani.net>,
Jan Burse <janb...@fastmail.fm> wrote:

> Well with your attitude you would not be
> fit to implement against a specfication.

Some specifications are better than others:

<http://geekfun.pl/pm_build_swing.gif>

Thank you for reminding me how much I value the critical feedback of my
colleagues and my manager's diligence in fostering peer review. I shall
bring baked goods to my next meeting.

Jan Burse

unread,
Feb 8, 2012, 2:23:17 PM2/8/12
to
John B. Matthews schrieb:
> In article<jgtd9s$lta$1...@news.albasani.net>,
> Jan Burse<janb...@fastmail.fm> wrote:
>
>> Well with your attitude you would not be
>> fit to implement against a specfication.
>
> Some specifications are better than others:
>
> <http://geekfun.pl/pm_build_swing.gif>
>
> Thank you for reminding me how much I value the critical feedback of my
> colleagues and my manager's diligence in fostering peer review. I shall
> bring baked goods to my next meeting.
>

The cookies will not make the people who should
understand or feedback on the spec more intelligent.
See what happended in this thread. I wrote:

On 11/26/2011 6:10 AM, Jan Burse wrote:
> I am not looking for a bounded memory stream, it
> should be unbounded, like (temporary) files are.
> You just write and write and write, thats it.
>
> Size control of the memory stream can be done
> via close truncate for example.

Then this clever guy who will never be able to code
or suggest solutions wrote:

On 02/07/2012 2:59 AM, Arne Vajhøj:
> Given that memory/available address space is
> bounded then what you can store in memory will
> be bounded no matter what.
>
> You will just need to code to handle it.

Was this a reminder from theoretical computer
science that we deal with finite machines, or
what? Thats new to me. (no)

Looks smart, but is just this relentless stupid
asking and commenting that would allow to expell
any person in some peer reviewer postion from the
team for fooling around and not working.

Bye

Gene Wirchenko

unread,
Feb 8, 2012, 4:18:04 PM2/8/12
to
On Wed, 08 Feb 2012 12:18:09 -0500, "John B. Matthews"
<nos...@nospam.invalid> wrote:

>In article <jgtd9s$lta$1...@news.albasani.net>,
> Jan Burse <janb...@fastmail.fm> wrote:
>
>> Well with your attitude you would not be
>> fit to implement against a specfication.
>
>Some specifications are better than others:
>
><http://geekfun.pl/pm_build_swing.gif>
>
>Thank you for reminding me how much I value the critical feedback of my
>colleagues and my manager's diligence in fostering peer review. I shall
>bring baked goods to my next meeting.

Bagels with cream cheese are good, too. Have them cut into
quarters so they are easy to handle and less likely to make a mess.

I, too, appreciate good peer review.

Sincerely,

Gene Wirchenko

Gene Wirchenko

unread,
Feb 8, 2012, 4:18:54 PM2/8/12
to
On Wed, 08 Feb 2012 20:23:17 +0100, Jan Burse <janb...@fastmail.fm>
wrote:

[snip]

>Looks smart, but is just this relentless stupid
>asking and commenting that would allow to expell
>any person in some peer reviewer postion from the
>team for fooling around and not working.

What team? Ask a question, and you go antago.

Sincerely,

Gene Wirchenko
0 new messages