sending EOF to a pipe reader?

[Mathieu Lonjaret]

Hello,

say I have an io.Pipe, and on the reading end, I'm reading with
ReadFrom into a bytes.Buffer because I don't know how much the writer
is going to send, kindof like this:

pr, pw := io.Pipe()
foo := "helloworld"
b := bytes.NewBuffer(make([]byte, 0))
go func() {
n, err := pw.Write([]byte(foo))
if err != nil {
print(err)
}
}()
b.ReadFrom(pr)

now this won't do, because ReadFrom is blocking, waiting for an EOF,
and I don't want that.
Is there any way for the writer to actually send an EOF, so I can
unblock the reader? or another way for that matter?
I don't want to close the pipe either because I'll need it again afterwards.

Cheers,
Mathieu

[Nigel Tao]

If this is for LZW, then you don't want to read the whole thing into a
bytes.Buffer, or else you'll have to allocate 1GB of memory to
compress a 1GB file. Just allocate a fixed size []byte and loop
calling pr.Read as many times as you have to.

[Mathieu Lonjaret]

Alright will do, thanks.
But out of curiosity, is it even possible to "unblock" the reader in
that situation without closing the pipe?

[roger peppe]

the whole point of EOF is that it is *end* of file - it can't happen more
than once. if you want to delimit a stream, you'll have to use
some other kind of signal - for instance by chunking the stream
into blocks and layering another Reader on top of that.

[Noah Evans]

I've run into your problem, usually I prefix the amount I'm sending
with a length(sent via a gob) and do an io.Copyn on the pipe for that
amount. Does that work for you?

Noah

[Mathieu Lonjaret]

yeah, I was doing something similar at first; the writer would first
send through a chan shared with the reader the length about to be
written in the pipe, so that the reader would know how to size its
receiving []byte. But as Nigel pointed out, in my case it's not the
way to go as this could result in a pretty big []byte if it's all read
in one go.
I'll remember your technique for another time though, thx

Cheers,
Mathieu

[Noah Evans]

In that case don't you really want to implement your datatype as a
type with methods satisfying io.Writer?

Noah

[Mathieu Lonjaret]

Yes, that's what I'm doing.

[Noah Evans]

In that case, the gob + Copyn would solve your problem, because
whatever buffering and whatnot would be hidden in your Writer right?
Or am I missing something really obvious?

Noah

On Wed, Dec 8, 2010 at 3:17 PM, Mathieu Lonjaret

[Mathieu Lonjaret]

To put the example I started with in context, the reading end of the
pipe would be hidden in the Writer, while the writing end is used by
the caller of the Write.
So to answer your question, I don't really need the gob + Copyn
because the part where I was having trouble is precisely getting the
"buffering and whatnot hidden in the Writer" done correctly.
But I think I'm good now, unless I'm the one now actually missing your point.
If you're interested I'll let you know with the details.

Cheers,
Mathieu

[Noah Evans]

Sure, send it offline.

Noah

On Wed, Dec 8, 2010 at 5:12 PM, Mathieu Lonjaret