sendfile() in Go
Roger Pau Monné
This is going to be short, Is there anything similar to sendfile
system call in Go?
http://linux.die.net/man/2/sendfile
Thanks, Roger.
Rob 'Commander' Pike
It can be written in a few lines of efficient user code.
The output must be a socket. Why not a general file descriptor? Is
this yet another example of ''all files are equal but networks are
more equal than others?" And then I see the input must be mmap-able.
What a monster!
The system call provides an extra type you can use to send (and
receive) other pieces of unspecified data. What's wrong with writev?
On BSD it's even got an unused parameter reserved for future expansion
that must be zero.
There appears to be an argument that it can be more efficient because
the data crosses the user/kernel boundary only once. That's just a
failure of the kernel's memory management. This is a clumsy way to fix
it.
Wow.
-rob
ron minnich
> That is one bizarre system call. I've never seen anything like it.
Would it surprise you to find that the inspiration (is that the right
word?) for this came from Windows?
My recollection is that the main reason it went in, years back, was to
make web benchmarks go faster.
ron
Norman Yarvin
>That is one bizarre system call. I've never seen anything like it.
To quote an old linux-kernel message about sendfile(),
"It was _very_ easy to implement, and can be considered a
5-minute hack to give a feature that fit very well in the MM
architecture, and that the Apache folks had already been using on
other architectures." -- Linus Torvalds
David Roundy
> There appears to be an argument that it can be more efficient because
> the data crosses the user/kernel boundary only once. That's just a
> failure of the kernel's memory management. This is a clumsy way to fix
> it.
I thought that the data crossed the user/kernel boundary zero times
with sendfile...
--
David Roundy
Roger Pau Monné
Thanks for the info, now I was wondering if I could be able to perform
concurrent io.Copy (or io.Copyn) with the same source file, or the
offset will go crazy? If so, what's the best way to do this (perform
concurrent io.Copyn at different offsets from the same source file)?
Thanks, Roger.
2010/9/5 David Roundy <rou...@physics.oregonstate.edu>:
roger peppe
> Hello again,
>
> Thanks for the info, now I was wondering if I could be able to perform
> concurrent io.Copy (or io.Copyn) with the same source file, or the
> offset will go crazy? If so, what's the best way to do this (perform
> concurrent io.Copyn at different offsets from the same source file)?
>
> Thanks, Roger.
you've got two options here: open the file multiple times,
or use ReadAt instead of Read, which would mean
copying and mutating io.Copy to use explicit offsets.
e.g.
type ReadAter interface {
ReadAt(b []byte, off int64) (n int, err os.Error)
}
func CopyThreadSafe(dst os.Writer, r ReadAter) {
...
}
in the above, it still uses os.Writer, so you can't
be writing to the same file concurrently, but i'd guess
that's what you'd want. thus CopyThreadSafe is
not a great name for it, but i hope you get the idea.
Roger Pau Monné
I will have to do the same inverse operation (reading from a single
thread net.Conn and writing to a concurrent writer), but I think I get
the idea.
Thanks!
2010/9/8 roger peppe <rogp...@gmail.com>:
Andrew Gerrand
which will use the File's ReadAt method and should behave nicely when
read from concurrently.
Andrew
Roger Pau Monné
io.SectionWriter will also be good, but I think this is much more
complicated.
2010/9/8 Andrew Gerrand <a...@golang.org>:
Andrew Gerrand
> So os.ReadAt (that uses syscall.Pread) is thread safe?
I believe so.
> I will have to do the same inverse operation (reading from a single
> thread net.Conn and writing to a concurrent writer), but I think I get
> the idea.
If you're writing a single stream to multiple files you might want to
check out io.MultiWriter:
http://golang.org/pkg/io/#Writer.MultiWriter
Andrew
Roger Pau Monné
> On 8 September 2010 20:16, Roger Pau Monné <roy...@gmail.com> wrote:
>> So os.ReadAt (that uses syscall.Pread) is thread safe?
>
> I believe so.
>
>> I will have to do the same inverse operation (reading from a single
>> thread net.Conn and writing to a concurrent writer), but I think I get
>> the idea.
>
> If you're writing a single stream to multiple files you might want to
> check out io.MultiWriter:
> http://golang.org/pkg/io/#Writer.MultiWriter
>
Nope, I'm reading from various net.Conn and writing to several
sections (offsets) of the same file at the same time. (well, now I'm
not doing it at the same time, but I would like to do it).
Andrew Gerrand
> 2010/9/8 Andrew Gerrand <a...@golang.org>:
>> On 8 September 2010 20:16, Roger Pau Monné <roy...@gmail.com> wrote:
>>> So os.ReadAt (that uses syscall.Pread) is thread safe?
>>
>> I believe so.
>>
>>> I will have to do the same inverse operation (reading from a single
>>> thread net.Conn and writing to a concurrent writer), but I think I get
>>> the idea.
>>
>> If you're writing a single stream to multiple files you might want to
>> check out io.MultiWriter:
>> http://golang.org/pkg/io/#Writer.MultiWriter
>>
>
> Nope, I'm reading from various net.Conn and writing to several
> sections (offsets) of the same file at the same time. (well, now I'm
> not doing it at the same time, but I would like to do it).
Oh, in that case use WriteAt. If you want to write a SectionWriter,
I'll review the CL. :-)
http://golang.org/doc/contribute.html
Andrew
Roger Pau Monné
I'm sorry to bother the list with this, I've spent 2 days trying to
figure this out, but I cannot understand why it doesn't work, so let's
see if someone knows what's happening (I'm sure it's just a stupid
error, but I haven't been able to see it).
I'm working on a bittorrent client in Go, and I have to send "piece"
messages, which consists of a small header (with message length and
piece index) and then a 2^14 payload that contains the data. I would
like to send the piece data using io.Copy, but I don't know why it
doesn't work.
Here's a code that works (without using io.Copy), packing the whole
message before sending (I'm sorry if it's a little difficult to
understand):
msg_byte = make([]byte, msg.length + 4) // Allocate memory for the whole message
binary.BigEndian.PutUint32(msg_byte[0:4], msg.length) // Copy message
length at the start
buffer := bytes.NewBuffer(msg_byte[0:4])
buffer.WriteByte(msg.msgId) // Write msg ID ("piece" -> 7)
buffer.Write(position) // Information about the piece
payLoad := make([]byte, msg.length - 9)
n, err := reader.Read(payLoad)
if err != nil {
log.Stderr("Wire -> Reading piece:", err, "read:", n)
return
}
buffer.Write(payLoad) // Append piece data to the end
log.Stderr("Wire -> Sending:", buffer.Bytes())
n, err = wire.conn.Write(buffer.Bytes()) // Send the peice
log.Stderr("Wire -> Sent:", n, "bytes")
if err != nil {
return
}
And If I use io.Copy or if I send the message in two parts it doesn't
work (the client on the other end closes the connection):
msg_byte = make([]byte, 13) // Allocate memory for the header only
binary.BigEndian.PutUint32(msg_byte[0:4], msg.length) // Write msg length
buffer := bytes.NewBuffer(msg_byte[0:4])
buffer.WriteByte(msg.msgId) // Write ID
buffer.Write(position) // Write info about the piece being send
log.Stderr("Wire -> Sending header:", buffer.Bytes())
n, err = wire.conn.Write(buffer.Bytes()) // Send the header
log.Stderr("Wire -> Sent:", n, "bytes")
if err != nil {
return
}
if msg.msgId == piece {
payLoad := make([]byte, msg.length - 9) // Allocate memory for the piece data
n, err := reader.Read(payLoad) // Read
if err != nil {
log.Stderr("Wire -> Reading piece:", err, "read:", n)
}
log.Stderr("Wire -> Sending piece")
n, err = wire.conn.Write(payLoad) // Write
if err != nil {
return
}
log.Stderr("Wire -> Sent:", n, "bytes")
}
reader is an io.SectionReader in both cases, with the exact length. I
don't use io.Copy in this example, but you can replace the contents of
the last "if" with "io.Copy(wire.conn, reader)" and it won't work
also. The number of bytes send is the same in both examples.
Could someone throw some light into this?
Thanks, Roger.
Roger Pau Monné
I've done some more research, and I get to the conclusion that I
cannot write the message using more than one call to Write(), I have
tried using bufio.Writer, but still the same problem, if I try to send
the message with two calls to Write, it fails, but if I do it with
only one call it works (either directly to the net.Conn or using
bufio.Writer). Any ideas?
Thanks, Roger.
2010/9/10 Roger Pau Monné <roy...@gmail.com>:
Ian Lance Taylor
> if msg.msgId == piece {
This is a difference between your two code fragments--the first one
didn't seem to have a similar test.
> payLoad := make([]byte, msg.length - 9) // Allocate memory for the piece data
> n, err := reader.Read(payLoad) // Read
How long does this take? Is it possible that the other end is timing
out before this is read and then written?
Unless the other end is using direct calls into the networking layer, I
don't see how it could even detect that you are using multiple Write
calls.
Ian