This is going to be short, Is there anything similar to sendfile
system call in Go?

http://linux.die.net/man/2/sendfile

Thanks, Roger.

It can be written in a few lines of efficient user code.

The output must be a socket. Why not a general file descriptor?  Is
this yet another example of ''all files are equal but networks are
more equal than others?"  And then I see the input must be mmap-able.
What a monster!

The system call provides an extra type you can use to send (and
receive) other pieces of unspecified data.  What's wrong with writev?

On BSD it's even got an unused parameter reserved for future expansion
that must be zero.

There appears to be an argument that it can be more efficient because
the data crosses the user/kernel boundary only once. That's just a
failure of the kernel's memory management. This is a clumsy way to fix
it.

Wow.

-rob

On Sat, Sep 4, 2010 at 3:48 PM, Rob 'Commander' Pike <r...@google.com> wrote:
> That is one bizarre system call. I've never seen anything like it.

Would it surprise you to find that the inspiration (is that the right
word?) for this came from Windows?

My recollection is that the main reason it went in, years back, was to
make web benchmarks go faster.

ron

On Sun, Sep 05, 2010 at 08:48:35AM +1000, Rob 'Commander' Pike wrote:
>That is one bizarre system call. I've never seen anything like it.

To quote an old linux-kernel message about sendfile(),

        "It was _very_ easy to implement, and can be considered a
        5-minute hack to give a feature that fit very well in the MM
        architecture, and that the Apache folks had already been using on
        other architectures." -- Linus Torvalds

On Sat, Sep 4, 2010 at 6:48 PM, Rob 'Commander' Pike <r...@google.com> wrote:
> There appears to be an argument that it can be more efficient because
> the data crosses the user/kernel boundary only once. That's just a
> failure of the kernel's memory management. This is a clumsy way to fix
> it.

I thought that the data crossed the user/kernel boundary zero times
with sendfile...
--
David Roundy

Thanks for the info, now I was wondering if I could be able to perform
concurrent io.Copy (or io.Copyn) with the same source file, or the
offset will go crazy? If so, what's the best way to do this (perform
concurrent io.Copyn at different offsets from the same source file)?

Thanks, Roger.

2010/9/5 David Roundy <rou...@physics.oregonstate.edu>:

On 8 September 2010 10:37, Roger Pau Monné <roy...@gmail.com> wrote:
> Hello again,
>
> Thanks for the info, now I was wondering if I could be able to perform
> concurrent io.Copy (or io.Copyn) with the same source file, or the
> offset will go crazy? If so, what's the best way to do this (perform
> concurrent io.Copyn at different offsets from the same source file)?
>
> Thanks, Roger.

you've got two options here: open the file multiple times,
or use ReadAt instead of Read, which would mean
copying and mutating io.Copy to use explicit offsets.

e.g.

type ReadAter interface {
    ReadAt(b []byte, off int64) (n int, err os.Error)
}

func CopyThreadSafe(dst os.Writer, r ReadAter) {
    ...
}

in the above, it still uses os.Writer, so you can't
be writing to the same file concurrently, but i'd guess
that's what you'd want. thus CopyThreadSafe is
not a great name for it, but i hope you get the idea.

>
> 2010/9/5 David Roundy <rou...@physics.oregonstate.edu>:
>> On Sat, Sep 4, 2010 at 6:48 PM, Rob 'Commander' Pike <r...@google.com> wrote:
>>> There appears to be an argument that it can be more efficient because
>>> the data crosses the user/kernel boundary only once. That's just a
>>> failure of the kernel's memory management. This is a clumsy way to fix
>>> it.
>>
>> I thought that the data crossed the user/kernel boundary zero times
>> with sendfile...
>> --
>> David Roundy
>>
>

I will have to do the same inverse operation (reading from a single
thread net.Conn and writing to a concurrent writer), but I think I get
the idea.

Thanks!

2010/9/8 roger peppe <rogp...@gmail.com>:

Andrew

2010/9/8 Andrew Gerrand <a...@golang.org>:

On 8 September 2010 20:16, Roger Pau Monné <roy...@gmail.com> wrote:
> So os.ReadAt (that uses syscall.Pread) is thread safe?

I believe so.

> I will have to do the same inverse operation (reading from a single
> thread net.Conn and writing to a concurrent writer), but I think I get
> the idea.

If you're writing a single stream to multiple files you might want to
check out io.MultiWriter:
  http://golang.org/pkg/io/#Writer.MultiWriter

Andrew

> On 8 September 2010 20:16, Roger Pau Monné <roy...@gmail.com> wrote:
>> So os.ReadAt (that uses syscall.Pread) is thread safe?
>
> I believe so.
>
>> I will have to do the same inverse operation (reading from a single
>> thread net.Conn and writing to a concurrent writer), but I think I get
>> the idea.
>
> If you're writing a single stream to multiple files you might want to
> check out io.MultiWriter:
>  http://golang.org/pkg/io/#Writer.MultiWriter
>

Nope, I'm reading from various net.Conn and writing to several
sections (offsets) of the same file at the same time. (well, now I'm
not doing it at the same time, but I would like to do it).

On 8 September 2010 20:29, Roger Pau Monné <roy...@gmail.com> wrote:
> 2010/9/8 Andrew Gerrand <a...@golang.org>:
>> On 8 September 2010 20:16, Roger Pau Monné <roy...@gmail.com> wrote:
>>> So os.ReadAt (that uses syscall.Pread) is thread safe?
>>
>> I believe so.
>>
>>> I will have to do the same inverse operation (reading from a single
>>> thread net.Conn and writing to a concurrent writer), but I think I get
>>> the idea.
>>
>> If you're writing a single stream to multiple files you might want to
>> check out io.MultiWriter:
>>  http://golang.org/pkg/io/#Writer.MultiWriter
>>
>
> Nope, I'm reading from various net.Conn and writing to several
> sections (offsets) of the same file at the same time. (well, now I'm
> not doing it at the same time, but I would like to do it).

Oh, in that case use WriteAt. If you want to write a SectionWriter,
I'll review the CL. :-)
  http://golang.org/doc/contribute.html

Andrew

I'm sorry to bother the list with this, I've spent 2 days trying to
figure this out, but I cannot understand why it doesn't work, so let's
see if someone knows what's happening (I'm sure it's just a stupid
error, but I haven't been able to see it).

I'm working on a bittorrent client in Go, and I have to send "piece"
messages, which consists of a small header (with message length and
piece index) and then a 2^14 payload that contains the data. I would
like to send the piece data using io.Copy, but I don't know why it
doesn't work.

Here's a code that works (without using io.Copy), packing the whole
message before sending (I'm sorry if it's a little difficult to
understand):

msg_byte = make([]byte, msg.length + 4) // Allocate memory for the whole message
binary.BigEndian.PutUint32(msg_byte[0:4], msg.length) // Copy message
length at the start
buffer := bytes.NewBuffer(msg_byte[0:4])
buffer.WriteByte(msg.msgId) // Write msg ID ("piece" -> 7)
buffer.Write(position) // Information about the piece
payLoad := make([]byte, msg.length - 9)
n, err := reader.Read(payLoad)
if err != nil {
        log.Stderr("Wire -> Reading piece:", err, "read:", n)
        return
}
buffer.Write(payLoad) // Append piece data to the end
log.Stderr("Wire -> Sending:", buffer.Bytes())
n, err = wire.conn.Write(buffer.Bytes()) // Send the peice
log.Stderr("Wire -> Sent:", n, "bytes")
if err != nil {
        return
}

And If I use io.Copy or if I send the message in two parts it doesn't
work (the client on the other end closes the connection):

msg_byte = make([]byte, 13) // Allocate memory for the header only
binary.BigEndian.PutUint32(msg_byte[0:4], msg.length) // Write msg length
buffer := bytes.NewBuffer(msg_byte[0:4])
buffer.WriteByte(msg.msgId) // Write ID
buffer.Write(position) // Write info about the piece being send
log.Stderr("Wire -> Sending header:", buffer.Bytes())
n, err = wire.conn.Write(buffer.Bytes()) // Send the header
log.Stderr("Wire -> Sent:", n, "bytes")
if err != nil {
        return
}
if msg.msgId == piece {
        payLoad := make([]byte, msg.length - 9) // Allocate memory for the piece data
        n, err := reader.Read(payLoad) // Read
        if err != nil {
                log.Stderr("Wire -> Reading piece:", err, "read:", n)
        }
        log.Stderr("Wire -> Sending piece")
        n, err = wire.conn.Write(payLoad) // Write
        if err != nil {
                return
        }
        log.Stderr("Wire -> Sent:", n, "bytes")
}

reader is an io.SectionReader in both cases, with the exact length. I
don't use io.Copy in this example, but you can replace the contents of
the last "if" with "io.Copy(wire.conn, reader)" and it won't work
also. The number of bytes send is the same in both examples.

Could someone throw some light into this?

Thanks, Roger.

I've done some more research, and I get to the conclusion that I
cannot write the message using more than one call to Write(), I have
tried using bufio.Writer, but still the same problem, if I try to send
the message with two calls to Write, it fails, but if I do it with
only one call it works (either directly to the net.Conn or using
bufio.Writer). Any ideas?

Thanks, Roger.

2010/9/10 Roger Pau Monné <roy...@gmail.com>:

> if msg.msgId == piece {

This is a difference between your two code fragments--the first one
didn't seem to have a similar test.

>         payLoad := make([]byte, msg.length - 9) // Allocate memory for the piece data
>         n, err := reader.Read(payLoad) // Read

How long does this take?  Is it possible that the other end is timing
out before this is read and then written?

Unless the other end is using direct calls into the networking layer, I
don't see how it could even detect that you are using multiple Write
calls.

Ian