How to handle write operation when the err is ErrShortWrite?

[Xiaobin She]

hi all,

I'm new to go, and I got some questions that I can't google it out.

I'm dealing with an socket programme, and when I'm writing the server, I encounter an problem.

The server will make an goroutine to handle the client connection, after read and handle the request from the client, the server will send the response back to the client, what confuses me is that how to handle the write operation with the tcpconn ?

As I can see from the api page, the write call will return the length of data has been writeen and the error that shows what's wrong.

And after a little dig into the source code of go, I found that when the returned lenght < len(buffer), the error var will be set to ErrShortWrite， so here comes my question, do I have to handle ErrShortWrite?

Do I have to write code like this :

                if isPkgComplete {
                    if err := svr.HandlePkg(readBuf[0:pkgLen], &writeBuf, &writeDataLen); err != nil {
                        commlog.LogErr("HandlePkg failed %s err %s", connFrom, err.Error())
                        return
                    }  

                    writePos := 0
                    for writePos < writeDataLen {
                        if writeLen, err := conn.Write(writeBuf[writePos:writeDataLen]); err != nil && err != io.ErrShortWrite {
                            commlog.LogErr("Write to %s failed err %s", connFrom, err.Error())
                            return
                        } else {
                            if writeLen > 0 {
                                writePos += writeLen
                            }  
                        }  
                    }  
                }

Is that for {}  needed ?

Or do I only need to check is the returned err var nil or not and ignore the returned length ?

Thanks!

[Dave Cheney]

Or do I only need to check is the returned err var nil or not and ignore the returned length ?

The io.Writer contract specifies that if less than len(buf) bytes are written, an error must be returned.

Critically it does _not_ make any guarantees about the content or form of that error message.

[Xiaobin She]

在 2014年10月8日星期三UTC+8下午9时52分20秒，Dave Cheney写道：

Or do I only need to check is the returned err var nil or not and ignore the returned length ?

The io.Writer contract specifies that if less than len(buf) bytes are written, an error must be returned.

Critically it does _not_ make any guarantees about the content or form of that error message.

Thank you very much for your reply.

So how to handle the situation that less than len(buf) bytes are written?
If written in C with non-blocking io, if the write system call return less than len(buf), we can add the write event to the epoll loop, and wait for the next write call.

As far as I known, go use non-blocking io for system calls, so do I need to handle this less than len(buf) bytes situation?
Like the for {} loop I wrote ? And if go does not make any guarantees about the content or form of that error message, how can I know which err I should deal with ?

Or does go guarantees that if less than len(buf) bytes returned,  I don't need to send the rest of the data?

 

[Tamás Gulácsi]

Go will write all the bytes you want, except in case of error.
In essence: go runtime includes the for loop.

[Xiaobin She]

在 2014年10月9日星期四UTC+8下午12时35分00秒，Tamás Gulácsi写道：

Go will write all the bytes you want, except in case of error.
In essence: go runtime includes the for loop.

So what I have to write is codes like the following one ? No need to check the returned lenght?
So what is that return value used for?


                    if writeLen, err := conn.Write(writeBuf[writePos:writeDataLen]); err != nil {

                        commlog.LogErr("Write to %s failed err %s", connFrom, err.Error())
                        return
                    } else {

                        //do something else
                    }
 

[Tamás Gulácsi]

It depends. There will be an error if written length is less than the wanted. You have to decide what to do based on the error and your goals.

[Xiaobin She]

在 2014年10月9日星期四UTC+8下午2时27分01秒，Tamás Gulácsi写道：

It depends. There will be an error if written length is less than the wanted. You have to decide what to do based on the error and your goals.

when the return length is less than len(buf), what I want is:
1. if the err indicates that I can call the write again, than I will call the write again to send the reset of the data
for example, in C code, if the errno is EAGAIN when using non-blocking IO, I will know that I can call the write system call again
Is there some err like this one ?

2. if the err indicates that it's some kind of other error (like the connection is closed by the peer), than I will close the connection and return fail

And what confuse me is that I don't know how to distinguish these two type of error.

I found it difficult to find the return err message from the api page, from the official website, all I can found is this:

        // Write writes data to the connection.
        // Write can be made to time out and return a Error with Timeout() == true
        // after a fixed time limit; see SetDeadline and SetWriteDeadline.
        Write(b []byte) (n int, err error)

So the Write call return an error var, and what does this error var mean?
How can I know what kind of error is?
After search the web, I found that this err var maybe implement the net.Error interface or maybe an net.OpError struce, but what message can I got from this interface or strut?

Take the net.Error interface for example, this is the definition of the interface:

type Error interface {
        error
        Timeout() bool   // Is the error a timeout?
        Temporary() bool // Is the error temporary?
}

From the interface, I can know that if the err var implements this interface, I can use the Timeout() or Temporary() func to know if the error is an timeout error or an temporary error.
I know what timeout means, but what does temporary error means?

Does it means I can call the Write call again? Where can I found the definition of Temporary?

And what about oter kind of error? Like the connection closed by the peer, how can I know this type of error?

And for net.OpError, this is the definition:
type OpError struct {
        // Op is the operation which caused the error, such as
        // "read" or "write".
        Op string

        // Net is the network type on which this error occurred,
        // such as "tcp" or "udp6".
        Net string

        // Addr is the network address on which this error occurred.
        Addr Addr

        // Err is the error that occurred during the operation.
        Err error
}


Again, from the comments of the definition, I don't know how to find the exact type of the error.
Do I have to use regex to find the pattern of the error string to find out what kind of error it is?

And I notice that the error can also be an syscall.Errno type, is this type enough for distinguish the error type?
And which func in the net package will return this type of error? Is there any document I can read from ?


thanks

 

[Dave Cheney]

On Thu, Oct 9, 2014 at 9:11 AM, Xiaobin She <xiaob...@gmail.com> wrote:
>
>
> 在 2014年10月9日星期四UTC+8下午2时27分01秒，Tamás Gulácsi写道：
>>
>> It depends. There will be an error if written length is less than the
>> wanted. You have to decide what to do based on the error and your goals.
>
>
>
> when the return length is less than len(buf), what I want is:
> 1. if the err indicates that I can call the write again, than I will call
> the write again to send the reset of the data
> for example, in C code, if the errno is EAGAIN when using non-blocking IO, I
> will know that I can call the write system call again
> Is there some err like this one ?

You cannot observe Go's use of non blocking IO from a Go program. From
the POV of a single goroutine Read or Write _always_ blocks.

>
> 2. if the err indicates that it's some kind of other error (like the
> connection is closed by the peer), than I will close the connection and
> return fail

What I and the other contributors are trying to say is you have two states

1. err == nil, everything worked
2. err != nil, something failed, that's all we know

You _could_ try to use a type assertion or type switch to interrogate
the value inside the error, but you'll find that few packages give any
suggestion of what error will returned.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/EebOLHrdAeo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Xiaobin She]

在 2014年10月9日星期四UTC+8下午4时58分08秒，Dave Cheney写道：

On Thu, Oct 9, 2014 at 9:11 AM, Xiaobin She <xiaob...@gmail.com> wrote:
>
>
> 在 2014年10月9日星期四UTC+8下午2时27分01秒，Tamás Gulácsi写道：
>>
>> It depends. There will be an error if written length is less than the
>> wanted. You have to decide what to do based on the error and your goals.
>
>
>
> when the return length is less than len(buf), what I want is:
> 1. if the err indicates that I can call the write again, than I will call
> the write again to send the reset of the data
> for example, in C code, if the errno is EAGAIN when using non-blocking IO, I
> will know that I can call the write system call again
> Is there some err like this one ?

You cannot observe Go's use of non blocking IO from a Go program. From
the POV of a single goroutine Read or Write _always_ blocks.

Dave, thank you very much for your explaination.
Still I have one question, in C code, when deal with blocking io, we will have the handle one situation, which is that the errno is EINTR.
If I call read or write and the errno is EINTR, I will call read/write again.

So do I have to handle this situation within GO ?

Do I have to wirte code like this?

func checkNetErr(err error) bool {                                                                                                                                      
    if ne, ok := err.(*net.OpError); ok {                                                                                                                               
        if errno, enok := ne.Err.(syscall.Errno); enok {                                                                                                                
            if errno == syscall.EINTR {                                                                                                                                 
                return true                                                                                                                                             
            }                                                                                                                                                           
        }                                                                                                                                                               
    }                                                                                                                                                                   
    return false                                                                                                                                                        
}


 if readLen, err := conn.Read(readBuf[readPos:maxReadBufSize]); err != nil {
                    if checkNetErr(err) {
                        continue
                    }
                    if err == io.EOF {
                        commlog.LogErr("maybe client disconnect %s %s", connFrom, err.Error())
                    } else {
                        commlog.LogErr("Read from %s failed err %s", connFrom, err.Error())
                    }
                    break
                }



 

>
> 2. if the err indicates that it's some kind of other error (like the
> connection is closed by the peer), than I will close the connection and
> return fail

What I and the other contributors are trying to say is you have two states

1. err == nil, everything worked
2. err != nil, something failed, that's all we know

I understand these two rules.
 

You _could_ try to use a type assertion or type switch to interrogate
the value inside the error, but you'll find that few packages give any
suggestion of what error will returned.

But what I don't know how to deal with is the second situation, which is err != nil.
I can't get an clear information with the err var, like you said, I can try to use a type assertion or type switch to interrogate
the value inside the error, but how to know exactly what kind of error it is, and how to deal with it ?
It seems that there is no clear or easy way.


 

[Tamás Gulácsi]

As I said the for cycle which retries writing where possible (the error is not an error, or eventually it will succeed), is in the Go runtime (for sockets).
So you won't get EAGAIN.

[Xiaobin She]

在 2014年10月9日星期四UTC+8下午8时18分39秒，Tamás Gulácsi写道：

As I said the for cycle which retries writing where possible (the error is not an error, or eventually it will succeed), is in the Go runtime (for sockets).
So you won't get EAGAIN.

So I don't need to handle EAGAIN, Go runtime will do the retry until timeout, and if I got an error, that means it's an error that can't be retry, all I have to do is deal with this error, like close the connection, is this understanding right?

If so, does this means I don't need to handle EINTR either？

Thank you very much for your patience!

[Dave Cheney]

Nope. Go handles that for you, we tell the signal handlers to restart any restartable syscall.s

At this point I think you need to step out of the theoretical and wrote some test code to validate the questions you have about Go.

[Xiaobin She]

在 2014年10月9日星期四UTC+8下午9时49分51秒，Dave Cheney写道：

To Dave and Tamás,

thanks for your patience and explaination, I think I understand the question now.

Actually all these questions coming out when I'm writting some code.
And to be honest, I think the Go doc page is a bit confusing or lack of information, if it can be as clear as the man page, then I think it would be much easyer to understand, writting some test code would be helpful but without an clear definition it would be just guessing.

Thanks very much for your time.

[Lars Seipel]

On Thu, Oct 09, 2014 at 08:51:51PM -0700, Xiaobin She wrote:
> And to be honest, I think the Go doc page is a bit confusing or lack of
> information, if it can be as clear as the man page, then I think it would
> be much easyer to understand

You can find the contract for an io.Writer alongside the documentation
for package io. Net.Conn just implements this contract. Still, even that
documentation might not be as specific as your system's man page for the
write system call.

The reason for that is as follows: the man page author generally knows
on what kinds of file descriptors (i.e. referring to sockets, local
files, ttys, …) write(2) can be called and just describes the union of
their quirks.

That's not possible with Go interfaces. You have the interface contract
and an unbounded number of possible implementations. There isn't any
answer more specific than Dave's while still being general, that is,
applicable to any given io.Writer.

Some implementations do give you more information that you can
programmatically check for. You already discovered the error types in
net or the possibility to return syscall.Errno values and the more
generic errors from package io. Most of the latter, I think, are newer
than the io.Writer contract itself. They might become the conventional
way of communicating IO errors.

Lars

[Lars Seipel]

On Fri, Oct 10, 2014 at 02:43:28PM +0200, Lars Seipel wrote:
> Net.Conn just implements this contract.

For the sake of correctness: net.Conn doesn't implement anything, it
just passes the contract on.