net.conn TCP connection

[ron.w...@gmail.com]

I am looking for a net.conn standard read that would return a data buffer the exact size of the read. I am trying to read an unknown amount of byte data from the connection. With the read i am using I am required to pre-allocate a buffer and pass that buffer to the read. I am looking for a read that works more like  the ReadString , but is for a byte slice.

// I want something similar to this read that returns the read string into the message string.

 message, err := bufio.NewReader(ServerConn).ReadString('\n')

                if ( err != nil ){

                        fmt.Println("RELAY: ERROR:  Reg Message read err:", err)

                        return 

                }

// had to preallocate a buffer, but I want a read to return me a buffer so I don't have to guess how big to make it.

 buf := make([]byte, 1024*32)

 // READ FROM CLIENT

 nBytes, err := Csrc.Read(buf)

Is this not possible, I have not seen any examples that would indicate that there is a standard library that would do something like what I am looking for.

thanks,

Ron

[Matthew Zimmerman]

https://golang.org/pkg/io/ioutil/#ReadAll

That might be what you want but if you don't know how big it's going to be, you're also easily able to run out of memory.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/96d2e64d-edce-4fb2-b4c2-970d04eedf14%40googlegroups.com.

[Ron Wahler]

I did look at ReadAll, but it won't return until it sees EOF.  I am trying to find something that would return when the standard read would return. I get the memory part and would manage that. Any other ideas ?

thanks,

Ron 

[Robert Engels]

You need a termination point. In the case of ReadString it is the line terminator. For an arbitrary read it is either a length or EOF - or you can read until the underlying socket has no more data but this is generally useless unless you are doing higher level buffering and protocol parsing. This latter mode is almost never needed in Go due to synchronous nature. 

On Dec 27, 2019, at 9:36 PM, Ron Wahler <ron.w...@gmail.com> wrote:



--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8ca83343-399d-47ef-aee0-0e173946c980%40googlegroups.com.

[Robert Engels]

The standard read will early return as soon as some of read is satisfied and a subsequent block would occur (or timeout) - so you have to decide when you want to stop reading...

On Dec 27, 2019, at 9:48 PM, Robert Engels <ren...@ix.netcom.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8F20C49C-7240-4E58-A25D-583BD0482115%40ix.netcom.com.

[Jake Montgomery]

On Friday, December 27, 2019 at 10:36:28 PM UTC-5, Ron Wahler wrote:

I did look at ReadAll, but it won't return until it sees EOF.  I am trying to find something that would return when the standard read would return. I get the memory part and would manage that. Any other ideas ?

Its unclear to me what the "standard read" is that you are referring to.

There are well established solutions to most networking communication and reading scenarios. However, it is not clear to me from your post exactly what your specific scenario is. For example what is the protocol? What kind of data are you retrieving? How are you defining a "standard read"?

With more information, you will have better luck getting a solution that works for your use case.

[Ron Wahler]

Jake,

Thanks for the reply. Csrc.Read is what I was referring to as the connection standard read, should not have used the words "standard read" sorry about that. The problem I am trying to solve is reading an unknown amount of byte data.  I am trying to understand what triggers the Csrc.Read(buf) to return when I send say 3 bytes to it with a client, I also keep the connection open and send a few bytes of characters with the netcat tool, the Csrc.Read returns, but the snip it below that with ReadAll does not return. I am trying to understand the underlying behavior of what triggers a return with the data in these two calls ?

on this read :

Csrc net.Conn

 buf := make([]byte, 1024*32)

 // READ FROM CLIENT

 nBytes, err := Csrc.Read(buf)

Csrc.Read(buf)  returns with a few bytes that I send to it.  It does not wait for the entire allocated buf size to return. This works great, but I am looking for a way to not preallocate a large buffer.

I am prototyping with ReadAll, see the following snip it, but when I send a few bytes to this call with a client, it does not return. The documentation is saying it may be  looking for an EOF which I do not send.

  buf, read_err := ioutil.ReadAll(Csrc)

thanks,

Ron

On Friday, December 27, 2019 at 5:11:42 PM UTC-7, Ron Wahler wrote:

[Robert Engels]

Use read and expand the buffer as needed (or write the chunks to a file). If at the end it is all going to be in memory, you might as well start with the very large buffer. There is nothing special about Go in this regard - it’s standard IO processing. 

On Dec 29, 2019, at 9:21 AM, Ron Wahler <ron.w...@gmail.com> wrote:



--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cd1f26d6-72aa-4239-83f2-18dc6220c2db%40googlegroups.com.

[Jake Montgomery]

It sounds like maybe you have some misconceptions about TCP. It is a stream protocol, there are no data boundaries that are preserved. If send 20 bytes via TCP in a single call, it is likely that those 20 will arrive together at the client. But it is NOT guaranteed. It is perfectly legitimate for 10 bytes to arrive first, then the next 10 sometime later. Obviously this is unlikely with only a few bytes, but becomes more likely as the size of the Write grows. Until the connection is closed, you never know if there is more data coming. So it may seem that there is a 1:1 correlation between conn.Write() and conn.Read(), but you can not count on it.

To answer you specific question, conn.Read() will return when it has filled up the buffer provided, or there is no more data ready to be read at that moment. ReadAll() will wait until EOF. Given that TCP is a stream, as I described above, it is still unclear what you hope to have happen without knowing more about the specific data being transmitted, and what you wan to do with it on the client side.

[Robert Engels]

ReadAll reads until buffer length or EOF. 

On Dec 30, 2019, at 11:04 AM, Jake Montgomery <jake...@gmail.com> wrote:



--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7e468329-2488-460c-9419-b4c55857b1eb%40googlegroups.com.

[Bruno Albuquerque]

But, to complicate things, you can create what is basically a TCp connection with packet boundaries using SOCK_SEQPACKET (as opposed to SOCK_STREAM or SOCK_DGRAM).

--

[Robert Engels]

That option requires proprietary protocols not standard tcp/udp. 

On Dec 30, 2019, at 12:04 PM, Bruno Albuquerque <b...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEd86TxqRRML-HDe7H8vESCTBOhBV3c%3DfY0gamEZrrb6XKgfzA%40mail.gmail.com.

[Bakul Shah]

I am trying to understand what triggers the Csrc.Read(buf) to return 

The Read call will eventually turn into a read() system call. For TCP it will return as long as there is at least one byte in the kernel's receive buffer for this connection, or if the buffer give to read() is filled up, or the connection is closed.

Do you know about ReadAtLeast? Read its documentation:

go doc io.ReadAtLeast

For example the following code will return exactly 32k bytes or return with an error.

buf := make([]byte, 32*1024)

...

n, err := io.ReadAtleast(Csrc, buf, len(buf))

....

I am guessing this may be what you want but your requirements are rather vague so I'm not sure. It'll help if you can show us some pseudo code that precisely describes what you expect to happen, possibly with pre and post conditions (but if ReadAtLeast is what you want, don't bother!).

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cd1f26d6-72aa-4239-83f2-18dc6220c2db%40googlegroups.com.

[Bruno Albuquerque]

Although I am no expert in the subject, I would doubt this assertion. It is there in the socket man page in a Ubuntu machine with no mention of anything specific being needed (other than the implicit fact that you need a TCP stack that supports it, which should be true for any modern version of Linux anyway).

[Robert Engels]

Im pretty sure I’m correct. It is a socket type not an option on TCP, which equates to a different protocol. If you use that option you get a SCTP transport not TCP. 

On Dec 30, 2019, at 4:06 PM, Bruno Albuquerque <b...@gmail.com> wrote:



[Robert Engels]

Oh, and I don’t think SCTP is natively supported on Windows yet. 

So your interoperability May vary...

On Dec 30, 2019, at 4:17 PM, Robert Engels <ren...@ix.netcom.com> wrote:



[Ron Wahler]

Thanks for all the great responses.

How much of the read behavior is in the golang underlying code on the read and how much is 

on the underlying OS driver that implements the behavior of the read. I understand the stream nature of the TCP connection and I handle that in my code, I was just looking to optimize the buffer allocation for the read on each packet  of the stream so I don't have to over allocate the buf if I scaled out my function to be concurrent. From the responses so far I think I just have to pick a buffer size for the Read() or the ReadAtLeast() and just process multiple buffers with my pre-allocated guess on the buffer size as I am already doing.

I was also looking to understand the read implementation of GoLand and how it utilizes the underlying drivers, so any info on that or where to read the code for golang integration with the drivers would be great.

cheers,

Ron

On Friday, December 27, 2019 at 5:11:42 PM UTC-7, Ron Wahler wrote:

[Robert Engels]

The important factors: 1) the larger the buffer the fewer system calls that need to be made 2) the larger the buffer the more memory use which can be significant with many simultaneous connections

You need to remember that with correct socket options the kernel is already buffering, and adding your own buffering on top may not be needed 3) really large user space (Go) buffers may be needed if processing a really fast fat pipe

On Dec 31, 2019, at 9:23 AM, Ron Wahler <ron.w...@gmail.com> wrote:



--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0e735f0e-dfb1-4e1f-9a2c-1b153be20514%40googlegroups.com.

[Robert Engels]

One other note, if you have a request / response type protocol with fairly defined lengths, you don’t need a buffer larger than the largest message if you don’t allow concurrent requests from the same client. 

On Dec 31, 2019, at 9:35 AM, Robert Engels <ren...@ix.netcom.com> wrote:



[Ron Wahler]

\\One other note, if you have a request / response type protocol with fairly defined lengths, you don’t need a buffer larger than the largest message if you don’t allow concurrent requests from the same client. 

Yes, understood, that was not the constraint, I have to process an unknown amount of bytes, so I have been just trying to make sure I had the best solution.  

I would still like to read the underlying GoLang code on Read() to see what assumptions it makes to the system calls.

thanks,

Ron

On Friday, December 27, 2019 at 5:11:42 PM UTC-7, Ron Wahler wrote:

[Robert Engels]

All of the source is there... but in general it is a bit more complex under the covers than most - as it uses select/poll to know when a socket is ready for IO then schedules the routine in the Read() to perform the IO. So Go has a bit of its own kernel code than you would see in typical synchronous C socket code. Ultimately Read() becomes a standard read() on the socket descriptor. 

On Dec 31, 2019, at 10:03 AM, Ron Wahler <ron.w...@gmail.com> wrote:



--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/24a6f6bd-8948-434c-bb83-c663d75bb23d%40googlegroups.com.

[Robert Engels]

Also, you might find something like nats.io simplifies you’re effort. I’m not sure if it supports extremely large message sizes off hand, but there is probably a chunking layer available. Doing simple TCP messaging is fairly easy, when you get into redundancy, fan-out, etc is can get complex fast, so using a messaging library may help. 

On Dec 31, 2019, at 10:37 AM, Robert Engels <ren...@ix.netcom.com> wrote:



[Jake Montgomery]

If you are really interested in how the Go code relates to the underlying socket system calls, the code is readily available. AFAICT, the TCPConn.Read() call on linux eventually comes down to the unix version poll.PD.Read(). See https://golang.org/src/internal/poll/fd_unix.go#L145. Before that is mostly just basic setup and then code to wrap the errors.

Unless you are on an embedded system or something, you will likely not find that memory usage is the real bottleneck in practice, assuming you choose reasonable constants and a reasonable number of workers. If GC thrashing is a problem, you can always use a strategy to reuse the buffers.

Good Luck.

[jvm...@gmail.com]

So I looked at the implementation of ReadAll, and found this interesting little nugget: 

	if int64(int(capacity)) == capacity {
    34  		buf.Grow(int(capacity))
    35  	}

To see it in context: https://golang.org/src/io/ioutil/ioutil.go?s=807:875#L18

Can anybody decipher what the point of the cast to `int` and then back to `int64` is? My best guess is it's guarding against integer overflow on a 32-bit arch, but it seems like there would be a more straightforward way to do that.. ?

On Friday, December 27, 2019 at 10:19:45 PM UTC-5, Matthew Zimmerman wrote:

https://golang.org/pkg/io/ioutil/#ReadAll

That might be what you want but if you don't know how big it's going to be, you're also easily able to run out of memory.

On Fri, Dec 27, 2019, 7:11 PM <ron....@gmail.com> wrote:

I am looking for a net.conn standard read that would return a data buffer the exact size of the read. I am trying to read an unknown amount of byte data from the connection. With the read i am using I am required to pre-allocate a buffer and pass that buffer to the read. I am looking for a read that works more like  the ReadString , but is for a byte slice.

// I want something similar to this read that returns the read string into the message string.

 message, err := bufio.NewReader(ServerConn).ReadString('\n')

                if ( err != nil ){

                        fmt.Println("RELAY: ERROR:  Reg Message read err:", err)

                        return 

                }

// had to preallocate a buffer, but I want a read to return me a buffer so I don't have to guess how big to make it.

 buf := make([]byte, 1024*32)

 // READ FROM CLIENT

 nBytes, err := Csrc.Read(buf)

Is this not possible, I have not seen any examples that would indicate that there is a standard library that would do something like what I am looking for.

thanks,

Ron

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

[Ian Lance Taylor]

On Tue, Dec 31, 2019 at 7:23 AM Ron Wahler <ron.w...@gmail.com> wrote:
>
> How much of the read behavior is in the golang underlying code on the read and how much is
> on the underlying OS driver that implements the behavior of the read. I understand the stream nature of the TCP connection and I handle that in my code, I was just looking to optimize the buffer allocation for the read on each packet of the stream so I don't have to over allocate the buf if I scaled out my function to be concurrent. From the responses so far I think I just have to pick a buffer size for the Read() or the ReadAtLeast() and just process multiple buffers with my pre-allocated guess on the buffer size as I am already doing.

You may be interested in the discussions at https://golang.org/issue/15735.

Ian

[Ian Lance Taylor]

On Tue, Dec 31, 2019 at 9:58 AM <jvm...@gmail.com> wrote:
>
> So I looked at the implementation of ReadAll, and found this interesting little nugget:
>
> if int64(int(capacity)) == capacity {
> 34 buf.Grow(int(capacity))
> 35 }
>
>
> To see it in context: https://golang.org/src/io/ioutil/ioutil.go?s=807:875#L18
>
> Can anybody decipher what the point of the cast to `int` and then back to `int64` is? My best guess is it's guarding against integer overflow on a 32-bit arch, but it seems like there would be a more straightforward way to do that.. ?

The Go standard library keeps file sizes as int64, and ioutil.ReadFile
wants to pass that size to readAll, so readAll takes the capacity as
an int64. However, in Go, the type int is the size of an object that
can be stored in memory, and on a 32-bit system this is of course a
32-bit type. Since a bytes.Buffer lives in memory, its size is type
int. To save time we want to grow the buffer to be large enough to
hold the expected file size. But of course converting a value like
0x200000000 to a 32-bit int will give us zero, and we don't want to
grow the buffer to size 0. So we only grow the buffer if we can
represent the capacity in type int. And it's very easy to test
whether an int64 value fits in type int by writing int64(int(x)) == x.
So that is what the code does.

I think you are asking whether there is a more straightforward way to
check whether an int64 value fits in an int, but I can't think of what
that might be. This approach seems straightforward to me: if you
convert to int and then back to int64, do you still have the same
value? (Although this isn't the most important criterion here, this
approach does have the advantage of compiling away on a system where
int is 64 bits.)

Ian