Why isn't golang input and output is buffered?

[nils...@gmail.com]

I have seen that python input and output is buffered, but golang's input and output is not buffered.

For those types of problems where i need to take large amounts of input, It's taking long time to just take input than to do processing.

When we have advantages with buffered input and output, why isn't it input taken in such a way while I'm using scanf function.

or else am I supposed to use bufio package whenever I want to take buffered input and buffered output.

Thank you.

[Lutz Horn]

> For those types of problems where i need to take large amounts of input,
> It's taking long time to just take input than to do processing.

Can you provide example code for how you read the input?

Lutz

[Ian Lance Taylor]

It's true that fmt.Scanf works on the unbuffered os.Stdin. That is
the correct choice when os.Stdin is used for different things. For
your case, use bufio.NewReader(os.Stdin) and passed the buffered
stream to fmt.Fscanf. Then of course you must make sure to always use
the buffered input stream, but plain os.Stdin.

Ian

[Nilsocket]

Can you provide example code for how you read the input?

Both of them were given same input size is:1000000, (i.e., 1 million)

https://play.golang.org/p/grB-muK7hw

// Time taken to read input : 9.840256889s

// Time taken to sort: 731.469604ms

I have implemented the same using bufio:

https://play.golang.org/p/1Aoxwwv-zo

// Time taken to read input : 377.038152ms

// Time taken to sort: 688.20638ms

[Michael Jones]

This can be much simpler...

https://play.golang.org/p/tMyipz6sYU

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones
michae...@gmail.com

[Patrick Smith]

That is simpler, but slower. Not nearly as slow as using unbuffered io though. Timings on my machine, reading 1e6 integers chosen randomly from the range [0,1e18):

Original code https://play.golang.org/p/grB-muK7hw

5.626974435s

155.367779ms

Original poster's optimized code https://play.golang.org/p/1Aoxwwv-zo

168.638597ms

150.923225ms

Michael's simpler code https://play.golang.org/p/tMyipz6sYU

954.543351ms

166.710399ms

So this is about 6 times slower. My guess is this is due to the use of reflection in fmt.Fscanf. But that is just a guess; I don't really have any evidence to back it up.

[Michael Jones]

good point. I was trying to show that the buffered stdin was "just like" normal scanning but the performance was less compared to the updated scanning code.

here is another version, this time with a data generator and since the input is line oriented, the default split function is fine.

https://play.golang.org/p/SgpAXyvsGs

read 1000000 65.362993ms

sort 1000000 187.092493ms

[peterGo]

Michael,

Your read times look slow to me. I used bytconv instead of strconv.

bytconv:
read 1000000 98.517584ms
sort 1000000 481.994354ms

strconv:
read 1000000 174.720883ms
sort 1000000 479.437831ms

bytconv is the missing Go standard library package. bytconv is the byte input analog of string input strconv.

Peter

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones
michae...@gmail.com

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[peterGo]

Michael,

n your code, you have :

const n = 1000 * 1000

for i := 0; i < n && scan.Scan(); i++ {
    d, _ := strconv.Atoi(scan.Text())
    array[i] = int64(d)
}

https://play.golang.org/p/SgpAXyvsGs

Here's a benchmark that demonstrates the fundamental issue, unnecessary string conversions.
,
BenchmarkAtoiBytconv-4       50000000     30.4 ns/op    0 B/op    0 allocs/op
BenchmarkAtoiStrconv-4       20000000    102 ns/op      8 B/op    1 allocs/op

https://play.golang.org/p/oSQ8RZeGL7

Peter

[Michael Jones]

Nice! Is "bytconv" shared somewhere?

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Jones]

By the way, if we're willing to do it all...we can take it down a little more:

https://play.golang.org/p/DNxQT01vy9

read 1000000 26.174853ms

sort 1000000 180.17782ms

[Nilsocket .]

Optimizations were really great.

Can somebody tell me, why isn't buffered i/o isn't used by default.

>It's true that fmt.Scanf works on the unbuffered os.Stdin.  That is
   the correct choice when os.Stdin is used for different things.

Ian Lance Taylor, have mentioned it, but it is too short for me.

Could some-one elaborate the answer.

Even though buffered i/o has considerable advantages, were there any other situations where in it really goes bad.

or,

Was it like providing a choice to the user.

Thank you.

--

from:- nilsocket

[as]

Once something is buffered, it can not be unbuffered. That is, the latency of the I/O operations permanently increase with respect to the properties of the buffering done underneath. Go does not buffer things by default because not everyone wants to buffer things by default. This is also why Nagle's algorithm is off by default in the network stack. These are good things: you can always buffer to get better throughput, but you can't decrease latency.