StringBuilder in Go with bytes.Buffer

[Tong Sun]

How best to construct bytes.Buffer gradually? with fmt.Fprint?

This is my code:

```

func getReqAddons(r Request) string {

ret := ""

if len(r.RequestPlugins.RequestPlugin) != 0 {

for _, v := range r.RequestPlugins.RequestPlugin {

ret += fmt.Sprintf("  R: (%s) %s\n", v.Name, minify(v.RuleParameters.Xml))

}

}

if len(r.ExtractionRules.ExtractionRule) != 0 {

for _, v := range r.ExtractionRules.ExtractionRule {

ret += fmt.Sprintf("  E: (%s: %s) %s\n", v.Name, v.VariableName, minify(v.RuleParameters.Xml))

}

}

if len(r.ValidationRules.ValidationRule) != 0 {

for _, v := range r.ValidationRules.ValidationRule {

ret += fmt.Sprintf("  V: (%s) %s\n", v.Name, minify(v.RuleParameters.Xml))

}

}

return ret + "\n"

}

```

This is still not efficient right? 

Having read that in Go, string is a primitive type, it's readonly, every manipulation to it will create a new string, I want to change above to something efficient, and learnt that bytes.Buffer (and fmt.Fprint) should be the way to go, but haven't found a good example showing how yet. 

So, how best to construct bytes.Buffer gradually? Is fmt.Fprint the best choice for my above case, or buf.WriteString(fmt.Sprintf(...))) is the best way to go? 

```

import "bytes"

...

var buf bytes.Buffer    // its zero value is usable as-is

buf.WriteString("one string.")

buf.Write([]byte(" more bytes"))

fmt.Fprint(buf, "%s more %s", buf, more)

return buf.String()

```

Thanks

[Tong Sun]

On Sunday, November 1, 2015 at 2:53:15 PM UTC-5, Tong Sun wrote:

How best to construct bytes.Buffer gradually? with fmt.Fprint?

This is my code:

```

func getReqAddons(r Request) string {

ret := ""

if len(r.RequestPlugins.RequestPlugin) != 0 {

for _, v := range r.RequestPlugins.RequestPlugin {

ret += fmt.Sprintf("  R: (%s) %s\n", v.Name, minify(v.RuleParameters.Xml))

}

}

if len(r.ExtractionRules.ExtractionRule) != 0 {

for _, v := range r.ExtractionRules.ExtractionRule {

ret += fmt.Sprintf("  E: (%s: %s) %s\n", v.Name, v.VariableName, minify(v.RuleParameters.Xml))

}

}

if len(r.ValidationRules.ValidationRule) != 0 {

for _, v := range r.ValidationRules.ValidationRule {

ret += fmt.Sprintf("  V: (%s) %s\n", v.Name, minify(v.RuleParameters.Xml))

}

}

return ret + "\n"

}

```

This is still not efficient right? 

Having read that in Go, string is a primitive type, it's readonly, every manipulation to it will create a new string, I want to change above to something efficient, and learnt that bytes.Buffer (and fmt.Fprint) should be the way to go,

Or, is it really matter? because if you look under the hood, `func (b *Buffer) WriteString(s string)` will call `b.grow(len(s))` any way. So maybe efficiency-wise, using bytes.Buffer won't be much different than my pretty intuitive code above? 

[Matt Harden]

Just do it in the way that is most natural. Usually this will be fast enough. Don't worry about the efficiency until you have a working program that's too slow for your use case. At that point, profile the code to find where the inefficiencies are, and fix those. They are often not where you think they will be.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Manlio Perillo]

Il giorno domenica 1 novembre 2015 20:53:15 UTC+1, Tong Sun ha scritto:

How best to construct bytes.Buffer gradually? with fmt.Fprint?

This is my code:

> [...]

 

If you want to know the difference between Fprintf and Sprintf, look at the source:

https://golang.org/src/fmt/print.go?s=4905:4982#L176

https://golang.org/src/fmt/print.go?s=4905:4982#L201

fmt always uses an internal buffer (a simple []byte).

With Sprintf there is an extra allocation, since []byte is converted to string; you can run a simple benchmark to see the real difference.

Regards  Manlio

[Tong Sun]

On Sun, Nov 1, 2015 at 3:41 PM, Manlio Perillo <manlio....@gmail.com> wrote:
> Il giorno domenica 1 novembre 2015 20:53:15 UTC+1, Tong Sun ha scritto:
>>
>>
>> How best to construct bytes.Buffer gradually? with fmt.Fprint?
>>
>> This is my code:
>>
>
>> [...]
>
> If you want to know the difference between Fprintf and Sprintf, look at the
> source:
> https://golang.org/src/fmt/print.go?s=4905:4982#L176
> https://golang.org/src/fmt/print.go?s=4905:4982#L201
>
> fmt always uses an internal buffer (a simple []byte).
> With Sprintf there is an extra allocation, since []byte is converted to
> string;

Wouldn't such differences be negligible, considering Fprintf have to
do the following?

fmt.Fprint(buf, "%s more %s", buf, more)

> you can run a simple benchmark to see the real difference.

Assuming I know how, and there isn't any conclusion already.

[Dan Kortschak]

You can just keep writing to the buffer and get the string on return. There is no need to get the string each time you append.

[Tong Sun]

Thanks Matt.

That string building would be the bulk part of my code though -- apart
for xml parsing, which is from standard lib, the rest of my code would
mostly be building strings piece by piece as shown.

That's why I want to know the established way of efficient string
building in Go.

[Tong Sun]

On Sun, Nov 1, 2015 at 4:07 PM, Dan Kortschak wrote:

>
> On 02/11/2015, at 7:25 AM, "Tong Sun" wrote:
>
>> Wouldn't such differences be negligible, considering Fprintf have to
>> do the following?
>>
>> fmt.Fprint(buf, "%s more %s", buf, more)
>

> You can just keep writing to the buffer and get the string on return. There is no need to get the string each time you append.

So are you endorsing the buf.WriteString(fmt.Sprintf(...))) way?

You know I have to use some kind of printf to do the appending right?

[Tong Sun]

Thanks, but how about the appending part?

On Sun, Nov 1, 2015 at 4:18 PM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
> No, if you need fmt functionality in your construction, us it: fmt.Fprintf(buf,...).

[Dan Kortschak]

http://play.golang.org/p/8Jq_dPuwNF

[Tong Sun]

Interesting, I thought doing that way will cause later output
overwriting previous ones. Interesting...

[Dan Kortschak]

How would that happen? The buffer is a single use value that conforms to io.Writer. All the semantics of writing to a file-like object hold here.

[Tong Sun]

I guess I'm too new to Go to understand how it work. I.e., each
`fmt.Fprintf(&buf` will advance buf magically, while in the end,
`return buf.String()` will magically return buf to the top of buffer
of the original place.

To much magical happening here for me to understand, but it is true --
http://play.golang.org/p/aG925WNos7

What's happening? I can't understand. How can buf advance its pointer
while maintaining the original place at the *same time*?

[Dan Kortschak]

No magic. Buffer.Write advances the write offset (just the length of the []byte used internally). Buffer.String returns the string conversion of the bytes from the read offset of the buffer to the current write offset - in the case here, the read offset is zero since no read has been done. (The String method is not a read operation BTW).

https://golang.org/src/bytes/buffer.go?s=4333:4384#L119 (see grow method for the update)
https://golang.org/src/bytes/buffer.go?s=1787:1819#L37

[Tong Sun]

On Sun, Nov 1, 2015 at 4:40 PM, Tong Sun <sunto...@gmail.com> wrote:
> I guess I'm too new to Go to understand how it work. I.e., each
> `fmt.Fprintf(&buf` will advance buf magically, while in the end,
> `return buf.String()` will magically return buf to the top of buffer
> of the original place.
>
> To much magical happening here for me to understand, but it is true --
> http://play.golang.org/p/aG925WNos7
>
> What's happening? I can't understand. How can buf advance its pointer
> while maintaining the original place at the *same time*?

I guess the short explanation is

buf is of type bytes.Buffer, which always points to the top of buffer;
whereas &buf is treated as the type of io.Writer. I.e., "All the

semantics of writing to a file-like object hold here".

So, the http://play.golang.org/p/8Jq_dPuwNF is the perfect solution
that I was looking for, while I can't possibly come up myself.

Thanks!

[Jan Mercl]

On Sun, Nov 1, 2015 at 10:40 PM Tong Sun <sunto...@gmail.com> wrote:

> What's happening? I can't understand. How can buf advance its pointer
while maintaining the original place at the *same time*?

I don't know what's mean by "its pointer", but maybe it could help to imagine that writes to a buffer work, ignoring implementattion details, like performing an append. Ie.

        fmt.Fprint(buf, "foo")

is very much like

        buf = append(buf, []byte("foo")...)

--

-j

[Tong Sun]

On Sun, Nov 1, 2015 at 4:51 PM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
> No magic. Buffer.Write advances the write offset (just the length of the []byte used internally). Buffer.String returns the string conversion of the bytes from the read offset of the buffer to the current write offset - in the case here, the read offset is zero since no read has been done. (The String method is not a read operation BTW).

Scratch my short explanation, this is the canonical one.

But my conclusion was right, the http://play.golang.org/p/8Jq_dPuwNF
by Dan is the perfect solution that I was looking for, which I can't
possibly come up myself.

Thanks again.

[Manlio Perillo]

Please, read the documentation.

https://golang.org/pkg/testing/#hdr-Benchmarks

So, you basically write a buf_test.go source file (the final _test is important), and write two Benchmark functions.

The first, let's call it BenchmarkSprintf, is defined as:

func BenchmarkSprintf(b *testing.B) {

    for i := 0; i < b.N; i++ {

        getReqAddonsSprintf()

    }

}

Where getReqAddonsSprintf is the function you originally posted.

The second, BenchmarkFprintf, is defined as:

func BenchmarkFprintf(b *testing.B) {

    for i := 0; i < b.N; i++ {

        getReqAddonsFprintf()

    }

}

Where getReqAddonsFprintf is the function defined in http://play.golang.org/p/8Jq_dPuwNF.

Here is a very simple benchmark example:

https://play.golang.org/p/JCCUxxemBc

Run it with:

go test -bench=. -benchmem buf_test.go

This will give you the conclusion.

Regards  Manlio

[Giulio Iotti]

On Sunday, November 1, 2015 at 9:53:15 PM UTC+2, Tong Sun wrote:

How best to construct bytes.Buffer gradually? with fmt.Fprint?

Having seen all the rest of the discussion: why do you need to create a full string from a buffer?

Usually it's more efficient to get a writer and write directly to it.

It's then the caller's responsibility to give you a buffered writer if necessary.

-- 

Giulio Iotti 

[Tong Sun]

On Mon, Nov 2, 2015 at 4:59 AM, Giulio Iotti <dullg...@gmail.com> wrote:
> On Sunday, November 1, 2015 at 9:53:15 PM UTC+2, Tong Sun wrote:
>>
>>
>> How best to construct bytes.Buffer gradually? with fmt.Fprint?
>
>
> Having seen all the rest of the discussion: why do you need to create a full
> string from a buffer?

FYI, this was the goal:

https://groups.google.com/d/msg/golang-nuts/LoUccTnv92M/FOUSXX9PDAAJ

I.e., "I need to write a program that writes either to a string buffer
or to a file, of which if file name is passed as "-", will write to
stdout instead"

All these questions are part of my architecture attempts.

> Usually it's more efficient to get a writer and write directly to it.
>
> It's then the caller's responsibility to give you a buffered writer if
> necessary.

Yep, I can say that as well *after* I understand all above.

[Dan Kortschak]

Where is your current working code? It would be easier to address issues
if a broader context were available.

[Dan Kortschak]

On Mon, 2015-11-02 at 09:54 -0500, Tong Sun wrote:

> I.e., "I need to write a program that writes either to a string buffer
> or to a file, of which if file name is passed as "-", will write to
> stdout instead"
>
> All these questions are part of my architecture attempts.
>

Given this, here is how I might go about it (caveats in comments and
note that this is based on no knowledge of the broader scheme you have
in mind): http://play.golang.org/p/dOtDw4IFK1

The case of writing to the buffer is handles by calling r.writeTo with a
*bytes.Buffer as the io.Writer. This allows the Buffer to be reused.