Proposal for implementation: format modifier for %v to print nested structs

[Johann Höchtl]

Rationale: During debugging, printing nested structs with fmt.Printf("%+v", data) would come handy.

[Alexey Palazhchenko]

Doesn't it work already? http://play.golang.org/p/FEO-mpmZ_T

–-–
Alexey "AlekSi" Palazhchenko

[Johann Höchtl]

Am Sonntag, 9. Dezember 2012 16:53:24 UTC+1 schrieb Alexey Palazhchenko:

Doesn't it work already? http://play.golang.org/p/FEO-mpmZ_T

Ok, what I realy meant are pointer values, your slightly modified goplay :
http://play.golang.org/p/jkUZFmHM8V

Not so useful.

–-–
Alexey "AlekSi" Palazhchenko

[Jesse McNelis]

On Mon, Dec 10, 2012 at 5:51 PM, Johann Höchtl <johann....@gmail.com> wrote:

Am Sonntag, 9. Dezember 2012 16:53:24 UTC+1 schrieb Alexey Palazhchenko:

Doesn't it work already? http://play.golang.org/p/FEO-mpmZ_T

Ok, what I realy meant are pointer values, your slightly modified goplay :
http://play.golang.org/p/jkUZFmHM8V

Not so useful.

What should fmt.Printf() do in the case of a self referential data structure? 

--
=====================
http://jessta.id.au

[Johann Höchtl]

At first step panic (once it will hit a recursion limit on the stack), or possibly at a later step, detect the cycle (as a garbage collector would too)

--
=====================
http://jessta.id.au

[Jan Mercl]

On Mon, Dec 10, 2012 at 7:51 AM, Johann Höchtl <johann....@gmail.com> wrote:
>
> Am Sonntag, 9. Dezember 2012 16:53:24 UTC+1 schrieb Alexey Palazhchenko:
>>
>> Doesn't it work already? http://play.golang.org/p/FEO-mpmZ_T
>>
> Ok, what I realy meant are pointer values, your slightly modified goplay :
> http://play.golang.org/p/jkUZFmHM8V
>
> Not so useful.

Actually very useful. Would the pointers be expanded then in the
general case of a graph formed by the (pointer) edges is not acyclic,
so what to print for:

type Node struct{ next *Node }
a := &Node(nil)
a.next = a
fmt.Println("%+v", *a)

?

-j

[Jan Mercl]

On Mon, Dec 10, 2012 at 8:02 AM, Johann Höchtl <johann....@gmail.com> wrote:
> At first step panic (once it will hit a recursion limit on the stack), or
> possibly at a later step, detect the cycle (as a garbage collector would
> too)

How to set the depth limit? 10? 100? There's no correct answer to
this. And detecting the cycle can use unbounded resources (possibly
bigger than the cyclic graph per se).

The status quo is reasonable, the desire to follow pointers in stdlib
printing is IMO not.

-j

[Alexey Palazhchenko]

What should fmt.Printf() do in the case of a self referential data structure? 

At first step panic (once it will hit a recursion limit on the stack), or possibly at a later step, detect the cycle (as a garbage collector would too)

It would be quite dangerous to program in Go if simple Printf function may panic. I think it should be different Dump method in other package.

–-–
Alexey "AlekSi" Palazhchenko

[Johann Höchtl]

&Node(0xdeadbeef){nil}

In my example it would print *string(0x12345678)("bar") so to retain the pointer information.

-j

[Johann Höchtl]

You are probably right, but existing packages, like serialization a cyclic struct to Json, now panic as well, and "%v" is used for primarily for debugging anyway.

–-–
Alexey "AlekSi" Palazhchenko

[roger peppe]

i'd like to see a deep pretty printer for Go data structures, descending
into pointers too. since it's for debugging purposes, it doesn't need
to be too efficient - to avoid infinite loops on cycles, it should keep
track of previously-seen pointers and print a reference to the previously
printed value if encounters one it's already seen.

i've done this before in another language and it wasn't hard - i've
been meaning to get around to it for Go, but haven't done it yet.
Go's interior pointers make things a little more awkward - it's
probably worth doing two passes through the data, once to work
out where things are located, and the other to print things out with
appropriate cross-references in place.

making the output format clear and concise is a good challenge.

On 9 December 2012 15:43, Johann Höchtl <johann....@gmail.com> wrote:
> Rationale: During debugging, printing nested structs with fmt.Printf("%+v",
> data) would come handy.
>

> --
>
>

[Ian Lance Taylor]

On Mon, Dec 10, 2012 at 6:14 AM, roger peppe <rogp...@gmail.com> wrote:
> i'd like to see a deep pretty printer for Go data structures, descending
> into pointers too. since it's for debugging purposes, it doesn't need
> to be too efficient - to avoid infinite loops on cycles, it should keep
> track of previously-seen pointers and print a reference to the previously
> printed value if encounters one it's already seen.
>
> i've done this before in another language and it wasn't hard - i've
> been meaning to get around to it for Go, but haven't done it yet.
> Go's interior pointers make things a little more awkward - it's
> probably worth doing two passes through the data, once to work
> out where things are located, and the other to print things out with
> appropriate cross-references in place.
>
> making the output format clear and concise is a good challenge.

I agree that this would be quite useful.

I'm less convinced that it should be in fmt.Printf. Either way it
seems to me that there should be a standalone function to do it. Once
we have that, we can consider whether to make it available as a format
specifier.

Ian

[minux]

On Mon, Dec 31, 2012 at 5:26 AM, Dave Collins <da...@davec.name> wrote:

On Monday, December 10, 2012 8:14:05 AM UTC-6, rog wrote:

i'd like to see a deep pretty printer for Go data structures, descending
into pointers too. since it's for debugging purposes, it doesn't need
to be too efficient - to avoid infinite loops on cycles, it should keep
track of previously-seen pointers and print a reference to the previously
printed value if encounters one it's already seen.

i've done this before in another language and it wasn't hard - i've
been meaning to get around to it for Go, but haven't done it yet.
Go's interior pointers make things a little more awkward - it's
probably worth doing two passes through the data, once to work
out where things are located, and the other to print things out with
appropriate cross-references in place.

making the output format clear and concise is a good challenge.

 
I have been working on a package that handles this as I find it useful as well.

One issue I've ran into though is that you can't invoke custom type Stringers on
unexported struct fields since you can't get an interface to them.
Unfortunately, that really hampers some of the usefulness of a deep pretty
printer for internal data structures while developing and debugging a package,
which is likely the main use case.

Here is some sample code demonstrating what I mean:

http://play.golang.org/p/bFAGv4M3Bf

Notice how the unexported field shows the raw uint8 value while the exported
field shows the pretty custom formatted version.  In the second printf, the
unexported field is passed directly and, naturally, is able to make use of the
Stringer interface.

Is there some trick I'm missing that would allow this capability?

you need to use unsafe.

0. optionally verify there is a Error/String() method on non-pointer receiver,

so that your debug package won't accidentally damage your user's data

1. dig out a pointer to that field (you might to dig it out by looking inside the

interface{} and utilizing structfield.Offset

2. use reflect.NewAt (can't handle unexported types), or better, just make an

interface{} value yourself by using unsafe.

3. cast it to Stringer and done.

somewhat involved procedure, but really there isn't a better alternative.

btw, i'd prefer the debug package to print out raw values, at least there should

be a mode to do this (ignoring String() and Error() methods).

[Dave Collins]

On Wednesday, January 2, 2013 2:03:07 AM UTC-6, minux wrote:

you need to use unsafe.

0. optionally verify there is a Error/String() method on non-pointer receiver,

so that your debug package won't accidentally damage your user's data

1. dig out a pointer to that field (you might to dig it out by looking inside the

interface{} and utilizing structfield.Offset

2. use reflect.NewAt (can't handle unexported types), or better, just make an

interface{} value yourself by using unsafe.

3. cast it to Stringer and done.

somewhat involved procedure, but really there isn't a better alternative.

btw, i'd prefer the debug package to print out raw values, at least there should

be a mode to do this (ignoring String() and Error() methods).

Thanks for the tips.  I was able to dig out a pointer as you suggested and get
it working.  I agree with you on making the invocation of Error/String optional
as sometimes you just want the raw value when debugging.

As a result of the pointer dance, I was also able to add the capability to
call Error/String methods on types bound to a pointer receiver when only a
value was provided.  I will, of course, make this optional since it could
theoretically mutate the data, though in practice Error and String shouldn't
be mutating data regardless of a pointer receiver.

After I get the code further along, I'll post some more details on how the
formatting looks for feedback.

[minux]

Great. Looking forward to your published package. I think it will be very useful. 

[Luis Alfonso Vega Garcia]

+1
I've been using JSON to debug nested structures with pointers.

-- Alfonso

Alfonso Vega-Garcia | Software Engineer | vegacom at gmail.com

> --
>
>

[Dave Collins]

I've finished and published the new package which handles this.  I've also written a blog entry about it here: https://blog.cyphertite.com/go-spew-a-journey-into-dumping-go-data-structures/

It is brand new, so I'm open to suggestions on any formatting issues or suggestions.

[Kevin Gillette]

Using text/tabwriter would be useful for scanning the output, so that types and values are aligned, not staggered.

[roger peppe]

On 9 January 2013 23:25, Dave Collins <da...@davec.name> wrote:
> I've finished and published the new package which handles this. I've also
> written a blog entry about it here:
> https://blog.cyphertite.com/go-spew-a-journey-into-dumping-go-data-structures/
>
> It is brand new, so I'm open to suggestions on any formatting issues or
> suggestions.

it's really nice to see this, thanks! i'll definitely be using it,
although i've only had a brief look so far, and that only at
the docs.

i definitely want to be able to choose custom formatting
options without disturbing the global Config variable, although
using the global for a default seems fine (it could work something
like the flag package in that respect).

it would be quite nice to be able to see when one slice aliases another,
although working out a good syntax to represent that is a challenge.

in general, i wonder if it would be nice to print any reference only
once - if there's a reference from one part of the data structure,
it could print the reference only.

i was surprised when %#v didn't print the types. i definitely want to have
the ability to see types.

finally, "spew" is a great name for this. instantly memorable!

[Miguel Pignatelli]

On 09/01/13 23:25, Dave Collins wrote:
> I've finished and published the new package which handles this. I've
> also written a blog entry about it here:
> https://blog.cyphertite.com/go-spew-a-journey-into-dumping-go-data-structures/
>
> It is brand new, so I'm open to suggestions on any formatting issues or
> suggestions.

This is very nice.
Could you (or Andrew) please link the article from
http://code.google.com/p/go-wiki/wiki/Articles
?

M;

[Dave Collins]

On Wednesday, January 9, 2013 5:34:24 PM UTC-6, Kevin Gillette wrote:

Using text/tabwriter would be useful for scanning the output, so that types and values are aligned, not staggered.

Thanks for the suggestion.  I played around with it a little bit, but ran into some issues with it not aligning things properly.   I'm going to create a ticket for it in the github issue tracker as I like the idea as an option.  I agree it would improve quickly scanning through large dumps.

[minux]

Done. 

[minux]

On Thu, Jan 10, 2013 at 7:25 AM, Dave Collins <da...@davec.name> wrote:

I've finished and published the new package which handles this.  I've also written a blog entry about it here: https://blog.cyphertite.com/go-spew-a-journey-into-dumping-go-data-structures/

Thank you!

I've only read the blog post. I think maybe you can write a follow-up blog post about implementation details

of some the non-obvious features.

[Dave Collins]

On Thursday, January 10, 2013 11:32:01 AM UTC-6, rog wrote:

i definitely want to  be able to choose custom formatting
options without disturbing the global Config variable, although
using the global for a default seems fine (it could work something
like the flag package in that respect).

it would be quite nice to be able to see when one slice aliases another,
although working out a good syntax to represent that is a challenge.

in general, i wonder if it would be nice to print any reference only
once - if there's a reference from one part of the data structure,
it could print the reference only.

i was surprised when %#v didn't print the types. i definitely want to have
the ability to see types.

finally, "spew" is a great name for this. instantly memorable!

 
Thanks for the feedback.  All great points on the current limitations.

The config per Formatter/Dumper instance and %#v are already on my agenda of things to implement.  I went with the global config as a first cut, but I wrapped it in a configState struct so it will be simple to apply individual config instances with fallback to the default config via pointers.

One of the reasons I didn't put %#v in the initial version is that, in the existing fmt package, %#v indicates Go syntax, but there really isn't a valid condensed Go syntax for declaring something like a double pointer to an int.  I was trying to decide between whether I should redefine the semantics of # when used with spew to simply add type information (this is likely what I will do barring objections), or instead use a different flag. The issue with a separate flag is that the fmt package doesn't pass on any flags which it doesn't already support to custom Formatters.

In regards to single references, I think that is a fine idea.  Currently, I only track the references down nested chains to detect cycles, but modifying it to support this should not be difficult.

I saved talking about slice aliases for last as I suspect it could be fairly tricky.  I haven't attempted to detect aliased slices before, but I assume I could simply check if the last element in the underlying array is at the same address.  The current code only does a single pass though, so it would likely have to do an initial pass to properly detect if an earlier slice is aliasing a later one. 

Syntax wise for denoting an aliased slice, perhaps showing the pointer address of the aliased slice along with the start and end positions (would have to figure out how to get these details)?  Something like (0x123123123)[start:end]int [4 5 6].

[Dave Collins]

On Thursday, January 10, 2013 11:32:01 AM UTC-6, rog wrote:

i definitely want to  be able to choose custom formatting
options without disturbing the global Config variable, although
using the global for a default seems fine (it could work something
like the flag package in that respect).

I've just implemented this feature.  Documentation and examples have been updated accordingly.

[Dan Kortschak]

This is great! I have just been playing around with it and it's going to
be very useful.

I have found a weird behaviour with nil pointers though. This snippet
illustrates the issue:

package main

import (
"github.com/davecgh/go-spew/spew"
)

type A struct {
P *A
}

func main() {
spew.Dump(A{})
spew.Dump(A{P: &A{}})
}

Output:
(main.A) {
P: (**main.A)()(<nil>)
}
(main.A) {
P: (*main.A)(0xf840043148)({
P: (**main.A)()(<nil>)
})
}

You can see that nil pointers are represented as though they are double
pointers. I can't see anywhere in the documentation that this is
expected.

Dan

[Dan Kortschak]

I've filed an issue and there is a patch that fixes this. Does not break
any tests, but I've not added tests for it. May come.

Dan

[Dave Collins]

Thanks for pointing this out.  I'll have this case fixed shortly and add tests for it.

[Dan Kortschak]

I was wondering why you have the ConfigState hidden behind a unexported
pointer field. It seems to me that you could get essentially the same
behaviour more simply (both go-spew code-wise and use-wise) with a
single type by fusing ConfigState and FuseState into a type (I guess
Spewer to fit the mould of the log package) which carries the
configuration and the methods. Is this just historical or is there
something I'm missing?

Dan

[Dan Kortschak]

This seems to be the answer. I had a closer look and it doesn't look
like you can get the default behaviour you have without doing what you
do.

[Dave Collins]

On Thursday, January 10, 2013 11:32:01 AM UTC-6, rog wrote:

i was surprised when %#v didn't print the types. i definitely want to have
the ability to see types.

I've now added %#v and %#+v support to the custom Formatter.  %#v adds type
information and the combination shows both type and pointer addresses.

This means it is possible to choose to display just the value with a
typeless indirection indicator (%v), the value with full pointer address
information (%+v), the value with type information (%#v), and the value with
type and full pointer address information (%#+v).

The following is an example of the output with the various flags:

%v: <**>5
%+v: <**>(0xf8400420d0->0xf8400420c8)5
%#v: (**uint8)5
%#+v: (**uint8)(0xf8400420d0->0xf8400420c8)5

There is also still the Dump method which displays using newlines and
customizable indentation.

[Dave Collins]

Kortschak,

Correct.  It was done that way to provide defaults that are not the same as the
zero value for the config struct.  However, considering your feedback and a
few others, I agree that the current syntax is a slightly more cumbersome than
it would be by just exposing the config struct with the methods on it directly.
So, I'm currently in the process of modifying it.

The new syntax will look like:

scs := spew.ConfigState{Indent: "\t"}
scs.Dump(whatever)

Of course, you will continue to be able to just use the default package-level
functions to use the defaults:

spew.Dump(whatever)

To cover the case of defaults on a new config instance, I'm thinking a separate
function like NewConfigState would get a new state populated with defaults.
That way the caller can either choose to just populate the struct directly as
above or use the function to get a state populated with defaults and modify it
from there.

Is this more in line with what you had in mind?

[Dan Kortschak]

Yes, I think so, but I'd need to see the implementation to be sure. Having said that, spew was extremely helpful in its current state for me last week.

Dan

--