Possible flag to enable better debugging

[David Chase]

We discussed this today and decided that we needed wider input.

https://go-review.googlesource.com/c/41770 supplies better debugging information for optimized code, so that users can (in theory) observe variables even when they've been allocated to registers.  Technically it meets the rules for inclusion into 1.9, if we can finish a problematic merge soon.

Unfortunately, right now it also slows down compilebench by 50%.  That might be a problem for some people.

We could put this off till 1.10, but:

- it's a large CL and will certainly go stale if we don't have tests checked in to help keep it alive;

- it's a prerequisite for allocating function/method arguments into registers, which we'd like to do for 1.10 but that's its own large job;

- we'd like to work with Delve and any other debugger developers to see if we can make this really work so we can run what we debug and debug what we run (and debug what we crash-dump);

- it's almost certain that we need user feedback.

- it's unclear (unlikely?) that it will run much faster then, in which case we'll have the very same problem.

So, since we're sure that a 50% slowdown in compile time is not likely to be popular but still want to expose the possibility to many people, we'd like to guard it with a flag.

Happily, the traditional "-g" (go build/install -gcflags -g ... ) is available for use.

That lets us write tests easily even if it's not on by default, allows debugger developers to play with it easily, and provides a low barrier to any users who want to play with it.

The problem with adding this flag is what we do a few releases down the road.  If we decide this is such a great idea that it should be on by default always, we've grown a semi-useless flag.  One option in that future is to change the -g=0 meaning of the flag to "don't emit debugging information (at all)".  Another is to reclaim the flag.

So, opinions? Suggestions?

[Brad Fitzpatrick]

I like the idea of submitting the code now but having it disabled for Go 1.9.

But I see no reason we need to use "-g". It could be something more verbose and ugly, and we can change the default later and remove the flag entirely in a few releases. We have no go1compat rules there.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Chase]

Can you provide an example of "more verbose and ugly"?

Would something as mild as 'gcflags -_g' be adequate?

[Josh Bleecher Snyder]

> https://go-review.googlesource.com/c/41770 supplies better debugging
> information for optimized code, so that users can (in theory) observe
> variables even when they've been allocated to registers. Technically it
> meets the rules for inclusion into 1.9, if we can finish a problematic merge
> soon.

It might technically qualify for 1.9, but it's a fairly substantive
change, and we're pretty close to the first beta.

> Unfortunately, right now it also slows down compilebench by 50%. That might
> be a problem for some people.

I'd love to see better debug data, but 50% is really a lot. For
context, that would undo well over a years' worth of collective work
to improve compile times.

It also appears to double binary sizes, which is also a pretty big
deal. DWARF compression might help with that, but that's yet another
big change to add during the freeze.

Also, have we tested thoroughly to make sure that it doesn't alter any
generated code, just DWARF?

Also, I'd like to take a close look from a concurrency-safety
perspective, and my time is fairly limited this week.

> We could put this off till 1.10, but:
>
> - it's a large CL and will certainly go stale if we don't have tests checked
> in to help keep it alive;

There shouldn't be many changes between now and release. If it's a
major concern and we want to continue to work on it during the freeze,
maybe a dev branch is called for.

> - it's a prerequisite for allocating function/method arguments into
> registers, which we'd like to do for 1.10 but that's its own large job;
> - we'd like to work with Delve and any other debugger developers to see if
> we can make this really work so we can run what we debug and debug what we
> run (and debug what we crash-dump);

Again, maybe these needs could be addressed with a dev branch.

> - it's almost certain that we need user feedback.
> - it's unclear (unlikely?) that it will run much faster then, in which case
> we'll have the very same problem.

Maybe. Although we'd have a lot more time to make sure. And add DWARF
compression. And fix whatever other downstream issue crop us. E.g.
does the debug/dwarf package work with DWARF 4? Should it, if we emit
DWARF 4? And I'm sure there'll be more such items.

> So, since we're sure that a 50% slowdown in compile time is not likely to be
> popular but still want to expose the possibility to many people, we'd like
> to guard it with a flag.
> Happily, the traditional "-g" (go build/install -gcflags -g ... ) is
> available for use.
> That lets us write tests easily even if it's not on by default, allows
> debugger developers to play with it easily, and provides a low barrier to
> any users who want to play with it.

An alternative here is to change the -dwarf flag from a bool to a
string, and allow values like "off", "default", and "full". The -dwarf
flag was added during the 1.9 cycle, so it is still open for easy
change. This should make changing defaults etc easier.

We would also want to consider whether the DWARF scopes work should
also be guarded by such a flag, instead of "-N -l".

> The problem with adding this flag is what we do a few releases down the
> road. If we decide this is such a great idea that it should be on by
> default always, we've grown a semi-useless flag. One option in that future
> is to change the -g=0 meaning of the flag to "don't emit debugging
> information (at all)". Another is to reclaim the flag.

Compiler flags aren't sacrosanct.


The one advantage not mentioned that I see to putting this in 1.9 is
that 1.9 should have significant compilation speed improvements, so
from a user perspective, there won't be as big a net slowdown. IIRC we
did this with write barriers when they were first introduced. But on
balance, I think putting it in now would be a mistake.

-josh

[Brad Fitzpatrick]

For example, in Go 1.5 we had an environment variable named GO15VENDOREXPERIMENT. In Go 1.9 we have GO19CONCURRENTCOMPILATION.

Both are super ugly names, but both say what they're about.

This is not me making a statement on environment variables vs. flags. I'm merely pointing out examples of effective yet ugly names.

[Derek Parker]

> - we'd like to work with Delve and any other debugger developers to see if we can make this really work so we can run what we debug and debug what we run (and debug what we crash-dump);

Happy to help with this in any way we (Delve team) can. 

[Florin Pățan]

Hi,

I can only speak on a few things on the list based on my limited experience in this matter, so please accept my input as it is.

 

It might technically qualify for 1.9, but it's a fairly substantive
change, and we're pretty close to the first beta.

If it doesn't affect any code path / production, speed etc when this is disabled then having it in the code and available via env vars, like Brad suggested, would be welcomed.

 

 

I'd love to see better debug data, but 50% is really a lot. For
context, that would undo well over a years' worth of collective work
to improve compile times.

Generally when you debug applications, slower compilation or execution is acceptable.

I understand that a lot of hard goes in every release to make things compile faster, but I believe that given the chance to vote, a majority of users would vote even for a 100% slow-down if it helps them avoid the tedious printf debugging in complex applications / workflows. Personally I would vote for it.

 

It also appears to double binary sizes, which is also a pretty big
deal. DWARF compression might help with that, but that's yet another
big change to add during the freeze.

Again, not a problem, at least not for servers and such. Might be an issue for devices with limited resources but having to trade-off debugging binary size for a better debugging experience would be a good deal.

The one advantage not mentioned that I see to putting this in 1.9 is
that 1.9 should have significant compilation speed improvements, so
from a user perspective, there won't be as big a net slowdown. IIRC we
did this with write barriers when they were first introduced. But on
balance, I think putting it in now would be a mistake. 

 

-josh

Another advantage would be that people could use it, report any issues, both with it and with the concurrent backend enabled and this would potentially lead to a lot more help from the community to catch the said issues. I think everyone will understand that experimental is experimental and I think it would be welcomed by gophers to see a move in this direction, with more experimental features being available (especially if it improves their quality of life when debugging).

Thank you for your time and all the great work everyone is putting into Go on each release.

Kind regards,

Florin 

[ron minnich]

We use on-demand compilation in u-root, and we were really happy to see the on-demand compilation performance of go 1.9 in u-root get back to go 1.2-like speed, so would be sorry to see this kind of slow-down. That said, we're a pretty unimportant user of Go, or at least maybe the only project that cares about compile speed, so I guess we'd just have to manage. But an environment variable or something to control it would be nice.

ron

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Josh Bleecher Snyder]

>> It might technically qualify for 1.9, but it's a fairly substantive
>> change, and we're pretty close to the first beta.
>
>
> If it doesn't affect any code path / production, speed etc when this is
> disabled then having it in the code and available via env vars, like Brad
> suggested, would be welcomed.

I started looking at the CL this morning and plan to continue to over
the coming week.

To be clear, I have several independent concerns here:

* Does it alter generated code when disabled? Flag integration hasn't
happened yet, so we don't know, but I suspect it's going to take some
work to get there. I'll feel a lot more comfortable with it in 1.9
once it passes toolstash-check when disabled. In my opinion, getting
to that point is the next key step.

* Does it substantively alter toolchain speed when disabled? It should
not, but I'd like to confirm.

* The CL is non-trivial and (of necessity) not very self-contained,
which increases risk and the probability of downstream fixes to
prepare for 1.9, each of which comes with the risk of impacting
non-DWARF code. Ultimately this is a judgment call, and I won't stand
in the way (and will contribute to getting it into good shape through
code reviews!), but it does feel large and late to me.

>> I'd love to see better debug data, but 50% is really a lot. For
>> context, that would undo well over a years' worth of collective work
>> to improve compile times.
>
>
> Generally when you debug applications, slower compilation or execution is
> acceptable.
> I understand that a lot of hard goes in every release to make things compile
> faster, but I believe that given the chance to vote, a majority of users
> would vote even for a 100% slow-down if it helps them avoid the tedious
> printf debugging in complex applications / workflows. Personally I would
> vote for it.

I've heard similar sentiments voiced both for and against, on
basically every front (compile time, generated code quality, binary
size, debugging, memory usage, simplicity, reliability, etc.). I'd be
very interested in a survey-based trade-off analysis about what
gophers value, but with respect, the anecdotal evidence is all over
the map.


Back to compiler flags, which I think we need to sort out regardless,
here's a new proposal:

-g is very appealing for this: short, familiar. I suggest that we make
-g an int flag. Like -l, its default value will be 1, a single -g sets
it to 0, and subsequent -g's count up starting at 2. -g == 0 means
disable DWARF, and eliminate the new -dwarf flag from 1.9 in favor of
plain -g. This guarantees that -g will continue to be useful on an
ongoing basis. -g == 1 means status quo. -g == 2 means include scopes,
and stop using -N -l as a proxy for that. -g == 3 means everything in
CL 41770. Document -g loosely, so that we can tweak what the levels
mean over time.

We can fruitfully do this whether or not CL 41770 goes into 1.9, and
probably should.

Opinions?

-josh

[hes...@google.com]

On Wednesday, May 24, 2017 at 3:24:23 PM UTC-4, Josh Bleecher Snyder wrote:

>> It might technically qualify for 1.9, but it's a fairly substantive
>> change, and we're pretty close to the first beta.
>
>
> If it doesn't affect any code path / production, speed etc when this is
> disabled then having it in the code and available via env vars, like Brad
> suggested, would be welcomed.

I started looking at the CL this morning and plan to continue to over
the coming week.

To be clear, I have several independent concerns here:

* Does it alter generated code when disabled? Flag integration hasn't
happened yet, so we don't know, but I suspect it's going to take some
work to get there. I'll feel a lot more comfortable with it in 1.9
once it passes toolstash-check when disabled. In my opinion, getting
to that point is the next key step.

You're right. I was expecting there to be no impact since it happens so late, but given that you were able to spot a problem with trim by inspection, clearly I need to spend some time testing. 

* Does it substantively alter toolchain speed when disabled? It should
not, but I'd like to confirm.

I'll do that testing at the same time. 

* The CL is non-trivial and (of necessity) not very self-contained,
which increases risk and the probability of downstream fixes to
prepare for 1.9, each of which comes with the risk of impacting
non-DWARF code. Ultimately this is a judgment call, and I won't stand
in the way (and will contribute to getting it into good shape through
code reviews!), but it does feel large and late to me.

I believe there will already be some very small changes necessary to x/debug/dwarf for the upgrade to DWARF3 which landed with the scopes change. The update to DWARF4 will be similar to that. For what it's worth, I agree that it's late. But we seem to be converging on having it disabled by default, and as long as that works I think it'll be okay.

>> I'd love to see better debug data, but 50% is really a lot. For
>> context, that would undo well over a years' worth of collective work
>> to improve compile times. 

>
> Generally when you debug applications, slower compilation or execution is
> acceptable.
> I understand that a lot of hard goes in every release to make things compile
> faster, but I believe that given the chance to vote, a majority of users
> would vote even for a 100% slow-down if it helps them avoid the tedious
> printf debugging in complex applications / workflows. Personally I would
> vote for it.  

I've heard similar sentiments voiced both for and against, on
basically every front (compile time, generated code quality, binary
size, debugging, memory usage, simplicity, reliability, etc.). I'd be
very interested in a survey-based trade-off analysis about what
gophers value, but with respect, the anecdotal evidence is all over
the map. 

Also, just to be clear, the CL specifically targets debugging with optimizations *on*. If you're willing to recompile your program with -N -l, as Delve does for you automatically, the debugging experience should be pretty good today and even better in 1.9 thanks to Alessandro. 

Back to compiler flags, which I think we need to sort out regardless,
here's a new proposal:

-g is very appealing for this: short, familiar. I suggest that we make
-g an int flag. Like -l, its default value will be 1, a single -g sets
it to 0, and subsequent -g's count up starting at 2. -g == 0 means
disable DWARF, and eliminate the new -dwarf flag from 1.9 in favor of
plain -g. This guarantees that -g will continue to be useful on an
ongoing basis. -g == 1 means status quo. -g == 2 means include scopes,
and stop using -N -l as a proxy for that. -g == 3 means everything in
CL 41770. Document -g loosely, so that we can tweak what the levels
mean over time.

It's not totally clear to me what good -g1 would be with optimizations on; it's no better than today, which isn't great.  Similarly, -g3 wouldn't mean much with optimizations off. My suggestion would be: -g1 means only symbols and line numbers -- good backtraces, basically. -g2 means variables and scopes, with whatever complexity is necessary, with the goal of making interactive debugging good. Maybe -N -l would imply -g2 to keep Delve working as it does today, maybe not.

[Austin Clements]

On Wed, May 24, 2017 at 4:42 PM, heschi via golang-dev <golan...@googlegroups.com> wrote:

On Wednesday, May 24, 2017 at 3:24:23 PM UTC-4, Josh Bleecher Snyder wrote:

Back to compiler flags, which I think we need to sort out regardless, 

here's a new proposal:

-g is very appealing for this: short, familiar. I suggest that we make
-g an int flag. Like -l, its default value will be 1, a single -g sets
it to 0, and subsequent -g's count up starting at 2. -g == 0 means
disable DWARF, and eliminate the new -dwarf flag from 1.9 in favor of

I don't think this is quite what you mean. Making a single "-g" disable debug info is not "familiar". :)

plain -g. This guarantees that -g will continue to be useful on an
ongoing basis. -g == 1 means status quo. -g == 2 means include scopes,
and stop using -N -l as a proxy for that. -g == 3 means everything in
CL 41770. Document -g loosely, so that we can tweak what the levels
mean over time.

It's not totally clear to me what good -g1 would be with optimizations on; it's no better than today, which isn't great.  Similarly, -g3 wouldn't mean much with optimizations off. My suggestion would be: -g1 means only symbols and line numbers -- good backtraces, basically. -g2 means variables and scopes, with whatever complexity is necessary, with the goal of making interactive debugging good. Maybe -N -l would imply -g2 to keep Delve working as it does today, maybe not.

The ultimate goal should be that whatever the default setting is produces enough debug info to make interactive debugging good. Ideally, the higher levels won't even exist in the long run. Of course, we could accomplish that by either changing the meaning of -g=1 over time, or by changing the default level so you can opt for lower levels.

[Derek Parker]

> The ultimate goal should be that whatever the default setting is produces enough debug info to make interactive debugging good.

This is ideal, especially as this is targeting better handling of optimized binaries, which are produced by default, the debug information should strive to match that. That way the expectation becomes that given a standard binary (without optimizations turned off, etc) still results in a positive debugging experience.

[oju...@gmail.com]

Speed is a hallmark of Go. Go compiles fast and runs fast. I am sure the gains in 1.9 will be much appreciated.

Better debugging information is very welcome, of course, but please don't slow down the default compiler configuration. If someone wants to debug a program, despite a slower compilation, let him opt in for that particular task only.

Don't forget we have a lot of tools and workflows that expect instant compilation, like "go run", "go test", etc.

[alessandr...@gmail.com]

On Wednesday, May 24, 2017 at 10:44:06 PM UTC+2, Heschi Kreinick wrote:

I believe there will already be some very small changes necessary to x/debug/dwarf for the upgrade to DWARF3 which landed with the scopes change.

We were actually thinking of switching back to debug/dwarf to avoid backporting stuff to x/debug/dwarf, and only keep a custom x/debug/dwarf/type.go around in a package.

 

Alessandro Arzilli.

[Josh Bleecher Snyder]

>> * Does it substantively alter toolchain speed when disabled? It should
>> not, but I'd like to confirm.
>
> I'll do that testing at the same time.

Great. If you've not found it already, github.com/josharian/compilecmp
might be helpful. I usually run it twice, once for a fast run to see
alloc + file size changes, which are generally very stable, and once
with memory profiling disabled to get better cpu timing:

compilecmp -n 5 -obj
compilecmp -n 50 -cpu

One subtle thing to watch for here is that creating a bunch of new
ssa.Values will likely slow down everything in a hard-to-pinpoint way,
since lots of things in the backend are O(#values).

>> * The CL is non-trivial and (of necessity) not very self-contained,
>> which increases risk and the probability of downstream fixes to
>> prepare for 1.9, each of which comes with the risk of impacting
>> non-DWARF code. Ultimately this is a judgment call, and I won't stand
>> in the way (and will contribute to getting it into good shape through
>> code reviews!), but it does feel large and late to me.
>
> I believe there will already be some very small changes necessary to
> x/debug/dwarf for the upgrade to DWARF3 which landed with the scopes change.
> The update to DWARF4 will be similar to that. For what it's worth, I agree
> that it's late. But we seem to be converging on having it disabled by
> default, and as long as that works I think it'll be okay.

Yep. And x/debug/dwarf isn't tied to a release cycle. We can continue
discussing the details on the CL.

> I don't think this is quite what you mean. Making a single "-g" disable debug info is not "familiar". :)

Touché. :)

Adapted proposal: default is -g=1. Additional -g flags count up from
there. Disabling DWARF can be done by setting g to zero explicitly:
-gcflags="-g=0".

> The ultimate goal should be that whatever the default setting is produces enough debug info to make interactive debugging good. Ideally, the higher levels won't even exist in the long run. Of course, we could accomplish that by either changing the meaning of -g=1 over time, or by changing the default level so you can opt for lower levels.

Absolutely agreed. It is similar to -l in this regard. Having -g just
gives us the means to do this gradually, to add experimental/expensive
features and get feedback on them, etcl.


-josh

[David Chase]

On Thu, May 25, 2017 at 10:26 AM, Josh Bleecher Snyder <josh...@gmail.com> wrote:

> I don't think this is quite what you mean. Making a single "-g" disable debug info is not "familiar". :)

Touché. :)

Adapted proposal: default is -g=1. Additional -g flags count up from
there. Disabling DWARF can be done by setting g to zero explicitly:
-gcflags="-g=0".

To be clear, this proposes (?):

default is -g=1

single "-g" implies -g=1

-g=0 implies (or will, in the future) skip unnecessary(*) debug info.

"-g -g" = "-g=2" = more debugging info (Heschi's CL, for now?)

This is not quite like -l, since default is "normal inlining" and "-l" is no inlining.

But I think it is more important to behave like other language compilers here

(I see no reason not to, other than paralleling use of a not-that-frequently-used

gc flag for inlining.)

(*) This is not an editorial remark about debug info. There may be necessary debug info.

[Josh Bleecher Snyder]

>> > I don't think this is quite what you mean. Making a single "-g" disable
>> > debug info is not "familiar". :)
>>
>> Touché. :)
>>
>> Adapted proposal: default is -g=1. Additional -g flags count up from
>> there. Disabling DWARF can be done by setting g to zero explicitly:
>> -gcflags="-g=0".
>
>
> To be clear, this proposes (?):
> default is -g=1
> single "-g" implies -g=1
> -g=0 implies (or will, in the future) skip unnecessary(*) debug info.
> "-g -g" = "-g=2" = more debugging info (Heschi's CL, for now?)
>
> This is not quite like -l, since default is "normal inlining" and "-l" is no
> inlining.
> But I think it is more important to behave like other language compilers
> here
> (I see no reason not to, other than paralleling use of a
> not-that-frequently-used
> gc flag for inlining.)
>
> (*) This is not an editorial remark about debug info. There may be necessary
> debug info.

Almost.

default is -g=1
single -g implies -g=2
-g=0 implies skip all debug info, to make compilation faster; it
replaces the existing (but new in 1.9) -dwarf=false compiler flag
-g -g implies -g=3, and so on

Yes, it is no longer like -l. That's fine. It's probably not exactly
like other compilers, and that's also fine.

I had in mind that (for now) -g=2 means add scopes and -g=3 means also
add Heschi's CL, but I'm not particular; -g=2 could do both if y'all
preferred.

The reason for -g=0 is that there are cases in which we know that we
don't need DWARF, and skipping it entirely speeds up compilation. See
CLs 40857, 40858, 40859, and 40865.

-josh

[Russ Cox]

I don't want ordinary users to be making choices about "how much debug info is enough". It's fine to have "no debug info" vs "debug info" but let's not introduce finer gradations, at least not in a way that ordinary users are expected to use (and that includes teaching delve about it). I think it's fine to add -dwarf / -dwarf=false to turn off generation entirely if you know you're linking with -w. (Dwarf used to be entirely done by the linker or else we would have introduced a compiler flag when we added the linker's -w.) But please let's not start adding -g=1 is basic, -g=2 is advanced etc, all in the name of faster builds. The way to speed is not by adding this kind of complexity. That kind of complexity is, if anything, an indication that we didn't finish the job of making what's useful also fast enough.

If the new dwarf code is suspected to be incorrect or you want to make it available in an unsupported form for early testing as part of Go 1.9, then it would be fine to only enable with with a compiler-debugging flag that is obviously not for general use, like -d=unstabledwarf, until it's ready for prime time. But once it is ready for prime time (say, in Go 1.10), it should just be the default, not something you have to opt in to. And part of being ready for prime time is being efficient enough to generate.

Thanks.

Russ

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.