Dynamic analysis tool for Go

[Dmitry Vyukov]

Hi Rob,

From http://gophercon.sourcegraph.com:

Q: If you could add one thing to the Go stdlib, what would it be?
A from Rob: We have a lot of good tools for static analysis. I’d like
to see a good tool for dynamic analysis. I haven’t seen any languages
with the kind of tool I want.

What tool do you want?

[Rob Pike]

Ha ha. I don't know yet, but I know I don't want a tool, I want a
toolkit that can examine running programs and answer questions about
them. Questions I don't know until the problem arises (that's why I
want a toolkit not a tool).

-rob

[Michael Jones]

How about a framework that when enabled for an executable at compile time, changes the code of function entry, exit, optimization, etc. such that robust and details adb/gdb work may be done with an executable using only a high-level API and RPCs or a socket of some kind.

My friend Charles Simonyi did something akin to this for Microsoft Word and Excel. There was a secret "other program" in them, that can be enabled, and when enabled, allows the developer to inspect data structures, trace program execution, monitor resource usage, and some amount of instruction-level GDB like activity.

I think this could be a cool thing in general. Most debugging/tracing tools approach the task without assistance at compile time other than "-O0." What if it was a collaboration? (Or what if, like Rob's program-rewriting line counter, it was done as a source rewrite rather than a compile change.)

--

---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones | Chief Technology Advocate  | m...@google.com |  +1 650-335-5765

[Jan Mercl]

On Sat, Apr 26, 2014 at 4:22 PM, Rob Pike <r...@golang.org> wrote:
> Ha ha. I don't know yet, but I know I don't want a tool, I want a
> toolkit that can examine running programs and answer questions about
> them. Questions I don't know until the problem arises (that's why I
> want a toolkit not a tool).

One thing I'm sometimes missing are exact per-type allocted/live/GC'ed
statistics. In some other language it was easy to hook in the
alloc/free and/or object.AfterInitialize/BeforeDestruction routines to
get this info in a few dozen of lines even when this particular
support was not there. (FPC I'm talking about, BTW).

-j

[Rob Pike]

Something like that, yes, although I'd like not to _require_
compilation-time changes. I'd like to come to an arbitrary program
while it's running.

-rob

[Rob Pike]

Also, on the word 'collaboration', I think the interesting
collaboration is not with the compiler but with the runtime.

-rob

[Gustavo Niemeyer]

When Rob said that it reminded me of Chrome's infrastructure for
digging into the JavaScript runtime. It would be really helpful to
have something along those lines available.


gustavo @ http://niemeyer.net

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

gustavo @ http://niemeyer.net

[Aram Hăvărneanu]

FWIW I frequently use DTrace to answer arbitrary questions about the
runtime in arbitrary conditions. A complication is that the non-C ABI
of Go binaries makes its use clunky, but I do plan to address this in
1.4 by adding CTF information and a DTrace provider in the binaries,
so that it will work well, by default.

Even in this limited and hard to use form, DTrace has replaced 97.5%
of my debugger usage.

PS: The DTrace language resembles awk and is really easy to use.

--
Aram Hăvărneanu

[Michael Jones]

On the Mac, the XCode tool Instruments is built on DTrace. Very nice visual interface to it.

--

---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[Rob Pike]

I don't want a UI, I want a programming interface above which I can
build whatever is needed, either generally-useful tools or ad hoc
probes. And the programming language I want to use for this is Go.

-rob

[Michael Jones]

I never doubted that. But if Aram loves DTrace, then he should know how beautifully Apple's freeware allows developer interaction with it.

[rh]

In your mind, would this look like Acid, for Go?

[mortdeus]

Im currently working on extensions to the gc toolchain and runtime that will emulate a virtual plan9 filesystem (somewhat similar to inferno in concept) at $GOROOT. ( https://code.google.com/p/chaos/ )

Part of my plan involves implementing a go specific procfs, which has proc fids that are generated dynamically by each running go process' runtime that was built against the new virtual $GOOS platform.

How do you guys feel about goprocfs as a dynamic interface into an arbitrary go process' runtime?     

[Dan Kortschak]

Aram, would you be prepared to write a short post - perhaps on the wiki - explaining how to use DTrace with Go programs? (What things you need to do to get it to play with Go binaries).

Dan

[Rob Pike]

I second that!

-rob

[andrey mirtchovski]

Please share. I have only been able to get segfaults trying to use
dtrace with Go programs. I do use dtrace daily, it would be invaluable
to have it work with Go binaries.

[mortdeus]

On Saturday, April 26, 2014 4:59:19 PM UTC-5, Rob Pike wrote:

I second that!

-rob

Are you praising my goprocfs idea, or Dan's request for Aram to write a dtrace-go doc? 

[Rob Pike]

I'm excited to see Aram's document.

-rob

[Sebastien Douche]

On Sat, Apr 26, 2014 at 11:55 AM, 'Dmitry Vyukov <dvy...@google.com>'
via golang-dev <golan...@googlegroups.com> wrote:
> Q: If you could add one thing to the Go stdlib, what would it be?
> A from Rob: We have a lot of good tools for static analysis.

But not (yet?) in the default distribution (vet, lint, oracle...).


--
Sebastien Douche <sdo...@gmail.com>
Twitter: @sdouche / G+: +sdouche

[Aram Hăvărneanu]

I'll write something up.

--
Aram Hăvărneanu

[Dmitry Vyukov]

Uh... fine-grained dynamic analysis w/o program recompilation is
somewhat above current state of the art for dynamic analysis.

What type of questions are you interested in?

The race detector is some kind of such toolkit -- it allows you to
execute some code on every function entry, exit and before every
memory accesses. I've built a Valgrind/memecheck-like tool on top of
it. But it requires recompilation.

It's possible to insert lots of NOPs into code, and then dynamically
patch it. This will bloat code size, and thus potentially decrease
execution speed.

It's possible to compile 2 versions of each function (normal and
instrumented) and then jump from the normal one to the instrumented
one depending on a global flag. This will cause 2.5x code bloat.

There is also dynamic binary instrumentation. But it usually has
astonishing complexity and fragility.

Moreover all these approaches can only answer questions about future,
e.g. tell me who will modify this variable.
Answering questions about program past, e.g. now that I got this
panic, tell me who modified this variable, requires memorization of
astonishing amount of information. So it can't run on live
applications and is usually specialized to answer very particular type
of questions.

[Aram Hăvărneanu]

> Uh... fine-grained dynamic analysis w/o program recompilation is
> somewhat above current state of the art for dynamic analysis.

I disagree.

> It's possible to insert lots of NOPs into code, and then dynamically
> patch it. This will bloat code size, and thus potentially decrease
> execution speed.

You don't have to insert NOPs. DTrace patches the code, and when
it traps it will emulate what it was overwritten. Very similar to
floating point emulation in the kernel. Programs are slowed down
only when traced, and in practice the overhead is very small,
fractions of a percent for usual scripts. I regularly use DTrace
to debug time-sensitive races. Even if you trace every function in
the program, the time overhead is less than 2x.

> Moreover all these approaches can only answer questions about future,
> e.g. tell me who will modify this variable. Answering questions
> about program past, e.g. now that I got this panic, tell me who
> modified this variable, requires memorization of astonishing amount
> of information.

That's why you can do speculative tracing, which is the biggest
architectural difference between DTrace and perf/ktap. You can
record a huge amount of data, and at some point in the future you
decide if you really need it or not. If you don't need it (e.g. the
code didn't exhibit the behavior you are interested in), the data
is discarded. It's not magic, you can't just ask "who modified this
variable in the past?", but you can tell it to record that information
and the volume of information is in practice manageable.

> So it can't run on live applications and is usually specialized to
> answer very particular type of questions.

Very much disagree, I run it on live production binaries to answer
arbitrary types of questions. For Go, it's not quite there yet;
e.g. for C I can just inspect any data using C-like sytax. I have
full access to types and everything. I can easily walk trees and
lists. For Go all the functionality is there, I can access the data,
but it's raw data. I have to a priory generate struct offsets and
write casts by hand. It's very ugly, that's why I want to make it
on par with C.

I'll try to find some time to write about this.

[Ingo Oeser]

On Sunday, April 27, 2014 12:37:56 PM UTC+2, Dmitry Vyukov wrote:

Moreover all these approaches can only answer questions about future,
e.g. tell me who will modify this variable.
Answering questions about program past, e.g. now that I got this
panic, tell me who modified this variable, requires memorization of
astonishing amount of information. So it can't run on live
applications and is usually specialized to answer very particular type
of questions.

Theoretically GDB can also go backwards from the panic.

https://www.sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution.html

But that kind of thing might be easier from a core dump. 

[Kevin Gillette]

On Sunday, April 27, 2014 8:14:53 AM UTC-6, Ingo Oeser wrote:

Theoretically GDB can also go backwards from the panic.

https://www.sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution.html

In some cases, computations can be inverted, but in many cases they can't, and stack frames would need to be memorized to achieve the effect. For example, you can't reverse compute a SHA1 hash back into its original input without storing the original input. 

[Dmitry Vyukov]

Can it actually reliably go far backwards in a concurrent program? I would be very surprised...

[Dmitry Vyukov]

I think we are talking about somewhat different things.
You call it "It's not magic", and such magic would be what I called
"above current state of the art".
If you have an episodically and unreprodicibly panicing program, it's
useful to attach to it at that point and ask "who had modified this
variable? what values did it have in the past?".

Can dtrace intercept all memory accesses? Such dynamic disassembling
is generally unsolvable problem (at least on x86), because it's not
possible to find instruction boundaries (and strictly saying you can
have overlapping instruction streams).

Note that to track accesses to a single variable, you need to
intercept all memory accesses in the program.

I think it's still fragile. You will discover that the dynamic
instrumentation does not support a new instruction only after release
cut when somebody tries to insert a particular type of instrumentation
into a particular place.

I think that the dynamic trace collection that you described is still
not feasible for all Go programs all the time.

[Ian Lance Taylor]

It's said to be pretty reliable, though I have not used it myself. It
basically records every instruction as it is executed. It's not full
speed.

https://sourceware.org/gdb/news/reversible.html
https://sourceware.org/gdb/wiki/ProcessRecord

Ian

[Michael Hudson-Doyle]

The figure for "not full speed" I heard was 1000x slower :-)

Cheers,
mwh

[Dmitry Vyukov]

Ah, if it records all instructions and the slowdown is around 1000x,
then I can believe it.

[2pau...@googlemail.com]

I've been keeping this thread in the back of my head. Aside from the deeper technical issues, which I think are far more non-trivial than some people think, there also appear to be open questions about licensing issues regarding Dtrace.  Ianal but from what I have read Dtrace is not compatible with the GPL by intention. That fact is suitable to raises some concerns and also to cause uncertainty. Given the source of that computer program and the total history of that vendors opensource double strategy, frankly I would not be inclined to touch anything coming from there with a 10 foot pole. But thats just me. 

[Aram Hăvărneanu]

Neither GPL nor Linux is the center of the universe. There is actually
non-Linux software out there. Who would have thought.

--
Aram Hăvărneanu