[RFC] Dynamically instrumentation Go binaries

[ju...@sqreen.io]

Hi,

I am working on dynamic instrumentation of Go programs at run time, possibly without static source-code instrumentation. As I would like a solution as close to Go and standard as possible, I was first thinking of using `go generate` to generate a file adding things `reflect` doesn't provide such as the list of packages, functions, global variables... That way, I should be able to use `reflect` to modify any dynamic calls by modifying the method tables of their underlying type representations.

But regarding statically linked calls, the less intrusive technique I found are uprobes, which is linux-specific. And at the opposite, there are user-space binary code instrumentation libraries such as dyninst that modify the code at run time...

So I am wondering if anyone here has any thoughts on this subject, that doesn't seem to be solved for Go programs.

Thanks!

Julio

[Robert Engels]

Go is statically compiled. Pretty sure you cant do dynamic instrumentation - like logging method entry and exit - much of this is even lost due to inlining. You could possibly have an instrumented stdlib and dynamically link to that, but that is still not dynamic instrumentation. The profiler works by using stack samples. 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steven Hartland]

dtrace support?

[Robert Engels]

AFAIK dtrace support is compiled in. If not enabled there are some tricks to make is essentially zero cost, but it is still there from the start. Could be wrong but that’s my understanding. 

Sent from my iPhone

[Michael Jones]

You could study Rob Pike’s coverage/profiling tool to see how he adds and exploits basic block counting.  Good work as always. 

Michael T. Jones
michae...@gmail.com

[robert engels]

Pike’s “coverage" uses source code rewriting and recompiling, so it is not dynamic instrumentation - at least how I would define it. Dynamic instrumentation to me means to instrument at runtime on an existing binary/distribution.

[Tristan Colgate]

Maybe you mean something like support for linux uprobes? 

There is some discussion here:

https://github.com/golang/go/issues/22008

Google for "golang uprobes" picks up a couple of other links:

http://www.brendangregg.com/blog/2017-01-31/golang-bcc-bpf-function-tracing.html

https://kinvolk.io/blog/2017/09/an-update-on-gobpf---elf-loading-uprobes-more-program-types/

[robert engels]

Can’t you just ptrace the process, and use the ptrace related tools ? 

[robert engels]

Even if you could get uprobes to work, I think you will still have problems with inlining, and if you are going to compile with inlining disabled, etc. you might as well use Pike’s “coverage” method.

[Damian Gryski]

One approach would be to augment the compiler to instrument the binary with the techniques outlined in:

XRay: A Function Call Tracing System

https://ai.google/research/pubs/pub45287

Damian

[Kevin Malachowski]

If you can't recompile the binary, an option is to use a generic dynamic instrumentation platform like DynamoRIO (which I think works for Go programs after a recent bugfix). It works on windows, mac, and linux, and has been used in the past for various security-based testing and verification. Plenty of papers available on the website: http://dynamorio.org/pubs.html

[Julio Guerra]

On Fri, Nov 16, 2018 at 12:06 AM Damian Gryski <dgr...@gmail.com> wrote:

One approach would be to augment the compiler to instrument the binary with the techniques outlined in:

XRay: A Function Call Tracing System

https://ai.google/research/pubs/pub45287

Yes, this is very likely the best option for performance and to avoid code relocation at run time... Unfortunately, the go toolchain currently doesn't have any "plugin" interface to add extra compilation stages.

--

Julio Guerra

[Julio Guerra]

On Thu, Nov 15, 2018 at 7:55 PM Michael Jones <michae...@gmail.com> wrote:

You could study Rob Pike’s coverage/profiling tool to see how he adds and exploits basic block counting.  Good work as always. 

 

Good idea, thanks. I had a look at the implementation of it, and at the way it is integrated into the go toolchain. Unfortunately, it doesn't seem possible to include extra compilation stages in the toolchain. I was hoping for a way to insert an extra step to be able to modify the AST to insert tracepoints.

So, for example, coverage is built inside `go test` by directly calling `go cover`, and it doesn't use any plugin interface :(

--

Julio Guerra

[Julio Guerra]

On Thu, Nov 15, 2018 at 9:57 PM Tristan Colgate <tcol...@gmail.com> wrote:

Maybe you mean something like support for linux uprobes? 

There is some discussion here:

https://github.com/golang/go/issues/22008

Google for "golang uprobes" picks up a couple of other links:

http://www.brendangregg.com/blog/2017-01-31/golang-bcc-bpf-function-tracing.html

https://kinvolk.io/blog/2017/09/an-update-on-gobpf---elf-loading-uprobes-more-program-types/

Ran a small benchmark using uprobes and there's no overhead for an io-bound function, for a cpu-bound function, it is twice slower. So it may be a very good solution for a first linux-only version, as I don't need return probes (uretprobes seem to be a problem with Go's dynamic stack management).

--

Julio Guerra