Running goanalyzer on golang process running inside kubernetes pod

[Siddhesh Divekar]

Hi,

Has anyone tried running goanalyzer on golang process running inside k8s pod.

If so can you point me to the steps.

--

-Siddhesh.

[Robert Engels]

You can map the port but might be easier to capture to a file via code and use goanalyzer on the file. 

On Sep 10, 2020, at 4:53 PM, Siddhesh Divekar <siddhesh...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BhUDqMRmE0mDp_gMTKWKT0Be8KgtBxuF2fYAGCs-dNoog%40mail.gmail.com.

[Siddhesh Divekar]

Hi Robert,

Laying down the steps to make sure I understood it correctly.

You can map the port 

What did you mean by above ? 

but might be easier to capture to a file via code and use goanalyzer on the file

In this case I will do the following.

a) Set up a webserver in my program for getting Go profiles (with import _ "net/http/pprof")
b) map a port from outside k8s so that i can run curl localhost:$PORT/debug/pprof/$PROFILE_TYPE to save a profile (say profile-file)

c) Use go tool pprof/goanalyzer to analyze given profile from the profile-file

I would prefer to run this outside my process as the issue we expect to happen is when my program is hung.

I am not sure if the http server would respond & generate profile files.

From the help of goanalyzer it's not very clear how I would pass the profile-file to it.

It's showing how to generate profile-file using go test/tool but not how to pass them to goanalyzer.

# ./goanalyzer -h
Usage of 'go tool trace':
Given a trace file produced by 'go test':
go test -trace=trace.out pkg

Open a web browser displaying trace:
go tool trace [flags] [pkg.test] trace.out

Generate a pprof-like profile from the trace:
    go tool trace -pprof=TYPE [pkg.test] trace.out

[pkg.test] argument is required for traces produced by Go 1.6 and below.
Go 1.7 does not require the binary argument.

Supported profile types are:
    - net: network blocking profile
    - sync: synchronization blocking profile
    - syscall: syscall blocking profile
    - sched: scheduler latency profile

Flags:
-http=addr: HTTP service address (e.g., ':6060')
-pprof=type: print a pprof-like profile instead
-d: print debug info such as parsed events

--

-Siddhesh.

[robert engels]

goanalyzer = go tool trace

The options to goanalyzer are the same as ‘go tool trace’ - the usage message is misleading in this way.

By ‘map’ I mean expose the pprof port as you would any other port, and yes you use the net/http/pprof to start the internal webserver - you don’t need to create your own.

[Siddhesh Divekar]

Is there any dependency on GOPATH while running `./goanalyzer binary trace-file` ?

Also my goanalyzer is built on mac and am trying to look at trace file generated on ubuntu (shouldn't be a problem).

./goanalyzer ~/workspace/binary  ~/workspace/trace
2020/09/10 23:27:32 Parsing trace...
2020/09/10 23:27:32 Splitting trace...
2020/09/10 23:27:32 Opening browser. Trace viewer is listening on http://127.0.0.1:55520

If I try to go to a particular go routing eg http://127.0.0.1:55520/trace?goid=9084 I see nothing on the browser.

Clicking on any of the graphs for a goroutine says the following.

failed to execute go tool pprof: exit status 1
failed to execute dot. Is Graphviz installed? Error: exec: "dot": executable file not found in $PATH

--

-Siddhesh.

[Robert Engels]

I would start with making sure you can get the standard ‘go tool trace’ and ‘pprof’ working. Once you have those working ‘goanalyzer’ is an enhanced version. 

The docs are far more complete and substantial on the standard tools. 

On Sep 11, 2020, at 1:37 AM, Siddhesh Divekar <siddhesh...@gmail.com> wrote:



[Siddhesh Divekar]

Ok, let me get them working first.

My concern with pprof is it has to be made part of my process.

We had an issue where user requests were not reaching our http server itself.

In this case what are my options if pprof server is not reachable when we hit the same issue again.

Are there any other tools which can be run independently?

--

-Siddhesh.

[Robert Engels]

Please read golang.org/cmd/trace

You need a trace file. There are many ways to capture one. 

On Sep 11, 2020, at 12:09 PM, Siddhesh Divekar <siddhesh...@gmail.com> wrote:



[Siddhesh Divekar]

I went through the suggested approaches and here are my thoughts.

- runtime/trace.Start - The example in the doc https://golang.org/pkg/runtime/trace/ suggests adding os.Create("trace.out"), trace.Start & 

defer trace.Stop in the main before the application program gets started.

In an environment like k8s if the executing program is the pod main program then the pod will terminate and the trace.out will be lost. 

- net/http/pprof package - This has the same problem where requests might not make to the http server.

- go test -trace - Works with only tests

Any thoughts/ideas of what we can do to get a trace file avoiding above issues ?

--

-Siddhesh.

[Robert Engels]

You need to write to a shared writable path that is global so when the pod terminated the file is avail. 

The http should work too if you map the ports properly. You can be able to access the port from outside the pod. 

On Sep 11, 2020, at 7:14 PM, Siddhesh Divekar <siddhesh...@gmail.com> wrote:



[Siddhesh Divekar]

You need to write to a shared writable path that is global so when the pod terminated the file is avail. 

That's a good idea, we can write to to a config map.

The http should work too if you map the ports properly. You can be able to access the port from outside the pod. 

This might not work. When the issue happened last time, prometheus was unable to pull any metrics from the pod.

For the entire period prometheus output was blank which told us that the pod was non-responsive.

--

-Siddhesh.

[Siddhesh Divekar]

In writing to a file option, we would periodically write the file as it might turn out to be a huge file

for the entire life of the pod to flush at once.

--

-Siddhesh.

[Mhd Shulhan]

Pada tanggal Sab, 12 Sep 2020 08.52, Siddhesh Divekar <siddhesh...@gmail.com> menulis:

In writing to a file option, we would periodically write the file as it might turn out to be a huge file

for the entire life of the pod to flush at once

Maybe, the obvious solution here is not using Kubernetes. Create a VM, deploy the binary, ssh into it and run the binary manually, redirect request to VM, and so on.

[Siddhesh Divekar]

Thanks Mhd and agree with the suggestion.

However, we are trying to catch a bug which happens in production on k8s &

hence trying to explore different options.