Looking for thoughts on how we might expand cgo pprof support to platforms other than Linux

[Tim Wright]

I have recently been working on aspects of FreeBSD support in Go and was looking into the above.

In attempting to enable the associated tests (calling testCgoPprof), they fail and when I look at the pprof output generated by the test, it's missing the header that details the executable and libraries, which causes the profile to be incomplete/incorrect.

Following the code, readMapping in runtime/pprof/proto.go is using /proc/self/maps which is Linux-specific. I'm looking for thoughts on how this might be extended to other platforms. It's not obvious that there's any portable equivalent here.

On FreeBSD there are at least two options. Code to parse /proc/curproc/map would be very easy to write based on the existing code, but the downside is that procfs isn't mounted by default on FreeBSD. The libprocstat code would work everywhere, but is sysctl-based and would require a fair bit more work, at least to convert it to Go code rather than relying on an external library.

On OSX, it's completely different again. There are Mach calls to get the info. Sample code can be found at e.g. http://www.newosxbook.com/src.jl?tree=listings&file=12-1-vmmap.c

Anyway, I'm curious if anybody has any ideas or opinions about this.

Thanks,

Tim

[Ian Lance Taylor]

For what it's worth, the old version of pprof that existed before the
rewrite into Go did support collecting memory region information on
various other systems. The old code for this can be seen at
https://github.com/gperftools/gperftools/blob/master/src/base/sysinfo.cc#L479
. For FreeBSD see, notably,
https://github.com/gperftools/gperftools/blob/master/src/base/sysinfo.cc#L647
. But it does depend on procfs. I don't know enough to suggest what
can or should be done if that is not available.

Ian

[Tim Wright]

Thanks Ian.

So that really confirms my suspicion that there isn't any commonality here that would enable much if any code-sharing cross-platform. Implementing the parsing of the FreeBSD procfs map is very straightforward with the caveat that the user needs to mount it. Implementing via the sysctl mechanism works even without /proc, but it's a lot more code at least to implement it cleanly in Go and not require libprocstat.,That being said, the interface also seems pretty stable. I'm quite happy to take either approach and if there's no strong preference, I certainly have no qualms about requiring the procfs mount given the history.

Tim

[paulzhol]

I've experimented with using parts of libc compiled as .syso before working on https://golang.org/cl/93156.
Maybe parts of libprocstat/libkvm can be reused that way to save the porting.
I did encounter a problem (not sure if it is even a bug) with the internal linker: it would not calculate relative offsets to data locations,
even C static variables declared in functions.

[Devon H. O'Dell]

There are (or at least were) a fair number of ports that required procfs mounted to function, so if it's documented, I don't think it's a huge caveat to require. That said, people running in jailed environments may not be able to mount procfs at all. I'm not aware of what level of FreeBSD VPS services run atop jails without procfs-mounting privileges, and what services are running using hypervisors (and whether anyone really cares).

--dho 

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Wright]

Interesting. That certainly is another possibility. Porting the libprocstat code to Go doesn't seem especially hard, at least the parts I'd need here, it's just additional effort.

[Tim Wright]

On Thursday, March 29, 2018 at 8:52:43 AM UTC-7, Devon H. O'Dell wrote:

There are (or at least were) a fair number of ports that required procfs mounted to function, so if it's documented, I don't think it's a huge caveat to require. That said, people running in jailed environments may not be able to mount procfs at all. I'm not aware of what level of FreeBSD VPS services run atop jails without procfs-mounting privileges, and what services are running using hypervisors (and whether anyone really cares).

--dho 

 

Thanks Devon,

that's helpful context. I personally like the procfs metaphor as "one-stop shopping" rather than having to make a large number of system calls to gather the same information (I don't honestly know how much it matters these days, but I've always preferred to make as few system call transitions as possible. Probably more relevant in the case of Meltdown mitigation). But resorting to the sysctl interface does have the advantage of working everywhere including jails if/where procfs isn't mounted.

Tim

[Bakul Shah]

On Thu, 29 Mar 2018 10:33:09 -0700 Tim Wright <teno...@gmail.com> wrote:
>
> But resorting to the sysctl interface does have the advantage
> of working everywhere including jails if/where procfs isn't mounted.

I too would prefer this. Looking at runtime/pprof/proto.go,
something like the following C code should do the job.

#define countof(vec) (sizeof(vec)/sizeof(vec[0]))

int mib[] = {CTL_KERN, KERN_PROC, KERN_PROC_VMMAP, getpid()};
size_t len = 0;

// find out how much space is needed to get the results
if (sysctl(mib, countof(mib), NULL, &len, NULL, 0) < 0) {
perror("sysctl kern.proc.vmmap"); exit(1);
}

char* buf = malloc(len);
if (sysctl(mib, countof(mib), buf, &len, NULL, 0) < 0) {
perror("sysctl kern.proc.vmmap"); exit(1);
}

for (size_t off = 0; off < len; ) {
struct kinfo_vmentry* v = (struct kinfo_vmentry*)&buf[off];
off += v->kve_structsize;
if (v->kve_type != KVME_TYPE_VNODE ||
(v->kve_protection & KVME_PROT_EXEC) == 0)
continue;
// extricate kve_{start,end,offset,path}
}
free(buf);

As for Go, looks like syscall.Sysctl() won't allow specifying
a pid. May be a variant of Sysctl is needed?

For example:

func client() ([]byte, error) {
...
mib, err := NameToMIB("kern.proc.vmmap") // <-- new func
if err != nil { return nil, err }
mib = append(mib, os.Getpid())
buf, err := SysctlByMIB(mib) // <-- new func
if err != nil { return nil, err }
// convert buf into successive kinfo_vmentry structs
...
}

[Bakul Shah]

> On Thu, 29 Mar 2018 10:33:09 -0700 Tim Wright <teno...@gmail.com> wrote:
> >
> > But resorting to the sysctl interface does have the advantage
> > of working everywhere including jails if/where procfs isn't mounted.

This turned out to be pretty simple. I extended syscall.Sysctl
to accept a string like "kern.proc.vmmap.1234", that adds a
process id. This allows you to extract per process vmmap. A
quick hack to demonstrate this below.

This is sufficient to implement method readMapping() in
runtime/pprof/proto.go for FreeBSD. kern.proc.vmmap.<pid>
works for OpenBSD & vm.proc.map.<pid>) for NetBSD.

$ git diff
diff --git a/src/syscall/syscall_bsd.go b/src/syscall/syscall_bsd.go
index d141a7de9d..8fb011b050 100644
--- a/src/syscall/syscall_bsd.go
+++ b/src/syscall/syscall_bsd.go
@@ -14,6 +14,8 @@ package syscall

import (
"runtime"
+ "strconv"
+ "strings"
"unsafe"
)

@@ -450,11 +452,21 @@ func Kevent(kq int, changes, events []Kevent_t, timeout *Timespec) (n int, err e
//sys sysctl(mib []_C_int, old *byte, oldlen *uintptr, new *byte, newlen uintptr) (err error) = SYS___SYSCTL

func Sysctl(name string) (value string, err error) {
+ num := -1
// Translate name to mib number.
+ i := strings.LastIndex(name, ".")
+ if i < len(name) && '0' <= name[i+1] && name[i+1] <= '9' {
+ num, err = strconv.Atoi(name[i+1:])
+ if err != nil { return "", err }
+ name = name[:i]
+ }
mib, err := nametomib(name)
if err != nil {
return "", err
}
+ if num >= 0 {
+ mib = append(mib, _C_int(num))
+ }

// Find size.
n := uintptr(0)


$ cat> foo.go<<'EOF'
package main

import (
"fmt"
"os"
"syscall"
)

/*
#include <sys/types.h>
#include <sys/user.h>
#include <sys/sysctl.h>
#include <string.h>

struct kinfo_vmentry* getNextVMEntry(char* b, int off) {
return (struct kinfo_vmentry*)(b+off);
}
*/
import "C"

func main() {
pid := os.Getpid()
r, err := syscall.Sysctl("kern.proc.vmmap."+fmt.Sprint(pid))
if err != nil { fmt.Fprintf(os.Stderr, "%v\n", err); os.Exit(1) }
p := C.CString(r)
for off := 0; off < len(r); {
v := C.getNextVMEntry(p, C.int(off))
off += int(v.kve_structsize)
fmt.Printf("%012x-%012x %o %012x %d %s\n",
v.kve_start, v.kve_end, v.kve_protection,
v.kve_offset, v.kve_structsize,
C.GoString(&v.kve_path[0]))
}
}
EOF

$ go build -o /tmp/foo foo.go && /tmp/foo
000000400000-000000517000 5 000000000000 152 /tmp/foo
000000717000-00000072a000 3 000000000000 152 /tmp/foo
00000072a000-000000800000 3 000000000000 144
000800717000-000800730000 5 000000000000 176 /usr/jails/basejail/libexec/ld-elf.so.1
000800730000-000800754000 3 000000000000 144
...