[telemetry] counter: the first version of the counter package
8 views
Skip to first unread message
Peter Weinberger (Gerrit)
May 29, 2023, 1:00:15 PM5/29/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Peter Weinberger has uploaded this change for review.
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
8 files changed, 1,562 insertions(+), 0 deletions(-)
diff --git a/counter/STATUS.md b/counter/STATUS.md
new file mode 100644
index 0000000..7b09a22
--- /dev/null
+++ b/counter/STATUS.md
@@ -0,0 +1,36 @@
+# Issues in the counter package
+
+## `Open()`
+
+`Open()` is an exported function that asociates counters with a
+ counter file. Should it remain exported? It is presently called in
+ `init()` in upload.go. [We don't want to call it if telemetry is off,
+ but it could be called conditionally in counter.init(), which would
+ create a disk file even if no counter is ever incremented.]
+
+## Generating reports and uploading
+
+The simplest story would be to generate and upload reports when the
+counter file is rotated, but uploads might fail, so that would not be
+enough. The proposed way is to start a separate command each time the
+counter package starts.
+
+The code could be in the counter package, or in a separate package, or
+in a separate command, for instance 'go telemetry upload'. The latter ties
+updates to the 'go' command release cycle, and separates the upload code from the
+counter package. Thus the code will be in the upload package.
+
+The init() function in upload.go handles this. It checks to see if the
+program was invoked with a single argument `__telemetry_upload__`, and if
+so, executes the code to generate reports and upload them. If not it spawns
+a copy of the current program with that argument.
+
+This commentary can be moved to the upload package when it is checked in.
+
+## TODOs
+
+There are a bunch of TODOs. Also there are many places in the upload code
+where log messages are written, but it's unclear how to recover from the
+errors. The log messages are written to files named `telemetry-<pid>.log`
+in `os.TempDir()`.
+
diff --git a/counter/bootstrap.go b/counter/bootstrap.go
new file mode 100644
index 0000000..a2e6b04
--- /dev/null
+++ b/counter/bootstrap.go
@@ -0,0 +1,27 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+//go:build compiler_bootstrap
+
+package counter
+
+import "fmt"
+
+func Add(string, int64) {}
+func Inc(string) {}
+func Open() {}
+
+type Counter struct{ name string }
+
+func New(name string) *Counter { return &Counter{name} }
+func (c *Counter) Add(n int64) {}
+func (c *Counter) Inc() {}
+func (c *Counter) Name() string { return c.name }
+
+type File struct {
+ Meta map[string]string
+ Count map[string]uint64
+}
+
+func Parse(filename string, data []byte) (*File, error) { return nil, fmt.Errorf("unimplemented") }
diff --git a/counter/counter.go b/counter/counter.go
new file mode 100644
index 0000000..3a9a065
--- /dev/null
+++ b/counter/counter.go
@@ -0,0 +1,330 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+//go:build !compiler_bootstrap
+
+package counter
+
+import (
+ "flag"
+ "fmt"
+ "os"
+ "runtime"
+ "strings"
+ "sync/atomic"
+)
+
+// Note: not using internal/godebug, so that internal/godebug can use internal/counter.
+var debugCounter = strings.Contains(os.Getenv("GODEBUG"), "countertrace=1")
+
+func debugPrintf(format string, args ...interface{}) {
+ if debugCounter {
+ if len(format) == 0 || format[len(format)-1] != '\n' {
+ format += "\n"
+ }
+ fmt.Fprintf(os.Stderr, "counter: "+format, args...)
+ }
+}
+
+// Inc increments the counter with the given name.
+func Inc(name string) {
+ New(name).Inc()
+}
+
+// Add adds n to the counter with the given name.
+func Add(name string, n int64) {
+ New(name).Add(n)
+}
+
+// FlagParse calls flag.Parse and increments the counter "flag/<name>" for
+// each of the flags that was found on the command line.
+// It also wraps flag.Usage with a function that increments the counter "flag/usage"
+// if flag.Usage is called.
+func FlagParse() {
+ usage := flag.Usage
+ flag.Usage = func() {
+ Inc("flag/usage")
+ usage()
+ }
+ flag.Parse()
+ flag.Visit(func(f *flag.Flag) {
+ Inc("flag/" + f.Name)
+ })
+}
+
+// IncFlags increments all the non-default flags in the flag set,
+// using "<prefix>flag/<flagname>" as the counter name.
+func IncFlags(prefix string, flags *flag.FlagSet) {
+ flags.Visit(func(f *flag.Flag) {
+ Inc(prefix + "flag/" + f.Name)
+ })
+}
+
+// A Counter is a single named event counter.
+// A Counter is safe for use by multiple goroutines simultaneously.
+//
+// Counters should typically be created using New
+// and stored as global variables, like:
+//
+// package mypackage
+// var errorCount = counter.New("mypackage/errors")
+//
+// (The initialization of errorCount in this example is handled
+// entirely by the compiler and linker; this line executes no code
+// at program startup.)
+//
+// Then code can call Add to increment the counter
+// each time the corresponding event is observed.
+//
+// Although it is possible to use New to create
+// a Counter each time a particular event needs to be recorded,
+// that usage fails to amortize the construction cost over
+// multiple calls to Add, so it is more expensive and not recommended.
+type Counter struct {
+ name string
+ file *file
+ next atomic.Pointer[Counter]
+ state counterState
+ ptr counterPtr
+}
+
+// Name returns the name of the counter.
+func (c *Counter) Name() string {
+ return c.name
+}
+
+type counterPtr struct {
+ m *mappedFile
+ count *atomic.Uint64
+}
+
+type counterState struct {
+ bits atomic.Uint64
+}
+
+func (s *counterState) load() counterStateBits {
+ return counterStateBits(s.bits.Load())
+}
+
+func (s *counterState) update(old *counterStateBits, new counterStateBits) bool {
+ if s.bits.CompareAndSwap(uint64(*old), uint64(new)) {
+ *old = new
+ return true
+ }
+ return false
+}
+
+type counterStateBits uint64
+
+const (
+ stateReaders counterStateBits = 1<<30 - 1
+ stateLocked counterStateBits = stateReaders
+ stateHavePtr counterStateBits = 1 << 30
+ stateExtraShift = 31
+ stateExtra counterStateBits = 1<<64 - 1<<stateExtraShift
+)
+
+func (b counterStateBits) readers() int { return int(b & stateReaders) }
+func (b counterStateBits) locked() bool { return b&stateReaders == stateLocked }
+func (b counterStateBits) havePtr() bool { return b&stateHavePtr != 0 }
+func (b counterStateBits) extra() uint64 { return uint64(b&stateExtra) >> stateExtraShift }
+
+func (b counterStateBits) incReader() counterStateBits { return b + 1 }
+func (b counterStateBits) decReader() counterStateBits { return b - 1 }
+func (b counterStateBits) setLocked() counterStateBits { return b | stateLocked }
+func (b counterStateBits) clearLocked() counterStateBits { return b &^ stateLocked }
+func (b counterStateBits) setHavePtr() counterStateBits { return b | stateHavePtr }
+func (b counterStateBits) clearHavePtr() counterStateBits { return b &^ stateHavePtr }
+func (b counterStateBits) clearExtra() counterStateBits { return b &^ stateExtra }
+func (b counterStateBits) addExtra(n uint64) counterStateBits {
+ const maxExtra = uint64(stateExtra) >> stateExtraShift
+ x := b.extra()
+ if x+n < x || x+n > maxExtra {
+ x = maxExtra
+ } else {
+ x += n
+ }
+ return b | counterStateBits(x)<<stateExtraShift
+}
+
+// New returns a counter with the given name.
+// New can be called in global initializers and will be compiled down to
+// linker-initialized data. That is, calling New to initialize a global
+// has no cost at program startup.
+func New(name string) *Counter {
+ // Note: not calling defaultFile.New in order to keep this
+ // function something the compiler can inline and convert
+ // into static data initializations, with no init-time footprint.
+ return &Counter{name: name, file: &defaultFile}
+}
+
+// New returns a counter with the given name, using the given file
+// (used only for testing)
+func (f *file) New(name string) *Counter {
+ return &Counter{name: name, file: f}
+}
+
+// Inc adds 1 to the counter.
+func (c *Counter) Inc() {
+ c.Add(1)
+}
+
+// Add adds n to the counter.
+func (c *Counter) Add(n int64) {
+ debugPrintf("Add %q += %d", c.name, n)
+
+ if n < 0 {
+ panic("Counter.Add negative")
+ }
+ if n == 0 {
+ return
+ }
+ c.file.register(c)
+
+ state := c.state.load()
+ for ; ; state = c.state.load() {
+ switch {
+ case !state.locked() && state.havePtr():
+ if !c.state.update(&state, state.incReader()) {
+ continue
+ }
+ // Counter unlocked or counter shared; has an initialized count pointer; acquired shared lock.
+ if c.ptr.count == nil {
+ for !c.state.update(&state, state.addExtra(uint64(n))) {
+ // keep trying - we already took the reader lock
+ }
+ debugPrintf("Add %q += %d: nil extra=%d\n", c.name, n, state.extra())
+ } else {
+ sum := c.satAdd(uint64(n))
+ debugPrintf("Add %q += %d: count=%d\n", c.name, n, sum)
+ }
+ c.releaseReader(state)
+ return
+
+ case state.locked():
+ if !c.state.update(&state, state.addExtra(uint64(n))) {
+ continue
+ }
+ debugPrintf("Add %q += %d: locked extra=%d\n", c.name, n, state.extra())
+ return
+
+ case !state.havePtr():
+ if !c.state.update(&state, state.addExtra(uint64(n)).setLocked()) {
+ continue
+ }
+ debugPrintf("Add %q += %d: noptr extra=%d\n", c.name, n, state.extra())
+ c.releaseLock(state)
+ return
+ }
+ }
+}
+
+func (c *Counter) releaseReader(state counterStateBits) {
+ for ; ; state = c.state.load() {
+ // If we are the last reader and havePtr was cleared
+ // while this batch of readers was using c.ptr,
+ // it's our job to update c.ptr by upgrading to a full lock
+ // and letting releaseLock do the work.
+ // Note: no new reader will attempt to add itself now that havePtr is clear,
+ // so we are only racing against possible additions to extra.
+ if state.readers() == 1 && !state.havePtr() {
+ if !c.state.update(&state, state.setLocked()) {
+ continue
+ }
+ debugPrintf("releaseReader %s: last reader, need ptr\n", c.name)
+ c.releaseLock(state)
+ return
+ }
+
+ // Release reader.
+ if !c.state.update(&state, state.decReader()) {
+ continue
+ }
+ debugPrintf("releaseReader %s: released (%d readers now)\n", c.name, state.readers())
+ return
+ }
+}
+
+func (c *Counter) releaseLock(state counterStateBits) {
+ for ; ; state = c.state.load() {
+ if !state.havePtr() {
+ // Set havePtr before updating ptr,
+ // to avoid race with the next clear of havePtr.
+ if !c.state.update(&state, state.setHavePtr()) {
+ continue
+ }
+ debugPrintf("releaseLock %s: reset havePtr (extra=%d)\n", c.name, state.extra())
+
+ // Optimization: only bother loading a new pointer
+ // if we have a value to add to it.
+ c.ptr = counterPtr{nil, nil}
+ if state.extra() != 0 {
+ c.ptr = c.file.lookup(c.name)
+ debugPrintf("releaseLock %s: ptr=%v\n", c.name, c.ptr)
+ }
+ }
+
+ if extra := state.extra(); extra != 0 && c.ptr.count != nil {
+ if !c.state.update(&state, state.clearExtra()) {
+ continue
+ }
+ sum := c.satAdd(extra)
+ debugPrintf("releaseLock %s: flush extra=%d -> count=%d\n", c.name, extra, sum)
+ }
+
+ // Took care of refreshing ptr and flushing extra.
+ // Now we can release the lock, unless of course
+ // another goroutine cleared havePtr or added to extra,
+ // in which case we go around again.
+ if !c.state.update(&state, state.clearLocked()) {
+ continue
+ }
+ debugPrintf("releaseLock %s: unlocked\n", c.name)
+ return
+ }
+}
+
+func (c *Counter) satAdd(n uint64) uint64 {
+ count := c.ptr.count
+ for {
+ old := count.Load()
+ sum := old + n
+ if sum < old {
+ sum = ^uint64(0)
+ }
+ if count.CompareAndSwap(old, sum) {
+ runtime.KeepAlive(c.ptr.m)
+ return sum
+ }
+ }
+}
+
+func (c *Counter) invalidate() {
+ for {
+ state := c.state.load()
+ if !state.havePtr() {
+ debugPrintf("invalidate %s: no ptr\n", c.name)
+ return
+ }
+ if c.state.update(&state, state.clearHavePtr()) {
+ debugPrintf("invalidate %s: cleared havePtr\n", c.name)
+ return
+ }
+ }
+}
+
+func (c *Counter) refresh() {
+ for {
+ state := c.state.load()
+ if state.havePtr() || state.readers() > 0 || state.extra() == 0 {
+ debugPrintf("refresh %s: havePtr=%v readers=%d extra=%d\n", c.name, state.havePtr(), state.readers(), state.extra())
+ return
+ }
+ if c.state.update(&state, state.setLocked()) {
+ debugPrintf("refresh %s: locked havePtr=%v readers=%d extra=%d\n", c.name, state.havePtr(), state.readers(), state.extra())
+ c.releaseLock(state)
+ return
+ }
+ }
+}
diff --git a/counter/counter_test.go b/counter/counter_test.go
new file mode 100644
index 0000000..a8842d0
--- /dev/null
+++ b/counter/counter_test.go
@@ -0,0 +1,321 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package counter
+
+// Missing any tests of concurrent uses of a counter.
+
+import (
+ "encoding/hex"
+ "fmt"
+ "os"
+ "path/filepath"
+ "reflect"
+ "strings"
+ "testing"
+ "time"
+
+ "golang.org/x/telemetry"
+)
+
+func TestBasic(t *testing.T) {
+ setup(t)
+ defer restore()
+ var f file
+ c := f.New("gophers")
+ c.Add(9)
+ f.rotate()
+ if f.err != nil {
+ t.Fatal(f.err)
+ }
+ t.Logf("prefix %s", f.namePrefix)
+ current := f.current.Load()
+ if current == nil {
+ t.Fatal("no mapped file")
+ }
+ c.Add(0x90)
+
+ name := current.f.Name()
+ t.Logf("wrote %s:\n%s", name, hexDump(current.data))
+
+ data, err := os.ReadFile(name)
+ if err != nil {
+ t.Fatal(err)
+ }
+ pf, err := Parse(name, data)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ want := map[string]uint64{"gophers": 0x99}
+ if !reflect.DeepEqual(pf.Count, want) {
+ t.Errorf("pf.Count = %v, want %v", pf.Count, want)
+ }
+}
+
+func TestLarge(t *testing.T) {
+ setup(t)
+ defer restore()
+ var f file
+ f.rotate()
+ for i := int64(0); i < 10000; i++ {
+ c := f.New(fmt.Sprint("gophers", i))
+ c.Add(i*i + 1)
+ }
+ for i := int64(0); i < 10000; i++ {
+ c := f.New(fmt.Sprint("gophers", i))
+ c.Add(i / 2)
+ }
+ current := f.current.Load()
+ if current == nil {
+ t.Fatal("no mapped file")
+ }
+ name := current.f.Name()
+ t.Logf("wrote %s:\n%s", name, hexDump(current.data))
+
+ data, err := os.ReadFile(name)
+ if err != nil {
+ t.Fatal(err)
+ }
+ pf, err := Parse(name, data)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ for i := uint64(0); i < 10000; i++ {
+ key := fmt.Sprint("gophers", i)
+ want := 1 + i*i + i/2
+ if n := pf.Count[key]; n != want {
+ t.Errorf("Count[%s] = %d, want %d", key, n, want)
+ }
+ }
+}
+
+func TestRepeatedNew(t *testing.T) {
+ setup(t)
+ defer restore()
+ var f file
+ f.rotate()
+ New("gophers")
+ c1ptr := f.lookup("gophers")
+ New("gophers")
+ c2ptr := f.lookup("gophers")
+ if c1ptr != c2ptr {
+ t.Errorf("c1ptr = %p, c2ptr = %p, want same", c1ptr, c2ptr)
+ }
+}
+
+func hexDump(data []byte) string {
+ lines := strings.SplitAfter(hex.Dump(data), "\n")
+ var keep []string
+ for len(lines) > 0 {
+ line := lines[0]
+ keep = append(keep, line)
+ lines = lines[1:]
+ const allZeros = "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
+ if strings.Contains(line, allZeros) {
+ i := 0
+ for i < len(lines) && strings.Contains(lines[i], allZeros) {
+ i++
+ }
+ if i > 2 {
+ keep = append(keep, "*\n", lines[i-1])
+ lines = lines[i:]
+ }
+ }
+ }
+ return strings.Join(keep, "")
+}
+
+func TestNewFile(t *testing.T) {
+ setup(t)
+ defer restore()
+ year, month, day := time.Now().Date()
+ now := time.Date(year, month, day, 0, 0, 0, 0, time.UTC)
+
+ // test that completely new files have dates well in the future
+ // Try 20 times to get 20 different random numbers.
+ for i := 0; i < 20; i++ {
+ var f file
+ c := f.New("gophers")
+ // shouldn't see a file yet
+ fi, err := os.ReadDir(telemetry.LocalDir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(fi) != 0 {
+ t.Fatalf("len(fi) = %d, want 0", len(fi))
+ }
+ c.Add(9)
+ // still shouldn't see a file
+ fi, err = os.ReadDir(telemetry.LocalDir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(fi) != 0 {
+ t.Fatalf("len(fi) = %d, want 0", len(fi))
+ }
+ f.rotate()
+ // now we should see a file
+ fi, err = os.ReadDir(telemetry.LocalDir)
+ if len(fi) != 1 {
+ t.Fatalf("len(fi) = %d, want 1", len(fi))
+ }
+
+ buf, err := os.ReadFile(filepath.Join(telemetry.LocalDir, fi[0].Name()))
+ if err != nil {
+ t.Fatal(err)
+ }
+ cf, err := Parse(fi[0].Name(), buf)
+ if err != nil {
+ t.Fatal(err)
+ }
+ then, err := time.Parse(time.RFC3339, cf.Meta["TimeEnd"])
+ if err != nil {
+ t.Fatal(err)
+ }
+ days := (then.Sub(now)) / (24 * time.Hour)
+ if days <= 7 || days > 14 {
+ t.Errorf("days = %d, want 7 < days <= 14", days)
+ }
+ // remove the file for the next iteration of the loop
+ os.Remove(filepath.Join(telemetry.LocalDir, fi[0].Name()))
+ }
+}
+
+func TestRotate(t *testing.T) {
+ year, month, day := time.Now().Date()
+ now := time.Date(year, month, day, 0, 0, 0, 0, time.UTC)
+ setup(t)
+ defer restore()
+ // pretend something was uploaded
+ os.WriteFile(filepath.Join(telemetry.UploadDir, "anything"), []byte{}, 0666)
+ t.Setenv("GOTELEMETRY", "local")
+ var f file
+ c := f.New("gophers")
+ c.Inc()
+ var modified int
+ for i := 0; i < 2; i++ {
+ // nothing should change on the second rotate
+ f.rotate()
+ fi, err := os.ReadDir(telemetry.LocalDir)
+ if err != nil || len(fi) != 1 {
+ t.Fatalf("err=%v, len(fi) = %d, want 1", err, len(fi))
+ }
+ x := fi[0].Name()
+ y := x[len(x)-len("2006-01-02")-len(".v1.count") : len(x)-len(".v1.count")]
+ us, err := time.ParseInLocation("2006-01-02", y, time.UTC)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // we expect today's date?
+ if us != now {
+ t.Errorf("us = %v, want %v", us, now)
+ }
+ fd, err := os.Open(filepath.Join(telemetry.LocalDir, fi[0].Name()))
+ if err != nil {
+ t.Fatal(err)
+ }
+ stat, err := fd.Stat()
+ if err != nil {
+ t.Fatal(err)
+ }
+ mt := stat.ModTime().Nanosecond()
+ if modified == 0 {
+ modified = mt
+ }
+ if modified != mt {
+ t.Errorf("modified = %v, want %v", mt, modified)
+ }
+ }
+ counterTime = func() time.Time { return now.Add(7 * 24 * time.Hour) }
+ f.rotate()
+ fi, err := os.ReadDir(telemetry.LocalDir)
+ if err != nil || len(fi) != 2 {
+ t.Fatalf("err=%v, len(fi) = %d, want 2", err, len(fi))
+ }
+
+}
+
+func TestStack(t *testing.T) {
+ setup(t)
+ defer restore()
+ Open()
+ c := NewStack("foo", 5)
+ c.Inc()
+ c.Inc()
+ names := c.Names()
+ if len(names) != 2 {
+ t.Fatalf("got %d names, want 2", len(names))
+ }
+ // each name should be 6 lines, and the two names should
+ // differ only in the second line. The last few lines should
+ // be blank.
+ n0 := strings.Split(names[0], "\n")
+ n1 := strings.Split(names[1], "\n")
+ if len(n0) != 6 || len(n1) != 6 {
+ t.Fatalf("got %d and %d lines, want 6 (%q,%q)", len(n0), len(n1), n0, n1)
+ }
+ for i := 0; i < 6; i++ { // loop generated by copilot (from comment?)
+ if i == 1 {
+ continue
+ }
+ if n0[i] != n1[i] {
+ t.Fatalf("line %d differs:\n%s\n%s", i, n0[i], n1[i])
+ }
+ }
+ oldnames := make(map[string]bool)
+ for _, nm := range names {
+ oldnames[nm] = true
+ }
+ for i := 0; i < 2; i++ {
+ f(t, 4, c)
+ }
+ newnames := make(map[string]bool)
+ for _, nm := range c.Names() {
+ if !oldnames[nm] {
+ newnames[nm] = true
+ }
+ }
+ // expect 5 new names, one for each level of recursion
+ if len(newnames) != 5 {
+ t.Fatalf("got %d new names, want 5", len(newnames))
+ }
+ // look inside. old names should have a count of 1, new ones 2
+ for _, ct := range c.Counters() {
+ if ct == nil {
+ t.Fatal("nil counter")
+ }
+ if ct.ptr.count == nil {
+ t.Errorf("%q has nil ptr.count", ct.Name())
+ continue
+ }
+ if oldnames[ct.Name()] && ct.ptr.count.Load() != 1 {
+ t.Errorf("old name %q has count %d, want 1", ct.Name(), ct.ptr.count.Load())
+ }
+ if newnames[ct.Name()] && ct.ptr.count.Load() != 2 {
+ t.Errorf("new name %q has count %d, want 2", ct.Name(), ct.ptr.count.Load())
+ }
+ }
+}
+
+func f(t *testing.T, n int, c *StackCounter) {
+ c.Inc()
+ if n > 0 {
+ f(t, n-1, c)
+ }
+}
+
+func setup(t *testing.T) {
+ telemetry.LocalDir = t.TempDir() + "/local"
+ telemetry.UploadDir = t.TempDir() + "/upload"
+ os.MkdirAll(telemetry.LocalDir, 0777)
+ os.MkdirAll(telemetry.UploadDir, 0777)
+ // os.UserConfigDir() is "" in tests
+ t.Setenv("GOTELEMETRY", "local")
+}
+
+func restore() {
+ counterTime = time.Now
+}
diff --git a/counter/doc.go b/counter/doc.go
new file mode 100644
index 0000000..6465136
--- /dev/null
+++ b/counter/doc.go
@@ -0,0 +1,29 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// The counter package implements a simple counter system for collecting
+// totally public telemetry data.
+//
+// There are two kinds of counters, simple counters and stack counters.
+// Simple counters are created by New(<counter-name>).
+// Stack counters are created by NewStack(<counter-name>, depth).
+// Both are incremented by calling Inc().
+//
+// Counter files are stored in LocalDir(). Their content can be accessed
+// by Parse().
+//
+// Simple counters are very cheap. Stack counters are not collected if
+// the envirnonment variable GOTELEMETRY is set to 'off'. (Stack counters
+// are implemented as a set of regular counters whose names
+// are the concatenation of the name and the stack trace. There is an upper
+// limit on the size of this name, about 256 bytes. If the name is too long
+// the counter will be silently ignored.)
+//
+// Counter files are turned into reports by the upload package.
+// This happens weekly, except for the first time a counter file is
+// created. Then it happens on a random day of the week more than 7 days
+// in the future. After that the counter files expire weekly on the same day of
+// the week.
+
+package counter
diff --git a/counter/file.go b/counter/file.go
new file mode 100644
index 0000000..59f900f
--- /dev/null
+++ b/counter/file.go
@@ -0,0 +1,608 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+//go:build !compiler_bootstrap
+
+package counter
+
+import (
+ "bytes"
+ "errors"
+ "fmt"
+ "math/rand"
+ "os"
+ "path/filepath"
+ "runtime"
+ "runtime/debug"
+ "strings"
+ "sync"
+ "sync/atomic"
+ "syscall"
+ "time"
+ "unsafe"
+
+ "golang.org/x/telemetry"
+)
+
+// A file is a counter file.
+type file struct {
+ // Linked list of all known counters.
+ // (Linked list insertion is easy to make lock-free,
+ // and we don't want the initial counters incremented
+ // by a program to cause significant contention.)
+ counters atomic.Pointer[Counter] // head of list
+ end Counter // list ends at &end instead of nil
+
+ mu sync.Mutex
+ namePrefix string
+ err error
+ meta string
+ prev *mappedFile
+ current atomic.Pointer[mappedFile] // can be read without holding mu
+}
+
+var defaultFile file
+
+// register ensures that the counter c is registered with the file.
+func (f *file) register(c *Counter) {
+ debugPrintf("register %s %p\n", c.name, c)
+
+ // If counter is not registered with file, register it.
+ // Doing this lazily avoids init-time work
+ // as well as any execution cost at all for counters
+ // that are not used in a given program.
+ wroteNext := false
+ for wroteNext || c.next.Load() == nil {
+ head := f.counters.Load()
+ next := head
+ if next == nil {
+ next = &f.end
+ }
+ debugPrintf("register %s next %p\n", c.name, next)
+ if !wroteNext {
+ if !c.next.CompareAndSwap(nil, next) {
+ debugPrintf("register %s cas failed %p\n", c.name, c.next.Load())
+ continue
+ }
+ wroteNext = true
+ } else {
+ c.next.Store(next)
+ }
+ if f.counters.CompareAndSwap(head, c) {
+ debugPrintf("registered %s %p\n", c.name, f.counters.Load())
+ return
+ }
+ debugPrintf("register %s cas2 failed %p %p\n", c.name, f.counters.Load(), head)
+ }
+}
+
+// invalidateCounters marks as invalid all the pointers
+// held by f's counters (because they point into m),
+// and then closes prev.
+//
+// invalidateCounters cannot be called while holding f.mu,
+// because a counter invalidation may call f.lookup.
+func (f *file) invalidateCounters() {
+ // Mark every counter as needing to refresh its count pointer.
+ if head := f.counters.Load(); head != nil {
+ for c := head; c != &f.end; c = c.next.Load() {
+ c.invalidate()
+ }
+ for c := head; c != &f.end; c = c.next.Load() {
+ c.refresh()
+ }
+ }
+}
+
+// lookup looks up the counter with the given name in the file,
+// allocating it if needed, and returns a pointer to the atomic.Uint64
+// containing the counter data.
+// If the file has not been opened yet, lookup returns nil.
+func (f *file) lookup(name string) counterPtr {
+ current := f.current.Load()
+ if current == nil {
+ debugPrintf("lookup %s - no mapped file\n", name)
+ return counterPtr{}
+ }
+ ptr := f.newCounter(name)
+ if ptr == nil {
+ return counterPtr{}
+ }
+ return counterPtr{current, ptr}
+}
+
+// ErrDisabled is the error returned when telemetry is disabled.
+var ErrDisabled = errors.New("counter: disabled by GOTELEMETRY=off")
+
+var (
+ errNoBuildInfo = errors.New("counter: missing build info")
+ errCorrupt = errors.New("counter: corrupt counter file")
+)
+
+func (f *file) init(begin, end time.Time) {
+ info, ok := debug.ReadBuildInfo()
+ if !ok {
+ f.err = errNoBuildInfo
+ return
+ }
+ if !telemetry.Enabled {
+ f.err = ErrDisabled
+ return
+ }
+ dir := telemetry.LocalDir
+
+ if err := os.MkdirAll(dir, 0777); err != nil {
+ f.err = err
+ return
+ }
+
+ goVers := info.GoVersion
+ if strings.Contains(goVers, "devel") || strings.Contains(goVers, "-") {
+ goVers = "devel"
+ }
+ prog := info.Path
+ if prog == "" {
+ prog = strings.TrimSuffix(filepath.Base(os.Args[0]), ".exe")
+ }
+ prog = filepath.Base(prog)
+ progVers := info.Main.Version
+ if strings.Contains(progVers, "devel") || strings.Contains(progVers, "-") {
+ progVers = "devel"
+ }
+ f.meta = fmt.Sprintf("TimeBegin: %s\nTimeEnd: %s\nProgram: %s\nVersion: %s\nGoVersion: %s\nGOOS: %s\nGOARCH: %s\n\n",
+ begin.Format(time.RFC3339), end.Format(time.RFC3339),
+ prog, progVers, goVers, runtime.GOOS, runtime.GOARCH)
+ if len(f.meta) > maxMetaLen { // should be impossible for our use
+ f.err = fmt.Errorf("metadata too long")
+ return
+ }
+ if progVers != "" {
+ progVers = "@" + progVers
+ }
+ prefix := fmt.Sprintf("%s%s-%s-%s-%s-", prog, progVers, goVers, runtime.GOOS, runtime.GOARCH)
+ f.namePrefix = filepath.Join(dir, prefix)
+}
+
+// filename returns the name of the file to use for f,
+// given the current time now.
+// It also returns the time when that name will no longer be valid
+// and a new filename should be computed.
+func (f *file) filename(now time.Time) (name string, expire time.Time, err error) {
+ year, month, day := now.Date()
+ begin := time.Date(year, month, day, 0, 0, 0, 0, now.Location())
+ incr := fileValidity()
+ end := time.Date(year, month, day+incr, 0, 0, 0, 0, now.Location())
+ if f.namePrefix == "" && f.err == nil {
+ f.init(begin, end)
+ debugPrintf("init: %#q, %v", f.namePrefix, f.err)
+ }
+ if f.err != nil {
+ return "", time.Time{}, err
+ }
+
+ name = f.namePrefix + now.Format("2006-01-02") + "." + fileVersion + ".count"
+ return name, end, nil
+}
+
+// fileValidity returns the number of days that a file is valid for.
+// It is 7, except for new clients.
+func fileValidity() int {
+ dir := telemetry.UploadDir
+ if c, err := os.ReadDir(dir); err == nil && len(c) > 0 {
+ return 7
+ }
+ dir = telemetry.LocalDir
+ if c, err := os.ReadDir(dir); err == nil && len(c) > 0 {
+ return 7
+ }
+ return 8 + rand.Intn(7)
+}
+
+// rotate checks to see whether the file f needs to be rotated,
+// meaning to start a new counter file with a different date in the name.
+// rotate is also used to open the file initially, meaning f.current can be nil.
+// In general rotate should be called just once for each file.
+// rotate will arrange a timer to call itself again when necessary.
+func (f *file) rotate() {
+ expire, cleanup := f.rotate1()
+ cleanup()
+ if !expire.IsZero() {
+ // TODO(rsc): Does this do the right thing for laptops closing?
+ time.AfterFunc(time.Until(expire), f.rotate)
+ }
+}
+
+func nop() {}
+
+var counterTime = time.Now // changed for tests
+
+func (f *file) rotate1() (expire time.Time, cleanup func()) {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+
+ name, expire, err := f.filename(counterTime())
+ if err != nil {
+ debugPrintf("rotate: %v\n", err)
+ return time.Time{}, nop
+ }
+ if name == "" {
+ return time.Time{}, nop
+ }
+
+ current := f.current.Load()
+ if current != nil && name == current.f.Name() {
+ return expire, nop
+ }
+
+ m, err := openMapped(name, f.meta)
+ if err != nil {
+ debugPrintf("rotate: openMapped: %v\n", err)
+ if current != nil {
+ if v, _, _, _ := current.lookup("counter/rotate-error"); v != nil {
+ v.Add(1)
+ }
+ }
+ return expire, nop
+ }
+
+ debugPrintf("using %v", m.f.Name())
+ f.current.Store(m)
+ return expire, f.invalidateCounters
+}
+
+func (f *file) newCounter(name string) *atomic.Uint64 {
+ v, cleanup := f.newCounter1(name)
+ cleanup()
+ return v
+}
+
+func (f *file) newCounter1(name string) (v *atomic.Uint64, cleanup func()) {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+
+ current := f.current.Load()
+ if current == nil {
+ return nil, nop
+ }
+ debugPrintf("newCounter %s in %s\n", name, current.f.Name())
+ if v, _, _, _ := current.lookup(name); v != nil {
+ return v, nop
+ }
+ v, newM, err := current.newCounter(name)
+ if err != nil {
+ debugPrintf("newCounter %s: %v\n", name, err)
+ return nil, nop
+ }
+
+ cleanup = nop
+ if newM != nil {
+ f.current.Store(newM)
+ cleanup = f.invalidateCounters
+ }
+ return v, cleanup
+}
+
+var mainCounter = New("counter/main")
+
+func Open() {
+ debugPrintf("Open")
+ mainCounter.Add(1)
+ defaultFile.rotate()
+}
+
+type mappedFile struct {
+ meta string
+ hdrLen uint32
+ zero [4]byte
+ closeOnce sync.Once
+ f *os.File
+ data []byte
+}
+
+func openMapped(name string, meta string) (_ *mappedFile, err error) {
+ hdr, err := mappedHeader(meta)
+ if err != nil {
+ return nil, err
+ }
+
+ f, err := os.OpenFile(name, os.O_RDWR|os.O_CREATE, 0666)
+ if err != nil {
+ return nil, err
+ }
+ // Note: using local variable m here, not return value,
+ // so that reutrn nil, err does not set m = nil and break the code in the defer.
+ m := &mappedFile{
+ f: f,
+ meta: meta,
+ }
+ runtime.SetFinalizer(m, (*mappedFile).close)
+ defer func() {
+ if err != nil {
+ m.close()
+ }
+ }()
+ info, err := f.Stat()
+ if err != nil {
+ return nil, err
+ }
+
+ // Establish file header and initial data area if not already present.
+ if info.Size() < minFileLen {
+ if _, err := f.WriteAt(hdr, 0); err != nil {
+ return nil, err
+ }
+ // Write zeros at the end of the file to extend it to minFileLen.
+ if _, err := f.WriteAt(m.zero[:], int64(minFileLen-len(m.zero))); err != nil {
+ return nil, err
+ }
+ info, err = f.Stat()
+ if err != nil {
+ return nil, err
+ }
+ if info.Size() < minFileLen {
+ return nil, fmt.Errorf("counter: writing file did not extend it")
+ }
+ }
+
+ // Map into memory.
+ data, err := mmap(int(f.Fd()), 0, int(info.Size()), syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED)
+ if err != nil {
+ return nil, err
+ }
+ m.data = data
+ if !bytes.HasPrefix(m.data, hdr) {
+ return nil, fmt.Errorf("counter: header mismatch")
+ }
+ m.hdrLen = uint32(len(hdr))
+
+ return m, nil
+}
+
+const (
+ fileVersion = "v1"
+ hdrPrefix = "# telemetry/counter file " + fileVersion + "\n"
+ recordUnit = 32
+ maxMetaLen = 512
+ numHash = 512 // 2kB for hash table
+ maxNameLen = 256
+ limitOff = 0
+ hashOff = 4
+ pageSize = 4096
+ minFileLen = 4096
+)
+
+func mappedHeader(meta string) ([]byte, error) {
+ if len(meta) > maxMetaLen {
+ return nil, fmt.Errorf("counter: metadata too large")
+ }
+ np := round(len(hdrPrefix), 4)
+ n := round(np+4+len(meta), 32)
+ hdr := make([]byte, n)
+ copy(hdr, hdrPrefix)
+ *(*uint32)(unsafe.Pointer(&hdr[np])) = uint32(n)
+ copy(hdr[np+4:], meta)
+ return hdr, nil
+}
+
+func (m *mappedFile) place(limit uint32, name string) (start, end uint32) {
+ if limit == 0 {
+ // first record in file
+ limit = m.hdrLen + hashOff + 4*numHash
+ }
+ n := round(uint32(16+len(name)), recordUnit)
+ start = round(limit, recordUnit) // should already be rounded but just in case
+ if start/pageSize != (start+n)/pageSize {
+ // bump start to next page
+ start = round(limit, pageSize)
+ }
+ return start, start + n
+}
+
+var mmap = syscall.Mmap
+var munmap = syscall.Munmap
+
+func (m *mappedFile) close() {
+ m.closeOnce.Do(func() {
+ if m.data != nil {
+ munmap(m.data)
+ m.data = nil
+ }
+ if m.f != nil {
+ m.f.Close()
+ m.f = nil
+ }
+ })
+}
+
+// hash returns the hash code for name.
+// The implementation is FNV-1a.
+// This hash function is a fixed detail of the file format.
+// It cannot be changed without also changing the file format version.
+func hash(name string) uint32 {
+ const (
+ offset32 = 2166136261
+ prime32 = 16777619
+ )
+ h := uint32(offset32)
+ for i := 0; i < len(name); i++ {
+ c := name[i]
+ h = (h ^ uint32(c)) * prime32
+ }
+ return (h ^ (h >> 16)) % numHash
+}
+
+func (m *mappedFile) load32(off uint32) uint32 {
+ if int64(off) >= int64(len(m.data)) {
+ return 0
+ }
+ return (*atomic.Uint32)(unsafe.Pointer(&m.data[off])).Load()
+}
+
+func (m *mappedFile) cas32(off, old, new uint32) bool {
+ if int64(off) >= int64(len(m.data)) {
+ panic("bad cas32") // return false would probably loop
+ }
+ return (*atomic.Uint32)(unsafe.Pointer(&m.data[off])).CompareAndSwap(old, new)
+}
+
+func (m *mappedFile) entryAt(off uint32) (name []byte, next uint32, v *atomic.Uint64, ok bool) {
+ if off < m.hdrLen+hashOff || int64(off)+16 > int64(len(m.data)) {
+ return nil, 0, nil, false
+ }
+ nameLen := m.load32(off+8) & 0x00ffffff
+ if nameLen == 0 || int64(off)+16+int64(nameLen) > int64(len(m.data)) {
+ return nil, 0, nil, false
+ }
+ name = m.data[off+16 : off+16+nameLen]
+ next = m.load32(off + 12)
+ v = (*atomic.Uint64)(unsafe.Pointer(&m.data[off]))
+ return name, next, v, true
+}
+
+func (m *mappedFile) writeEntryAt(off uint32, name string) (next *atomic.Uint32, v *atomic.Uint64, ok bool) {
+ if off < m.hdrLen+hashOff || int64(off)+16+int64(len(name)) > int64(len(m.data)) {
+ return nil, nil, false
+ }
+ copy(m.data[off+16:], name)
+ atomic.StoreUint32((*uint32)(unsafe.Pointer(&m.data[off+8])), uint32(len(name))|0xff000000)
+ next = (*atomic.Uint32)(unsafe.Pointer(&m.data[off+12]))
+ v = (*atomic.Uint64)(unsafe.Pointer(&m.data[off]))
+ return next, v, true
+}
+
+func (m *mappedFile) lookup(name string) (v *atomic.Uint64, headOff, head uint32, ok bool) {
+ h := hash(name)
+ headOff = m.hdrLen + hashOff + h*4
+ head = m.load32(headOff)
+ off := head
+ for off != 0 {
+ ename, next, v, ok := m.entryAt(off)
+ if !ok {
+ return nil, 0, 0, false
+ }
+ if string(ename) == name {
+ return v, headOff, head, true
+ }
+ off = next
+ }
+ return nil, headOff, head, true
+}
+
+func (m *mappedFile) newCounter(name string) (v *atomic.Uint64, m1 *mappedFile, err error) {
+ if len(name) > maxNameLen {
+ return nil, nil, fmt.Errorf("counter name too long")
+ }
+ orig := m
+ defer func() {
+ if m != orig {
+ if err != nil {
+ m.close()
+ } else {
+ m1 = m
+ }
+ }
+ }()
+
+ v, headOff, head, ok := m.lookup(name)
+ for !ok {
+ // Lookup found an invalid pointer,
+ // perhaps because the file has grown larger than the mapping.
+ limit := m.load32(m.hdrLen + limitOff)
+ if int64(limit) <= int64(len(m.data)) {
+ // Mapping doesn't need to grow, so lookup found actual corruption.
+ debugPrintf("corrupt1\n")
+ return nil, nil, errCorrupt
+ }
+ newM, err := openMapped(m.f.Name(), m.meta)
+ if err != nil {
+ return nil, nil, err
+ }
+ if m != orig {
+ m.close()
+ }
+ m = newM
+ v, headOff, head, ok = m.lookup(name)
+ }
+ if v != nil {
+ return v, nil, nil
+ }
+
+ // Reserve space for new record.
+ // We are competing against other programs using the same file,
+ // so we use a compare-and-swap on the allocation limit in the header.
+ var start, end uint32
+ for {
+ // Determine where record should end, and grow file if needed.
+ limit := m.load32(m.hdrLen + limitOff)
+ start, end = m.place(limit, name)
+ debugPrintf("place %s at %#x-%#x\n", name, start, end)
+ if int64(end) > int64(len(m.data)) {
+ newM, err := m.extend(end)
+ if err != nil {
+ return nil, nil, err
+ }
+ if m != orig {
+ m.close()
+ }
+ m = newM
+ continue
+ }
+
+ // Attempt to reserve that space for our record.
+ if m.cas32(m.hdrLen+limitOff, limit, end) {
+ break
+ }
+ }
+
+ // Write record.
+ next, v, ok := m.writeEntryAt(start, name)
+ if !ok {
+ debugPrintf("corrupt2 %#x+%d vs %#x\n", start, len(name), len(m.data))
+ return nil, nil, errCorrupt // more likely our math is wrong
+ }
+
+ // Link record into hash chain, making sure not to introduce a duplicate.
+ // We know name does not appear in the chain starting at head.
+ for {
+ next.Store(head)
+ if m.cas32(headOff, head, start) {
+ return v, nil, nil
+ }
+
+ // Check new elements in chain for duplicates.
+ old := head
+ head = m.load32(headOff)
+ for off := head; off != old; {
+ ename, enext, v, ok := m.entryAt(off)
+ if !ok {
+ return nil, nil, errCorrupt
+ }
+ if string(ename) == name {
+ next.Store(^uint32(0)) // mark ours as dead
+ return v, nil, nil
+ }
+ off = enext
+ }
+ }
+}
+
+func (m *mappedFile) extend(end uint32) (*mappedFile, error) {
+ end = round(end, pageSize)
+ info, err := m.f.Stat()
+ if err != nil {
+ return nil, err
+ }
+ if info.Size() < int64(end) {
+ if m.f.WriteAt(m.zero[:], int64(end)-int64(len(m.zero))); err != nil {
+ return nil, err
+ }
+ }
+ return openMapped(m.f.Name(), m.meta)
+}
+
+// round returns x rounded up to the next multiple of unit,
+// which must be a power of two.
+func round[T int | uint32](x T, unit T) T {
+ return (x + unit - 1) &^ (unit - 1)
+}
diff --git a/counter/parse.go b/counter/parse.go
new file mode 100644
index 0000000..a406649
--- /dev/null
+++ b/counter/parse.go
@@ -0,0 +1,96 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+//go:build !compiler_bootstrap
+
+package counter
+
+import (
+ "bytes"
+ "fmt"
+ "strings"
+ "time"
+ "unsafe"
+)
+
+type File struct {
+ Meta map[string]string
+ Count map[string]uint64
+}
+
+func Parse(filename string, data []byte) (*File, error) {
+ if !bytes.HasPrefix(data, []byte(hdrPrefix)) || len(data) < pageSize {
+ if len(data) < pageSize {
+ return nil, fmt.Errorf("%s: file too short (%d<%d)", filename, len(data), pageSize)
+ }
+ return nil, fmt.Errorf("%s: wrong hdr (not %q)", filename, hdrPrefix)
+ }
+ corrupt := func() (*File, error) {
+ return nil, fmt.Errorf("%s: corrupt counter file", filename)
+ }
+
+ f := &File{
+ Meta: make(map[string]string),
+ Count: make(map[string]uint64),
+ }
+ np := round(len(hdrPrefix), 4)
+ hdrLen := *(*uint32)(unsafe.Pointer(&data[np]))
+ if hdrLen > pageSize {
+ return corrupt()
+ }
+ meta := data[np+4 : hdrLen]
+ if i := bytes.IndexByte(meta, 0); i >= 0 {
+ meta = meta[:i]
+ }
+ m := &mappedFile{
+ meta: string(meta),
+ hdrLen: hdrLen,
+ data: data,
+ }
+
+ lines := strings.Split(m.meta, "\n")
+ for _, line := range lines {
+ if line == "" {
+ continue
+ }
+ k, v, ok := strings.Cut(line, ": ")
+ if !ok {
+ return corrupt()
+ }
+ f.Meta[k] = v
+ }
+ if f.Meta["TimeBegin"] == "" {
+ // Infer from file name.
+ if !strings.HasSuffix(filename, ".v1.count") || len(filename) < len("-2022-11-19") {
+ return corrupt()
+ }
+ short := strings.TrimSuffix(filename, ".v1.count")
+ short = short[len(short)-len("2022-11-19"):]
+ t, err := time.ParseInLocation("2006-01-02", short, time.UTC)
+ if err != nil {
+ return nil, fmt.Errorf("%s: invalid counter file name", filename)
+ }
+ f.Meta["TimeBegin"] = t.Format(time.RFC3339)
+ // TODO(pjw): 1 isn't correct. 7?, but is this ever executed?
+ f.Meta["TimeEnd"] = t.AddDate(0, 0, 1).Format(time.RFC3339)
+ }
+
+ for i := uint32(0); i < numHash; i++ {
+ headOff := hdrLen + hashOff + i*4
+ head := m.load32(headOff)
+ off := head
+ for off != 0 {
+ ename, next, v, ok := m.entryAt(off)
+ if !ok {
+ return corrupt()
+ }
+ if _, ok := f.Count[string(ename)]; ok {
+ return corrupt()
+ }
+ f.Count[string(ename)] = v.Load()
+ off = next
+ }
+ }
+ return f, nil
+}
diff --git a/counter/stackcounter.go b/counter/stackcounter.go
new file mode 100644
index 0000000..d2a557c
--- /dev/null
+++ b/counter/stackcounter.go
@@ -0,0 +1,115 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package counter
+
+import (
+ "fmt"
+ "runtime"
+ "strings"
+ "sync"
+
+ "golang.org/x/telemetry"
+)
+
+// On the disk, and upstream, stack counters look like sets of
+// regular counters with names that include newlines.
+
+// a StackCounter is the in-memory knowledge about a stack counter.
+// StackCounters are more expensive to use than regular Counters,
+// requiring, at a minimum, a call to runtime.Callers.
+type StackCounter struct {
+ name string
+ depth int
+
+ mu sync.Mutex
+ // as this is a detail of the implementation, it could be replaced
+ // by a more efficient mechanism
+ stacks []stack
+}
+
+type stack struct {
+ pcs []uintptr
+ counter *Counter
+}
+
+func NewStack(name string, depth int) *StackCounter {
+ return &StackCounter{name: name, depth: depth}
+}
+
+// Inc increments a stack counter. It computes the caller's stack and
+// looks up the corresponding counter. It then increments that counter,
+// creating it if necessary.
+func (c *StackCounter) Inc() {
+ if !telemetry.Enabled {
+ return
+ }
+ pcs := make([]uintptr, c.depth)
+ n := runtime.Callers(2, pcs) // caller of Inc
+ pcs = pcs[:n]
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ for _, s := range c.stacks {
+ if eq(s.pcs, pcs) {
+ s.counter.Inc()
+ return
+ }
+ }
+ // have to create the new counter's name, and the new counter itself
+ locs := make([]string, c.depth)
+ frs := runtime.CallersFrames(pcs)
+ for i := 0; ; i++ {
+ fr, more := frs.Next()
+ _, pcline := fr.Func.FileLine(pcs[i])
+ entryptr := fr.Func.Entry()
+ _, entryline := fr.Func.FileLine(entryptr)
+ locs[i] = fmt.Sprintf("%s:%d", fr.Function, pcline-entryline)
+ if !more {
+ break
+ }
+ }
+
+ name := c.name + "\n" + strings.Join(locs, "\n")
+ if len(name) > maxNameLen {
+ return // fails silently, every time
+ }
+ ctr := New(name)
+ c.stacks = append(c.stacks, stack{pcs: pcs, counter: ctr})
+ ctr.Inc()
+}
+
+// Names reports all the counter names associated with a StackCounter.
+func (c *StackCounter) Names() []string {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ names := make([]string, len(c.stacks))
+ for i, s := range c.stacks {
+ names[i] = s.counter.Name()
+ }
+ return names
+}
+
+// Counters returns the known Counters for a StackCounter.
+// There may be more in the count file.
+func (c *StackCounter) Counters() []*Counter {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ counters := make([]*Counter, len(c.stacks))
+ for i, s := range c.stacks {
+ counters[i] = s.counter
+ }
+ return counters
+}
+
+func eq(a, b []uintptr) bool {
+ if len(a) != len(b) {
+ return false
+ }
+ for i := range a {
+ if a[i] != b[i] {
+ return false
+ }
+ }
+ return true
+}
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 29, 2023, 1:32:08 PM5/29/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #2 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Peter Weinberger, TryBot-Result-1 by Gopher Robot
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
8 files changed, 1,567 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 29, 2023, 2:07:21 PM5/29/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #3 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Peter Weinberger, TryBot-Result-1 by Gopher Robot
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
8 files changed, 1,570 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 11:03:06 AM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #4 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Peter Weinberger, TryBot-Result-1 by Gopher Robot
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
There is a version of mmap modified from src/cmd/go/internal/mmap.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
A mmap/mmap.go
A mmap/mmap_other.go
A mmap/mmap_unix.go
A mmap/mmap_windows.go
12 files changed, 1,700 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 11:11:01 AM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #5 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Peter Weinberger, TryBot-Result-1 by Gopher Robot
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
There is a version of mmap modified from src/cmd/go/internal/mmap.
This package does not work for systems that do not have an mmap.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
A mmap/mmap.go
A mmap/mmap_other.go
A mmap/mmap_unix.go
A mmap/mmap_windows.go
12 files changed, 1,700 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 11:17:24 AM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #6 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Peter Weinberger, TryBot-Result-1 by Gopher Robot
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
There is a version of mmap modified from src/cmd/go/internal/mmap.
This package does not work for systems that do not have an mmap.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
M go.mod
M go.sum
A mmap/mmap.go
A mmap/mmap_other.go
A mmap/mmap_unix.go
A mmap/mmap_windows.go
14 files changed, 1,704 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 1:37:19 PM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #7 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Peter Weinberger, TryBot-Result-1 by Gopher Robot
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
There is a version of mmap modified from src/cmd/go/internal/mmap.
The windows version of unmap needs to be implemented.
This package does not work for systems that do not have an mmap.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
M go.mod
M go.sum
A mmap/mmap.go
A mmap/mmap_other.go
A mmap/mmap_unix.go
A mmap/mmap_windows.go
14 files changed, 1,725 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 1:52:22 PM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #8 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Peter Weinberger, TryBot-Result-1 by Gopher Robot
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
There is a version of mmap modified from src/cmd/go/internal/mmap.
The windows version of unmap needs to be implemented.
This package is not intended for systems that do not have an mmap.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
M go.mod
M go.sum
A mmap/mmap.go
A mmap/mmap_other.go
A mmap/mmap_unix.go
A mmap/mmap_windows.go
14 files changed, 1,716 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 1:55:08 PM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #9 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Peter Weinberger
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
There is a version of mmap modified from src/cmd/go/internal/mmap.
The windows version of unmap needs to be implemented.
This package is not intended for systems that do not have an mmap.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
M go.mod
M go.sum
A mmap/mmap.go
A mmap/mmap_other.go
A mmap/mmap_unix.go
A mmap/mmap_windows.go
14 files changed, 1,717 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 2:24:33 PM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #10 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Peter Weinberger, TryBot-Result-1 by Gopher Robot
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
There is a version of mmap modified from src/cmd/go/internal/mmap.
The windows version of unmap needs to be implemented.
This package is not intended for systems that do not have an mmap.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
M go.mod
M go.sum
A mmap/mmap.go
A mmap/mmap_other.go
A mmap/mmap_unix.go
A mmap/mmap_windows.go
14 files changed, 1,721 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 2:59:12 PM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #11 to this change.
14 files changed, 1,723 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 3:10:14 PM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #12 to this change.
14 files changed, 1,729 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 3:50:58 PM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #13 to this change.
14 files changed, 1,734 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 30, 2023, 3:56:41 PM5/30/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #14 to this change.
14 files changed, 1,735 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 31, 2023, 7:38:49 AM5/31/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #15 to this change.
14 files changed, 1,789 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 31, 2023, 7:53:34 AM5/31/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #16 to this change.
14 files changed, 1,817 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 31, 2023, 8:11:54 AM5/31/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #17 to this change.
14 files changed, 1,828 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 31, 2023, 5:01:46 PM5/31/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #18 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Peter Weinberger, TryBot-Result-1 by Gopher Robot
counter: the first version of the counter package
This is a large CL. The code is pretty much as Russ wrote it,
with the addition of stackcounter.go. The package provides
counters and stack counters, and the code to maintain and
rotate counter files.
There is a version of mmap modified from src/cmd/go/internal/mmap.
This package is not intended for systems that do not have an mmap.
Change-Id: I7921345ce162a01f8cfc87cd5f9bc5d30a3b0c4f
---
A counter/STATUS.md
A counter/bootstrap.go
A counter/counter.go
A counter/counter_test.go
A counter/doc.go
A counter/file.go
A counter/parse.go
A counter/stackcounter.go
M go.mod
M go.sum
A mmap/mmap.go
A mmap/mmap_other.go
A mmap/mmap_unix.go
A mmap/mmap_windows.go
14 files changed, 1,834 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 31, 2023, 5:10:27 PM5/31/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #19 to this change.
14 files changed, 1,835 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 31, 2023, 5:50:45 PM5/31/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #20 to this change.
14 files changed, 1,838 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 31, 2023, 7:44:11 PM5/31/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #21 to this change.
14 files changed, 1,873 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
May 31, 2023, 8:05:27 PM5/31/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #22 to this change.
14 files changed, 1,883 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
Jun 1, 2023, 3:20:59 PM6/1/23
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Peter Weinberger.
Peter Weinberger uploaded patch set #23 to this change.
14 files changed, 1,884 insertions(+), 0 deletions(-)
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Peter Weinberger (Gerrit)
Jun 1, 2023, 3:54:03 PM6/1/23
to goph...@pubsubhelper.golang.org, Gopher Robot, golang-co...@googlegroups.com
Peter Weinberger abandoned this change.
Abandoned
it has served its purpose of getting to minimal test failures
To view, visit change 499095. To unsubscribe, or for help writing mail filters, visit settings.
Reply all
Reply to author
Forward
0 new messages