golang mem leak investigation help

[Josh Jaques]

Hey one of my production processes has a slow leak of goroutines buiding in the grpcsync CallbackSerializer function.

So I thought maybe it was leaking a grpc connection somewhere and the associated balancer wrapper but so far I've been unable to determine where it could be happening. 

I thought maybe there would be associated sockets that were leaked but I haven't been able to find any sign of them as all socket counts on the host look normal.

Running `curl "localhost:6060/debug/pprof/goroutine?debug=2"` produces output like the following which also doesn't show me much detail for the actual source of where my process created these connections:


goroutine 12 [select, 1254 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000991100, {0x3d0a388, 0xc0009297c0})
        /Users/me/go/pkg/mod/google.golang.org/gr...@v1.58.3/internal/grpcsync/callback_serializer.go:83 +0x12a
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
        /Users/me/go/pkg/mod/google.golang.org/gr...@v1.58.3/internal/grpcsync/callback_serializer.go:55 +0x138

goroutine 13 [select, 1254 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000991220, {0x3d0a388, 0xc000929900})
        /Users/me/go/pkg/mod/google.golang.org/gr...@v1.58.3/internal/grpcsync/callback_serializer.go:83 +0x12a
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
        /Users/me/go/pkg/mod/google.golang.org/gr...@v1.58.3/internal/grpcsync/callback_serializer.go:55 +0x138


Wondering if anyone can suggest additional tips to try debugging this? Thanks for checking

[Easwar Swaminathan]

Could you please file an issue at https://github.com/grpc/grpc-go/issues/new/choose and the team will be able to help you better there. Thanks.