adding context.Context to new code

[meirf...@gmail.com]

I'm adding tracing to an existing code base with many packages and it seems the best way to have context's passed around is to just have every method take a context.Context. 

Is there any tooling for converting a code base/package to have:

(a) context.Context as the first parameter in each function - ctx context.Context

(b) for any function that has changed, have its callers (within that package) pass ctx as the first arg

The difficulty here seems to differentiate intra package calls from calls to standard/3rd party libraries which shouldn't be having new param.

[mhh...@gmail.com]

I want something similar too.

Automatic and smart insertion of context args in a chain of calls.

Methods signature updates are easy, but how to appropriately insert context check in the ast  ?
I m not sure yet.

>The difficulty here seems to differentiate intra package calls from calls to standard/3rd party libraries which shouldn't be having new param.

That does not sound too difficult, from the pkg identifier, lookup for the import path, for every import path, exists in GOROOT ?

Please put updates here anything you want to share.

At that moment i m using this package to help me with ast,
https://github.com/mh-cbon/astutil

might be a start even though it needs refactoring.

[Sameer Ajmani]

The eg tool can execute simple refactoring steps, but automating context plumbing through a chain of calls is an open problem. Alan Donovan put some thought into this a few years ago, and I've done a limited form of this using awk ;-)

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mhh...@gmail.com]

> I've done a limited form of this using awk ;-)

if you have a minute,

can you tell more about what limited you
in your attempts and which trade made you stop (guessing),
if any ?

Do you still think it be awesome ?
Or have you made your mind to an opposite position ?
if so, For which reasons?

My tool is very poor, consider it as on going, a place for inspiration to get started from absolutely no idea to lets get a dirty prototype.
not sure yet how long is going to be the road, still digging :)

[Sameer Ajmani]

Our approach was to identify function calls that consume a Context (indicated by a call to context.TODO in the function body) and function calls that provide a Context (such as RPC and HTTP handlers). Then we use the guru callgraph tool to find all call paths from context providers to context consumers. These are the call paths that need to have a context plumbed through them.

Starting from a context consumer, we can work upward to the context provider, adding a context parameter to reach intervening function signature. Replace context.TODO in the consumer function body with the new ctx parameter, then update all the places that call the consumer to pass context.TODO. Now we have a new set of context consumers. Repeat until you reach the context providers (if you reach main or a Test function, pass context.Background instead).

This works OK for static function calls but gets messy for dynamic calls. If you need to add a context parameter to an interface method, now you have to update all implementations of that method, too (guru can find these for you). And if that interface is outside your control (like io.Writer), you cannot change its signature, so you have to pass the context some other way (such as via the method receiver).

This gets yet more complicated if you cannot make atomic changes to all callers of your functions, because callers may be in other repositories. In this case, you must do an incremental refactoring in multiple steps: each change to a function signature involves adding a new function that has the context parameter, then changing all existing calls to use the new function, while preventing new calls to the old function, so that you can finally delete it.

Inside Google, we ended up not needing to build all this: Context was introduced early enough that Go users could plumb it manually where needed. I think a context plumbing tool could still be interesting and useful to other Go users. I'd love to see someone build it!

S

[mhh...@gmail.com]

Thanks a lot!

Might i guess and try to generalize your explanations into
"we tried to write a plumber for all cases possible"

Which matters a lot, in my humble opinion.

At least for the various reasons you put there,
simply put,
because it seems not technically achievable.

Still i m happy you gave me those details, they are of interest indeed.

I rather intent to solve the problem on a smaller surface
with more predictability, less impossible solution to find.
I hope so.
And if it saves 90% of the time, that s already a win, imho.

May i ask another question,
have you considered to create the plumbing
at runtime rather than statically ?
With some intents from the syntax, necessarily
(and yeah go 1 compat will be a problem, let s forget it for 1 minute).

I suspect in some aspects in might be handier,
because it might be all about chained func calls and not type system handling,
and it might be easier to interleave the ctx.check in the flaw of ops,
I don t know enough to realize for sure.

[Sameer Ajmani]

I think you are asking whether we considered integrating Context into the runtime, so that it does not need to be passed explicitly. Yes, we discussed this, but decided against it. The flow of request processing in Go may include multiple goroutines and may move across channels; we decided an explicit Context made this much easier to get right. C++ and Java use thread-local Contexts, and while this is convenient, it is often a source of mysterious bugs.

[mhh...@gmail.com]

thanks,

..integrating Context into the runtime..

50% runtime, 50% syntax with explicit contextualization.

..The flow of request processing in Go may include multiple goroutines and may move across channels;

yes big ? mark here. might the 50% of an handy and explicit syntax help with it?

C++ and Java use thread-local Contexts, and while this is convenient, it is often a source of mysterious bugs.

thanks! I don't know them,

I quickly checked according to this
https://dzone.com/articles/painless-introduction-javas-threadlocal-storage
I may have totally wrong, the syntax does not look like, not even from a 100km,
to how i represent go contexts in my head.

This is more like the actor pattern dave cheney talks about in
https://dave.cheney.net/2016/11/13/do-not-fear-first-class-functions
(search for  Let’s talk about actors)

Is the dzone link correctly describe
what you mentioned as being go context equivalent in java ?

sorry my questions are so basic.

[Sameer Ajmani]

Hmm, I'm not finding good open-source examples of ThreadLocal context propagation in C++ and Java.  My experience with this is based on what Google uses internally.  Perhaps someone more familiar with context use (tracing?) outside Google can chime in? +Jaana Burcu Dogan 

[Henrik Johansson]

With the whole "Reactive" movement thread locals have started to vanish at least in the Java ecosystem.
I agree with Sameer that, while convenient, it comes with a whole set of obscure bugs.

[mhh...@gmail.com]

A reference i found about "reactive programming"
https://github.com/dsyer/reactive-notes/blob/master/intro.adoc

Hope this introduced the concept correctly, thanks for pointing that.

Following are only some thoughts,

Two things surprising here,
1/ a programmer prefers not to solve a problem
2/ one failed attempt mean the end of all attempts**

That being said, my little understanding so far is that current context is used for two purposes,
- cancellation
- data (transmission? not sure) (+/- ts, have not checked i expect so in regards to previous referenceS cited)

While cancellation is a feature i really want to take advantage of,
the other one is much more arguable in the sense that
it can remains expressed purely by the end user without bloating
his productivity and quality (you don t really need it everywhere, just where it is needed),
it seems to be highly software context dependent.

Whereas cancellation, while it looks likes simple, is maybe more subtle.
After all it is about assigning uuid to a chain of call and
appropriately propagate, disjoint/rejoin new uuids with the previous one,
so that we can ask to stop execution of a sub selection of a chain of calls
via an handle.
Am i too simplistic?

Its difficulty reside in its requirement to be passed absolutely everywhere,
That reveals an important fact about the nature of cancellation,
it is there, it is everywhere, at all time (...within the program lifetime),
it is somewhere in the background of every ops since the very beginning the program began,
but not necessarily enabled, and certainly not a straight line.

That is where the syntax might help to establish the plots
that the runtime consumes to connect the dots
and support what the developers want to achieve,
in my understanding so far.

My best comparison to cancellation is
request rays tracing in micro services oriented architecture,
on both end it is a multi-tenant,
it always start with one ray,
the ray always split into multiple rays,
because we do programming, we need ways
to create, control distribute existing/new rays,
and possibly let the userland introduce a new behavior for the rays.

So yeah basically main has a ray,
if you process an http request,
you to create a new ray
to be able to cancel only that request,
but it needs to be connected to the main ray
because if main should be canceled,
that signals should propagate to all rays connected with it,
probably.

** i want to put emphasis because in the description provided
by Sameer, as i tried to summarize, It has been tried to handle
the problem via the type system as if it was
a completely defined set of types.
Which in go is wrong (yeah i m very affirmative here :),
from my understanding, that might be correct in other oop languages.
Technically it is possible to rewrite interfaces, methods signatures, taken individually,
as a whole, and from the consumer pov,
i d say it is an impossible task in go because it it goes against its nature.
And i confirm/understand that by reading to Sameer feedback.

Notes: i m a strong lover of go type system (not talking about values and pointers ;)

[Peter Weinberger (温博格)]

Hi. I was one of the people who failed in an attempt to auto-insert contexts in all of google3. I no longer remember all the obstacles I failed to overcome, but would encourage others to take on the project.

One issue was libraries that were used both in paths from http requests (so they needed the context propagated through them) but were also used in places where their callers (starting at main) didn't have any contexts at all. Sitting by my coffee this morning this no longer seems like an insuperable obstacle (e.g., pass nils and change them to context.Background() inside, or, like appengine's log, split packages into one that requires contexts and one that doesn't). But I had lots of what I thought were plausible ideas that broke on the hard rock of google3, and I suspect these would too.

(The project wasn't a total failure. It did plumb contexts through a bunch of packages. But the general case was too hard for me.)

[Sameer Ajmani]

Generally I'd suggest passing context.Background() when calling functions that need a context from main or tests.

[Sameer Ajmani]

Specifically: don't pass nil Contexts. They are not valid, and most code that uses Contexts don't check for nil and will panic.

[Peter Weinberger (温博格)]

the comment was for libraries without context parameters but which would need to have them added.

[mhh...@gmail.com]

+1 for any feedback.

> One issue was libraries that were used both in paths from http requests (so they needed the context propagated through them) but were also used in places where their callers (starting at main) didn't have any contexts at all

yup.
Imagine also in the middle of the call you invoke a third party package without context support.

The problem mlight be even worst because of the weaknesses of the current type system.
You can t simply declare a type mypackage.TContexted of foreign.T and hope it will work.
I suspect it might need a whole re-factoring/re-integration of the foreign package and its dependencies.

> or, like appengine's log, split packages into one that requires contexts and one that doesn't)

yes i think its an approach that can work if taken appropriately.

Taken in its basic form:
- its expensive, money costs * 2
- its prone to bug, duplicated code, duplicated bugs
- its not genuine nor magic, it still needs lots of handcrafted work

________________________________________________________

A sarcastic question that bugs me,
and probably wrong in its term as is,

Is go the language of the 21th century,
or the last language of the 20th century
?

[Nazri Ramliy]

I'm about to embark on doing the same, and my first pit stop is at
using the callgraph to generate the caller-callee data but stumbled
upon this error while trying out the example given by the "callgraph"
command when run without arguments:

$ callgraph -format '{{.Caller.Pkg.Object.Path}} ->
{{.Callee.Pkg.Object.Path}}' /usr/local/go/src/net/http/triv.go | sort
| uniq
callgraph: template: -format:1:9: executing "-format" at
<.Caller.Pkg.Object.P...>: can't evaluate field Object in type
*ssa.Package

any idea what I'm doing wrong?

nazri

[Nazri Ramliy]

For those who stumble upon similar error, here's what works for me:

$ callgraph -format '{{.Caller}} -> {{.Callee}}'
/usr/local/go/src/net/http/triv.go | sort | uniq
bufio.init -> bytes.init
bufio.init -> errors.init
bufio.init -> errors.New
bufio.init -> io.init
bufio.init -> unicode/utf8.init
bufio.NewReader -> bufio.NewReaderSize
...

The documentation printed out by callgraph (when run without any
arguments) says:

Caller and Callee are *ssa.Function values, which print as
"(*sync/atomic.Mutex).Lock"

That seems to no longer be true with the latest version of callgraph.

nazri

[Yash Bansal]

@nazri Were you able to achieve this? Adding context to every function. Is there any repository/library available for the same. 

I have been one of the unaware ones who started writing a new service without adding context explicitly. 

::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.