Dapper

[David Robinson]

Hi all,

I'm interested in using the net/rpc library in a distributed system, but would like to be able to trace requests using a system essentially the same as the Dapper tracing system from Google [1]. A question was asked [2], several years ago, regarding the status of a tracing system for Go, and at the time a solution hadn't been investigated. I'm curious whether there has been any advances in this area since?

I hacked up (literally - it's ugly) the net/rpc code to support passing an identifier as part of a request which, unless I'm mistaken, solves the lack of thread/goroutine local variables problem. My implementation is terrible, but is the concept/technique sound? Code is here:

https://gist.github.com/zxvdr/5624054

[1] http://research.google.com/pubs/pub36356.html
[2] https://groups.google.com/forum/?fromgroups#!topic/golang-dev/8sXBJuH8feM

--

Dave

[Andrew Wilkins]

Hi Dave,

I've forgotten most of the specifics of the paper, and my office's network is too rubbish to download it in a reasonable time, so take this with a grain of salt...

IIRC, the TLS variable was used to transparently record the ID so existing users of the RPC client didn't need to be modified. Sure, you can modify net/rpc to pass it in manually, but then you have to modify all of the callers as well.

Apart from that, making an additional RPC call will hurt at scale. I'd shoehorn the ID into the existing request, but that will require modifying the codec.

Cheers,
Andrew

[David Robinson]

Hi Andrew,

The ID is shoehorned into the request header, so there isn't any additional RPC. The tricky part is not necessarily transporting the ID, but storing it somewhere useful, since there's no thread/goroutine local storage, which is what's used in C++/Java IIRC. Storing it in the exported object seems like a neat trick, that way it can be accessed without polluting the method's arguments and it's local to the goroutine. AFAICT the method object isn't used for anything, it only exits to allow reflection of exported methods, their arguments and return value.

I suspect you could even avoid manually passing on the ID - if the object implemented a Call method it could do remote RPC calls and forward on the ID found in the object. Adding a Call method may complicate things though (is it an exported method that can be called remotely, or not). Either way, I don't think it's impossible to solve with this "store it in the exported object" technique.

--

Dave

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Andrew Wilkins]

On Wed, May 22, 2013 at 10:41 AM, David Robinson <zxvd...@gmail.com> wrote:

The ID is shoehorned into the request header, so there isn't any additional RPC.

So it is. Sorry, I misread the bit that was calling Trace on the server-side as being on the client-side.

The tricky part is not necessarily transporting the ID, but storing it somewhere useful, since there's no thread/goroutine local storage, which is what's used in C++/Java IIRC. Storing it in the exported object seems like a neat trick, that way it can be accessed without polluting the method's arguments and it's local to the goroutine. AFAICT the method object isn't used for anything, it only exits to allow reflection of exported methods, their arguments and return value.

Right; sorry, I didn't mean to suggest my (invalid) performance concern was the difficult part.

I suspect you could even avoid manually passing on the ID - if the object implemented a Call method it could do remote RPC calls and forward on the ID found in the object. Adding a Call method may complicate things though (is it an exported method that can be called remotely, or not). Either way, I don't think it's impossible to solve with this "store it in the exported object" technique.

My point was not that your method is not workable, but unless you have some place like TLS to store it, it will require modification of the code for more complicated use-cases. TLS can be used to trade an assumption about thread locality for transparency.

In the past (and still from time to time) I worked on something very similar to Dapper for doing end-to-end transaction tracing. If you have multiple RPC calls chained together, then you'll want to correlate them somehow. Using TLS simplifies this in the face of multiple, mutually-ignorant components. This is assuming you can make the assumptions that rsc mentions don't hold for Go.

e.g. A calls B, B calls C. At A you send a unique ID to B, B stores it in TLS; B grabs ID out of TLS and sends it to C. The TLS bit is only necessary if A->B and B->C are handled by two different code paths that can't easily know about the other.

Cheers,
Andrew