Cloud Trace API on AppEngine - latency and other questions

[Alexander Bertram]

Hi there,

I've started to look at using the new CloudTrace API to add information on CloudSQL requests to traces on AppEngine requests, which we desperately need, but I encountered two challenges:

First, even batching all the traces for a request together in a single API call adds about 100 ms latency to the request (that's the /urlfetch.Fetch bar below), which I don't think I can stomach in production. Is there anyway to piggy back custom trace data onto AppEngine's existing reporting mechanism, which seems to run independently of requests? 

Second, is there any way to detect which requests AppEngine has sampled for tracing so that we only spend the time to report traces for those that are sampled for RPC traces?

Thanks,

Alex

[Adam (Cloud Platform Support)]

You could try wrapping the Cloud Trace API call in a push task and executing it asynchronously at the end of the request which would save you needing to wait for the URL Fetch to complete inside the request handler. There isn't a way to 'piggy back' on top of the RPCs used internally (that is documented at least - you could always try poking around in the ApiProxy code in the SDK). There exists the RPC API for Cloud Trace however the service definitions for generating gRPC client code are not yet available.

[Brad Abrams]

Great thanks Alexander... We really appreciate your feedback. 

We (the trace team) are working on solutions to improve this latency. Meanwhile, the eng team would suggest the your use task queue to schedule the cloud trace API call outside of user requests. As Adam suggests...

 

Second, is there any way to detect which requests AppEngine has sampled for tracing so that we only spend the time to report traces for those that are sampled for RPC traces?

Not sure which runtime the you are is using. In Managed VM, we already make trace context available to external developers as a HTTP header “X-Cloud-Trace-Context” (value format: traceid/spanid;o=options).

We are working to expose the same header in Google App Engine.

Trace context tells the user whether the request is sampled. Here is the sample code that parses the trace context header.

https://github.com/liqianluo/gcloud-trace-java/blob/master/cloud-trace-sdk-java-core/src/main/java/com/google/cloud/trace/sdk/TraceContext.java

Hope that helps!

..brad

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/0f0433fc-8351-427b-8ae8-8fcdf54d1943%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alexander Bertram]

Thanks for the response!

As I mentioned to Qi, this is very helpful, just would like to add that in general, it seems like overkill to report traces over HTTP from within AppEngine, could the traces not just be parsed from the AppEngine logs?