Observability + Google Cloud Functions

[Sam Stern]

Hey all,

I've been looking into ways to get more detailed observability information from my Node.js GCF backend. I was able to get Cloud Trace working although only after cobbling together advice from the GCP docs (which were sparse) and the Dynatrace docs which had advice on using OpenTelemetry + GCF.

Originally I wanted to use DataDog for this (we use them for other things) but they seem to have no support for GCF or even for fully managed Cloud Run. Strangely they do have support for AWS Lambda ... ¯\_(ツ)_/¯

If anyone else has gone down this road and found something they like, I'd love to hear about it!

- Sam

[Daniel Lee]

Hey Sam!

Good to hear from you.

I agree with you that we don't have any great documentation around on integration Cloud Trace with GCP Serverless product offerings like Google Cloud Functions.

Yes - Open Telemetry is suppose to be the new hotness, but there are several foot guns that makes it challenging to use with GCF/Cloud Run:

1) GCF/Functions Framework does offer any "lifecycle hooks" that ensures that generated traces are exported before the container is shutdown. Github Issue discussion.

2) Reporting traces AFTER request completes, i.e. network request outside of the incoming request context, leads to forced container shutdown. This means that you'll see a lot of cold starts. Github issue discussion.

3) Note that Cloud Run/GCF automatically traces at most 0.1 request/s [doc]. You might see orphaned spans if you configured to have your own sampling rate. I think https://www.npmjs.com/package/@google-cloud/opentelemetry-cloud-trace-propagator is suppose to fix this, but it still is strange (to me) that you have no control over the overall sampling rate.

Last time I tried hooking up GCF + Cloud Trace, I found this code snippet helpful. There are some compatibility issues w/ otel-js and GCP's otel exporter, but that might be fixed now.

tl;dr - It's a wild west out there.

I'd be interested in hearing from anyone else who have experience in this area.

Thanks,

Daniel

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/4aa7e484-40f6-4989-b473-fb7853d5c7f5n%40googlegroups.com.

[Sam Stern]

Thanks Daniel! This is a lot of great information and it's making me consider some of the hacks I was using to get tracing, I did not consider that there could be negative consequences.

- Sam

You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/UdxZnwss8w0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAF53BK5%3DxQL_ze9DDLBC9dcy1Uxh%2BAgEtuD33-R6NnruoqHhYw%40mail.gmail.com.