Metrics + GAE

[Mark Mandel]

Hi!

I'm having some fun trying to integrate some metrics into our GAE application, so we can build alerting and dashboard around them. Some of these metrics are performance (which will eventually get replaced by the new Google Cloud Monitoring), and some which are business Analytics.

I was looking at https://github.com/rcrowley/go-metrics for metrics needs - but this will need to remain in memory while the application is running. I'm wondering if that is going to be a problem? (Obviously it could get cut short as a VM goes down)

Then it usually does something like (https://github.com/rcrowley/go-metrics/blob/master/librato/librato.go#L40-L53)

func (self *Reporter) Run() {

ticker := time.Tick(self.Interval)

for now := range ticker {

               // send metrics

}

}

Within a go block so it runs concurrently.

Is this sort of approach even a good idea on GAE? I'm wondering if it's a good idea or not. Instances can come and go any time, and the go runtime is limited to 1 thread.

The other options I'm floating is running a small StatsD server on a Cloud VM instance, and just sending it my data, which can then be sent on wherever I want, but that could potentially be more work, and has to be monitored, etc.

Feedback would be great, I'm not quite sure what the best approach is here.

Thanks!

Mark

--
E: mark....@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

2 Devs from Down Under Podcast

http://www.2ddu.com/

[Brandon Thomson]

I assume you're wanting to run this on frontend instances (backends are totally different and the advice below does not apply).

Storing your metrics in instance memory could work as long as you don't mind losing any recent data when the instance goes offline. But this kinda goes against the GAE way of doing things, which is usually to write everything to persistent storage before the http request has completed.

I would not recommend keeping volatile data in instance memory unless it is really necessary for performance or cost reasons. Better to write the metrics to datastore immediately as they are generated, or else ship them off to external service like stathat or Google Analytics immediately.

Also, requests have a time limit (I think around 90 seconds?), so time.Tick() may not work like you expect.

[Mark Mandel]

On Sat, Jun 28, 2014 at 9:58 PM, Brandon Thomson <b...@brandonthomson.com> wrote:

I would not recommend keeping volatile data in instance memory unless it is really necessary for performance or cost reasons. Better to write the metrics to datastore immediately as they are generated, or else ship them off to external service like stathat or Google Analytics immediately.

Thanks for the feedback. The issue I'm hitting is all the services I've been looking at (datadog for example) have some kind of 'collector' between them and the service I need. I really don't want to write my own if I can avoid it.

You are right in that I could write my data to datastore (which I currently do anyway), and then process it myself, as a cron job (or similar) every n seconds, and then send it off. That could potentially work quite well actually.

Much appreciated.

[Mark Mandel]

Thanks for the link to stathat too - this looks like it may well be another good answer. I didn't find them on my travels. They don't have an external collector, but it looks very easy to their API in Go to WaitUntilFinished() (which could well be queued off), without having to run some sort of collector.

Mark

[David Byttow]

FWIW, we (Secret) do two things:

1. We have some "varz" which are simply day timestamped named metrics (e.g., varz:2014-6-30:foo-counter=24252) backed by memcache. Every minute or so, a cron job runs which collects them and sends them to our monitoring service (Librato in this case)

2. For more fancy stuff, we collect them in memory and flush them periodically to the taskqueue and batch upload them there.

Kind of wonky, but it works.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Mandel]

Sounds very similar to what I was thinking.

Thanks for sharing!

Mark