Monitoring uptime for a single app from all its dynos

[Tom Buckley-Houston]

Hi Everyone,

I've got a working version of uptime monitoring and I was wondering what you thought of the approach;

Firstly, rukorun uses setInterval() to send a socket message to dynohost every 60 seconds.

Then, dynohost sends a request to the API server with the instance_id of the heartbeat.

Finally, the API server cross references the instance_id to find the parent app_id and increments a field in the app table called 'heartbeats'.

I had previously considered using the transitioned_at date on the instance_state table, but this just felt like it was too vulnerable to going out of sync, what if an instance doesn't correctly report its state change? So, my approach just seems like the simplest way for a basic working version at the moment. The only problem I can foresee is scalability, perhaps if there are tens of thousands of instances then it might incur a non-trivial load on the database, but even then it doesn't seem too bad, the heartbeat is only once a minute after all.

I looked into it and the UPDATE command is atomic so there shouldn't be any race conditions when multiple instances are sending heartbeats to the same app.

Thanks,

tom :)

[Romain]

It simple to understand, it's perfect :)

Two things, if you merge it with openruko :

 * could you add some tests?

 * in api server, if you change the schema, create a new file with a new semver version : 0.2.0.do.app.pgsql for exemple

ALTER TABLE app ADD COLUMN heartbeats integer;

And in the package.json update the version to 0.2.0

It will simplify db migrations.

Great feature, it's nice to see it implemented ;)

2013/2/10 Tom Buckley-Houston <t...@tombh.co.uk>

--
You received this message because you are subscribed to the Google Groups "OpenRuko" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openruko+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tom Buckley-Houston]

Thanks Romain :D

I will definitely write tests and add the migration, yes :)

First I will merge your dibigrator work on apiserver and all the changes we've been making to chef-openruko into slotbox.

tom :)