Comments on: Meanwhile: More Meta-Metrics

By: prasana

prasana — Mon, 19 Oct 2009 22:40:00 +0000

@admob we have since the summer started mining nagios alert logs across different dimensions,
1. application [easily done in our env because we pointed them to different contactgroups]
2. colo
as with any of this data visualization [and good one at that] is a worthwhile investment and a has yielded for us the result that we do get to focus on things besides the high runners only.

great post and really like your take on the metrics as a way of doing the “showme” and leading to a better engineered system. thanks and keep up the good work.

By: Daily Links #111 | CloudKnow

Daily Links #111 | CloudKnow — Sat, 10 Oct 2009 16:58:35 +0000

[...] John Allspaw: More Meta-Metrics [...]

By: Dan Ackerson

Dan Ackerson — Wed, 07 Oct 2009 18:10:32 +0000

Thanks for the trackback, John!

It seems like we have all the fundamental tools for monitoring server problems (nagios), site outtages (pingdom) and release metrics (cruisecontrol). What we’re missing is your “meta-metrics” layer that provides another layer of abstraction on top of these. Performing some intelligent data-mining to give us some better insights into how we are _really_ doing operational wise.

I’ve got a damn Excel pivottable in my head for some reason : cruisecontrol release metrics vs. nagios alerts vs. pingdom warnings … now just add some Splunk trending to the mix. Very sexy tool indeed!

By: Tom Cignarella

Tom Cignarella — Tue, 06 Oct 2009 17:55:07 +0000

At Clickability we do something similar where we track and publish a monthly metric on “total alerts sent” and “number of night’s on-call was woken up”. Clearly the later is critical to keeping an operations team sane and happy. We’ve not gone down the path yet of further analyzing the results rather we spend our time making sure that every single alert is an actionable item that requires human intervention, If it does not we create automation or set it to not page.

I would love to be able to utilize an open source page that digs deeper in Nagios to understand alerts better. Thank you for suggesting it.

By: Abe Hassan

Abe Hassan — Mon, 05 Oct 2009 20:54:28 +0000

At Six Apart we’ve put together a bunch of scripts that analyze the Nagios logs for alert notifications; we have a couple twiddles to only display critical alerts, or only nighttime alerts, or both. We get a nightly report with a breakdown of top alerts, broken down by host (and then by service within that host) and a breakdown by service (and then by host — useful for seeing if a check is broken or misconfigured, or if there was some more widespread problem).

It gives me a great view into not just what people are woken up for, but the general pulse of our systems overall. (I’d be happy to clean it up and share it with the world … perhaps I should.) And really, at some point, something that sits in a “warning” state for an entire weekend — never actually paging someone — is just as actionable and just as important as a false positive critical alert: it either needs to be silenced or it’s indicative of a real problem.

By: Vladimir Vuksan

Vladimir Vuksan — Mon, 05 Oct 2009 20:21:38 +0000

That is certainly a sore point. Lots (most?) places will alert for everything and anything without much consideration for the human component. What’s even worse if there is ever a major outage even more alerts will be added draining the human capital even further.

In the past I have mostly tried to identify those alerts that are truly critical ie. are worth of waking someone up however the idea of graphing them is an interesting one. I don’t believe there are any tools whether open source or commercial that are available for such a purpose. I’d definitely be interested in one.