Thoughts on capacity planning and web operations.

The minimum cycle of questions you should be asking yourself. As brought up by @debuggist and @benjaminblack.

{ 1 comment }

I think I’ve said before somewhere that working in the field of web operations prepared me somewhat for being a parent. I thought the other day that I should write down some of this reasoning, because it’s pretty often that I’m reminded of similarities:

High availability

Having redundant infrastructure is WebOps 101. For my kids’ most prized possessions, their sleeping  ‘loveys’ there is no reason to have a SPOF, under any circumstances. We have at least 4 backups for each on any trip that we go on, as well as a couple of trusted stuffed animals who might meet unfortunate fates.

Capacity planning

This applies to both disposable diapers (a.k.a. consumable capacity) and episodes of the few TV shows we allow them to watch, on the Tivo. My daughter, at 3 and a half, knows every detail from every of the 49 episodes of The Backyardigans. Having some of them on ipods and iphones can make a 6 hour drive to L.A. feel like 4, not 12.

Documentation

Since I’m already used to writing down observations and techniques learned ‘in the field’, then I was totally prepared:

Allspaw Baby Soothing Method, v1

and in case I ever forgot what my most successful swaddling method was:

Architecture and design

It’s unfortunate that I was so sleep-deprived that I never got a photo of the RadioShack remote-control truck that I turned into a cam-driven Moses basket automatic rocker mechanism. But you understand what I’m talking about.

There is one other thing that I learned from working at Flickr which turned out to be useful new parent advice: expect the unexpected, and never rely on past behaviors as an indication of what can happen in the future. They’re kids, not applications.

{ 2 comments }

Wow, how did I miss this until now? In June, some smart people gathered in Barcelona for the First Workshop on Automated Control for Datacenters and Clouds (ACDC09) and jeez it looked like it was a good time, from a glance at the program.

One of the cooler papers is “Automatic exploration of datacenter performance regimes” in which the smart folks over at the RAD Lab at UCB tackle the idea of:

1. Gathering up real usage metrics in production
2. Taking that data to feed a resource allocation (”auto-scaling”) controller

The bits about coming up with an exploration policy is where the juicy stuff comes in, building in safety factors driven by external SLAs. You should read the whole thing to see how thoughtful their method was, which includes taking into account effects such as cold ramping, which you almost never see accounted for in simulated situations.  Rock on, RAD Lab: this is the stuff that brings the academia smarts to the real world. Kudos.

FYI: I’m not just saying the paper is cool because they cite my book as a resource in it.

{ 10 comments }

I’ve said it before that I’ve always been a huge fan of SystemImager, for super simple imaging. It has some shortcomings for config management, but those are solved with things like Chef or Puppet.

With all of the great things being talked about surrounding ‘Automated Infrastructure’, I’ll point to something insanely cool: 1,190 nodes installed from bare metal to all done in 15 minutes.

That’s One Thousand One Hundred and Ninety nodes. Completely installed in: Fifteen. Fucking. Minutes.

{ 7 comments }

Excellent. Good work, Ben:

ah, the mighty service level agreement! the tooth and claw by which the wily customer brings the vendor to heel. get the SLA right and you, the customer, can sit back and relax, safe in the knowledge that should there be an outage, you are covered. your business is protected from harm by the warm, experienced embrace of a big, stable telco. pinch me, i must be dreaming.

go read the whole thing.

{ 0 comments }

I can’t tell you how ripped I get when people say things like this:

“cloud computing means getting rid of ops”

If by “ops” you mean “people in data centers racking servers, installing OSes, running cables, replacing broken hardware, etc.” then sure, cloud computing aims to relieve you of those burdens. If you really think ‘ops’ is just that, then you really should put down your Nick Carr book and pay attention to the real world for a change.

The reality is, if your ops team is spending a lot of time doing that, then you’re either:

1. Too big to use someone *else’s* cloud, because you basically have your own (Yahoo, Amazon, Google, etc.)
2. Stuck in 1999.

If you deal with any of these things:

• handling site issues/incidents
• building and maintaining tools to monitor and gather systems and application-level metrics
• program abilities to adapt infrastructure to changing system or application-level conditions (usage, failure, degradation, etc.)
• implements, and maintains deployment systems (code, config management, etc.)
• capacity planning (no, really)

then you’re doing “ops”, by my definition. In some environments, these things are done by “developers”. But my definition says those devs are performing ops functions.

Cloud computing isn’t going to make ‘ops’ go away, it’s relieving of ops (and dev) of a bunch of pain-in-the-ass things so they can focus on the real work needed. Namely: your application.

Last I checked, clouds don’t perform the tasks listed above, because those things (done right) are application-specific. And while cloud computing enables (in an excellent way) the efficient resource allocation (or de-allocation) for an application, it doesn’t get rid of the need to do the above things.

{ 8 comments }