Kitchen Soap » Capacity Planning

Some Things We Did Today

allspaw — Thu, 05 Mar 2009 20:59:34 +0000

Moving one of our eight photoserving farms from hardware Layer7 URL hash balancing (expensive, has limits) to L4 DSR balancing with CARP (cheap and simple) and figuring out how to juggle 18,000 requests/second while we do it.
Built yet some more automated query analysis reporting (with some yummy MySQLProxy)
Added yet another aggregated graph of database queries, broken down by type and cluster
Bunch of cfg mgmt changes (polishing up IO scheduling and filesystem tunings in a 2nd datacenter, more caching of search results)
Review of the higher priority to-dos in the Ops open bug queue (only 155 open! )
Finding new capacity ceilings for the image processing, given some recent optimizations)

Speaking at Web2.0 Expo 2009

allspaw — Fri, 20 Feb 2009 01:22:43 +0000

Looks like I’m gonna talk about even more nerdy things at the Web2.0 Expo in April.

You don’t have to wait for a recession to tighten up your operations. Squeezing more oomph out of your servers (or instances!) is always a good thing, and streamlining how you handle site issues is too. We’ll will talk about what we’ve been doing at Flickr to get more out of less from both our machines and our humans.

Capacity Hacks: diagonal scaling, tuning opportunities, and some other stupid performance tricks.

Ops “runbook” Hacks: Server and process self-healing, application-level measurement, ops communication tools, and some worst-case scenario tricks to have in your back pocket.

I wrote a book about common sense.

allspaw — Mon, 06 Oct 2008 15:34:34 +0000

Whew. That took longer than I thought.

Todd Hoff over at the High Scalability blog has an email interview with me about a book that I wrote, called “The Art of Capacity Planning: Scaling Web Resources“. I’m still just happy that I got it done at all, seeing how it was due the same week that my son was born.

This book happened because of a LOT of people. You know who you are, because you’re in the Acknowledgements.

More back-of-envelope-math…

allspaw — Thu, 18 Sep 2008 16:21:45 +0000

Via kottke: some good examples of doing rough math in your head, causing you to guess about assumptions all along the way.

IMHO, being able to do this is one of the things that makes a good web ops person. The examples might be “useless”, but the process is invaluable.

Internet-Scale Efficiency

allspaw — Tue, 16 Sep 2008 23:32:42 +0000

James Hamilton’s excellent LADIS 2008 presentation has lots of great stuff in it about internet scale bits. Cool stats.

Four chapters of the new book on RoughCuts…

allspaw — Fri, 13 Jun 2008 22:04:31 +0000

So now there’s chapters 1-4 on Safari RoughCuts. Which means if you don’t mind shelling out the dough, you can take a look at what I’ve been getting up early for every day for the past few months. The working title is “The Art of Capacity Planning” and it’s meant to be a no-nonsense description of the capacity planning process and considerations for web operations.

I still have two chapters to go before it’s all finished, but if you’re nice enough to take a look at what I’ve got thus far, I’d appreciate any feedback. I’m sure there could be typos and some graphs misaligned, but such is life with “drafts”.

Slides from Web 2.0 Expo2008

allspaw — Tue, 29 Apr 2008 17:15:39 +0000

Here they are.

Too big to use utility computing ?

allspaw — Thu, 28 Feb 2008 04:39:27 +0000

Dear users of S3, EC2, and other ‘utility’ computing stuffs:

Here’s a crude and completely oversimplified evolution of infrastructure needs of a growing website, with an assumption:

Have you ‘outgrown’ your original use of utility computing, for whatever reason ? If so, what was the reason? Financial? Technical?

Why I’m asking:

I’m in the process of writing a book on the topic of capacity planning for web architectures, so I’m interested in what you’ve got to say.

Datacenter Operating Systems

allspaw — Wed, 20 Feb 2008 16:16:07 +0000

I’m probably late in getting to this, but seeing the article in the WSJ about the RAD project made me stop to take a look. It appears to be a collection of different projects, all relating to infrastructure deployment/management and various research topics surrounding it. Looks cool so far.

Loving Dashboard Spy.

allspaw — Sun, 17 Feb 2008 21:20:35 +0000

I’m probably very late to this party, but I just discovered Dashboard Spy. Given the amount of “data porn” that folks in webops look at on a daily basis, this sort of stuff is pretty damn interesting.

I’m especially loving the current trend of developing ‘business’ dashboards, since it can fit in quite nicely with infrastructure statistics. Quite often when I need to make capacity justifications, I pull forecasts from both the higher-level metrics (i.e. photos uploaded) and the lower-level metrics (i.e. disk space consumed by photos) and have to marry those two bits together.

In fact, I love that stuff so much that I’m writing a book about it.