Needless to say, I jumped at the chance to help out. Over the following months, Jesse Robbins and I wrangled a bunch of topics we thought were integral to the field and authors who could cover them. It’s in the final stages of being published as we speak, but I think it came out pretty damn good.

These folks cranked out great chapters while still doing their day jobs, and it shows. It’s a great collection of war stories, advice, and hard-earned lessons.

Here is a list of the chapters:

“The Web Ops Career Path” Theo Schlossnagle
“Cloud Computing At Picnik: Lessons Learned” Justin Huff
“Infrastructure and Application Metrics” Matt Massie and myself
“Continuous Deployment” Eric Ries
“Infrastructure as Code” Adam Jacob
“Monitoring” Patrick Debois
“How Complex Systems Fail” Dr. Richard Cook
“Community Management and Web Operations” Heather Champ
“Dealing With Unexpected Traffic Spikes” Brian Moon
“Dev and Ops Cooperation and Collaboration” Paul Hammond
“How Your Visitors Feel: User-Facing Metrics” Alistair Croll and Sean Power
“Relational Database Strategy and Tactics For The Web” Baron Schwartz
“The Art and Science of Postmortems” Jacob Loomis
“Managing Web Storage” Anoop Nagwani
“Nonrelational Datastores” Eric Florenzano
“Agile Infrastructure” Andrew Clay Shafer
“Things That Go Bump In The Night (And How To Sleep Through Them)” Michael Christian

Royalties from the sales will go to the national 826 Valencia organization, which is dedicated to supporting students ages 6 to 18 with their writing skills. They do this by offering free drop-in tutoring at eight different locations around the country, as well as special events, student publishing, and scholarships.

Let me start with this: I don’t think I can overstate how right-on this paper is, with respect to the challenges, solutions, observations, and concerns involved with operating a medium to large web infrastructure. I found this via @benjaminblack, and I agree with him 100%: this should be considered required reading for anyone in our industry. I’m not sure if Cook ever thought that his paper would apply to web infrastructure, but I think it can and does. Please take 30 minutes right now and read it.

There are a number of salient points in the paper that I’d like to comment on. Again, this is through the lens of failures of complex systems as it pertains to web operations:

7) Post-accident attribution accident to a ‘root cause’ is fundamentally wrong.

I’m going to guess that this portion may be viewed as controversial in the prevailing webops wisdom, where post-mortems are for sure necessary, but whose content may or may not be effective in preventing similar types of failure. I do value the process of a post-mortem, because I think the human element of understanding complex failures is important and doing whatever you can to put in place safety is good, modulo what is said in section #16 of the paper. I believe that even a rudimentary process of “5 Whys” has value. But at the same time, I also think that there is something in the spirit of this paragraph, which is that there is a danger in standing behind a single underlying cause when there are systemic failures involved. Doing this can lead to the false belief that you’ve got this mode covered, you’ve found the silver bullet that made the whole mountain crumble, and jeez what a relief because that will never bite us again.

14) Change introduces new forms of failure.

I totally agree with this point. However, I often see this as a rallying point for operations teams to say “No!” to change, when instead they should be working alongside development (and product owners) with a goal of reducing the risk of failure associated with each change. I do not believe that ‘release early, release often’ in and of itself can reduce that risk. I believe that the real (and only) way to do this is both technical and cultural. But I’ve spoken about this before.

16) Safety is a characteristic of systems and not of their components

Emphasis on “Safety cannot be purchased or manufactured; it is not a feature that is separate from the other components of the system.” Real safety comes from smart people doing smart things to the entire shebang, not the individual guts.

and I think the point I love the most, with all of my heart:

18) Failure free operations require experience with failure.

Fear is a strong emotion. I believe it can be used as a strong motivator for ensuring safety in the face of constant change, instead of a reason to push back on the very idea of change. Embrace fear of outages and degradation. Use it to guide your architecture, your code, your infrastructure. So lean into it.

There are a lot of great points in the paper, and I could go on, but you get the idea.

What:

…did we do before (historical trending, etc)
…is going on right now? (troubleshooting, health, etc.)
…is coming down the road (capacity planning, new feature adoption, etc.)
…can we do to make things better (business intelligence, user-behavior, etc.)

All of which, of course, should be considered mandatory in order to help your business increase its awesome. Yay metrics!

Some time ago, Matthias wrote great a blog post about some of the metrics that can reasonably profile the effectiveness of web operations, taken from the ITIL primer, VisibleOps.

In my opinion, there’s nothing on that list of things that isn’t valuable, as long as the cost of gathering those metrics isn’t too behaviorally, technically, or organizationally expensive. The topics included in that list of metrics and the context they live in is fodder for many, many blog posts.

But in the category of historical trending, I’m more and more fascinated by gathering what I’ll call “meta-metrics”, which is data about how you respond to the changes your system is experiencing.

One of the best examples of this is gathering information about operational disruptions. Collecting information about how many times your on-call rotation was alerted/paged/woken-up, during what times, and for what service(s) can be enlightening to say the least.  We’ve been tracking the volume of alerts a lot closer recently, and even with the level of automation we’ve got at Flickr, it’s still something you have to keep on top of, especially if you’re always finding new things to measure and alert on.

Now ideally, you have an alerting system that only communicates conditions that need resolvable action by a human. Which means every alert is critically important, and you’re not ignoring or dismissing any pages for any reasons that sound like “oh, that’s ok, that cluster always does that…it’ll clear up, I’ll just acknowledge the page so I can shut up nagios.” In other words, our goal is to have a zero-noise alerting system. Which means that all alerts are actionable, not ignorable, and require a human to troubleshoot or fix. Over time, you push as much of this work as you can to the robots. In the meantime, save humans for the yet-to-be-automated work, or the stuff that isn’t easily captured by robots.

Why is this important to us? I may be stating the obvious, but it’s because interrupting humans with alerts that don’t require action has a mental and physical context switching cost (especially if the guy on-call was sleeping), and it increases the likelihood of missing a truly critical page in a slew of non-critical ones.

Of course in the reality of evolving and growing web applications, even if we could reach a 100% noise-free alerting system, it’s impossible to sustain for any extended period of time, because your application, usage, and failure modes are constantly changing. So in the meantime, knowing how your alerts affect the team is a worthwhile thing to do for us. In fact, I think it’s so important that it’s worth collecting and displaying next to the rest of your metrics, and exposing these metrics to the entire dev and ops groups.

Something like this: (made-up numbers)

Tracking Critical Alerts

Gathering up info about these alerts should give us a better perspective on where we can improve. So, things like:

• How many critical alerts are sent on a daily/hourly/weekly basis?
• What does a time histogram of the alerts look like? Do you get more or less alerts during nighttime or non-peak hours?
• How much (if any) correlation is there between critical alerts and:

- code deploys?
- software upgrades?
- feature launches?
- open API abuse?

• What does a breakdown of the alerts look like, in terms of: host type, service type, and frequency of each in a given time period?

and maybe the most important ones:

• How many of those alerts aren’t actually critical or demand human attention?
• How many of them always self-recover?
• How many (and which) don’t matter in their role context (like, a single node in a load-balanced cluster) and could be turned into an aggregate check?

We’ve built our own stuff to track and analyze these things. My question to the community is: I’m not aware of any open-source tool that is dedicated to analyzing these metrics. Do they exist? Nagios obviously has host/hostgroup/cluster warning and critical histories, and those can be crunched to find critical alert statistics, but I’m not aware of any comprehensive crunching. Of course, until I find one, we’re just building our own.

Thoughts, lazyweb?

That was a blast! I had never done a ‘duet’ talk before. Here are the slides:

UPDATE: Gil Raphaelli has posted his python bindings he wrote for our libyahoo2 use in our Ops IM Bot.

There was something that I left out of my slides, mostly because I didn’t want to distract from the main topic, which was optimization and efficiencies.

While I used our image processing capacity at Flickr as an example of how compilers and hardware can have some significant influence on how fast or efficient you can run, I had wondered what the Magical Cloud™ would do with these differences.

So I took the tests I ran on our own machines and ran them on Small, Medium, Large, Extra Large, and Extra Large(High) instances of EC2, to see. The results were a bit surprising to me, but I’m sure not surprising to anyone who uses EC2 with any significant amount of CPU demand.

For the testing, I have a script that does some super simple image resizing with GraphicsMagick. It splits a DSLR photo into 6 different sizes, much in the same way that we do at Flickr for the real world. It does that resizing on about 7 different files, and I timed them all. This is with the most recent version of GraphicsMagick, 1.3.5, with the awesome OpenMP bits in it.

Here is the slide of the tests run on different (increasingly faster) dedicated machines:

and here is the slide that I didn’t include, of the EC2 timings of the same test:

Now I’m not suggesting that the two graphs should look similar, or that EC2 should be faster. I’m well aware of the shift in perspective when deploying capacity within the cloud versus within your own data center. So I’m not surprised that the fastest test results are on the order of 2x slower on EC2. Application logic, feature designs (synchronous versus asynchronous image processing, for example) can take care of these differences and could be a welcome trade-off in having to run your own machines.

What I am surprised about is the variation (or lack thereof) of all but the small instances. After I took a closer look at vmstat and top, I realized that the small instances consistently saw about 50-60% CPU stolen from it, the mediums almost always saw zero stolen, and the Large and ExtraLarges saw up to 35% CPU stolen from it during the jobs.

So, interesting.

I’m lucky enough to be on the program committee this year, and I think the conference is a huge opportunity to spread the ops love on all kinds of topics. There’s a list on the O’Reilly page to get you thinking about what might make for a good submission:

- How to tie web performance and operations to the bottom line
- Real-world incident management – getting “tight like a pit crew”
- Making websites as fast and reliable as desktop apps
- Networking, DNS, and load balancing
- Profiling’s not just on the backend: JavaScript, CSS, and the network
- Managing web services – flaming disasters you survived and lessons learned
- The intersection between performance and design
- Wicked cool (and actionable) metrics
- Ads, ads, ads – the performance killer?
- Troubleshooting in production
- How to scale and be fast on the social web
- Capacity planning and load testing
- Establishing performance and operations best practices within your organization
- Configuration management best (and worst) tools and practices
- Monitoring and instrumentation: Open Source, as a service, commercially supported solutions
- Using multiple CDNs to improve customer experience and reduce cost

Think for a minute: Do you have a bunch of sweet ops hacks that you’re really proud of? Do you and your dev teams collaborate on making things easy to manage? Do you face unique challenges that others don’t which ops folks can learn from?

If so, don’t be lame: submit a proposal!

Our automated config management system is called Gemstone, but conceptually you can think of it as a pretty extensible SystemImager/Puppet/cfengine-style system. In the animation, the dots are changes made by the ops person shown.  The legend is:

transforms: this is what cluster should have what packages, files, actionable scripts, etc.
raw: these are actual files, like apache/memcached/squid configs, which get munged depending on what cluster they might be in
conf: this is what boxes/clusters are subsets or supersets of which clusters
code: ops-written tools/utilities
Misc: stuff that doesn’t fit into the above.

IMHO, being able to do this is one of the things that makes a good web ops person. The examples might be “useless”, but the process is invaluable.