While searching around for something else, I came across this note I sent in late 2009 to the executive leadership of Yahoo’s Engineering organization. This was when I was leaving Flickr to work at Etsy. My intent on sending it was to be open to the rest of Yahoo about what how things worked at Flickr, and why. I did this in the hope that other Yahoo properties could learn from that team’s process and culture, which we worked really hard at building and keeping.

The idea that Development and Operations could:

• Share responsibility/accountability for availability and performance
• Have an equal seat at the table when it came to application and infrastructure design, architecture, and emergency response
• Build and maintain a deferential culture to each other when it came to domain expertise
• Cultivate equanimity when it came to emergency response and post-mortem meetings

Ben Rockwood said something last December about the re-emergence of the Systems Engineer and I agree with him, 100%.

NASA Systems Engineering Handbook, 2007

To add to that, I’d like to quote the excellent NASA Systems Engineering handbook’s introduction. The emphasis is mine:

Systems engineering is a methodical, disciplined approach for the design, realization, technical management, operations, and retirement of a system. A “system” is a construct or collection of different elements that together produce results not obtainable by the elements alone. The elements, or parts, can include people, hardware, software, facilities, policies, and documents; that is, all things required to produce system-level results. The results include system-level qualities, properties, characteristics, functions, behavior, and performance. The value added by the system as a whole, beyond that contributed independently by the parts, is primarily created by the relationship among the parts; that is, how they are interconnected. It is a way of looking at the “big picture” when making technical decisions. It is a way of achieving stakeholder functional, physical, and operational performance requirements in the intended use environment over the planned life of the systems. In other words, systems engineering is a logical way of thinking.

Systems engineering is the art and science of developing an operable system capable of meeting requirements within often opposed constraints. Systems engineering is a holistic, integrative discipline, wherein the contributions of structural engineers, electrical engineers, mechanism designers, power engineers, human factors engineers, and many more disciplines are evaluated and balanced, one against another, to produce a coherent whole that is not dominated by the perspective of a single discipline.

Systems engineering seeks a safe and balanced design in the face of opposing interests and multiple, sometimes conflicting constraints. The systems engineer must develop the skill and instinct for identifying and focusing efforts on assessments to optimize the overall design and not favor one system/subsystem at the expense of another. The art is in knowing when and where to probe. Personnel with these skills are usually tagged as “systems engineers.” They may have other titles—lead systems engineer, technical manager, chief engineer— but for this document, we will use the term systems engineer.

The exact role and responsibility of the systems engineer may change from project to project depending on the size and complexity of the project and from phase to phase of the life cycle. For large projects, there may be one or more systems engineers. For small projects, sometimes the project manager may perform these practices. But, whoever assumes those responsibilities, the systems engineering functions must be performed. The actual assignment of the roles and responsibilities of the named systems engineer may also therefore vary. The lead systems engineer ensures that the system technically fulfills the defined needs and requirements and that a proper systems engineering approach is being followed. The systems engineer oversees the project’s systems engineering activities as performed by the technical team and directs, communicates, monitors, and coordinates tasks. The systems engineer reviews and evaluates the technical aspects of the project to ensure that the systems/subsystems engineering processes are functioning properly and evolves the system from concept to product. The entire technical team is involved in the systems engineering process.

I would imagine that successful organization understands this concept of systems engineering, but I don’t think I’ve ever seen it put so well.

NASA’s engineers have both common and conflicting goals, just like we do in web operations. They weigh trade-offs in efficiency and thoroughness, and wade into the constraints of better, cheaper, faster, and hopefully: more resilient.

This re-emergence of the systems engineering (or “full-stack” engineering) notion is excellent and exciting to me, and I’m hoping that everyone in our field, when they hear “DevOps” (and/or how Theo says *Ops) what they mean is taking a systems engineering view.

One of my apprehensions in moving to New York from San Francisco was a common concern: why would I move from the ‘epicenter’ of the web to a place where it’s not? There’s been lots written about startup hub cities, and innovative web metro areas, but the fact of the matter is that New York hasn’t historically been a hotbed of web growth and innovation. Not compared to the Bay Area or Seattle, anyway.

I do, of course, think this is changing as of recently. The punch line is that I obviously did take the job, despite my misgivings about not being surrounded by people who are constantly thinking about my industry. One of the reasons I got over not being in the ‘epicenter’ is that Fred Wilson and Albert Wenger did an insanely good job at convincing me it was a good idea. Another reason is that I think Etsy is basically a Bay Area company that just happens to be in Brooklyn. I mean that as a compliment.

So while I always had some inkling of what ‘epicenter of the web’ means, I was never really sure how that could be measured. Indeed.com has indirectly measured it by the # of job listings.  O’Reilly did something similar for the # of startup jobs in 2006.

Number of jobs is interesting, but I thought it might be fun to measure it by locations of headquarters as seen through the lens of monthly unique users. So, I took the Quantcast “Top 100″ sites, found the latitude and longitude of the headquarters of each site via Crunchbase’s API, as well as other bits around the web, and Aaron helped out with the excellent Modest Maps to make this:

Quantcast Top 100 plotted on U.S. Map, radius = monthly uniques

Like I said, this doesn’t change my thoughts about the new job, or what I think ‘epicenter of the web’ means. But, still interesting, dontcha think?

UPDATE: Here’s a link to the raw data: http://spreadsheets.google.com/pub?key=tLwD1C5mghn9U3XJj_yqyjw&output=html

If there’s anything wrong, lemme know.

I guess I’m late on getting to this, but How Complex Systems Fail by Richard Cook is excellent.

Let me start with this: I don’t think I can overstate how right-on this paper is, with respect to the challenges, solutions, observations, and concerns involved with operating a medium to large web infrastructure. I found this via @benjaminblack, and I agree with him 100%: this should be considered required reading for anyone in our industry. I’m not sure if Cook ever thought that his paper would apply to web infrastructure, but I think it can and does. Please take 30 minutes right now and read it.

There are a number of salient points in the paper that I’d like to comment on. Again, this is through the lens of failures of complex systems as it pertains to web operations:

7) Post-accident attribution accident to a ‘root cause’ is fundamentally wrong.

I’m going to guess that this portion may be viewed as controversial in the prevailing webops wisdom, where post-mortems are for sure necessary, but whose content may or may not be effective in preventing similar types of failure. I do value the process of a post-mortem, because I think the human element of understanding complex failures is important and doing whatever you can to put in place safety is good, modulo what is said in section #16 of the paper. I believe that even a rudimentary process of “5 Whys” has value. But at the same time, I also think that there is something in the spirit of this paragraph, which is that there is a danger in standing behind a single underlying cause when there are systemic failures involved. Doing this can lead to the false belief that you’ve got this mode covered, you’ve found the silver bullet that made the whole mountain crumble, and jeez what a relief because that will never bite us again.

14) Change introduces new forms of failure.

I totally agree with this point. However, I often see this as a rallying point for operations teams to say “No!” to change, when instead they should be working alongside development (and product owners) with a goal of reducing the risk of failure associated with each change. I do not believe that ‘release early, release often’ in and of itself can reduce that risk. I believe that the real (and only) way to do this is both technical and cultural. But I’ve spoken about this before.

16) Safety is a characteristic of systems and not of their components

Emphasis on “Safety cannot be purchased or manufactured; it is not a feature that is separate from the other components of the system.” Real safety comes from smart people doing smart things to the entire shebang, not the individual guts.

and I think the point I love the most, with all of my heart:

18) Failure free operations require experience with failure.

Fear is a strong emotion. I believe it can be used as a strong motivator for ensuring safety in the face of constant change, instead of a reason to push back on the very idea of change. Embrace fear of outages and degradation. Use it to guide your architecture, your code, your infrastructure. So lean into it.

There are a lot of great points in the paper, and I could go on, but you get the idea.

Domas makes something more useful than I bet most would think: http://mituzas.lt/2009/06/26/uncache/

Via kottke: some good examples of doing rough math in your head, causing you to guess about assumptions all along the way.

IMHO, being able to do this is one of the things that makes a good web ops person. The examples might be “useless”, but the process is invaluable.

So now there’s chapters 1-4 on Safari RoughCuts. Which means if you don’t mind shelling out the dough, you can take a look at what I’ve been getting up early for every day for the past few months. The working title is “The Art of Capacity Planning” and it’s meant to be a no-nonsense description of the capacity planning process and considerations for web operations.

I still have two chapters to go before it’s all finished, but if you’re nice enough to take a look at what I’ve got thus far, I’d appreciate any feedback. I’m sure there could be typos and some graphs misaligned, but such is life with “drafts”.

Awhile back, I said I’d love to have a tool that would allow me to peek inside filesystem cache and tell me what files (or pages of files) are inside. Well Peter Zaitsev points to the fincore tool, which comes pretty damn close: you give it a file, and it will tell you which pages of a particular file are in core memory.

Rock. Thanks, David Plonka.

I’m probably late in getting to this, but seeing the article in the WSJ about the RAD project made me stop to take a look. It appears to be a collection of different projects, all relating to infrastructure deployment/management and various research topics surrounding it. Looks cool so far.

Ok, Geva Perry tagged me so here goes:

1. After graduating from Umass/Amherst with a Mechanical Engineering degree, I worked at the Volpe National Transportation Systems Center, doing side-impact crashworthiness research for the NHTSA. We ran transient dynamic finite-element analysis simulations using LS-DYNA and rigid-body simulations with MADYMO3D. This is where I learned UNIX, because we ran the simulations on big SGI Origin compute servers, T3 Cray machines, Intel Paragons, and even AIX RS/6000s.

2. I’ve been playing guitar since I was about 13 years old, and after playing live many bluegrass, metal, jamband covers, and pop songs, I still can’t seem to do anything with jazz improvisation except faking it.

3. In high school I was a major gearhead, and tricked out all of my 3 1980s VW GTIs for extra horsepower. My introduction to automotive repairs was replacing the front left transaxle on my 1977 VW Scirocco in my driveway. This enthusiasm for cars also led me to be one ‘bad driving’ point away from the Massachusetts Registry of Motor Vehicles seizing my license. (I drive slower now)

4. In 1999, I moved to San Francisco and after a short stint at Genentech, I went to work at Salon.com for Chad Dickerson, who continues to be my all-time favorite manager. I learned a lot from Chad, and am lucky enough to work with him at Yahoo!.

5. I was born in The Ozarks, then moved to north suburban Boston. Depending on how much I’ve had to drink, I either have a Boston accent, or a slight southern-ish accent.

And now, as Norby points out…I tag Sara Wood, Aaron Cope, Dathan, Kellan, and George.