Some Things We Did Today

by allspaw on March 5, 2009

Moving one of our eight photoserving farms from hardware Layer7 URL hash balancing (expensive, has limits) to L4 DSR balancing with CARP (cheap and simple) and figuring out how to juggle 18,000 requests/second while we do it.
Built yet some more automated query analysis reporting (with some yummy MySQLProxy)
Added yet another aggregated graph of database queries, broken down by type and cluster
Bunch of cfg mgmt changes (polishing up IO scheduling and filesystem tunings in a 2nd datacenter, more caching of search results)
Review of the higher priority to-dos in the Ops open bug queue (only 155 open! )
Finding new capacity ceilings for the image processing, given some recent optimizations)

{ 4 comments… read them below or add one }

Justin Huff March 5, 2009 at 1:24 pm: What are you guys using for the L4 load balancing?
allspaw March 5, 2009 at 2:27 pm: Foundry ServerIrons, for now.
Matthew Rothenberg March 5, 2009 at 4:56 pm: This is a small part of why I want to kiss John Allspaw every morning when I walk into work.
Chad Dickerson March 7, 2009 at 4:54 am: The list seems a little sparse. What else did you do?