Some Things We Did Today

  • Moving one of our eight photoserving farms from hardware Layer7 URL hash balancing (expensive, has limits) to L4 DSR balancing with CARP (cheap and simple) and figuring out how to juggle 18,000 requests/second while we do it.
  • Built yet some more automated query analysis reporting (with some yummy MySQLProxy)
  • Added yet another aggregated graph of database queries, broken down by type and cluster
  • Bunch of cfg mgmt changes (polishing up IO scheduling and filesystem tunings in a 2nd datacenter, more caching of search results)
  • Review of the higher priority to-dos in the Ops open bug queue (only 155 open! :) )
  • Finding new capacity ceilings for the image processing, given some recent optimizations)

I like making things go! At the moment, I'm SVP of Infrastructure and Operations at Etsy, and I'm currently pursuing a Master's degree in Human Factors and Systems Safety at Lund University.

4 comments

  1. Justin Huff   •  

    What are you guys using for the L4 load balancing?

  2. allspaw   •     Author

    Foundry ServerIrons, for now. :)

  3. Matthew Rothenberg   •  

    This is a small part of why I want to kiss John Allspaw every morning when I walk into work.

  4. Chad Dickerson   •  

    The list seems a little sparse. What else did you do? ;-)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>