The key is not to come up with “the” number, but “a” number. You run the tests all the time and watch for changes in trends. If you are measuring load time constantly, you’ll be able to see pretty much immediately when things change. Even if tool A says, “2.5 seconds” and tool B says, “3.5 seconds”, if the time doubles when you deploy, I think it’s pretty clear something went wrong.

-/\/

Caught your presentation at Velocity, loved it. I had a question, one that I can’t seem to answer locally in Oz at the moment.

We’re using a number of tools to measure site load time – from Gomez to http://www.webpagetest.org/ . In the past we’ve used Keynote and Webmetrics and probably more that I’ve forgotten.

What I struggle with – and I’d love to know your approach is – every single tool gives a different load time. Even if we try to run the same test at the same time on the same day of the week I get a different number.

How do I come up with a definitive number? How do you do it?

It makes it very hard to judge whether a code change has improved site load time, or merely that we happened to run the test at a time when the test gave a quick response.

How do you guys do it?

Cheers
Ed