7 Nov 2006
(updated 8 Nov 2006 at 02:40 UTC) »
You were in Boston before, right? Welcome to the Bay Area...
Thank you! In response to your solving one of my long-standing
about the recentlog, I'll try to
post something interesting to it.
Load test grapher
Background: I've been slowly playing with a suite of software for
looking at performance. It's one of those problems that I think few people
look at, because even though there's real computer science involved, any
project relating to the word "test" is sneered off by most people as QA work.
(Another project I've been playing with is a monitoring system for sysadmins;
that's basically the same
way. Sysadmin tools suck.) It's possible there's a little more computer science
necessary because lately I've been wanting to do a little less software
engineering and a little more computer science...
No release, documentation, or even a name yet, but there's a little bit of
in my Subversion repository here.
Today's problem: I made a graph of transaction duration vs. time of a
quick Apache load test.
It's worthless. You have basically no idea how long the average
takes because the scale is blown by a few outliers. I tried a couple obvious
things to solve this:
- switching to log scale
- setting ymax to be some function of the mean and standard deviation
but wasn't happy with the result. What I really wanted was what network
people do for billing: look at the 95th percentile. The obvious way to do that
is to sort the data and pick element n/.95. But my program works with huge
data sets - this graph was generated from 1.3 million transactions.
Everything else uses (nearly) linear time and not much disk. I don't want to
Off to the ACM Digital Library:
- The P^2 algorithm
for dynamic calculation of quantiles and histograms without storing
online computation of quantile summaries
These are really cool algorithms to estimate quantiles like the median or
percentile. The first has no defined error bounds but uses O(1) space and
time. The second returns a value whose rank r' is within [r-εN, r
+εN] (with ε
chosen in advance). It runs in a single pass using O(1/ε log
Neither requires you to know N ahead of time.
I implemented the first one. A really simple test:
r = random.Random()
e = QuantileEstimator(0.5)
numbers = range(100)
for i in numbers:
self.assertTrue(40 < e.get() < 60)
Though there aren't well-defined error bounds, it seems to do quite well
on my data set. That range of 40 to 60 in the test case is crazy liberal; I just
wanted to catch my blatant implementation errors.
I fed that into my graph tool to bound the graph at five times the 95%.
Now I have a general-purpose statistics class I can use all over the
place. Maybe I'll plug it into Axamol SQL Library's SQL statement timing to get
the median and 95%.