It's been a neat day.
Some time ago, I wrote some fairly simple routines to front
end a spreadsheet to a database to simplify some fairly
complex calculations. The goal here was to develop a way to
rapidly get at info that would otherwise take far too long
to calculate manually, or would require a phenomenal handle
of SQL. I'm not yet such a person, but that may yet happen.
I found myself re-writing one of the main functions merging
disparate data, and dammit, if the rewritten version wasn't
a heckuva lot more logical than the first one. Trouble was,
tho, that both versions were really, really expensive for
disk calls. Two am this morning I realized that I wouldn't
get the numbers if I relied on my current machine only. By
tonight, it will have crunched 18 k rows of the data set,
and there are twice as many records in the set.
A couple of email last night secured some time on a couple
of kick-ass boxes locally, and I spent most of the day going
from place to place to set up the db and program. End
result: I have my 36 k rows crunched, and can finish things
up tonight. At this rate, I might even sleep :P I'm going to
have to, I can't really work as it stands.
This solution ("More power! Ar! Ar! Ar!") worked fine this
time, but this is clearly not workable for the future. The
problem with the spreadsheet approach is that it will make
repeated calls for a similar - but not identical - subset of
data, in no particular order - caching may not be effective
in this case. I have to think about this more carefully.