It's been a neat day.
Some time ago, I wrote some fairly simple routines to front end a spreadsheet to a database to simplify some fairly complex calculations. The goal here was to develop a way to rapidly get at info that would otherwise take far too long to calculate manually, or would require a phenomenal handle of SQL. I'm not yet such a person, but that may yet happen.
I found myself re-writing one of the main functions merging disparate data, and dammit, if the rewritten version wasn't a heckuva lot more logical than the first one. Trouble was, tho, that both versions were really, really expensive for disk calls. Two am this morning I realized that I wouldn't get the numbers if I relied on my current machine only. By tonight, it will have crunched 18 k rows of the data set, and there are twice as many records in the set.
A couple of email last night secured some time on a couple of kick-ass boxes locally, and I spent most of the day going from place to place to set up the db and program. End result: I have my 36 k rows crunched, and can finish things up tonight. At this rate, I might even sleep :P I'm going to have to, I can't really work as it stands.
This solution ("More power! Ar! Ar! Ar!") worked fine this time, but this is clearly not workable for the future. The problem with the spreadsheet approach is that it will make repeated calls for a similar - but not identical - subset of data, in no particular order - caching may not be effective in this case. I have to think about this more carefully. After sleep. :)