12 Mar 2001 SeaDragons   » (Observer)

apm (Andrew McKinlay) posted an article about Suneido recently, and, well, I can't reply to it. So I'm replying here with my comments.

I had recently decided to undertake an Obj-C project which would require a database backend. While I was thinking about file formats (before deciding not to reinvent the wheel, and while I thought I might implement the database part myself), I thought I'd dispense with file and record locking by having a single data manager which was messaged by (and messaged) an arbitrary number of client-interface-managers. Since only one object did read/write, file locking was unnecessary. I was excited to see Suneido use the same idea (no file/record locking, because a single database manager had exclusive read/write to files); not having studied databases, I thought it was interesting I'd come up with the same solution to simplify the ssytem. (ie, I was proud of myself for guessing a solution that turned out to be in existence)

BUT: what happens when the single database manager gets so many requests for data and instructions to write data that one manager running on one machine can't possibly withstand the load? If you scale this to enough clients, it will eventually happen regardless how small and quick the manager is. And if the manager is doing complicated search logic, these requests could potentially be so burdensome that even a relatively small number would kill it. Some possibilities that come to mind are:
(a) have several managers acting as peers to service requests filtered through a load-balancer; each manager has equal access to a shared drivespace ... but now they need to either
    (i) use file locking not to overwrite one another with simultaneous write requests,
    (ii) message one another to be sure nobody else is working on a simultaneous request that interferes, which could be a problem by causing delays while the managers message one another about individual files and waiting for the results instead of just getting the results, or
    (iii) the load balancer keeps all requests for any particular record going to the same database manager ... but if the clients don't know which records they're looking for until the search logic is processed, which is likely in lots of database uses, this can't be done without additional access by the load-balancer to the data, which makes the "database managers" irrelevant since the work is being done by the load-balancer .... or (hopefully not)
(b) it doesn't scale beyond the capabilities of the box containing the database manager, so either use a big, butch box or a non-intense request load.

Have I missed something? Please write. I'm trying to deal with the same problem myself, and am interested in either (a) using your solution as a backend for handling my problem, or (b) using your solution to enlighten my search for my solution.

I'm interested in creating a piece of open-source-ware which is capable of handling enterprise-level demands, so these concerns are fairly serious to me. Personally I will not soon need anything requiring load balancing of requests, but I think this is better built in than bolted on (like security). Also: Suneido's site offers binaries built for operating systems which offer apps the Win32 API ... does Suneido require this API? (I haven't looked at the code.) Assuming the guts of Suneido are mostly code for handling Suneido's OO language and its search info, there's little obvious reason the Win32 API need be invoked ... although the code for controlling MS "Windows" UI elements certainly would have to be.

Will Suneido be portable to non Win32 environments? Will it turn up on *nix boxes where I will want to use it? And mostly: How do you answer extreme request loads? (ie, assuming you put the database manager module on a dedicated box and this is not enough to satisfy the requests?) Is this a concern for Suneido?

Best wishes!

Edited Later: Andrew McKinlay posted this in Suneido's forums for discussion, and answered there that scalability was (at least temporarily) a sacrifice made for performance and for expediance (to get a product working, so people don't get tired cosing forever without the payoff of seeing the code doing work) http://www.suneido.com

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!