On thursday it was decided to move my friend Mico's Micoi photo community to its new server, after living on Rackspace for three years. Rackspace is a really neat hosting prodiver: their customer support really is good and they have always been a stable and assuring provider. However, Micoi is a non-profit hobby and the server was three years old, suffering from bitrot and quite lame hardware, so now it was time to move to a cheaper provider with a new machine and, most importantly, bigger disks (digital cameras tend to produce larger image files now than four years ago!).

First, some background history: Micoi was created in November, 1999, after Mico and I had a burger at Burger King, discussing his never-ending stream of ideas. He had mentioned that he wanted his own photo website where he could upload his digital camera photos and perhaps even let people comment and upload their own images. We made a few sketches on a greasy paper and during the following weekend I coded it all up using Pike and RXML (Roxen Markup Language) on the Roxen platform. I think Roxen was in the 1.2 versions back then. This is basically the same code that is still running today, with only minor modifications and a few features added. When Roxen 2.0 arrived, some parsing rules and syntax in RXML changed just about enough to break everything I had coded on Micoi. So we set out to see if we could rewrite the beast for Roxen 2.0. Eventually, it didn't work out (or I was lazy), so we resorted to using a fork of Roxen 1.3, called Caudium, which continued the pre-2.0 Roxen traditions. Roxen 2.0 had its merits, though, so we moved all dynamic image handling to Roxen, while Caudium got to serve the HTML.

This duo of web serving has been very faithful, and in practice no upgrades has been done over time, unless there has been a critical security advisory concerning them. And now it was time to revisit what had become of them.

I learned about Roxen during 1998 during my first employment, and quickly grew fond of its easy-to-use web interface and RXML which could be used for most simpler tasks, and the powerful Pike programming language which could (and can!) solve virtually any problem. During that following year, I spent much time developing Roxen-based web sites and solutions. Thus, choosing Roxen for Micoi was just natural. While I am now discouraging the use of some of the development techiques encouraged by Roxen (such as their error-prone but indeed very convenient <SQLOUTPUT> and <FORMOUTPUT> tags), I still think it is a really cool platform to build web sites on.

Roxen, in the ancient times called Spinner, was originally developed by a Linköping-based company called Informationsvävarna (Infovav) and later Idonex and even later Roxen Internet Software. I remember that when having problems, one could log onto the Roxen Chat and quite often, one of the core Roxen developers would be there and sometimes even take the time to answer questions. I have even gotten custom patches mailed to me solving my problems. Checking out their Roxen community website, I noticed that the chat is actually still there, and there is a little, but not much, activity there. The Roxen Web Server has advanced to version "4.0.172-release3". Roxen Internet Software does no longer lead the development of the Pike programming language (in which the entire Roxen web server is built), instead the Computer Science instition at Linköping University has taken over the development. The build and installation was very straight-forward, and the source package included Pike 7.4, which I used to avoid version conflicts. The only hassle was that this new Pike was a little stricter when it came to variable declaration and thus noticed a few mistakes I had made there I had duplicate variable declarations.

The Caudium installation had only one problem too; as Caudim doesn't ship with a bundled Pike distribution, I downloaded the latest Pike tarball and installed it, only to discover that Caudium requires an older Pike version. No biggie, though, I just installed 7.2.239 and configured --with-pike.

For both Caudium and Roxen, it was a very pleasant view to log into their web interfaces and set up the web servers, just like in the Good Old Days. With a few mouse clicks, my old Pike modules was loaded, file systems mounted, and everything configured. Thanks to the fact that both Roxen and Caudium doesn't treat "unopenable ports" (such as trying to open a port number below 1024) as a fatal error, I could do the entire configuration as non-root, including setting up port 80, and then just give the sysadmin instructions for how to start the web servers when I was done. I set up test-ports for testing purposes, and when I had tested most of it, I simply shut down the web servers and when the sysadmin woke up, he started the web servers and they bound port 80 and then changed UID to a nonprivileged user.

Excluding the large bulk of data transfer (containing the actual user images, about 60 GB data and was done the night earlier), the entire move to the new server took about four hours, including testing. And the pleasure with working with small web sites (Micoi has about 8000 users, and around 800 unique user logins each month) is that no one actually noticed the down time (which should only have been a few minutes anyway, before old cached DNS values expired). :-)

In contrast, I recently moved a busy customer web site to a new server. This was about 5 o'clock in the morning, when the traffic was expected to be low, but even then the logs showed over one thousand unique IP-adresses on the old web server still receiving the requests from clients with cached DNS entries.

The dynamic dedicated server hosting market

The markets of web hosting are very dynamic - or at least they should be. Me and a couple of friends - we are all independent computer consultants - run a small business hosting web sites for our customers. Many customers develop very specialized web sites thus giving the need for custom software and components installations, which a standard web host won't provide. Still, they don't want the hassle of employing a system administrator and running their own web servers (which must have good connectivity, et.c.). Thus grows the need for "customized hosting", which in turn opens up the market opportunity for people like us, that can provide a flexible server platform while still taking away the expensive system administration duties from the customer. We, in turn, sign up for a dedicated server from one of the many dedicated server hosting providers around the world. This business, while still small scale, has been running for over four years and over the time we have noticed a few interesting facts and tendencies in this business model.

Good servers are cheap - this has been especially true the last couple of years, when low-priced providers such as Ev1Servers has surfaced, selling insanely cheap dedicated servers with good network connectivity and bandwidth. (In Europe, providers in USA is especially promising thanks to the cheap dollar - since the year 2000, our spendings has decreased noticably only because of the falling USD - in contrast we have ditched British provides because they would be billing us in GBP, which is considerably more expensive.) The low price, of course, comes at a price: in general, these cheaper providers have a lower level of customer support than, for example Verio or Rackspace. But prices being so much lower, is it really worth it to pay so much more for a little extra customer service? It is very easy to say "yes", as being able to call a competent customer service always gives a very strong psychological boost. But my next observation may show that this extra cost is unnecessary.

Customer support isn't critical - with the low price of servers, you can mostly optimize away many of the emergency calls to customer support by having spare fail-over servers. Because, when is it you, instinctively, want to have a very responsive customer support? It is, of course, in the worst case scenario: when the server crashes completely beyond remote restore. But even with the most enthustiastic (or "fanatical" as Rackspace call it) customer support, it will still take them quite a while to restore a server with, for example, a garbled file system or broken hardware. In these cases, you would want a hot spare that instantaneously can take over the responsibilities of the crashed server. And if you have a setup that can do this (with everything pre-configured and all data rsync'd very often), suddenly you have eliminated the need for that emergency customer support call at 4 am. When your fail-over server has started operating, which should be only a matter of minutes if you have done everything right, it doesn't matter much if it takes 45 minutes or 8 hours to get the old server up and running. You can even have the fail-over server running at an entirely different hosting provider, in another state or country, to cover for natural disasters or other site-local failures. Replicating server content using 'rsync' is fairly easy and quick enough to be done several times a day, even over transatlantic Internet connections.

Bandwidth has become even cheaper - many providers charge you per gigabyte transfer, or on most cases, increments of 10 or 100 gigabyte of transferred data. This way they can give everyone a 100 mbps connection, and with the 100 GB or so included in the basic montlhy fee, most small customers are happy while the large customers pay for their usage allowing the provider to upgrade their backbone connections if need be. However, with the price of raw bandwidth falling (?), these new cheap-o providers provide over 1000 GB of monthly transfer included in a basic $100/mo package, giving even pretty busy sites a comfortable room to grow in. In contrast, Rackspace provided 100 GB per month and charges quite a bit for any excess bandwidth.

You are never forced to stick with a bad provider - although most providers have some kind of minimum lease time or a lengthy contract termination period of notice, when signing for a new provider, we always calculate the worst case as if we would discover that their service is inferor and want to terminate the contract immediately. Thus, if a provider wants us to sign a 12-month $150/month contract, we will assume the worst case being that we have spent $1800 with no gain. Generally, providers don't force you to sign up for such long periods, but anywhere up to 6 months is common. With a French provider, we recently had an experience like that. The first few weeks worked flawlessly, but when the server became heavily loaded, it always froze - almost once a day - forcing us to do a hard reboot. We couldn't figure out the cause, and to add to it all, the provider had a major air conditioning breakdown in the server room, leading the server to be down for somewhere about five ours. That provider didn't last long in our books, and luckily, we only signed up for one month at a time, basically giving us a very marginal cost of one months lease. (We filed a ticket asking them to investigate the spontaneus freeze problems, but as far as I know, they never came back with a proper response.) Since it's so easy to change providers - just rsync the data, configure your software, update your DNS - all providers must be very careful to keep a good track record. One big outage and you risk losing a lot of customers.

By renting the servers, you have an incentive to clean up your own backyard - and you are never tempted to use obsolete hardware since it is very easy to just rent a new server and ditch the old one. Keeping an old server is bad economics: generally, you pay as much after three years as when the server was brand new. And for a server load that grows with a relatively slow pace, you will find that when it is time to get some more disk or RAM, your provider most likely has a new server offering with the specs you need - at the same price (or cheaper) than your old server. This way, switching servers every year or so isn't expensive, and each time you switch, you automatically get a fresh install and an opportunity to upgrade everything that you haven't yet gotten around to upgrade (perhaps in fear of breaking the things running on the server). After a few of these moves, you (hopefully!) have documented the installation procedures and quirks enough to be able to move to a fresh install very quickly. Many of the headaches of system administration goes away with the habit of starting from scratch every now and then.