Older blog entries for pipeman (starting at number 5)

11 Nov 2004 (updated 12 Nov 2004 at 05:35 UTC) »
I am Amiga OS

According to this quiz, I am Amiga OS, and I am flattered.

My head hurts

Spent the night out yesterday. A friend of a friend played at a pub where you can get a 33 cl bottle of Strongbow cider for 20 SEK (which is under 3 USD, which is really cheap by Swedish standards). Now that is not healthy, and after that we went to the local night club where you get two beers for the price of one. I ended up in a friend's apartment drinking rosé and Beck's. After I left, he apparently jumped from the balkony and wandered away. His girlfriend called me and we went looking for him, without any luck. Even later, she called and told me she'd found him sitting somewhere downtown, totally lost and confused. She dropped by earlier tonight and told me he had no recollection of what had happened after we left the night club. As for me, I spent the entire day sleeping and ignoring the constantly ringing cell phone that was disturbingly out of reach. For some reason, I had placed the laptop next to my bed, though. I guess Internet addictiveness doesn't wear off even when I'm hopelessly hammered.

SWT

I got an e-mail from someone using my program Nular Shell (Swedish only) on Linux. He gave me a very good bug report with screenshots showing how the program (not) worked in Linux/GTK. This gave me the incentive I needed to recode parts of the SWT GUI, which suffered badly from me not being very good at SWT programming at the time I wrote it. I couldn't figure out some layout tweaks, so I resorted to fixed positioning using setBounds(), which made the app really ugly on Linux since all the widgets had different sizes in SWT. Also, some widgets differ in functionality, too - for example, when using the ToolBar, you can create a SEPARATOR item and assign a widget to it (using setControl()) in order to embed it into the toolbar. This, however, doesn't seem to work in the GTK2 port of SWT, so I had to make a Linux-specifik hack for a small text box I wanted in the toolbar. Now, however, I rearranged the main window a bit, so I don't have to embed the text box in either Windows nor Linux (before and after), and with no stupid fixed-positioning code, it should look decent in Linux too.
Update: it seems that the SWT GTK2 ToolBar widget doesn't support the SWT.VERTICAL style either, and the Button widget in GTK2 lacks SWT.DOWN. Annoying.

Java class loaders

When publishing my development version of Nular to the web, I learned that you must explicitly assign a parent class loader when creating new ones, in order for class searches to be delegated upwards. I discovered this because the new version of Nular actually has a plugin API (which no one will ever use) which uses some class loader hacks to dynamically load class files dropped in a "modules" directory, either as JAR archives (with a property file in the JAR specifying which class to be loaded), or as plain .class files. In the latter case, the class must be in the anonymous package, or the class loader will fail. I never figured out how to determine a class' package from it's .class file.

Fiddler and MSN search

I did some random ego-searching on Microsoft's new search engine and discovered that the author of Fiddler had tried to communicate with me regarding my Fiddler plugin. I kind of expected that all those web boards sends an e-mail when someone replies to your post, so I never bothered to go back and check the forum.

Speking of MSN search, apparently they are getting back at Google. Makes you wonder if it really is a bizarre result of search engine ranking algorithms or playful engineers in action.

Other

I'm thinking about registering a Freshmeat entry for web hammer, but I'm kind of hesitant since I tend to never update those things anyway. We'll see about that. The Freshmeat entry for Duper has 10 subscribers, which is kind of weird, but it sort of proves that some people actually wants me to tell them about new versions via Freshmeat. Compare to the Roxen Web Server which has 20 subscribers and conclude that my software is at least half as popular as an obscure web server. :-) Staffan is doing an impressive work with lagen.nu, a collection of all Swedish laws and more. MrFriday will (hopefully) proudly sponsor lagen.nu with web hosting when it gets out of beta.

7 Nov 2004 (updated 7 Nov 2004 at 03:14 UTC) »
Simple webserver stress tool

A few days ago, I wrote a small utility program to help out stress test a client's web site. I wanted something small and simple, that I could send to (non-tech) clients and colleagues to allow them to run the tests from their own computers. Tonight, I tidied up the user interface a bit and packaget it for Java Web Start, so that the user only has to click on a WWW link to start the program. Once started, he or she only fills out an URL pointing to a file describing which URLs to stress test.

I haven't released the source yet (it's time to get some sleep), but I plan on doing that soon.

Oh, and it's available here.

On another note: anyone know who MikeLowe is (removed link, thanks ncm for pointing out my stupidity)? Seems to me he's just Google-whoring. Haven't seen that on Advogato before, is it common?

2 Nov 2004 (updated 2 Nov 2004 at 23:33 UTC) »
Testing Advogato's XMLRPC interface

I am currently implementing a tiny SOAP middleware for a client, and while reading up on SOAP I realized that Advogato has some simple XML-RPC services. So using Apache XML-RPC, and a smallish Java hack, my homepage now includes the latest diary entry from Advogato. Sweet! This is how it works: the code checks if /tmp/diary.cache exists. If it doesn't, it sets off to asynchronously fetch the latest diary from Advogato, while displaying a short note informing the user that the data is unavailable and an auto-refresh Javascript snippet reloading the page after a few seconds. When the XML-RPC requests are completed, the diary entry is stored in that file, and upon subsequent requests to the home page, the code will simply include the HTML found in the cache file. Every once in a while, a cron job deletes the cache. This way, I don't have to poll Advogato needlesssly to see if there is new data; I will only make the request if someone actually access the page. Yes, there are probably some race conditions and such, but this was what I could wrap up while waiting for the coffee to finish. :-)
update: I noticed that the time from Advogato that my Java code printed out was in some weird time zone, and somewhat surprised assumed that my JSP container had a broken locale/time zone setup. So I went about to manually point it to the time zone I wanted to use, but that didn't help: the time stamps were still many hours apart from my own local clock (which is UTC+1). After checking into what was actually sent over the wire behind the scenes of my XML-RPC calls, I noticed that the ISO 8601 time stamps didn't contain any time zone data. The XML-RPC client thus assumed that the received dateTime.iso8601 strings (eg. 20041102T11:32:32) were in the local time zone of the client computer. Appearantly, they are not, but rather in what I assume is some US time such as EST. I guess I should point this out to raph or whoever it was that developed the XML-RPC backend for Advogato, as it would be very helpful for people in different time zones to know what offset the returned time has. My guess is that all that has to be done is adding something like "-0500" (or whatever timezone Advogato uses) to the iso8601 string in order to make it possible for the XML-RPC client to correctly translate into something like a timezone-neutral DateTime object.

Securing my WLAN, sort of

Earlier this year, my WLAN PC Card for my laptop broke, since hardware hates me. I had been using a D-Link WLAN card together with a combined D-Link firewall/NAT gateway/WLAN access point until then, and since I don't exactly live in a high tech area where war driving is common, I had settled on using the 256-bit WEP encryption that the D-Link hardware provided (however aware that WEP in general isn't secure all, it probably keeps my average neighbor from peeking into my network).

When faced with being WLAN-less, I got to borrow an old spare card from a friend. Tre problem was, that the card in question was a stoneage-ish Lucent WaveLAN Bronze - i.e., quite slow and no encryption. So I was forced to accept that my wireless network would be wide open to anyone (or buy a new network adapter, which would cost me at least ten beers and is thus out of the question). Also, when trying out my new-old WLAN card, I noticed that one of my neighbors, too, had set up a wireless access point. No encryption, but a very friendly DHCP server and all - I could use his/her AP for my Internet surfing with a few mouse clicks.

Anyway, so I set out to secure my network with the assumption that it was untrusted. I have briefly mentioned this before: the solution was found in OpenVPN. Set up was a breeze - I configured my (wired) desktop computer as a server and a new private RFC1918 subnet, and let it hand out IP dynamic adresses in that range to the clients. So with OpenVPN, I had a secure authenticated channel between the laptop and the desktop, regardless of whether I was using my unencrypted home WLAN or an office broadband connection (well, assuming a firewall in the way doesn't block UDP port 5000). The next step was to assure that no war driver could use my WLAN for sending spam. That was quite easy: on my NAT box, I can configure the built-in firewall rules quite flexibly (it's not iptables or pf, but hey, it works): I can configure the rules per interface (WAN or LAN), source and destination adress and ports, and finally IP protocol. The end result looked something like this:

  1. ALLOW from SERVER to ANY/ANY
  2. ALLOW from LAN to GATEWAY port 53/UDP
  3. ALLOW from ANY 0.0.0.0/0 TO SERVER port 5000/UDP
  4. DENY FROM ANY TO ANY

Here, SERVER represent IP adress of the desktop computer and LAN includes both WLAN and Ethernet interfaces. The packets that travel from the WLAN interface to the Ethernet network isn't filtered at all. Basically, I only allow DNS to the gateway (which includes a DNS relay) and UDP port 5000 to the OpenVPN server. This way, someone connecting to my WLAN may use DNS (and can potentially use that through IP over DNS, but if they go through that hassle, they're worth the few kbit/sec they can get from it), and can connect to my desktop computer (which hopefully is quite secured even though it's a Windows 2000 box), but they can't leech warez or send spam using my broadband connection. Meanwhile, I connect to the SOCKS proxy on the VPN interface on the server machine (the SOCKS software denies connections on other interfaces), and through that proxy connect to anywhere on the Internet, with OpenVPN protecting my privacy for the parts that are wireless.

As an extra bonus, I can let friends that I trust take part in my VPN if we want a secure way of sharing files via otherwise insecure means (such as SMB) - all they have to do is set up a minimal OpenVPN configuration and get me to sign their public key. Also, an added level of obscureness, my laptop now connects to my Waste network through the VPN via the Waste node running on my desktop computer. As a friend put it: you should PGP-encrypt some files written in pig latin before sending them over the Waste network running on top of the VPN to be really sure Echelon gets something to chew on. :-)

Unreal Tournament - 1999 Game Of The Year

I just installed Unreal Tournament - 1999 Game Of The Year edition. It's been couple of years since I played it. At my old employer, Halogen (PBUH), we hade a period of some serious UT festing during the years 1999-2001. It was real fun.

I had completely forgotten about the tremendous joy of blowing people up with a huge rocket launcher.

24 Oct 2004 (updated 24 Oct 2004 at 05:00 UTC) »
Revisiting old friends

On thursday it was decided to move my friend Mico's Micoi photo community to its new server, after living on Rackspace for three years. Rackspace is a really neat hosting prodiver: their customer support really is good and they have always been a stable and assuring provider. However, Micoi is a non-profit hobby and the server was three years old, suffering from bitrot and quite lame hardware, so now it was time to move to a cheaper provider with a new machine and, most importantly, bigger disks (digital cameras tend to produce larger image files now than four years ago!).

First, some background history: Micoi was created in November, 1999, after Mico and I had a burger at Burger King, discussing his never-ending stream of ideas. He had mentioned that he wanted his own photo website where he could upload his digital camera photos and perhaps even let people comment and upload their own images. We made a few sketches on a greasy paper and during the following weekend I coded it all up using Pike and RXML (Roxen Markup Language) on the Roxen platform. I think Roxen was in the 1.2 versions back then. This is basically the same code that is still running today, with only minor modifications and a few features added. When Roxen 2.0 arrived, some parsing rules and syntax in RXML changed just about enough to break everything I had coded on Micoi. So we set out to see if we could rewrite the beast for Roxen 2.0. Eventually, it didn't work out (or I was lazy), so we resorted to using a fork of Roxen 1.3, called Caudium, which continued the pre-2.0 Roxen traditions. Roxen 2.0 had its merits, though, so we moved all dynamic image handling to Roxen, while Caudium got to serve the HTML.

This duo of web serving has been very faithful, and in practice no upgrades has been done over time, unless there has been a critical security advisory concerning them. And now it was time to revisit what had become of them.

I learned about Roxen during 1998 during my first employment, and quickly grew fond of its easy-to-use web interface and RXML which could be used for most simpler tasks, and the powerful Pike programming language which could (and can!) solve virtually any problem. During that following year, I spent much time developing Roxen-based web sites and solutions. Thus, choosing Roxen for Micoi was just natural. While I am now discouraging the use of some of the development techiques encouraged by Roxen (such as their error-prone but indeed very convenient <SQLOUTPUT> and <FORMOUTPUT> tags), I still think it is a really cool platform to build web sites on.

Roxen, in the ancient times called Spinner, was originally developed by a Linköping-based company called Informationsvävarna (Infovav) and later Idonex and even later Roxen Internet Software. I remember that when having problems, one could log onto the Roxen Chat and quite often, one of the core Roxen developers would be there and sometimes even take the time to answer questions. I have even gotten custom patches mailed to me solving my problems. Checking out their Roxen community website, I noticed that the chat is actually still there, and there is a little, but not much, activity there. The Roxen Web Server has advanced to version "4.0.172-release3". Roxen Internet Software does no longer lead the development of the Pike programming language (in which the entire Roxen web server is built), instead the Computer Science instition at Linköping University has taken over the development. The build and installation was very straight-forward, and the source package included Pike 7.4, which I used to avoid version conflicts. The only hassle was that this new Pike was a little stricter when it came to variable declaration and thus noticed a few mistakes I had made there I had duplicate variable declarations.

The Caudium installation had only one problem too; as Caudim doesn't ship with a bundled Pike distribution, I downloaded the latest Pike tarball and installed it, only to discover that Caudium requires an older Pike version. No biggie, though, I just installed 7.2.239 and configured --with-pike.

For both Caudium and Roxen, it was a very pleasant view to log into their web interfaces and set up the web servers, just like in the Good Old Days. With a few mouse clicks, my old Pike modules was loaded, file systems mounted, and everything configured. Thanks to the fact that both Roxen and Caudium doesn't treat "unopenable ports" (such as trying to open a port number below 1024) as a fatal error, I could do the entire configuration as non-root, including setting up port 80, and then just give the sysadmin instructions for how to start the web servers when I was done. I set up test-ports for testing purposes, and when I had tested most of it, I simply shut down the web servers and when the sysadmin woke up, he started the web servers and they bound port 80 and then changed UID to a nonprivileged user.

Excluding the large bulk of data transfer (containing the actual user images, about 60 GB data and was done the night earlier), the entire move to the new server took about four hours, including testing. And the pleasure with working with small web sites (Micoi has about 8000 users, and around 800 unique user logins each month) is that no one actually noticed the down time (which should only have been a few minutes anyway, before old cached DNS values expired). :-)

In contrast, I recently moved a busy customer web site to a new server. This was about 5 o'clock in the morning, when the traffic was expected to be low, but even then the logs showed over one thousand unique IP-adresses on the old web server still receiving the requests from clients with cached DNS entries.

The dynamic dedicated server hosting market

The markets of web hosting are very dynamic - or at least they should be. Me and a couple of friends - we are all independent computer consultants - run a small business hosting web sites for our customers. Many customers develop very specialized web sites thus giving the need for custom software and components installations, which a standard web host won't provide. Still, they don't want the hassle of employing a system administrator and running their own web servers (which must have good connectivity, et.c.). Thus grows the need for "customized hosting", which in turn opens up the market opportunity for people like us, that can provide a flexible server platform while still taking away the expensive system administration duties from the customer. We, in turn, sign up for a dedicated server from one of the many dedicated server hosting providers around the world. This business, while still small scale, has been running for over four years and over the time we have noticed a few interesting facts and tendencies in this business model.

Good servers are cheap - this has been especially true the last couple of years, when low-priced providers such as Ev1Servers has surfaced, selling insanely cheap dedicated servers with good network connectivity and bandwidth. (In Europe, providers in USA is especially promising thanks to the cheap dollar - since the year 2000, our spendings has decreased noticably only because of the falling USD - in contrast we have ditched British provides because they would be billing us in GBP, which is considerably more expensive.) The low price, of course, comes at a price: in general, these cheaper providers have a lower level of customer support than, for example Verio or Rackspace. But prices being so much lower, is it really worth it to pay so much more for a little extra customer service? It is very easy to say "yes", as being able to call a competent customer service always gives a very strong psychological boost. But my next observation may show that this extra cost is unnecessary.

Customer support isn't critical - with the low price of servers, you can mostly optimize away many of the emergency calls to customer support by having spare fail-over servers. Because, when is it you, instinctively, want to have a very responsive customer support? It is, of course, in the worst case scenario: when the server crashes completely beyond remote restore. But even with the most enthustiastic (or "fanatical" as Rackspace call it) customer support, it will still take them quite a while to restore a server with, for example, a garbled file system or broken hardware. In these cases, you would want a hot spare that instantaneously can take over the responsibilities of the crashed server. And if you have a setup that can do this (with everything pre-configured and all data rsync'd very often), suddenly you have eliminated the need for that emergency customer support call at 4 am. When your fail-over server has started operating, which should be only a matter of minutes if you have done everything right, it doesn't matter much if it takes 45 minutes or 8 hours to get the old server up and running. You can even have the fail-over server running at an entirely different hosting provider, in another state or country, to cover for natural disasters or other site-local failures. Replicating server content using 'rsync' is fairly easy and quick enough to be done several times a day, even over transatlantic Internet connections.

Bandwidth has become even cheaper - many providers charge you per gigabyte transfer, or on most cases, increments of 10 or 100 gigabyte of transferred data. This way they can give everyone a 100 mbps connection, and with the 100 GB or so included in the basic montlhy fee, most small customers are happy while the large customers pay for their usage allowing the provider to upgrade their backbone connections if need be. However, with the price of raw bandwidth falling (?), these new cheap-o providers provide over 1000 GB of monthly transfer included in a basic $100/mo package, giving even pretty busy sites a comfortable room to grow in. In contrast, Rackspace provided 100 GB per month and charges quite a bit for any excess bandwidth.

You are never forced to stick with a bad provider - although most providers have some kind of minimum lease time or a lengthy contract termination period of notice, when signing for a new provider, we always calculate the worst case as if we would discover that their service is inferor and want to terminate the contract immediately. Thus, if a provider wants us to sign a 12-month $150/month contract, we will assume the worst case being that we have spent $1800 with no gain. Generally, providers don't force you to sign up for such long periods, but anywhere up to 6 months is common. With a French provider, we recently had an experience like that. The first few weeks worked flawlessly, but when the server became heavily loaded, it always froze - almost once a day - forcing us to do a hard reboot. We couldn't figure out the cause, and to add to it all, the provider had a major air conditioning breakdown in the server room, leading the server to be down for somewhere about five ours. That provider didn't last long in our books, and luckily, we only signed up for one month at a time, basically giving us a very marginal cost of one months lease. (We filed a ticket asking them to investigate the spontaneus freeze problems, but as far as I know, they never came back with a proper response.) Since it's so easy to change providers - just rsync the data, configure your software, update your DNS - all providers must be very careful to keep a good track record. One big outage and you risk losing a lot of customers.

By renting the servers, you have an incentive to clean up your own backyard - and you are never tempted to use obsolete hardware since it is very easy to just rent a new server and ditch the old one. Keeping an old server is bad economics: generally, you pay as much after three years as when the server was brand new. And for a server load that grows with a relatively slow pace, you will find that when it is time to get some more disk or RAM, your provider most likely has a new server offering with the specs you need - at the same price (or cheaper) than your old server. This way, switching servers every year or so isn't expensive, and each time you switch, you automatically get a fresh install and an opportunity to upgrade everything that you haven't yet gotten around to upgrade (perhaps in fear of breaking the things running on the server). After a few of these moves, you (hopefully!) have documented the installation procedures and quirks enough to be able to move to a fresh install very quickly. Many of the headaches of system administration goes away with the habit of starting from scratch every now and then.

20 Oct 2004 (updated 20 Oct 2004 at 03:24 UTC) »

I thought I'd try to use this diary thing since I never seem to find a blog tool that I like.

Today's missing Firefox feature: I'd like to be able to disable proxy usage on a per protocol basis, much as I can do with specific subnets and hosts. This is because the main reason I'm using a proxy is that I'm using an unencrypted WLAN for my laptop, and I don't want people sneaking in on my dirty web surfing. So I have set up a SOCKS proxy on a landline-connected computer, with which my laptop communicates using OpenVPN 2.0 beta11. There is, however, a considerable network latency to the SOCKS server from my laptop, so when possible without compromising too much of my privacy, I'd like to bypass the proxy. And one such example is SSL encrypted web pages. Consequently, I'd like a checkbox or something next to each protocol field in the Firefox proxy setup saying "Don't proxy this protocol".

Speaking of OpenVPN, I'm quite impressed by the simplicity of setting it up. Of course, it took me a couple of hours before I finally realized (as always) that reading the manual would be a Good Idea. But even before that, I managed to get it up and running with self-signed certificates and all quite easily. It was just when I wanted to add a third computer to my VPN I actually had to learn something. Now, the configuration file for the clients are just about 8 rows, and it just works. Wonderful. Now lets hope it's actually secure, too.

Todays almost a free lunch: When at the Central Station in Stockholm yesterday, I had a spare half hour before the train departure, so I switched on the laptop only to notice that there were no less than three visible WLANs there. Not very surprising in itself, I guess: they were all commercial "hot spot" WLAN operators providing paid-for Internet access . As such, they of course wouldn't let me use their net without fishing up a credit card or otherwise buy access time. No free surf for me, thus. But I poked around on the networks and noticed that of the three different nets, two of them allowed both unrestricted DNS and ICMP pings (but not routing anything else). This should allow for some free wireless surfing using IP over DNS, and it seems as if there has been a release as late as june 2004, so the project doesn't seem entirely dead. I'd love to see a Windows implementation (at least on the client side), too, as no Linux distribution has yet to survive attempting to acccess my crappy PC-Card slot (they just freeze), and consequently I can't use WLAN in Linux. Perhaps you could merge it into the transport layer of OpenVPN, making it an alternative to the traditional TCP and UDP transports it offers today? I'll offer a beer to anyone who implements it. :-)

Today's missing Azureus feature: I'd like to be able to request that a specific piece # is downloaded as soon as possible - or, for that matter, to have a better method than the current "priority" selection to be able to force Azureus to download a specific file contained within a torrent more or less immediately. As it is now, is seems that if I tag a file with "high" priority, it often gets stuck at 99,9% for some reason, presumably missing only a piece or to. With the first suggestion, I could then figure out which pieces are missing and manually request that Azureus try to download them. However, at the moment there seem to be no way to find out the exact piece numbers.

Today's I-like-to-browse-my-logfiles: I checked out the latest stats for my web site, noticing that there has been 89 downloads of my silly little program Duper during the last seven days, although the web page has only gotten 82 hits. It is interesting, though, that there has been just under two hundred downloads since its release three weeks ago, although a "file duplicate finder" must be one of the most over-implemented utility software category on the web today, and I doubt that my attempt at it is among the better ones. Also, Duper requires Java, which I though was something that scared people off. Well, maybe not anymore. Even more surprising (to me, at least) is that there has been over one hundred downloads (138 to be exact) of the source tarballs, showing that at least half of the users were actually interested in the source code. Which, in the case of the latest version, won't even build (without the modified fast-md5 package, see the next paragraph).

Today's confession: I have been in violation of the LGPL. The binary (JWS) distribution of Duper comes with a modified version of a quite fast Java MD5 implementation. I wanted to ensure efficient use of memory-mapped IO, so I modified it slightly to work with ByteBuffers instead of a regular byte[]. I never completed the modifications though (I couldn't get the native library to work with the ByteBuffer mania, but the pure Java version worked alright and seemed fast enough), and I couldn't decide on how to ship the modifications without being forced to bundle my own "fast-md5" distribution, which also essentially is a fork. So I just closed my eyes to it all in order to get the thing out on the wild at all. Now, however, I just put up my own modified version of "MD5.java" from the fast-md5 package at http://dll.nu/duper/MD5.java. Presumably that should be enough, and anyway, I don't think anyone cares either way. :-)

Today's Firefox/Gecko annoyance slash finally: this bug has been bugging me for over a year (more?) and is still present in the pre-release of Firefox 1.0. According to bugzilla, it's been fixed now.

I seem to have a fetish for writing redundant software. When browsing my website stats, I also noticed that my seemingly redundant Gzip plugin for Fiddler is also receiving a small but persistent stream of visitors (of which about 40 looked at the (minimal) source code during the last week). But there has been only five downloads of the zip file. Yay.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!