Older blog entries for jpick (starting at number 62)

16 Jan 2005 (updated 16 Jan 2005 at 21:56 UTC) »

I haven't blogged for a bit - so I'll have to catch up...

The server: I signed up with simpli.biz as the colo for the new server - $99/month for 200GB of transfer, with the server physically located at AboveNet in downtown San Jose. Somebody on the ops list at communitycolo.net gave me the lead. I put it in on the 7th - they seem like a really good bunch to deal with.

Unfortunately, I upgraded the firmware on the Supermicro IPMI 2.0 board just before putting the server in (to version 2.0.1), and the IPMI board hasn't behaved nicely since. At one point, the board stopped responding altogether. It was also doing lots of other non-responsive things. Since I want to have confidence in the board, I went in this Friday, and downgraded the firmware. It was still acting in ways I couldn't fathom, so I decided to pull the server and take it into work so I could investigate it further. With the 2.0 firmware, and using ipmitool and the command line serial-over-lan tool, I deduced that I couldn't contact the board whenever Linux creates or tears down it's ethernet , and during reboot - it's nearly impossible to get to the BIOS, which is something I could do regularily when I had the server at home. It may have something to do with how all the IP/MAC hackery interacts with the ethernet switch I'm plugging into. I think I may try using a cross-over cable to see if that makes the problems go away.

New toys: I ordered a SlimDevices Squeezebox for home. I've playing around with their open source software - it's great. I'm using the Softsqueeze emulator (written in Java) for my desktop instead of xmms now.

I also placed an order for a Mac Mini which I plan to hook up to my TV in my living room. :-)

Wrist Pain: I'm feeling a lot better, but I still feel the occasional dull pain in my left wrist. I'm going to lay off the video games until I'm 100%.

Japanese: I started taking an introductory conversational Japanese course at Soko Gakuen in San Francisco. It's quite a good deal - our sensei is excellent, the classes are inexpensive, and the school is well established (the school has existed since 1915, behind the Japanese Buddhist church near Japantown). Hajimemashite. Jim desu. Dozo yoroshiku. :-)

20 Dec 2004 (updated 21 Dec 2004 at 01:53 UTC) »

Ouch. I seem to have given myself a typing injury (RSI/Carpal Tunnel/etc.). Not good. It's probably due to doing too much typing on my laptop in odd positions, and I probably aggravated it further with too much Gameboy playing. So I've been attempting to stay away from keyboards and mice as much as possible the last 4-5 days. My hands and wrists are feeling much better now, but not as good as I'd like. As a diversion, I've been reading Asimov's Foundation novels, which I'm quite enjoying.

I tried to get the server swap accomplished before Christmas holidays, but I ran into a snag. Apparently, the racks at the non-profit colo I'm using are already maxed out for power. They're not really well set up for more power-hungry servers - the rack I'm currently in only has a 15A circuit. I bought a neato Seasonic Power Angel power meter from Fry's for $40, and measured the power usage of the server. Idling, it draws 220W, or 1.96A. Fully loaded, it draw 278W, or 2.51A, according to the meter. In 22 hours of idling, the meter says the server drew 4.89 KWH of power.

Anyways, since the power draw on the new server is probably about 1-1.5 amps higher than the old server, the colo people can't let me stick the server in the same place as the current one, and since they have stopped expanding, they have no new racks where I can put it. I'll probably have to go in search of a new colo in the new year.

7 Dec 2004 (updated 7 Dec 2004 at 23:15 UTC) »
Jeremy Fitzhardinge at work corrected some of my bad assumptions about Xen, so I decided to give it a try. Wow, it worked right out of the box on the new server, even with the RAID and LVM setup I had. It sounds like I can do just about everything I was planning to do with UML/QEMU, with the same basic filesystem layout I was planning on, without the 40% performance hit. Cool - I think it's a keeper for the new server.

Cal doesn't get to go to the Rose Bowl - what a rip-off. So it looks like I won't even be able to watch them over new years up in Canada. Anyways, at least we keep coach Tedford for another five years. Tedford is god, as they say in Berkeley.

I spent two days figuring out how to build a quarantine network/VPN using Linux and a UML session. So now, I can grant people access to the new server before I deploy it. I had to resort to using policy-based routing, and all sorts of funky iptables tricks. It's probably something I could have done a lot easier using Xen, if only I had tried that earlier.

I'm going to shoot for getting the server deployed this weekend.

18 Nov 2004 (updated 18 Nov 2004 at 21:03 UTC) »

Lots of little things to post... My TV cable company, Comcast, finally deployed VOD (Video on Demand) here in Berkeley. I can finally see what all the fuss is about. We do VOD development at work, but it's nice to see what it really looks like deployed in the wild. It's actually really cool - there are hundreds of free TV shows available. There's a lot more content than I was expecting. I was pleasantly surprised to find some more fringey content that isn't normally broadcast (eg. Japanese anime clips). Also they put in cool content that works really cool via random access, eg. guitar lessons, cooking recipes, individual video game reviews, and cinema trailers. I now feel a lot better about paying $50 per month for cable. I've got a Tivo, but the VOD has a whole different feel - it's a lot easier to sample content. The content does seem to be highly compressed, and the Motorola interface is somewhat lacking. I might even be tempted to rent a VOD movie -- they're only $3.99. I can't wait until the Moxi box is available from Comcast in my market - it's got such a better interface.

Speaking of Moxi -- we won the Emmy! Yay. We had champagne at work to celebrate. :-)

Slashdot recently had an article: Bill Gates Proclaims End of Passwords. It's about Axalto's .NET Smart Card, which Microsoft is going to issue to all it's employees. Nobody else mentioned it, I guess it's a secret, but the virtual machine is actually Hive Minded's Nectar .NET virtual machine -- that's what Tim Wilkinson has been working on for the last two years (along with Simon Rabe-Hesketh, another Transvirtual ex-boss/friend).

I bought some Google stock at $170. I hope it goes up -- or I'll have to bug my friends that work there.

I've got the new Kaffe.org server on a separate physical subnet at home. Hopefully this weekend, I'll get a tunnel set up to the outside, and I'll hand out some accounts.

It's the Big Game this weekend. Go Bears!

14 Nov 2004 (updated 14 Nov 2004 at 00:48 UTC) »
Work has been busy. Our HDTV cable settop box / DVR product is finally being deployed into thousands of homes, and it looks like it is a hit. We've even been nominated for an Emmy Award! I would definitely get one of our settop boxes for home if Comcast offered it here - it just does a whole lot more than my Tivo. Or maybe I'll just sign up for the employee home trial, and suffer through the pre-release testing cycle and QA surveys.

I helped set up a 20 machine distcc+ccache compile farm. It's fun seeing the compile time go from 80 minutes to just 20 minutes. The tricky part was modifying all the Makefiles for our product so it will go in parallel. It's a bit like a game of "whack-a-mole" -- setting up the Makefiles to go faster, and then fixing all the missing dependencies that turn up as random build failures. Now that I know what I'm doing, I'll have to see if I can get some other things (eg. Kaffe, gcj) to build on the cluster. It should really speed up some regression testing investigations. :-)

I encountered a bizarre GNU make bug where a particular Makefile I have will drain all the jobserver tokens, and the "make -j20" build will degrade into a serial build. I can reproduce it, but only after running the build for 20 minutes. It's also a "heisenbug", which goes away when I try to add some printf statements. Ugh. It was easy to fix up the Makefile to avoid it, so I'm not sure if I'll ever have enough time to fully debug it and file a bug report/patch.

I finally found some time to start working on the new server for Kaffe.org. It turns out the built-in Intel ICH5R SATA RAID is really just a joke, it's not really hardware RAID at all (I knew this when I bought it). I ended up turning it off in the BIOS, and going with the Linux software RAID in the kernel, which is really quite cool -- this is the first I've ever gotten to play with it (all the other servers I use have hardware RAID). I got so enthused, I wanted to try out RAID5, so I went out and bought a 3rd 250GB hard driver, so now the machine has 0.75TB of storage. The new drive is a Western Digital just like the others, but it's a EIDE drive, whereas the others are SATA. I ran the bonnie++ benchmark, and there wasn't much difference in speed at all. I partitioned each drive into 15 partitions (the max for the SCSI subsystem), and setup about half of them as RAID-5. I set up the other half with RAID-0 (striped) sets -- there's no disk space overhead for parity, and it's theoretically faster, but it has no redundancy, so I'll have downtime if a disk fails. I'm running LVM2 on top of the RAID sets, so I can easily create and move around logical volumes. I also moved the boot partition to a RAID-1 (Mirrored) volume. And I did lots of GRUB magic, all remotely over the network using IPMI2. It's all very cool.

The machine came with Fedora Core 2 -- the x86_64 version. I spent a lot of time removing RPMs so I now just have a minimal set. I think I'll attempt to upgrade that to Fedora Core 3 somehow. I'm not too keen to run their installer though - I really only want a minimal setup. I think I'll also try out Debian's AMD64 port -- however, that hasn't been released yet. The name for their port seems wrong, since I'm running Intel Xeon EM64T chips. I'd somewhat leery of it, since it I don't know what decisions they've made with regards to the porting issues. My loyalties are split - I maintain RPMs all day long at work, but I'm also a Debian "emeritus" developer. It's hard to make any decisions without having hard benchmark numbers for performance. I've got access to some benchmark suites myself (SPEC CPU 2000, SPEC JVM 98), so maybe I'll have to do some experiments.

Anyways, I'm almost ready to set up a separate subnet for the machine, and tunnel it to the existing kaffe.org server, so I can start migrating some services over. :-)

I'm way behind on my other kaffe.org commitments -- eg. getting testing going for the release process, some website reorganization, JIT4 merge, etc. Hopefully, I'll get this server stuff tucked away soon, and I can move on to the fun stuff. :-)

I just went to see "The Incredibles". Excellent! I really had fun picking out all the local East Bay references -- Pixar is located next door in Emeryville, where I saw the movie. Anyways, I think it's the best Pixar film yet, so go see it. :-)

Well, there goes another crazy, divisive, not-immediately-clear-who-is-the-victor election here in the good ol' USA. As a Canadian citizen who's just working here temporarily, all I could do was sit on the sidelines and watch the madness.

It's very interesting to see the ways that new technologies, the "Internets", and free software really are starting to change the way that ordinary people can interact with the political process. As with all technology, there are good things about this, and there are bad things as well. It's also clear that the existing batch of technologies aren't accessable to enough people, don't scale well, and are prone to censorship.

We all now live in very "interesting times", and things continue to change at an ever increasing pace. In the end, all politics is local. I am an optimist, and I believe that people are essentially good, and will always work to improve their own lives, and the lives of those they love and those they must live with.

Economic policies, the environment, and war are all examples of global issues that are abstract to us all, except when the issues "come home". Our politicians and institutions are supposed to represent us, and act in our best interest. Different communities of people will have different ideas of what must be done based on their worldviews. We're all learning, and it's a arduous process of losing our precious innocence as we all grow older. We can only work and hope that humankind gets to a better place in the end.

New Server: I finally got the new server on Friday. Yay! I picked it up at ASA Computers in Sunnyvale. I was really impressed - they were a much larger operation than I had imagined. I got a mini-tour of one of their warehouses where they were getting ready to ship out a bunch of hardware to Fermilab.

Anyways, they were very pleasant to deal with -- I had no problems. I'd recommend them based on my experience so far -- of course, I haven't had to deal with returning hardware -- hopefully I won't have to deal with that ever.

Anyways, the hardware is pretty nice. Here are some photos (a bit blurry, it was still dark out). It's also extremely LOUD! It sounds like a jet engine. That's not surprising since it's a 1U intended for a colo facility, and has two blower fans to keep the two Xeons cool. I set it up in my office at home, and after an hour or so, I couldn't handle the noise anymore, and evacuated to my living room to escape the earsplitting noise.

It came with Fedora Core 2 (x86-64 version) installed. It seems pretty solid, but I'm planning on moving it over to Debian, primarily because I'm more comfortable with it, particularly when upgrading it remotely. Overall, I'm going to install very little in the root image, and I'm going to make extensive use of User Mode Linux (and other VMs, like Kaffe) to subdivide things up.

I built a custom 2.6.8.1 kernel for it, and installed the latest IPMI patches from the OpenIPMI project. After a few hours of fooling around, I finally figured out that I had to run a DOS application to reset the IP address that the IPMI board was using. After that, I was able to use SuperMicro's IPMIView application (a Java Swing app) to remotely control the power on the box, and control the BIOS and console. The trick with the BIOS was to use the option to redirect the output to COM2, and it worked with the IPMIView application.

IPMI so cool! I should never ever have to make the drive to the colo again when the server crashes, since I can now restart it and control it remotely -- unless of course the hardware itself dies. I've had to do that routine literally dozens of times of the years, so I think this is the greatest thing ever!

A nice side effect is that I can now work on BIOS-level configuration stuff from my living room in my apartment, so I don't have to listen to the server!

My current plan is to take a week or two setting up the server just the way I want it, and then I'll do the server swap at the colo.

Backups: they've always been one of those things I have done infrequently, and basically procrastinated on. Why do something today when you can do it tomorrow?

I was burnt about a month ago when I bought 2 new external 250GB Maxtor Firewire/USB2 drives. They're quite nice, but one died on me after about 3 weeks, and I lost some work. If you've been wondering where berkeleysignal.com went, that's where (it'll be back soon). Fortunately, I didn't lose too much, but I decided to really get my act together with regards to backups.

All the machines I run tend to be geographically distributed. To make matters worse, I'm gradually moving everything over to many, many User-mode Linux virtual servers.

I did some research using my favourite tool, and decided to try out Duplicity. It's got some nice features -- it's quite simple to use, standalone, generates binary diffs and small incremental backup sets, and doesn't need any special setup on the server side.

After backporting it to Debian stable, I discovered it wasn't that efficient over the network -- particularily to my home. When doing an incremental backup of the kaffe server using scp to a share at home, it would start by attempting to fetch a 90MB index file, which takes forever, since I've only got 256kbps upstream on my cable modem.

I owe the patent rights on the final solution to Paul Fisher, who's been crashing at my place in Berkeley until he can move into his new apartment. So the approach I'm using now is to rsync the whole filesystem image to a remote machine, and then generate the backup sets using duplicity. I built a custom User-mode linux image with the appropriate ssh keys and 40GB of disk space, and it's working great.

I'm now doing daily backups of kaffe.org, developer.classpath.org, and several of my own internal servers and UML images. The nice thing is that I can now restore files back in history since I'm keeping the incremental sets. They're a convenient size to burn onto DVD-R, so I should be able to keep a complete history. :-)

Having good backups is going to become really useful when I get the new server for kaffe.org, since I'm going to be doing a lot of re-arranging. I'm also going to look at building a system for generating "hot spare" images from the backups. Generally speaking, I'll be much more comfortable rolling out a whole bunch of new services, since I know I will be able to rely on having backups.

I still kick the scripts off manually, since things still tend to break (eg. not enough disk space, etc.). Eventually, I think I'll stick a web interface on the thing, and publish the UML disk image. :-)

14 Oct 2004 (updated 14 Oct 2004 at 21:53 UTC) »

I finally scraped together enough money to buy the new server for kaffe.org (and some other projects).

I placed the order yesterday. It's going to have 2 Xeon 2.8GHz EM64T processors, 2GB PC2700 DDR memory, a SuperMicro X6DHR-XIG motherboard, a SuperMicro 1U case, 2 250GB Western Digital SATA drivers, plus a slimline CD and floppy. Also, I'm going to get the SuperMicro IPMI card, which will let me monitor the box via the network, as well as reboot it, and access the BIOS and VGA console!

I'm ordering it from ASA Computers in Santa Clara -- they seemed knowledgeable. Hopefully, they'll have it assembled sometime next week.

53 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!