Older blog entries for jpick (starting at number 60)

7 Dec 2004 (updated 7 Dec 2004 at 23:15 UTC) »
Jeremy Fitzhardinge at work corrected some of my bad assumptions about Xen, so I decided to give it a try. Wow, it worked right out of the box on the new server, even with the RAID and LVM setup I had. It sounds like I can do just about everything I was planning to do with UML/QEMU, with the same basic filesystem layout I was planning on, without the 40% performance hit. Cool - I think it's a keeper for the new server.

Cal doesn't get to go to the Rose Bowl - what a rip-off. So it looks like I won't even be able to watch them over new years up in Canada. Anyways, at least we keep coach Tedford for another five years. Tedford is god, as they say in Berkeley.

I spent two days figuring out how to build a quarantine network/VPN using Linux and a UML session. So now, I can grant people access to the new server before I deploy it. I had to resort to using policy-based routing, and all sorts of funky iptables tricks. It's probably something I could have done a lot easier using Xen, if only I had tried that earlier.

I'm going to shoot for getting the server deployed this weekend.

18 Nov 2004 (updated 18 Nov 2004 at 21:03 UTC) »

Lots of little things to post... My TV cable company, Comcast, finally deployed VOD (Video on Demand) here in Berkeley. I can finally see what all the fuss is about. We do VOD development at work, but it's nice to see what it really looks like deployed in the wild. It's actually really cool - there are hundreds of free TV shows available. There's a lot more content than I was expecting. I was pleasantly surprised to find some more fringey content that isn't normally broadcast (eg. Japanese anime clips). Also they put in cool content that works really cool via random access, eg. guitar lessons, cooking recipes, individual video game reviews, and cinema trailers. I now feel a lot better about paying $50 per month for cable. I've got a Tivo, but the VOD has a whole different feel - it's a lot easier to sample content. The content does seem to be highly compressed, and the Motorola interface is somewhat lacking. I might even be tempted to rent a VOD movie -- they're only $3.99. I can't wait until the Moxi box is available from Comcast in my market - it's got such a better interface.

Speaking of Moxi -- we won the Emmy! Yay. We had champagne at work to celebrate. :-)

Slashdot recently had an article: Bill Gates Proclaims End of Passwords. It's about Axalto's .NET Smart Card, which Microsoft is going to issue to all it's employees. Nobody else mentioned it, I guess it's a secret, but the virtual machine is actually Hive Minded's Nectar .NET virtual machine -- that's what Tim Wilkinson has been working on for the last two years (along with Simon Rabe-Hesketh, another Transvirtual ex-boss/friend).

I bought some Google stock at $170. I hope it goes up -- or I'll have to bug my friends that work there.

I've got the new Kaffe.org server on a separate physical subnet at home. Hopefully this weekend, I'll get a tunnel set up to the outside, and I'll hand out some accounts.

It's the Big Game this weekend. Go Bears!

14 Nov 2004 (updated 14 Nov 2004 at 00:48 UTC) »
Work has been busy. Our HDTV cable settop box / DVR product is finally being deployed into thousands of homes, and it looks like it is a hit. We've even been nominated for an Emmy Award! I would definitely get one of our settop boxes for home if Comcast offered it here - it just does a whole lot more than my Tivo. Or maybe I'll just sign up for the employee home trial, and suffer through the pre-release testing cycle and QA surveys.

I helped set up a 20 machine distcc+ccache compile farm. It's fun seeing the compile time go from 80 minutes to just 20 minutes. The tricky part was modifying all the Makefiles for our product so it will go in parallel. It's a bit like a game of "whack-a-mole" -- setting up the Makefiles to go faster, and then fixing all the missing dependencies that turn up as random build failures. Now that I know what I'm doing, I'll have to see if I can get some other things (eg. Kaffe, gcj) to build on the cluster. It should really speed up some regression testing investigations. :-)

I encountered a bizarre GNU make bug where a particular Makefile I have will drain all the jobserver tokens, and the "make -j20" build will degrade into a serial build. I can reproduce it, but only after running the build for 20 minutes. It's also a "heisenbug", which goes away when I try to add some printf statements. Ugh. It was easy to fix up the Makefile to avoid it, so I'm not sure if I'll ever have enough time to fully debug it and file a bug report/patch.

I finally found some time to start working on the new server for Kaffe.org. It turns out the built-in Intel ICH5R SATA RAID is really just a joke, it's not really hardware RAID at all (I knew this when I bought it). I ended up turning it off in the BIOS, and going with the Linux software RAID in the kernel, which is really quite cool -- this is the first I've ever gotten to play with it (all the other servers I use have hardware RAID). I got so enthused, I wanted to try out RAID5, so I went out and bought a 3rd 250GB hard driver, so now the machine has 0.75TB of storage. The new drive is a Western Digital just like the others, but it's a EIDE drive, whereas the others are SATA. I ran the bonnie++ benchmark, and there wasn't much difference in speed at all. I partitioned each drive into 15 partitions (the max for the SCSI subsystem), and setup about half of them as RAID-5. I set up the other half with RAID-0 (striped) sets -- there's no disk space overhead for parity, and it's theoretically faster, but it has no redundancy, so I'll have downtime if a disk fails. I'm running LVM2 on top of the RAID sets, so I can easily create and move around logical volumes. I also moved the boot partition to a RAID-1 (Mirrored) volume. And I did lots of GRUB magic, all remotely over the network using IPMI2. It's all very cool.

The machine came with Fedora Core 2 -- the x86_64 version. I spent a lot of time removing RPMs so I now just have a minimal set. I think I'll attempt to upgrade that to Fedora Core 3 somehow. I'm not too keen to run their installer though - I really only want a minimal setup. I think I'll also try out Debian's AMD64 port -- however, that hasn't been released yet. The name for their port seems wrong, since I'm running Intel Xeon EM64T chips. I'd somewhat leery of it, since it I don't know what decisions they've made with regards to the porting issues. My loyalties are split - I maintain RPMs all day long at work, but I'm also a Debian "emeritus" developer. It's hard to make any decisions without having hard benchmark numbers for performance. I've got access to some benchmark suites myself (SPEC CPU 2000, SPEC JVM 98), so maybe I'll have to do some experiments.

Anyways, I'm almost ready to set up a separate subnet for the machine, and tunnel it to the existing kaffe.org server, so I can start migrating some services over. :-)

I'm way behind on my other kaffe.org commitments -- eg. getting testing going for the release process, some website reorganization, JIT4 merge, etc. Hopefully, I'll get this server stuff tucked away soon, and I can move on to the fun stuff. :-)

I just went to see "The Incredibles". Excellent! I really had fun picking out all the local East Bay references -- Pixar is located next door in Emeryville, where I saw the movie. Anyways, I think it's the best Pixar film yet, so go see it. :-)

Well, there goes another crazy, divisive, not-immediately-clear-who-is-the-victor election here in the good ol' USA. As a Canadian citizen who's just working here temporarily, all I could do was sit on the sidelines and watch the madness.

It's very interesting to see the ways that new technologies, the "Internets", and free software really are starting to change the way that ordinary people can interact with the political process. As with all technology, there are good things about this, and there are bad things as well. It's also clear that the existing batch of technologies aren't accessable to enough people, don't scale well, and are prone to censorship.

We all now live in very "interesting times", and things continue to change at an ever increasing pace. In the end, all politics is local. I am an optimist, and I believe that people are essentially good, and will always work to improve their own lives, and the lives of those they love and those they must live with.

Economic policies, the environment, and war are all examples of global issues that are abstract to us all, except when the issues "come home". Our politicians and institutions are supposed to represent us, and act in our best interest. Different communities of people will have different ideas of what must be done based on their worldviews. We're all learning, and it's a arduous process of losing our precious innocence as we all grow older. We can only work and hope that humankind gets to a better place in the end.

New Server: I finally got the new server on Friday. Yay! I picked it up at ASA Computers in Sunnyvale. I was really impressed - they were a much larger operation than I had imagined. I got a mini-tour of one of their warehouses where they were getting ready to ship out a bunch of hardware to Fermilab.

Anyways, they were very pleasant to deal with -- I had no problems. I'd recommend them based on my experience so far -- of course, I haven't had to deal with returning hardware -- hopefully I won't have to deal with that ever.

Anyways, the hardware is pretty nice. Here are some photos (a bit blurry, it was still dark out). It's also extremely LOUD! It sounds like a jet engine. That's not surprising since it's a 1U intended for a colo facility, and has two blower fans to keep the two Xeons cool. I set it up in my office at home, and after an hour or so, I couldn't handle the noise anymore, and evacuated to my living room to escape the earsplitting noise.

It came with Fedora Core 2 (x86-64 version) installed. It seems pretty solid, but I'm planning on moving it over to Debian, primarily because I'm more comfortable with it, particularly when upgrading it remotely. Overall, I'm going to install very little in the root image, and I'm going to make extensive use of User Mode Linux (and other VMs, like Kaffe) to subdivide things up.

I built a custom 2.6.8.1 kernel for it, and installed the latest IPMI patches from the OpenIPMI project. After a few hours of fooling around, I finally figured out that I had to run a DOS application to reset the IP address that the IPMI board was using. After that, I was able to use SuperMicro's IPMIView application (a Java Swing app) to remotely control the power on the box, and control the BIOS and console. The trick with the BIOS was to use the option to redirect the output to COM2, and it worked with the IPMIView application.

IPMI so cool! I should never ever have to make the drive to the colo again when the server crashes, since I can now restart it and control it remotely -- unless of course the hardware itself dies. I've had to do that routine literally dozens of times of the years, so I think this is the greatest thing ever!

A nice side effect is that I can now work on BIOS-level configuration stuff from my living room in my apartment, so I don't have to listen to the server!

My current plan is to take a week or two setting up the server just the way I want it, and then I'll do the server swap at the colo.

Backups: they've always been one of those things I have done infrequently, and basically procrastinated on. Why do something today when you can do it tomorrow?

I was burnt about a month ago when I bought 2 new external 250GB Maxtor Firewire/USB2 drives. They're quite nice, but one died on me after about 3 weeks, and I lost some work. If you've been wondering where berkeleysignal.com went, that's where (it'll be back soon). Fortunately, I didn't lose too much, but I decided to really get my act together with regards to backups.

All the machines I run tend to be geographically distributed. To make matters worse, I'm gradually moving everything over to many, many User-mode Linux virtual servers.

I did some research using my favourite tool, and decided to try out Duplicity. It's got some nice features -- it's quite simple to use, standalone, generates binary diffs and small incremental backup sets, and doesn't need any special setup on the server side.

After backporting it to Debian stable, I discovered it wasn't that efficient over the network -- particularily to my home. When doing an incremental backup of the kaffe server using scp to a share at home, it would start by attempting to fetch a 90MB index file, which takes forever, since I've only got 256kbps upstream on my cable modem.

I owe the patent rights on the final solution to Paul Fisher, who's been crashing at my place in Berkeley until he can move into his new apartment. So the approach I'm using now is to rsync the whole filesystem image to a remote machine, and then generate the backup sets using duplicity. I built a custom User-mode linux image with the appropriate ssh keys and 40GB of disk space, and it's working great.

I'm now doing daily backups of kaffe.org, developer.classpath.org, and several of my own internal servers and UML images. The nice thing is that I can now restore files back in history since I'm keeping the incremental sets. They're a convenient size to burn onto DVD-R, so I should be able to keep a complete history. :-)

Having good backups is going to become really useful when I get the new server for kaffe.org, since I'm going to be doing a lot of re-arranging. I'm also going to look at building a system for generating "hot spare" images from the backups. Generally speaking, I'll be much more comfortable rolling out a whole bunch of new services, since I know I will be able to rely on having backups.

I still kick the scripts off manually, since things still tend to break (eg. not enough disk space, etc.). Eventually, I think I'll stick a web interface on the thing, and publish the UML disk image. :-)

14 Oct 2004 (updated 14 Oct 2004 at 21:53 UTC) »

I finally scraped together enough money to buy the new server for kaffe.org (and some other projects).

I placed the order yesterday. It's going to have 2 Xeon 2.8GHz EM64T processors, 2GB PC2700 DDR memory, a SuperMicro X6DHR-XIG motherboard, a SuperMicro 1U case, 2 250GB Western Digital SATA drivers, plus a slimline CD and floppy. Also, I'm going to get the SuperMicro IPMI card, which will let me monitor the box via the network, as well as reboot it, and access the BIOS and VGA console!

I'm ordering it from ASA Computers in Santa Clara -- they seemed knowledgeable. Hopefully, they'll have it assembled sometime next week.

Time to update the blog again... I'm still not in the habit.

I finally set up a User-mode Linux virtual server for Mark Wielaard and the Classpath project on the Kaffe server. Hopefully it holds up well. I should have enough cash next month to upgrade the server to something decent (eg. a dual-Opteron).

I've been busy attending to real-life stuff -- eg. work, Disneyland, taxes, visitors from Boston (Paul Fisher), sailing, etc. I haven't done too much Kaffe stuff lately, other than keep the server running. That's going to change fairly quickly - I've got a lot of testing ideas I'm going to drop on the mailing list.

I've got some other cool little projects on the go too - stay tuned.

23 Jul 2004 (updated 23 Jul 2004 at 19:57 UTC) »

A friend gave me a Gmail invite. It's pretty cool. I like the web interface - it's quite nice for reading mail list stuff. I don't plan to use it for my primary email, but I'll probably steal some ideas from it for a little free software project I'm working on.

Kaffe looks like it's going great guns. The regression thingy I'm working on is still coming...

51 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!