Older blog entries for wsanchez (starting at number 10)

I'm checking out a new project called Gale. At MIT, we used this messaging system called Zephyr. Zephyr was cool because it was ubiquitous on campus. If you were looking for someone, there was always a good chance of fiding them online, or someone who knew where they are. You could subscribe and send to different categories of messages, some for given student groups, others for generic help (in any area) and discussion, others more specific, like one for NetBSD users. It's sort of like email and mailing lists, but real-time, a lot more dynamic, and less "formal" (if that makes sense). It was also tied into system services, so when the printer is done with your job, or your fileserver is going down for maintenance, you'd get a message about it.

Anyway, I still log in to MIT to use zephyr every day, and I wish I could do something similar with a broader group; say all of my friends, or the Darwin community, etc. I've looked at AIM and ICQ, but they stink at the group dynamic. They are OK for messaging specific people, but not for group discussions.

Unfortunately, I think Zephyr is pretty much dead. I mean MIT will probably use it for the foreseeable future because they've always used it, but the last Zephyr release (2.0.4) was in 1996. There are all sort of features that Zephyr could have, and there are patches around for some of them, but it really seems like MIT isn't doing anything with Zephyr any more, so I'm giving up on it. The need for a kerberos realm to authenticate message is also a bit of a pain. I do use kerberos, but it's not trivial to deploy and manage a realm, and it's not really the right thing for an open user base such as the Darwin community.

Gale is a new thing which aims to replace Zephyr (and has at Caltech). Not I hate to plug Caltech over MIT, given my academic roots, but there you have it. Gale does what zephyr does plus it fixes a bunch of things up, claims to be faster, leaner, and meaner, and looks like a good shot at picking up where zephyr left off.

The one drag I've noticed so far is that it's under the GPL instead of a free license, which tends to be a nuissance at best; but it's a common faux-pas, and one can't expect perfection.

I've also started porting OpenSSH to Darwin, since ssh is an important tool, and we all know the ssh1/ssh2 license fiasco. I was working with the OpenBSD code, but a couple of folks pointed out that now there is a cross-platform version that uses autoconf and so on, so I'm now using that. That's coming along more nicely, but it seems Darwin lacks /dev/random, though, so we need to fix that in order to make the OpenSSH build happy.

I'm taking a strong interest in a new version control system called inversion which is being designed from scratch. CVS seems to be the best thing available right now, but it really stinks in some important ways. Inversion is looking like The Right Thing is every way that bugged me about CVS, so I'm very excited about it. I don't know how much I'll be able to contribute to the project, but I hope I can do some useful work on it, since I think it's a very important problem to solve.

Today Darwin 1.0.2 rolled out.

1.0.2 is a minor update in terms of content. We added gdb and binutils. I didn't get enough time to work out some details with getting the new versions of these into CVS and I didn't want to do any hacks, so they didn't make 1.0.1, but now they're ready.

Also, while testing the new installer on Old World (Beige) machines, we found a recently-introduced SCSI driver bug which was causing a message to be printed to the console every second. Since the console is the only UI you get, this was very annoying. The bug had already been fixed; I had just grabbed a snapshot on the wrong day, it seems. So we patched xnu (the kernel) to fix this (a one-liner) bug. New World users who downloaded 1.0.1 won't be affected by it, so they needn't update their kernels.

For 1.0.1 we had two installers; one for New World machines, and another for Old World. (We found some installation problems with Old World so we didn't ship it on the launch date.) The reason we needed two installers is that New World machines have a new booting architecture which allowed us to restore Darwin into a single partition and boot that, but Old World machines lack this feature. We figured it was better to have two installers, since it would at least let people with newer hardware install into a partition without having to wipe a whole drive out.

It turns out that our crack team of boot-manglers at Apple are pretty clever and after much hackery and magic they found a way to get Old World machines to boot without a second partition for the booter, and they built a new version of ASR to install using the new scheme.

So Darwin 1.0.2 also features a spiffy new ASR which happily installs onto both Old World and New World machines.

All of this cost us a couple of days, but I think the result is well worth it. One installer fits all.

I got my truck back on yesterday.

Apparently a bunch of the girls from church decided to play games with me. They sent a package with photos of my Jeep in various unhappy places, and then sent a ransom note with a list of demands:

1-Peanut, pickle sandwich.
1-Pair nylons, color: Nude, size: B
1-Visitor badge for Apple Computers site.
1-Chopsticks from Mr. Chau's
10-Mini coffee creamers from McDonald's
1-MULTIPACK box of tampons. (Tampax brand)
1-New N'sync CD. (title: No Strings Attached)

In an attempt to be a sport about the whole thing, I went shopping. The sandwich sounds gross. Ick. And while I can buy the girly stuff under the premise that I'm married, I really had no excuse whatsoever for the N'sync CD. It was very embarrasing. The store clerk looked at me funny and what could I say? "Yeah I'm a fan."

I cheated on a couple of things. I didn't have a pickle (don't really care for them), so the sandwich turned out to be a peanut butter sandwich (not so yucky). And I never went to McDonald's for the creamers.

I got instructions to go to Vasona Park in Los Gatos, and find this tree with the big red X on it. There was a paper bag there with a walkie-talkie in it.

"Show us the peanut pickle sandwich."

So I lifted up the sandwich, then the nylons, and so on until they were satisfied. Then I left everything in the bag, which I left on the side of the road, and after I left they told me the truck was at Starbucks, where it should have been a week and a half ago.

They have yet to fess up to their identities, though there are some obvious suspects.

I was hoping the Old World installer would be ready today (oops, it's past midnight... yesterday), but now we found a system bug that happened to show up in the snapshot of the kernel I chose to build which only comes up on older machines. It ends up making the console rather unusable, and since that's the primary UI for Darwin at the moment, we're going to fix that first. I already have the fix tested, so we'll start working on getting the new installer created tomorrow. It takes a while to master these things, though (it's still pretty manual), and then we have to test it, so I can't promise it in time for the weekend, unfortunately.

I've gotten a lot of supportive email lately, which I'm grateful for. My apologies if I don't respond to all of them, but it's hard to keep up with mail sometimes.

My thanks go to Ernie for making the whole open source thing at Apple fly with management; Klee for writing the tools that make the builds go and for making all of the 0.1 (internal test), 0.2, and 0.3 releases happen last year; Ron and Brett for dealing with all the management hooey; and Joyce for being the one lawyer who gets it. It's been a long year for Darwin, and these folks really dove in and made it happen. Also Justin for covering the hard questions on the mailing lists, as well the rest of the Core OS team members who stepped in when I needed help. It's a great bunch of people I work with every day.

Darwin 1.0 launched today, where today is the 5th, but I guess editing an entry changes the date on it. Oops.

So after I started this diary, the press latched onto it as "news" and things got a little nutty. The Intel port is something I've promising for about a year now, so I was kind of surprised that everyone got all excited about it.

Once again- I compiled for Intel, but I did that on a PowerPC Mac. I have not yet booted Darwin on Intel, and I'd be quite surprised if Darwin 1.0 worked unmodified on Intel. There is still work to be done there.

By the way, it should be noted that I think it's a dissapointing that this "Intel thing" was made into a big deal, and yet nobody seemed to care that my truck was kidnapped. The news never mentions the important stuff.

Anyway... Last week, I updated Apache in Darwin to 1.3.12, a long overdue update, and I updated Perl to 5.6.0. I also ported automake, because it was really easy. Each sqeaked their way into the 1.0 release. The Perl update is a big milestone for me. Many Darwin projects come from 3rd parties, and I've managed to get our diffs up to the upstream guys for most of these, but Perl has been a difficult one. The skanky-nasty Configure script Perl uses was the biggest obstacle. But after 2 years, it's finally done, and 5.6.0 support Mac OS X Server fully, and has my current patches for Mac OS X, so we're finnally sync'ed up. I have to thank Jarkko for his help (and prodding) in making this happen.

We've started to get some more patches for Darwin lately. I hope this is a sign that activity will be picking up. Detlev Droege contributed a patch to automount which let it read flat-file mount maps, which it did in OpenStep, but broke way back in Rhapsody. We were happy to see this get fixed, since it's one of those "we'll get around to it someday" bugs that's been around for a long time.

The Darwin 1.0 image turned out to be a lot bigger than expected, around 220 MB. The 0.3 image was around 60 MB. We have a lot more components in 1.0, and the fact that all binaries have code for two architectures now are reasons for it.

The www.publicsource.apple.com web server has been quite busy the past couple of days, and apparently we've been sucking up a lot of Apple's network bandwidth, which isn't so nice. I've got the server set to a max of 256 clients, and during the day, it's pinned at 256 connections. The cool thing is that this is being served up by just one machine, a B&W G3 running Mac OS X Server 1.0 plus an updated kernel. We're marketing Mac OS X Server as a low-to-medium-end web server, but this machine is holding up really well under a pretty heavy load, and it's moving a lot of data out pretty quickly. Uptime is 98 days. This may not impress the FreeBSD user, but for a first attempt at a server OS from our group, I'm pretty jazzed about it.

We ended up needing two Darwin installers. One is the same as the Darwin 0.3 net download and requires that you wipe out a whole disk (the 0.3 CD installer used yet another mechanism which let you install onto a partition). This one is required for what we call "Old World" machines, which turns out to be everything Beige. On "New World" machines (iMac, colorful PowerMac, newer PowerBooks, iBooks) we can get away with a single-partition install. Unfortunately, we found a bug in the Old World installer, so only the New World image was available for the launch.

Sunday I flew with my friends Pat (my pilot "big buddy") and Dave (mentioned previously) to South Lake Tahoe. That's a pretty cool trip, my first over scary looking mountains. We watched Pat lose $40 at a casino and then we zipped on back.

I have this radio-controlled Jeep which I drive around Apple. Each day I drive it to the coffee shop in building One and it hauls the coffee in its bed back to building Two for me. It's a ritual. It's The Way My Coffee Gets Around.

Well, after the Sunday trip I went to church (I go in an evening service) and I'm pretty late because of the Tahoe trip so the service is over and people are just hanging out and I decide to bust out with the Jeep, which was in my trunk because I was supposed to be going on a retreat with this very group over the weekend, and I was going to bring the truck, but I'm kinda dumb and I missed the whole thing, instead spending the weekend hacking on Darwin and flying to Tahoe, both of which you know about by now.

Anyway, I have the Jeep and the batteries fizz out, so I put it on a table by the exit and go chat with some people. On my way out, there is no Jeep on the table. Well, it must be in the Happy Lost and Found Place, so I'm looking around, and I get some people to look and we notice a paper plate on the formerly Jeep-posessing table. It has a note on it which basically said that for $2.5M I could pick it up at 5PM Wed.

Now... OK, so someone kidnapped my truck, at church, no less. And they want $2.5M. So far, I'm OK with that. After all, it is a nice truck, and I should have been keeping an eye on it. Sure, it's church, but hey, we welcome all sorts at our church, and I guess one of the sorts liked my truck. Fine.

But for crying out loud, Starbucks?!?!?!? There's one of those every five feet between here and Boston, and I have to be at the right one on Wednessday? Sheesh, talk about mean. And 5PM to boot. That's rush hour. I go to work at about 11AM and leave pretty late or pretty early depending on several factors, but the point is I never drive during rush hour. I hate rush hour, and I only live a few miles from work. The church is in Los Gatos, and that's a bit more of a trek, and I have to assume the Starbucks in question is near the church, or I don't stand a chance.

That was Sunday.

So yesterday, (Wed) I'm at Starbucks, but I'm half an hour late, and my Jeep and it's captor were not to be found. Once again, perhaps my fault. After all, the note said 5PM not 5:30. But I think I deserved a little slack for the rush hour thing.

I guess I'll find out more on Sunday.

Well... another world-build, this time with the kernel that's checked into the Darwin repository, finished successfully. The only glitches were in the AppleTalk library, and (as a result) in the AppleTalk commands project. The new kernel headers required a small change in the library, and updating to the latest version fixed that right up. I had expected to see more problems, since when you get near the end of a project, that's when everything is supposed to blow up, but it's looking pretty good.

So it looks like the full OS builds are going quite well, and all that's left is making sure it all works OK, which I will start working on tomorrow.

I posted on darwin-development that I started this here diary to keep people up-to-date on my progress (I got this idea from Greg at ApacheCon, by the way; good call, dude) and my good friend Dan (from my last gig at Disney Online) sent me a note saying that it seems to have made a splash on Slashdot, where much meaningful commentary on the future of Apple and Darwin is underway.

Today another big milestone has come up. I imported the sources to the kernel into the Darwin CVS repository today, which means that at this point all of the sources needed to build Darwin are available externally for the first time.

This kernel is slightly newer than the one I've been testing builds with, so I am starting yet another complete world-build to make sure that it builds and everything else build against it (it installs a mondo portion of the system headers). I will probably need to do one or two more world-builds after this one before one completes without a single build failure. In the meantime, I can begin installing and testing the system to make sure that the results are good.

Wednessday I reached an important milestone in getting Darwin 1.0 released. For the first time, I managed to build all of the components together. The build system we use for Darwin (which was written by Klee Dienes and uses the Debian packaging tools) seeds a build root for each project with the software required to build that project, and it gets the seed packages from the last build. So this milestone is a very important one because everything built using Darwin 1.0-built packages, which means we're also testing a large portion of the software as we go along.

It should be noted that I am building all components fat (for both PowerPC and Intel) by default. The only component that is build for PowerPC only is emacs. This is because it wants to run and dump core as part of the build, so it needs to build natively and can't be cross-built. Once we have an Intel system running I can figure out how to build emacs natively on each platform and merge the binaries so it is fat as well.

Apparently a lot of people are under the impression that Apple isn't going to help out with reviving the Intel port of Darwin. This is false.

Getting everything built fat is a big step, but a lot of work remains. The next thing is to get installation bootstrapped so we can get Darwin onto an Intel system, and then to get the kernel running, since we haven't tested the new kernel on Intel yet, and there is limited driver support for Intel PC devices. I have a high degree of confidence that most of the user-space software will work without problems, particularly since a majority of it comes from the BSD world where Intel is the primary platform, but also because we've seen it work before in Rhapsody.

Also last week: David Zarzycki, who's been an intern at Apple since the Stone Age, most recently working in the Core OS Networking group on IPSec and IPv6, is now a full-timer at Apple. His new job will be to support Darwin, which means that I won't be doing this mostly alone any more. Dave will be working on some developer infrastructure that we've been wishing we had, and in general just making things happen that need to happen. This is great news, and I'm quite happy to have Dave on board. Some of you may have met Dave at MacHack, FreeBSDCon, or this month's ApacheCon.

1 older entry...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!