Older blog entries for wsanchez (starting at number 9)

I'm taking a strong interest in a new version control system called inversion which is being designed from scratch. CVS seems to be the best thing available right now, but it really stinks in some important ways. Inversion is looking like The Right Thing is every way that bugged me about CVS, so I'm very excited about it. I don't know how much I'll be able to contribute to the project, but I hope I can do some useful work on it, since I think it's a very important problem to solve.

Today Darwin 1.0.2 rolled out.

1.0.2 is a minor update in terms of content. We added gdb and binutils. I didn't get enough time to work out some details with getting the new versions of these into CVS and I didn't want to do any hacks, so they didn't make 1.0.1, but now they're ready.

Also, while testing the new installer on Old World (Beige) machines, we found a recently-introduced SCSI driver bug which was causing a message to be printed to the console every second. Since the console is the only UI you get, this was very annoying. The bug had already been fixed; I had just grabbed a snapshot on the wrong day, it seems. So we patched xnu (the kernel) to fix this (a one-liner) bug. New World users who downloaded 1.0.1 won't be affected by it, so they needn't update their kernels.

For 1.0.1 we had two installers; one for New World machines, and another for Old World. (We found some installation problems with Old World so we didn't ship it on the launch date.) The reason we needed two installers is that New World machines have a new booting architecture which allowed us to restore Darwin into a single partition and boot that, but Old World machines lack this feature. We figured it was better to have two installers, since it would at least let people with newer hardware install into a partition without having to wipe a whole drive out.

It turns out that our crack team of boot-manglers at Apple are pretty clever and after much hackery and magic they found a way to get Old World machines to boot without a second partition for the booter, and they built a new version of ASR to install using the new scheme.

So Darwin 1.0.2 also features a spiffy new ASR which happily installs onto both Old World and New World machines.

All of this cost us a couple of days, but I think the result is well worth it. One installer fits all.

I got my truck back on yesterday.

Apparently a bunch of the girls from church decided to play games with me. They sent a package with photos of my Jeep in various unhappy places, and then sent a ransom note with a list of demands:

1-Peanut, pickle sandwich.
1-Pair nylons, color: Nude, size: B
1-Visitor badge for Apple Computers site.
1-Chopsticks from Mr. Chau's
10-Mini coffee creamers from McDonald's
1-MULTIPACK box of tampons. (Tampax brand)
1-New N'sync CD. (title: No Strings Attached)

In an attempt to be a sport about the whole thing, I went shopping. The sandwich sounds gross. Ick. And while I can buy the girly stuff under the premise that I'm married, I really had no excuse whatsoever for the N'sync CD. It was very embarrasing. The store clerk looked at me funny and what could I say? "Yeah I'm a fan."

I cheated on a couple of things. I didn't have a pickle (don't really care for them), so the sandwich turned out to be a peanut butter sandwich (not so yucky). And I never went to McDonald's for the creamers.

I got instructions to go to Vasona Park in Los Gatos, and find this tree with the big red X on it. There was a paper bag there with a walkie-talkie in it.

"Show us the peanut pickle sandwich."

So I lifted up the sandwich, then the nylons, and so on until they were satisfied. Then I left everything in the bag, which I left on the side of the road, and after I left they told me the truck was at Starbucks, where it should have been a week and a half ago.

They have yet to fess up to their identities, though there are some obvious suspects.

I was hoping the Old World installer would be ready today (oops, it's past midnight... yesterday), but now we found a system bug that happened to show up in the snapshot of the kernel I chose to build which only comes up on older machines. It ends up making the console rather unusable, and since that's the primary UI for Darwin at the moment, we're going to fix that first. I already have the fix tested, so we'll start working on getting the new installer created tomorrow. It takes a while to master these things, though (it's still pretty manual), and then we have to test it, so I can't promise it in time for the weekend, unfortunately.

I've gotten a lot of supportive email lately, which I'm grateful for. My apologies if I don't respond to all of them, but it's hard to keep up with mail sometimes.

My thanks go to Ernie for making the whole open source thing at Apple fly with management; Klee for writing the tools that make the builds go and for making all of the 0.1 (internal test), 0.2, and 0.3 releases happen last year; Ron and Brett for dealing with all the management hooey; and Joyce for being the one lawyer who gets it. It's been a long year for Darwin, and these folks really dove in and made it happen. Also Justin for covering the hard questions on the mailing lists, as well the rest of the Core OS team members who stepped in when I needed help. It's a great bunch of people I work with every day.

Darwin 1.0 launched today, where today is the 5th, but I guess editing an entry changes the date on it. Oops.

So after I started this diary, the press latched onto it as "news" and things got a little nutty. The Intel port is something I've promising for about a year now, so I was kind of surprised that everyone got all excited about it.

Once again- I compiled for Intel, but I did that on a PowerPC Mac. I have not yet booted Darwin on Intel, and I'd be quite surprised if Darwin 1.0 worked unmodified on Intel. There is still work to be done there.

By the way, it should be noted that I think it's a dissapointing that this "Intel thing" was made into a big deal, and yet nobody seemed to care that my truck was kidnapped. The news never mentions the important stuff.

Anyway... Last week, I updated Apache in Darwin to 1.3.12, a long overdue update, and I updated Perl to 5.6.0. I also ported automake, because it was really easy. Each sqeaked their way into the 1.0 release. The Perl update is a big milestone for me. Many Darwin projects come from 3rd parties, and I've managed to get our diffs up to the upstream guys for most of these, but Perl has been a difficult one. The skanky-nasty Configure script Perl uses was the biggest obstacle. But after 2 years, it's finally done, and 5.6.0 support Mac OS X Server fully, and has my current patches for Mac OS X, so we're finnally sync'ed up. I have to thank Jarkko for his help (and prodding) in making this happen.

We've started to get some more patches for Darwin lately. I hope this is a sign that activity will be picking up. Detlev Droege contributed a patch to automount which let it read flat-file mount maps, which it did in OpenStep, but broke way back in Rhapsody. We were happy to see this get fixed, since it's one of those "we'll get around to it someday" bugs that's been around for a long time.

The Darwin 1.0 image turned out to be a lot bigger than expected, around 220 MB. The 0.3 image was around 60 MB. We have a lot more components in 1.0, and the fact that all binaries have code for two architectures now are reasons for it.

The www.publicsource.apple.com web server has been quite busy the past couple of days, and apparently we've been sucking up a lot of Apple's network bandwidth, which isn't so nice. I've got the server set to a max of 256 clients, and during the day, it's pinned at 256 connections. The cool thing is that this is being served up by just one machine, a B&W G3 running Mac OS X Server 1.0 plus an updated kernel. We're marketing Mac OS X Server as a low-to-medium-end web server, but this machine is holding up really well under a pretty heavy load, and it's moving a lot of data out pretty quickly. Uptime is 98 days. This may not impress the FreeBSD user, but for a first attempt at a server OS from our group, I'm pretty jazzed about it.

We ended up needing two Darwin installers. One is the same as the Darwin 0.3 net download and requires that you wipe out a whole disk (the 0.3 CD installer used yet another mechanism which let you install onto a partition). This one is required for what we call "Old World" machines, which turns out to be everything Beige. On "New World" machines (iMac, colorful PowerMac, newer PowerBooks, iBooks) we can get away with a single-partition install. Unfortunately, we found a bug in the Old World installer, so only the New World image was available for the launch.

Sunday I flew with my friends Pat (my pilot "big buddy") and Dave (mentioned previously) to South Lake Tahoe. That's a pretty cool trip, my first over scary looking mountains. We watched Pat lose $40 at a casino and then we zipped on back.

I have this radio-controlled Jeep which I drive around Apple. Each day I drive it to the coffee shop in building One and it hauls the coffee in its bed back to building Two for me. It's a ritual. It's The Way My Coffee Gets Around.

Well, after the Sunday trip I went to church (I go in an evening service) and I'm pretty late because of the Tahoe trip so the service is over and people are just hanging out and I decide to bust out with the Jeep, which was in my trunk because I was supposed to be going on a retreat with this very group over the weekend, and I was going to bring the truck, but I'm kinda dumb and I missed the whole thing, instead spending the weekend hacking on Darwin and flying to Tahoe, both of which you know about by now.

Anyway, I have the Jeep and the batteries fizz out, so I put it on a table by the exit and go chat with some people. On my way out, there is no Jeep on the table. Well, it must be in the Happy Lost and Found Place, so I'm looking around, and I get some people to look and we notice a paper plate on the formerly Jeep-posessing table. It has a note on it which basically said that for $2.5M I could pick it up at 5PM Wed.

Now... OK, so someone kidnapped my truck, at church, no less. And they want $2.5M. So far, I'm OK with that. After all, it is a nice truck, and I should have been keeping an eye on it. Sure, it's church, but hey, we welcome all sorts at our church, and I guess one of the sorts liked my truck. Fine.

But for crying out loud, Starbucks?!?!?!? There's one of those every five feet between here and Boston, and I have to be at the right one on Wednessday? Sheesh, talk about mean. And 5PM to boot. That's rush hour. I go to work at about 11AM and leave pretty late or pretty early depending on several factors, but the point is I never drive during rush hour. I hate rush hour, and I only live a few miles from work. The church is in Los Gatos, and that's a bit more of a trek, and I have to assume the Starbucks in question is near the church, or I don't stand a chance.

That was Sunday.

So yesterday, (Wed) I'm at Starbucks, but I'm half an hour late, and my Jeep and it's captor were not to be found. Once again, perhaps my fault. After all, the note said 5PM not 5:30. But I think I deserved a little slack for the rush hour thing.

I guess I'll find out more on Sunday.

Well... another world-build, this time with the kernel that's checked into the Darwin repository, finished successfully. The only glitches were in the AppleTalk library, and (as a result) in the AppleTalk commands project. The new kernel headers required a small change in the library, and updating to the latest version fixed that right up. I had expected to see more problems, since when you get near the end of a project, that's when everything is supposed to blow up, but it's looking pretty good.

So it looks like the full OS builds are going quite well, and all that's left is making sure it all works OK, which I will start working on tomorrow.

I posted on darwin-development that I started this here diary to keep people up-to-date on my progress (I got this idea from Greg at ApacheCon, by the way; good call, dude) and my good friend Dan (from my last gig at Disney Online) sent me a note saying that it seems to have made a splash on Slashdot, where much meaningful commentary on the future of Apple and Darwin is underway.

Today another big milestone has come up. I imported the sources to the kernel into the Darwin CVS repository today, which means that at this point all of the sources needed to build Darwin are available externally for the first time.

This kernel is slightly newer than the one I've been testing builds with, so I am starting yet another complete world-build to make sure that it builds and everything else build against it (it installs a mondo portion of the system headers). I will probably need to do one or two more world-builds after this one before one completes without a single build failure. In the meantime, I can begin installing and testing the system to make sure that the results are good.

Wednessday I reached an important milestone in getting Darwin 1.0 released. For the first time, I managed to build all of the components together. The build system we use for Darwin (which was written by Klee Dienes and uses the Debian packaging tools) seeds a build root for each project with the software required to build that project, and it gets the seed packages from the last build. So this milestone is a very important one because everything built using Darwin 1.0-built packages, which means we're also testing a large portion of the software as we go along.

It should be noted that I am building all components fat (for both PowerPC and Intel) by default. The only component that is build for PowerPC only is emacs. This is because it wants to run and dump core as part of the build, so it needs to build natively and can't be cross-built. Once we have an Intel system running I can figure out how to build emacs natively on each platform and merge the binaries so it is fat as well.

Apparently a lot of people are under the impression that Apple isn't going to help out with reviving the Intel port of Darwin. This is false.

Getting everything built fat is a big step, but a lot of work remains. The next thing is to get installation bootstrapped so we can get Darwin onto an Intel system, and then to get the kernel running, since we haven't tested the new kernel on Intel yet, and there is limited driver support for Intel PC devices. I have a high degree of confidence that most of the user-space software will work without problems, particularly since a majority of it comes from the BSD world where Intel is the primary platform, but also because we've seen it work before in Rhapsody.

Also last week: David Zarzycki, who's been an intern at Apple since the Stone Age, most recently working in the Core OS Networking group on IPSec and IPv6, is now a full-timer at Apple. His new job will be to support Darwin, which means that I won't be doing this mostly alone any more. Dave will be working on some developer infrastructure that we've been wishing we had, and in general just making things happen that need to happen. This is great news, and I'm quite happy to have Dave on board. Some of you may have met Dave at MacHack, FreeBSDCon, or this month's ApacheCon.

Some background to kick off this diary:

I've been a minor contributor to open source projects for a while as an undergraduate at MIT and afterwards. More recently, when I became part of the Core OS group at Apple and began work on what is now to become Mac OS X, I became a lot more active and started putting significant work into several projects, first by porting to Apple's new platform, and then by addressing the various issue that came us in our usage on the software. In particular, I became a NetBSD developer and an active contributor Apache soon into that process.

A lot has happened since I started at Apple. In particular, I was managing to get more and more of the Core OS source put onto the developer CD and eventually we decided as a company that this open source thing has some promise, and we committed to open sourcing the entire Core OS, which we now call Darwin.

We released the Darwin code one year ago, and that was followed by our Streaming Server technology and a network toolkit for game developers called OpenPlay.

One year later, we're about to roll out Darwin 1.0, the first version based on the new kernel (based on Mach 3.0 and some of FreeBSD 3.2) which includes our new C++ driver toolkit (IOKit) and a bunch of other new goodies. This is the release our developers have been waiting for, since it includes the CoreOS that we are building the final Mac OS X product on. Though Darwin's role in the world is as the guts of Mac OS X, it is shaping up to be a viable BSD platform on its own. It take advantage of some of the great work underway in the NetBSD, FreeBSD, and GNU/Linux communities, as well as several of the neat things we're working on at Apple.

On other fronts: I am now a member of the Apache Software Foundation, which is a wonderful group of people; I'm a FreeBSD developer as well as NetBSD; I'm a developer on the MIT Kerberos 5 distribution, and I've contributed code to a pile of other projects, including the GNU suite, Perl, and Sendmail.