Older blog entries for wsanchez (starting at number 17)

Last week was mostly boring. I spent a lot of time cleaning up random things now that DP4 is out. One was to get /etc/rpcs, /etc/services, and /etc/protocols matches up with the current IANA assignments. Should have been easy, but for one, IANA doesn't provide BSD format versions of the assignments (or I'm a bozo and didn't look hard enough), so I had to munge the files they do have, and then I had to get all of that into NetInfo, which was easier given Mark's new nicl tool in DP4 and the script I wrote for parsing the flat files and sending that to nicl, but there's a bug in my script, so the services file just loads forever or something. Anyway, the IANA file is pretty mondo, so I'm a little worried that this is a bit of overkill.

I started that while poking around in the FreeBSD /etc sources, which I'm doing in preparation for the Big FreeBSD Library Merge (TM) which I hope will start real soon. The goal there is to finally sync up Darwin's BSD libraries to more current code, and set it up so we have an easy way to keep that up.

I met Rohit Khare just recently. Ever since ApacheCon, I've noticed that a sizeable number of the people I know know Rohit, and while we probably crossed paths a while ago, I didn't. So now that's fixed, and I feel much better now.

Still recovering from WWDC. Lots of fun, but rather exhausting.

The BSD session at WWDC on Friday was a big success. Even though it was a late Friday session, and many people had already gone home, the room was packed in. I introduced the audience to BSD and what it's role in Mac OS X is, and I gave people an update on what's changed in DP4 since DP3. We have a Unified Buffer Cache, which gets rid of the separation of caches between open files and mapped files. We have POSIX semaphores and shared memory. We added OpenSSL into the release. We are also working on including a complete Kerberos library (initial work already checked into the Darwin CVS tree) and updating Tcl.

At one point, I was talking about the BSD libraries which comprise the system framework, and when I got to curses I explained that curses is the "Cocoa of the command line." Apparently that's a lot funnier than I had expected, because the whole room busted up laughing.

Jordan then came on stage and gave everyone an intro to FreeBSD, how it's organized, resources, goals, etc., and in the Q&A answered some questions from developers.

My little demo was a big hit. The demo was a small program called DropScript I wrote using Cocoa which basically just calls a shell script in its application bundle, passing it as arguments any files it was asked to open. So you can drag documents onto the app, and it will run the script on those files. The script that is built in accepts a single file which should be a shell script; the script clones DropScript, replaces the build-in script with the new script, and renames the new app with the script name. So you can write a one-line shell script called GZip.sh:

#!/bin/sh
gzip -9 "$*";

and drag that script onto DropScript. DropScript will then create a new app called DropGZip which will compress any files dropped onto it. All you need to write is the script code, and Cocoa will handle the AppleEvents and so on. One of the cool features of MacPerl in Mac OS 9 is that you can make drop apps from perl scripts. Now you can do the same on Mac OS X using the unmodified Perl 5.6 for Unix.

Well, the Open Source session at WWDC was today, and it went really well. Most of the Darwin Developers were there, which was great. David was the man today. Last week, he worked on putting together some great demos for the Darwin part of the session, and he delivered.

The first demo was Darwin running on an Intel PC. This validates that the whole BSD stack, from kernel to login shell, is still ready-to-go on Intel. Details will follow next week when we are done with the conference, so the folks out there woring on the Intel bringup can get a jump.

The second demo was X11 on Darwin. John did a port of XFree 3 to Mac OS X Server using the Interceptor API a while ago. Dave sat down and got the XFree 4 working with the I/O Kit in Darwin 1.0. This is a good step to improving the usability of Darwin as a standalone BSD system. It also bodes well for the possiblity of a port to Quartz, so that X apps can run on a full Mac OS X install.

We also saw a demo of NetSprocket running as a client of OpenPlay on both Mac OS and Window, showing that we now have a working cross-platform NetSprocket implementation. OpenPlay development is rolling along nicely, and we are seeing some companies using it in their software now. Joe remarked that the name "OpenPlay" is a bit of a misnomer, since it is a general cross-platform network transport API, and isn't limited to gaming. Apparently, there is a medical imaging company looking at OpenPlay as their networking layer as well.

It was a great session, and it also covered HeaderDoc and the Streaming Server.

I'm be an attendee until Friday, when I'll be presenting at the BSD session. I talked to Andy Stone, who write some of the coolest Cocoa software imaginable. He's going to help me spiff up my demo for Friday.

Today was also the Apple Masters event, with Herbie Hancock, Gregory Hines, and Brian Adams performing. Much fun all around.

WWDC is almost here. We did some set-up at the San Jose Convention Center today for the "Open Source at Apple" session on Monday. Dave will have some cool stuff to demo Monday.

I wrote up a neat little demo for Friday's "BSD in Mac OS X" session. I'm really happy with this one, because it makes a great case for why having BSD commands around would be a good thing even in an environment that has no interactive command line available.

More info on both will be available after each session. In the meantime, we'll keep the folks going to the show in suspense.

I spent most of the week getting ready for WWDC. Other than that, I updated some 3rd-party code in Darwin: groff 1.15, patch 2.5.4, GNU make 3.79, GNU grep 2.4.2, GNU tar 1.13, and Kerberos 1.1.

Since the export control situation has improved, the kerberos source is available in the external Darwin repository. We'll also be shipping a Kerberos framework (shared library) now. Previously, we only released the base clients and the server, which statically linked in the the library. With the public framework in place, we can start looking at kerberizing more system services, and include support for encrypted network sessions, etc.

Also in that vein, I've been porting OpenSSH, which is also in the Darwin repository. It builds happily, but it also blows up due to a bus error I haven't had time to debug. I'm using egd in lieu of /dev/random in order to make progress, but we'll also need to get /dev/random working.

I'm in the middle of updatng Tcl, but got bogged down in Makefile hooey and then I got distracted. I get back to it after I get emacs, tcsh, and sendmail updated, which I think will bring most of the upstream goodies in Darwin back up to current versions.

I've been generally "unproductive" this week. That is, I've done a bunch of little stuff like making sure the Darwin servers are getting syncronized and backed up as expected, tweaking the web content a bit, etc. Nothing really exciting, so it feel like nothing useful is going on.

I'm also getting ready for Apple's World-Wide Developer's Conference, which is in two weeks. This is a big deal event. As with the past two years, this is the biggest and most important WWDC in Apple's history, because the past few years have seen some phenomenal progress for the Mac platform. This year will not dissapoint. It's all very exciting, and everyone is working hard to make sure we have good news for our developers.

I love this conference, because we get to meet with people who are all excited about all the neat stuff we're working on, and they want to know more about how to use it and so on. We also get our share of flammage when people don't hear what they want, but that's useful, too. I remember meeting Andy Stone two years ago at WWDC. He's always got something cool to show you, some app he's been hacking on with neat graphics and whatnot, and he's always all jazzed about some new thing he discovered in AppKit that he can do crazy things with. I love that stuff.

I'll be talking just briefly at the "Open Source at Apple" session on Monday, and then I'm be giving a talk about BSD in Mac OS X on Friday. I've asked Jordan Hubbard from the FreeBSD Project to join us Friday and give us a little update on what's exciting in FreeBSD. I still need to come up with a cool demo. You have to have a cool demo at WWDC, or your talk isn't any good, which is a little tricky when you have this Unix stuff with a command-line UI, which doesn't really make for a sexy demo. Last year, my demo bombed out completely because we couldn't get the whole A/V thing working with the machine I brought in that morning; really dissapointing.

It turns out the USENIX paper wasn't really due last week. I got tricked by some email I got from USENIX which didn't really apply to me. The good news is that I did get a lot of info from a bunch of people and I have some others to follow up with (after WWDC), so the confusion at least got me into gear.

I've spent the past few days working on my USENIX 2000 presentation. "Challenges of Integrating the Mac OS and Unix Environments." I thought at first that it would be a really short thing, like the paper I did for FREENIX last year, but there is a lot of wacky stuff we've done when you look back at it, and so there is a lot of good material to draw from. I decided to distill it down to filesystem and multi-user issues so that I could get a handle on it. See, I've procrastinated, and the paper is due to the conference committee tommorrow, so I didn't give myself much time. Anyway, it turned out to be a view from the Unix side of things, since that's my problem space, so I'm going to need to talk to the Carbon folks a bit for their take on it.

In the process of talking to some people about this, I regained some respect for Mac OS. It's really easy in hindsight to look at it today given the kinds of computers we have now, and say that there were some bad decisions made early on, but that's totally not the case. Mac OS is an impressive bit of work. The thing is that it was really designed to be a one-user-one-computer system, and that was a good call given the hardware they started with, and the whole point of this newish "personal computer" idea. And it worked really well for a good long time. When you consider that it ran in less memory than my Palm Pilot has... well, that's not bad at all.

It's also interesting to see how implementation details turn into design further down the road. When you look at Mac OS and Unix, which evolved quite independantly, and see how things progressed, you can see it happen in both cases, and its easy to take for granted that one solution is The Right Thing because it so clearly works for you, but there could be a completely different thing that is actually just as good, but you would never have done it that way, because your situation was different.

I hope to capture some of this stuff in the paper, which will be online after the conference, which is in June.

Andy Stone has been helping me with some of the stuff I need to get Gale working. Porting gc took a bit of poking around and we're playing with ADNS now. ADNS looks prett cool. Too bad it's GPL'ed, or I'd look at adding it to Darwin proper. We can use GPL'ed stuff sometimes, but for libraries, it's not a good idea; we don't have to provide an API to our developers that they might use and then realize they've tainted their programs with GPL requirements. Oops, I already complained about the GPL last time. I guess Gale's use of ADNS might explain why it's GPL'ed as well. Anyway, it is a neat library.

I'm checking out a new project called Gale. At MIT, we used this messaging system called Zephyr. Zephyr was cool because it was ubiquitous on campus. If you were looking for someone, there was always a good chance of fiding them online, or someone who knew where they are. You could subscribe and send to different categories of messages, some for given student groups, others for generic help (in any area) and discussion, others more specific, like one for NetBSD users. It's sort of like email and mailing lists, but real-time, a lot more dynamic, and less "formal" (if that makes sense). It was also tied into system services, so when the printer is done with your job, or your fileserver is going down for maintenance, you'd get a message about it.

Anyway, I still log in to MIT to use zephyr every day, and I wish I could do something similar with a broader group; say all of my friends, or the Darwin community, etc. I've looked at AIM and ICQ, but they stink at the group dynamic. They are OK for messaging specific people, but not for group discussions.

Unfortunately, I think Zephyr is pretty much dead. I mean MIT will probably use it for the foreseeable future because they've always used it, but the last Zephyr release (2.0.4) was in 1996. There are all sort of features that Zephyr could have, and there are patches around for some of them, but it really seems like MIT isn't doing anything with Zephyr any more, so I'm giving up on it. The need for a kerberos realm to authenticate message is also a bit of a pain. I do use kerberos, but it's not trivial to deploy and manage a realm, and it's not really the right thing for an open user base such as the Darwin community.

Gale is a new thing which aims to replace Zephyr (and has at Caltech). Not I hate to plug Caltech over MIT, given my academic roots, but there you have it. Gale does what zephyr does plus it fixes a bunch of things up, claims to be faster, leaner, and meaner, and looks like a good shot at picking up where zephyr left off.

The one drag I've noticed so far is that it's under the GPL instead of a free license, which tends to be a nuissance at best; but it's a common faux-pas, and one can't expect perfection.

I've also started porting OpenSSH to Darwin, since ssh is an important tool, and we all know the ssh1/ssh2 license fiasco. I was working with the OpenBSD code, but a couple of folks pointed out that now there is a cross-platform version that uses autoconf and so on, so I'm now using that. That's coming along more nicely, but it seems Darwin lacks /dev/random, though, so we need to fix that in order to make the OpenSSH build happy.

I'm taking a strong interest in a new version control system called inversion which is being designed from scratch. CVS seems to be the best thing available right now, but it really stinks in some important ways. Inversion is looking like The Right Thing is every way that bugged me about CVS, so I'm very excited about it. I don't know how much I'll be able to contribute to the project, but I hope I can do some useful work on it, since I think it's a very important problem to solve.

Today Darwin 1.0.2 rolled out.

1.0.2 is a minor update in terms of content. We added gdb and binutils. I didn't get enough time to work out some details with getting the new versions of these into CVS and I didn't want to do any hacks, so they didn't make 1.0.1, but now they're ready.

Also, while testing the new installer on Old World (Beige) machines, we found a recently-introduced SCSI driver bug which was causing a message to be printed to the console every second. Since the console is the only UI you get, this was very annoying. The bug had already been fixed; I had just grabbed a snapshot on the wrong day, it seems. So we patched xnu (the kernel) to fix this (a one-liner) bug. New World users who downloaded 1.0.1 won't be affected by it, so they needn't update their kernels.

For 1.0.1 we had two installers; one for New World machines, and another for Old World. (We found some installation problems with Old World so we didn't ship it on the launch date.) The reason we needed two installers is that New World machines have a new booting architecture which allowed us to restore Darwin into a single partition and boot that, but Old World machines lack this feature. We figured it was better to have two installers, since it would at least let people with newer hardware install into a partition without having to wipe a whole drive out.

It turns out that our crack team of boot-manglers at Apple are pretty clever and after much hackery and magic they found a way to get Old World machines to boot without a second partition for the booter, and they built a new version of ASR to install using the new scheme.

So Darwin 1.0.2 also features a spiffy new ASR which happily installs onto both Old World and New World machines.

All of this cost us a couple of days, but I think the result is well worth it. One installer fits all.

8 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!