Older blog entries for wsanchez (starting at number 20)

I've been working on merging some FreeBSD code in the past couple of weeks. First, I pulled down FreeBSD's make code and /usr/share/mk suite, merged that into bsdmake in Darwin, and then tossed PB makefiles and got it building with its own makefiles. So now bsdmake builds with itself, so we know that it works for building BSD commands in Darwin. The goal here is to facilitate building BSD libraries without having to rewrite the makefiles, as I have been doing with the BSD commands. Munging the makefiles makes updating the code more troublesome, so I'd like to avoid that going forward.

I got libtelnet ported from FreeBSD as well, which was a good starting point, since it only has two clients in Darwin and it's static. Libc is probaby going to have to be next, which won't be as easy. Since it's going into the System framework, it can also build static, but making sure that existing programs don't all fall over will be difficult. Also, the fact that BSD keeps library headers separate from code (it's in include/, instead of lib/) makes it harder to keep things organized. We'll need some additional bsdmake/mk tweaks to get things going, and Chuck Swiger has been working on that some, which I'll take a look at next week.

I also talked to Jordan recently about giving people better access to the Darwin code, since right now you need to register to use CVS, even for the non-APSL parts of the repository. So I've been experimenting with a way to make the non-APSL parts visible anonymously via a side repository. We're also kicking around ideas for how to share code in "real time" rather than merging back and forth.

This week I'm at the O'Reilly Open Source Conference in Monterey. Yesterday I went to Prof. McKusick's tutorial on the BSD kernel, which was great, since I work almost exclusively outside of the kernel, and getting a better picture of what's inside is useful stuff.

Tonight, I'll be at the MacPerl BoF, so I can get a feel for what the MacPerl guys would like to bring forward to Mac OS X. The Apache beerfest is also tonight, but I'll probably miss it due to the BoF. Thursday I'm be on a panel on the Future of BSD, representing Darwin/Mac OS X.

I've gotten a lot of mail and feedback about my USENIX paper, mostly positive, some occasionally critical. Several people were hoping to find out different things, such as how administration of the BSD subsystem will be presented to the user. There are certainly many other interesting topics in the Mac OS X space and the relationship of the Mac OS user experience and the BSD underpinnings. Hopefully other people will find time to write papers on these various topics. I'd love to expand on the material in the paper I did write, and write up some new stuff altogether, but that takes a fair bit of time and effort. For the time being, I need to get back to other things.

Some in the Slashdot crowd apparently think I'm an idiot because I didn't spellcheck the paper. Actually, I did, but making lots of last-minute changes didn't help, I guess. Certainly I'm happy to correct such things, if for no other reason than my professors some college might read some of this stuff, but the goal was to communicate some issues we ran into, not to pass a writing test. My target audience seems to understand this, so I'll not lose sleep over it. My apologies for the typos and bad grammar in my diary; I know there are plenty of examples here as well.

Also some people think I'm being too generous to the original Mac development team. Now I don't program for Classic Mac OS--I never have--and that's largely because I find the APIs too bulky, I "grew up" in a POSIX world, and I don't like some of the fundamentals of the system. Those things are the reason we have Mac OS X in development, and why I have a job at Apple. That said, in talking to some of the people that really understand why things are the way they are, I did develop a lot more respect for the decisions that were made, and I do think that they were quite reasonable given the state of computing at the time. My friend Dan B. (as opposed to Dan R. who needs to turn his modem off of sometime so I can call him) pointed out that Amiga did manage to made preemption in the same timeframe as the Mac launch. That's certainly true, though I don't know how long each product was in development. But cooperative tasks could in theory be more efficient than preemptive tasks, and while the practical value of preemption seems obvious now, it wasn't necessarily a proven thing then. Memory was really the big issue, and the even Amiga übercomputer didn't protect memory.

One of the key points I was trying to make is that the priority set between the Unix and Mac worlds were very different. One a system where a one-to-one mapping between computers and users exists simply has a different notion of what "security" is than a system where many users share a computer, particularly if they do so simultaneously. it's not the case that security isn't needed in both systems, but the requirements are quite different.

Anyway, I firmly believe that the Mac team has nothing at all to apologize for, given their accomplishments.

I presented my paper on "The Challenges of Integrating the Unix and Mac OS Environments" at USENIX on Thursday morning. The session went really well. It was well-attended, and I fielded about half an hour of questions afterward. The general feeling I got at the conference was that many people in the Unix community are excited about the potential of Mac OS X.

Wednesday night was the "Night of the Living BSDs"; there were back-to-back Birds-of-a-Feather sessions for NetBSD, OpenBSD, FreeBSD, and BSDi, followed by a BSD coctail party. I met up with a bunch up people there, and it was a fun night. Jordan gave Mac OS X a nice plug at the beginning of the FreeBSD BoF.

Right after my talk, I flew to Detroit for Mac Hack, which I'll at now. This year's Mac Hack is packed with lots of sessions, and there is a lot of Mac OS X hacking this year. I'm taking the DropScript demo I whipped up for WWDC and wiring up a pack/unpack tool pair called "ShoveIt Deluxe". Pretty wimpy, really, but I only had one day instead of two, and I wanted to go to some sessions and see what everyone else is doing this year. I also ran into someone who was looking to hack up SystemStarter to display marching icons at the bottom of the screen as done in Classic Mac OS. It turns out he was asking about it nearby, and I overheard him mention some problems he was having (we don't public all of the API he needs yet), so I let him know I wrote SystemStarter and helped him out with it. I love Mac Hack; this stuff is just fun.

Josh and David and I spent some time in the past couple of weeks rewriting the hack we did last year, "Shagadellic Telly". Josh and Dave bough a "WinTV" TV tuner card and figure we could fix the "Win" bug in the name by writing a Mac OS X driver. We have a much cleaner driver this time. I wrote the Cocoa application (the Telly) which controls the driver and tells it where to blast onto the screen. This version, though still incomplete, works rather well, and has a bunch of neat features. The driver and app are in the "AppleBt8xx" driver project in Darwin, if you're ihterested in playing with it.

Last week was mostly boring. I spent a lot of time cleaning up random things now that DP4 is out. One was to get /etc/rpcs, /etc/services, and /etc/protocols matches up with the current IANA assignments. Should have been easy, but for one, IANA doesn't provide BSD format versions of the assignments (or I'm a bozo and didn't look hard enough), so I had to munge the files they do have, and then I had to get all of that into NetInfo, which was easier given Mark's new nicl tool in DP4 and the script I wrote for parsing the flat files and sending that to nicl, but there's a bug in my script, so the services file just loads forever or something. Anyway, the IANA file is pretty mondo, so I'm a little worried that this is a bit of overkill.

I started that while poking around in the FreeBSD /etc sources, which I'm doing in preparation for the Big FreeBSD Library Merge (TM) which I hope will start real soon. The goal there is to finally sync up Darwin's BSD libraries to more current code, and set it up so we have an easy way to keep that up.

I met Rohit Khare just recently. Ever since ApacheCon, I've noticed that a sizeable number of the people I know know Rohit, and while we probably crossed paths a while ago, I didn't. So now that's fixed, and I feel much better now.

Still recovering from WWDC. Lots of fun, but rather exhausting.

The BSD session at WWDC on Friday was a big success. Even though it was a late Friday session, and many people had already gone home, the room was packed in. I introduced the audience to BSD and what it's role in Mac OS X is, and I gave people an update on what's changed in DP4 since DP3. We have a Unified Buffer Cache, which gets rid of the separation of caches between open files and mapped files. We have POSIX semaphores and shared memory. We added OpenSSL into the release. We are also working on including a complete Kerberos library (initial work already checked into the Darwin CVS tree) and updating Tcl.

At one point, I was talking about the BSD libraries which comprise the system framework, and when I got to curses I explained that curses is the "Cocoa of the command line." Apparently that's a lot funnier than I had expected, because the whole room busted up laughing.

Jordan then came on stage and gave everyone an intro to FreeBSD, how it's organized, resources, goals, etc., and in the Q&A answered some questions from developers.

My little demo was a big hit. The demo was a small program called DropScript I wrote using Cocoa which basically just calls a shell script in its application bundle, passing it as arguments any files it was asked to open. So you can drag documents onto the app, and it will run the script on those files. The script that is built in accepts a single file which should be a shell script; the script clones DropScript, replaces the build-in script with the new script, and renames the new app with the script name. So you can write a one-line shell script called GZip.sh:

#!/bin/sh
gzip -9 "$*";

and drag that script onto DropScript. DropScript will then create a new app called DropGZip which will compress any files dropped onto it. All you need to write is the script code, and Cocoa will handle the AppleEvents and so on. One of the cool features of MacPerl in Mac OS 9 is that you can make drop apps from perl scripts. Now you can do the same on Mac OS X using the unmodified Perl 5.6 for Unix.

Well, the Open Source session at WWDC was today, and it went really well. Most of the Darwin Developers were there, which was great. David was the man today. Last week, he worked on putting together some great demos for the Darwin part of the session, and he delivered.

The first demo was Darwin running on an Intel PC. This validates that the whole BSD stack, from kernel to login shell, is still ready-to-go on Intel. Details will follow next week when we are done with the conference, so the folks out there woring on the Intel bringup can get a jump.

The second demo was X11 on Darwin. John did a port of XFree 3 to Mac OS X Server using the Interceptor API a while ago. Dave sat down and got the XFree 4 working with the I/O Kit in Darwin 1.0. This is a good step to improving the usability of Darwin as a standalone BSD system. It also bodes well for the possiblity of a port to Quartz, so that X apps can run on a full Mac OS X install.

We also saw a demo of NetSprocket running as a client of OpenPlay on both Mac OS and Window, showing that we now have a working cross-platform NetSprocket implementation. OpenPlay development is rolling along nicely, and we are seeing some companies using it in their software now. Joe remarked that the name "OpenPlay" is a bit of a misnomer, since it is a general cross-platform network transport API, and isn't limited to gaming. Apparently, there is a medical imaging company looking at OpenPlay as their networking layer as well.

It was a great session, and it also covered HeaderDoc and the Streaming Server.

I'm be an attendee until Friday, when I'll be presenting at the BSD session. I talked to Andy Stone, who write some of the coolest Cocoa software imaginable. He's going to help me spiff up my demo for Friday.

Today was also the Apple Masters event, with Herbie Hancock, Gregory Hines, and Brian Adams performing. Much fun all around.

WWDC is almost here. We did some set-up at the San Jose Convention Center today for the "Open Source at Apple" session on Monday. Dave will have some cool stuff to demo Monday.

I wrote up a neat little demo for Friday's "BSD in Mac OS X" session. I'm really happy with this one, because it makes a great case for why having BSD commands around would be a good thing even in an environment that has no interactive command line available.

More info on both will be available after each session. In the meantime, we'll keep the folks going to the show in suspense.

I spent most of the week getting ready for WWDC. Other than that, I updated some 3rd-party code in Darwin: groff 1.15, patch 2.5.4, GNU make 3.79, GNU grep 2.4.2, GNU tar 1.13, and Kerberos 1.1.

Since the export control situation has improved, the kerberos source is available in the external Darwin repository. We'll also be shipping a Kerberos framework (shared library) now. Previously, we only released the base clients and the server, which statically linked in the the library. With the public framework in place, we can start looking at kerberizing more system services, and include support for encrypted network sessions, etc.

Also in that vein, I've been porting OpenSSH, which is also in the Darwin repository. It builds happily, but it also blows up due to a bus error I haven't had time to debug. I'm using egd in lieu of /dev/random in order to make progress, but we'll also need to get /dev/random working.

I'm in the middle of updatng Tcl, but got bogged down in Makefile hooey and then I got distracted. I get back to it after I get emacs, tcsh, and sendmail updated, which I think will bring most of the upstream goodies in Darwin back up to current versions.

I've been generally "unproductive" this week. That is, I've done a bunch of little stuff like making sure the Darwin servers are getting syncronized and backed up as expected, tweaking the web content a bit, etc. Nothing really exciting, so it feel like nothing useful is going on.

I'm also getting ready for Apple's World-Wide Developer's Conference, which is in two weeks. This is a big deal event. As with the past two years, this is the biggest and most important WWDC in Apple's history, because the past few years have seen some phenomenal progress for the Mac platform. This year will not dissapoint. It's all very exciting, and everyone is working hard to make sure we have good news for our developers.

I love this conference, because we get to meet with people who are all excited about all the neat stuff we're working on, and they want to know more about how to use it and so on. We also get our share of flammage when people don't hear what they want, but that's useful, too. I remember meeting Andy Stone two years ago at WWDC. He's always got something cool to show you, some app he's been hacking on with neat graphics and whatnot, and he's always all jazzed about some new thing he discovered in AppKit that he can do crazy things with. I love that stuff.

I'll be talking just briefly at the "Open Source at Apple" session on Monday, and then I'm be giving a talk about BSD in Mac OS X on Friday. I've asked Jordan Hubbard from the FreeBSD Project to join us Friday and give us a little update on what's exciting in FreeBSD. I still need to come up with a cool demo. You have to have a cool demo at WWDC, or your talk isn't any good, which is a little tricky when you have this Unix stuff with a command-line UI, which doesn't really make for a sexy demo. Last year, my demo bombed out completely because we couldn't get the whole A/V thing working with the machine I brought in that morning; really dissapointing.

It turns out the USENIX paper wasn't really due last week. I got tricked by some email I got from USENIX which didn't really apply to me. The good news is that I did get a lot of info from a bunch of people and I have some others to follow up with (after WWDC), so the confusion at least got me into gear.

I've spent the past few days working on my USENIX 2000 presentation. "Challenges of Integrating the Mac OS and Unix Environments." I thought at first that it would be a really short thing, like the paper I did for FREENIX last year, but there is a lot of wacky stuff we've done when you look back at it, and so there is a lot of good material to draw from. I decided to distill it down to filesystem and multi-user issues so that I could get a handle on it. See, I've procrastinated, and the paper is due to the conference committee tommorrow, so I didn't give myself much time. Anyway, it turned out to be a view from the Unix side of things, since that's my problem space, so I'm going to need to talk to the Carbon folks a bit for their take on it.

In the process of talking to some people about this, I regained some respect for Mac OS. It's really easy in hindsight to look at it today given the kinds of computers we have now, and say that there were some bad decisions made early on, but that's totally not the case. Mac OS is an impressive bit of work. The thing is that it was really designed to be a one-user-one-computer system, and that was a good call given the hardware they started with, and the whole point of this newish "personal computer" idea. And it worked really well for a good long time. When you consider that it ran in less memory than my Palm Pilot has... well, that's not bad at all.

It's also interesting to see how implementation details turn into design further down the road. When you look at Mac OS and Unix, which evolved quite independantly, and see how things progressed, you can see it happen in both cases, and its easy to take for granted that one solution is The Right Thing because it so clearly works for you, but there could be a completely different thing that is actually just as good, but you would never have done it that way, because your situation was different.

I hope to capture some of this stuff in the paper, which will be online after the conference, which is in June.

Andy Stone has been helping me with some of the stuff I need to get Gale working. Porting gc took a bit of poking around and we're playing with ADNS now. ADNS looks prett cool. Too bad it's GPL'ed, or I'd look at adding it to Darwin proper. We can use GPL'ed stuff sometimes, but for libraries, it's not a good idea; we don't have to provide an API to our developers that they might use and then realize they've tainted their programs with GPL requirements. Oops, I already complained about the GPL last time. I guess Gale's use of ADNS might explain why it's GPL'ed as well. Anyway, it is a neat library.

11 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!