Older blog entries for wsanchez (starting at number 25)

James Duncan Davidson did come by Apple to repeat his talk from ApacheCon, and it was, as before, and excellent talk followed by some lively Q&A. It was great to have someone else with a similar perspective on open source come and share ideas. Them, of course, James had to tell us about what he lives about Mac OS X and what doesn't work for him, which was immediately followed by some of the authors of the offending software showing him that he can, indeed, do what we wants with our tools. So that went well, too. :)

I've spent a bunch of time recently trying to clean up some BSD compatibility issues in Darwin. Specifically, there are a few BSD-ish libraries which we install as frameworks.

For those of you that don't know a framework is a directory bundle which contains a library, it's headers, and any associated resources such as images and documentation. The cool thing about frameworks is that everything associated with it is in one place; it's all in this bundle which can easily be treaed as a single object. This has some nice advantages over installing in a typical BSD layout: you put the library in /usr/lib, the docs in /usr/share/man, the headers in /usr/include; if you have image or text resources, those go in /usr/share, and so on. The BSD system has advantgaes as well, but it's more geared at managing the operating system, rather than at managing components.

Anyway, the upshot is that we've been mangling libraries into frameworks because frameworks are what we prefer in Darwin, but of course then you end up needing to build clients in a slightly different way when building on Darwin, which creates an annoying portability problem. So the new plan for now is that new stuff should be frameworks, but let's not monkey around with BSD/Unix/POSIX style software so that porting code is easier. We can't rule the world all at once, after all.

So that works is all in CVS now. Affected projects include libSystem (!), zlib, tcl, and kerberos. I still need to do OpenSSL, when I return from vacation.

Speaking of which, I'll be taking off for vacation tonight until the start of MacWorld Expo SF. If you go to the expo, don't forget to swing by the Mac OS X booth and see what's up. In the meantime, happy holidays, all.

Today, I'm back in California, having arrived yesterday from my ApacheCon trip.

But first, two weeks ago was BSDCon. I met up with a bunch of people there and synched up some ideas we've been kicking around for a while.

I mentioned before (18 Jul 2000) that I had been talking to Jordan about sharing mode code in real time between Darwin and FreeBSD. At the O'Reily Conference, I sat down with Jordan and Charles Hannum (from NetBSD) to talk about actually sharing repositories for code like /bin/cp and /bin/ls, which we can probably agree on a common implementation. The general consensus was that this is a good thing, probably doable with some infrastructure work and a bit of political ironing, but none of us has had the time to really dig into it.

So then comes Chris Coleman and the Open Packages idea, which is a similar idea, except for the ports collections, rather than the base system software. This is a great thing, because there is already some good technology in place which can be tweaked to work accross BSDs (and other platforms as well), and the ports are already pretty neutral ground, so consensus is generally easier (though not trivial) to accomplish. So I'm very excited about this, because success here might be something we can extend further down into the system software, which for me is the jackpot. (From the Darwin point of view, most of the BSD subsystem is already a "port," since we're already drawing the code from NetBSD and FreeBSD; if we can find a way to share code uniformly accross our systems, that's just too cool.)

To make things even better, Keith Bostic gave us a great keynote at the conference, and one of his questions to the community was "Why are we sharing the code to cp?", which got more people thinking about it as well.

Last week, I was in London for ApacheCon. We started with a Hackathon, where the ASF members and some guests got together to work our some issues while we have everyone in one place. We had about 30 people sharing a 56K net connection, which was rather interesting. Mostly it was a good chance to meet everyone before the conference and hack out some code with all of the relevant experts nearby. I spent a lot of time trying to wrap my brain around libtool, and thanks to some help from Sander (from Covalent) and a kick start from Sascha, I got libtool working well enough that PHP will build on Mac OS X correctly. And, as a bonus, it even works. Rasmus hooked me up with a PHP CVS account, so I'll probably be committing the fixes this week, and I'll need to send a new patch to the libtool folks.

ApacheCon itself went very well. Duncan gave a fabulous talk about how Sun came to open source Tomcat via Apache, and the hurdles they had to get past to make it work, which I'm going to try to get him to repeat at Apple some time. It's a great success story for a big company starting an open source project well. Douglas Adams wrapped up the conference with a fun talk about the unpredicatable evolution of computing. For such a young conference (this was #3), it's going very quickly, and it's packed with good talks and all the right people there.

While we were there, I got to check out London with some of the ASF guys. Roy and Greg and I were staying in this small Bed-and-Breakfast type hotel, and we played tourist all over the city. I have to say I really liked it. The weather was pretty reasonable for walking about. The only thing that I didn't like was the itty bitty portions you get at restaurants (it's like I was on a diet for a week).

I've been getting poked at by a few people because I haven't updated my diary in a while.

One of the big things that has happened in the past month is that we shipped Mac OS X Public Beta, which is the first public release of the software I've been working on for the past three years. The reception has so far been very positive, and I'm pretty happy about the results. We still have a long way to go, but looking back, we've actually gotten a lot done.

One reason I've not posted here in a while is that I've been working on an internal project lately, to help with deploying Mac OS X at Apple so that more of the company is using it now that it's fairly stable. Unfortunately, it's not all that interesting to talk about.

One great thing is that the Darwin community is really picking up some steam. There has been a fair bit of code pouring in this month. Luke Howard is cranking on adding PAM support, which he started working on by back-porting the current linker to Rhapsody DR2, as all he had was a PC at the time. Fortunately, he now has a shiny new G4 do his work on. Check out his diary for some status. Rob Braun has been doing a bit of a code audit in parts of the system and has committed a large pile of fixes to clean up things from possible buffer overruns to nicer syslog output. Dave Z. is working on getting a Darwin 1.2 release out the door to coorespond with Public Beta. Rob's also started working on getting the FreeBSD libc I imported, which is the critical and most difficult software sync we have to do in order to get the rest of the system software updated. There has also been some work on creating a Darwin logo; I'm a big fan of Hexley.

Tonight I'll be driving to Monterey for BSDCon 2000, and Friday I'll be flying to London for ApacheCon 2000 Europe, so it'll be a good couple of weeks for conferences. I'll be repeating my USENIX talk at BSDCon; and I figure I'll get the PHP4 port finished up, which I've been putting off, as well as Apache 2.0 while I'm at ApacheCon.

Chris Coleman from Daemon News has been working for some time on getting the various BSD distributions to collaborate on a shared ports collection, so that we can all benefit from each others' work. This is a wonderful goal to strive for, so when he called asking for some support from Apple, we agreed.

Note that Darwin is still at a stage where we are trying to get the system itself built and released in a timely fashion, which puts concerns about a ports system somewhat out in the future, but certainly it's going to be important once we're rolling along, and we would love to be able to take advantage of the already existing infrastructure pioneered by FreeBSD and expanded on by NetBSD. So while it might take some time before we're actively using and contributing to the new unified ports collection, David and I are both on the mailing list and we'll try to make sure that we'll be ready to jump in as soon as we're ready, so that Darwin and Mac OS X users can all participate.

OpenSSH somehow managed to sneak itself into the Mac OS X build train (wonder how that happened...); expect to see ssh support built into Mac OS X Public Beta. A tip of the hat to the OpenBSD folks for getting us an excellent ssh implementation under a free license, and to the Unix porting team that made the port to Darwin rather straightforward. Going forward, I want to start deprecating the use of telnet, ftp, (especially) rsh and other clear protocols in Darwin, replacing them with ssh, rsync-over-ssh, and so on.

Last week, I went to my first Burning Man festival. Now that's an experience. The desert environment and weather was at times quite brutal, but fortunately the attending citizens are overwhelmingly supporting of each other, and there was much borrowing and loaning of gear and supplies with our neighbors, then off to explore the city and much fun. I went for pretty much the whole week (took off Thursday and Friday to pick up some more people), which was a good thing. Early in the week, when the population is smaller and everything is getting built up, it was much mellower. Apparently while I was back in the Bay Area, the winds torn down some of the cooler camps, so had I only come at the end, I'd not have gotten to see them at all. Plus the weather in the latter half of the week was pretty cold, even during the day.

I had hoped to fly in, but my flight school didn't like the idea of dust all over the inside of their plane so much. I saw about 20 planes there by the end of the week, though, so it seems reasonable... I'll try harder next year. We camped next to the Black Light Light Brite camp, which had a giant Light Brite made of wood and lit up with UV lamps. It was excellent, and the guys camped there the most helpful bunch we could have camped next to. I had a blast, and next year I'll be more prepared.

Earlier this week, I was at LinuxWorld, on the BSD BoF panel. There were plenty of questions, but it was rather uneventful overall.

The BSD code merge has been stagnating while I've been distracted by some other things. I need to get back to working on the libraries.

Some good news: OpenSSH finally works, thanks to a good catch from Melissa E. O'Neil.

I've been putting some work into GNU libtool in order to get PHP4 building properly on OS X. I've got a bunch of good changes ready in libtool, and a few in PHP, which got it to the point where it is compiling and linking without errors, but we're not linking in all of the files that get built, so you get a runtime linker error when you try to use it in Apache. Not sure whether this is a PHP makefiles thing or a libtool thing. Libtool's also not passing down linker flags like I expect, so that'll need some cleanup as well. I'm new to libtool, so I need to learn a few more things before I can move further. After php4 will come mod_perl, which I'm sure will have its own fun quirks.

There has been some very lively discussion on the Darwin development list about SystemStarter, which is a program I wrote to bootrap Mac OS X (it will be a replacement for /etc/rc). We're working out some issues about adding shutdown sequence support, some ideas about incremental startup, and parallelizing the startup sequence.

We finally finished renaming the Darwin servers so, for example instead of www.publicsource.apple.com, we're using www.opensource.apple.com, which is much easier for people to deal with (the old names still work, naturally).

I've been working on merging some FreeBSD code in the past couple of weeks. First, I pulled down FreeBSD's make code and /usr/share/mk suite, merged that into bsdmake in Darwin, and then tossed PB makefiles and got it building with its own makefiles. So now bsdmake builds with itself, so we know that it works for building BSD commands in Darwin. The goal here is to facilitate building BSD libraries without having to rewrite the makefiles, as I have been doing with the BSD commands. Munging the makefiles makes updating the code more troublesome, so I'd like to avoid that going forward.

I got libtelnet ported from FreeBSD as well, which was a good starting point, since it only has two clients in Darwin and it's static. Libc is probaby going to have to be next, which won't be as easy. Since it's going into the System framework, it can also build static, but making sure that existing programs don't all fall over will be difficult. Also, the fact that BSD keeps library headers separate from code (it's in include/, instead of lib/) makes it harder to keep things organized. We'll need some additional bsdmake/mk tweaks to get things going, and Chuck Swiger has been working on that some, which I'll take a look at next week.

I also talked to Jordan recently about giving people better access to the Darwin code, since right now you need to register to use CVS, even for the non-APSL parts of the repository. So I've been experimenting with a way to make the non-APSL parts visible anonymously via a side repository. We're also kicking around ideas for how to share code in "real time" rather than merging back and forth.

This week I'm at the O'Reilly Open Source Conference in Monterey. Yesterday I went to Prof. McKusick's tutorial on the BSD kernel, which was great, since I work almost exclusively outside of the kernel, and getting a better picture of what's inside is useful stuff.

Tonight, I'll be at the MacPerl BoF, so I can get a feel for what the MacPerl guys would like to bring forward to Mac OS X. The Apache beerfest is also tonight, but I'll probably miss it due to the BoF. Thursday I'm be on a panel on the Future of BSD, representing Darwin/Mac OS X.

I've gotten a lot of mail and feedback about my USENIX paper, mostly positive, some occasionally critical. Several people were hoping to find out different things, such as how administration of the BSD subsystem will be presented to the user. There are certainly many other interesting topics in the Mac OS X space and the relationship of the Mac OS user experience and the BSD underpinnings. Hopefully other people will find time to write papers on these various topics. I'd love to expand on the material in the paper I did write, and write up some new stuff altogether, but that takes a fair bit of time and effort. For the time being, I need to get back to other things.

Some in the Slashdot crowd apparently think I'm an idiot because I didn't spellcheck the paper. Actually, I did, but making lots of last-minute changes didn't help, I guess. Certainly I'm happy to correct such things, if for no other reason than my professors some college might read some of this stuff, but the goal was to communicate some issues we ran into, not to pass a writing test. My target audience seems to understand this, so I'll not lose sleep over it. My apologies for the typos and bad grammar in my diary; I know there are plenty of examples here as well.

Also some people think I'm being too generous to the original Mac development team. Now I don't program for Classic Mac OS--I never have--and that's largely because I find the APIs too bulky, I "grew up" in a POSIX world, and I don't like some of the fundamentals of the system. Those things are the reason we have Mac OS X in development, and why I have a job at Apple. That said, in talking to some of the people that really understand why things are the way they are, I did develop a lot more respect for the decisions that were made, and I do think that they were quite reasonable given the state of computing at the time. My friend Dan B. (as opposed to Dan R. who needs to turn his modem off of sometime so I can call him) pointed out that Amiga did manage to made preemption in the same timeframe as the Mac launch. That's certainly true, though I don't know how long each product was in development. But cooperative tasks could in theory be more efficient than preemptive tasks, and while the practical value of preemption seems obvious now, it wasn't necessarily a proven thing then. Memory was really the big issue, and the even Amiga übercomputer didn't protect memory.

One of the key points I was trying to make is that the priority set between the Unix and Mac worlds were very different. One a system where a one-to-one mapping between computers and users exists simply has a different notion of what "security" is than a system where many users share a computer, particularly if they do so simultaneously. it's not the case that security isn't needed in both systems, but the requirements are quite different.

Anyway, I firmly believe that the Mac team has nothing at all to apologize for, given their accomplishments.

I presented my paper on "The Challenges of Integrating the Unix and Mac OS Environments" at USENIX on Thursday morning. The session went really well. It was well-attended, and I fielded about half an hour of questions afterward. The general feeling I got at the conference was that many people in the Unix community are excited about the potential of Mac OS X.

Wednesday night was the "Night of the Living BSDs"; there were back-to-back Birds-of-a-Feather sessions for NetBSD, OpenBSD, FreeBSD, and BSDi, followed by a BSD coctail party. I met up with a bunch up people there, and it was a fun night. Jordan gave Mac OS X a nice plug at the beginning of the FreeBSD BoF.

Right after my talk, I flew to Detroit for Mac Hack, which I'll at now. This year's Mac Hack is packed with lots of sessions, and there is a lot of Mac OS X hacking this year. I'm taking the DropScript demo I whipped up for WWDC and wiring up a pack/unpack tool pair called "ShoveIt Deluxe". Pretty wimpy, really, but I only had one day instead of two, and I wanted to go to some sessions and see what everyone else is doing this year. I also ran into someone who was looking to hack up SystemStarter to display marching icons at the bottom of the screen as done in Classic Mac OS. It turns out he was asking about it nearby, and I overheard him mention some problems he was having (we don't public all of the API he needs yet), so I let him know I wrote SystemStarter and helped him out with it. I love Mac Hack; this stuff is just fun.

Josh and David and I spent some time in the past couple of weeks rewriting the hack we did last year, "Shagadellic Telly". Josh and Dave bough a "WinTV" TV tuner card and figure we could fix the "Win" bug in the name by writing a Mac OS X driver. We have a much cleaner driver this time. I wrote the Cocoa application (the Telly) which controls the driver and tells it where to blast onto the screen. This version, though still incomplete, works rather well, and has a bunch of neat features. The driver and app are in the "AppleBt8xx" driver project in Darwin, if you're ihterested in playing with it.

Last week was mostly boring. I spent a lot of time cleaning up random things now that DP4 is out. One was to get /etc/rpcs, /etc/services, and /etc/protocols matches up with the current IANA assignments. Should have been easy, but for one, IANA doesn't provide BSD format versions of the assignments (or I'm a bozo and didn't look hard enough), so I had to munge the files they do have, and then I had to get all of that into NetInfo, which was easier given Mark's new nicl tool in DP4 and the script I wrote for parsing the flat files and sending that to nicl, but there's a bug in my script, so the services file just loads forever or something. Anyway, the IANA file is pretty mondo, so I'm a little worried that this is a bit of overkill.

I started that while poking around in the FreeBSD /etc sources, which I'm doing in preparation for the Big FreeBSD Library Merge (TM) which I hope will start real soon. The goal there is to finally sync up Darwin's BSD libraries to more current code, and set it up so we have an easy way to keep that up.

I met Rohit Khare just recently. Ever since ApacheCon, I've noticed that a sizeable number of the people I know know Rohit, and while we probably crossed paths a while ago, I didn't. So now that's fixed, and I feel much better now.

Still recovering from WWDC. Lots of fun, but rather exhausting.

The BSD session at WWDC on Friday was a big success. Even though it was a late Friday session, and many people had already gone home, the room was packed in. I introduced the audience to BSD and what it's role in Mac OS X is, and I gave people an update on what's changed in DP4 since DP3. We have a Unified Buffer Cache, which gets rid of the separation of caches between open files and mapped files. We have POSIX semaphores and shared memory. We added OpenSSL into the release. We are also working on including a complete Kerberos library (initial work already checked into the Darwin CVS tree) and updating Tcl.

At one point, I was talking about the BSD libraries which comprise the system framework, and when I got to curses I explained that curses is the "Cocoa of the command line." Apparently that's a lot funnier than I had expected, because the whole room busted up laughing.

Jordan then came on stage and gave everyone an intro to FreeBSD, how it's organized, resources, goals, etc., and in the Q&A answered some questions from developers.

My little demo was a big hit. The demo was a small program called DropScript I wrote using Cocoa which basically just calls a shell script in its application bundle, passing it as arguments any files it was asked to open. So you can drag documents onto the app, and it will run the script on those files. The script that is built in accepts a single file which should be a shell script; the script clones DropScript, replaces the build-in script with the new script, and renames the new app with the script name. So you can write a one-line shell script called GZip.sh:

#!/bin/sh
gzip -9 "$*";

and drag that script onto DropScript. DropScript will then create a new app called DropGZip which will compress any files dropped onto it. All you need to write is the script code, and Cocoa will handle the AppleEvents and so on. One of the cool features of MacPerl in Mac OS 9 is that you can make drop apps from perl scripts. Now you can do the same on Mac OS X using the unmodified Perl 5.6 for Unix.