[ Home | Articles | Account | People | Projects | FAQ ] 23 Sep 2004 (updated 24 Sep 2004 at 14:07 UTC) »
The sheer volume of bugs is just too staggering for one person to handle on their own, and so many just sit there forever simply because there aren't enough man hours to go around to cover every single issue reported. I'm sure this situation is common to all software out there, and probably all developers, wether they're working for a Linux vendor, or just an open source project. I think it's safe to state the following law about bug trackers: "The number of bugs open in a given project/vendor's bug tracker at any given point in time, will exceed the project/vendor's available man hour resources in order to investigate all of the issues".
I've thought about this over time, and like many, have had some weird utopian idea that "some day, the bug count will go down". Well, after the count hit 400 or so, and has steadily inclined since then, I realized that my utopian fantasy was just not very real-world realistic, so I gave up hope of ever seeing the total number of open bugs ever drop in any significant way, since it is just the nature of computer software to be imperfect, and something the size of XFree86/X.Org is just too massive to ever really experience a major decline in the number of overall bugs/problems. Since the number of Linux users steadily increases over time, it stands to reason that the number of people finding and reporting bugs will also increase over time, and so it seems natural that the bug tracker would have a net steady increase in open issues over time as well.
Several times when I've had some spare time, I did a massive "bug attack", where I tried to examine as many issues as possible and do some kind of triage, to try to find as many bogus issues, or ancient bugs as possible and determine if they were valid/invalid and try to close out as many issues as possible that seemed fixed now, invalid, or some other resolution. That helped somewhat, but it was realistically a losing battle due to time constraints and the sheer number of bugs.
A year or so later, we hired a second X developer, in order to be able to scale up the number of issues we could resolve in a given time frame. This turned out to be very helpful, as some of the issues reported in bugzilla can end up taking a great many days or even a week or two to investigate and track down the problem and fix it. He lessened the burden somewhat by taking over quite a number of these D-Day type of bugs, in particular on IA64 architecture which is a very strange beast which tends to find more obscure problems in the X code than you can shake a stick at. Having a second person working on X was a definite blessing.
The number of bugs continued to be more staggering than our finite man-hours available to investigate and fix, which is expected with something such as X. I doubt that *any* OS vendor out there has the resources to have enough X11 engineers to effectively handle all bugs reported at the rate they're reported, although we've done a fairly reasonable job considering all factors.
Fast forward a year and a half or so after that, and the bug count for X and related technology is almost tripled. A lot of these bugs are now old and stale... but which ones? It takes a lot of man hours just to read them all and try to find bugs to close out and triage. Not the best use of engineer time. So, they continue to pile up.
This year, we expanded our X Development team by an additional 3 members, and had one person switch out of X development into other areas of OS development, so we now have an X Development team of 4, which is really starting to take off the ground now. Having the larger team, allows much more work to get done, but also allows a lot more discussions and team collaboration on issues. It allows multiple viewpoints to work together, and I believe the net result of the team is greater than the sum of the individual parts.
We are now looking into ways where we can improve our team policies and procedures to "the next level", and we have a great many ideas that have come out of brainstorming sessions. Some of these we've implemented, some we're working the details out and experimenting with, and others are still in the idea stage. The future does look bright now though, as I can see the light at the end of the tunnel! ;o)
One of the things we've decided is of high priority, is to try to get a handle on bugzilla. We're currently trying to focus on developing some standard policies and procedures for handling X related bug reports, and to come up with stock polite/proactive responses to give to people for common situations (much like the GNOME bug squad uses). We believe this will help us to be more productive, and will also be greatly appreciated by users and customers. We've also decided to start doing "bug days", where we sit in IRC for a full day (or more) and just triage bugs, add comments, and try to make a solid concrete decision about each bug, rather than having so many bugs in "limbo" states or falling between the cracks into the bugrot zone.
There are about 500 or more bugs currently to go through and triage. I believe once we pass through these, we can probably cut off about 100-200 easily, and work our way through another 100 or so. A lot of others are issues that we probably wont ever have the resources to investigate, and should thus be reported upstream to X.Org or wherever. Personally, I believe that if we know 100% that we will never be able to schedule time for a particular issue, there's no sense in us leaving it open forever "until we get around to it", because quite frankly, that is not likely to happen. It is much better to come with concrete DECISIONS on issues than to let them rot.
I've devised a method for analyzing/triaging bugs which I believe will help tremendously in this area. It is based off of a decision making method called "The 4-D approach" in the book "The Power of Focus" by Jack Canfield. My method seems to have a lot of merit in theory, and we're going to try it out in the X Devel team now and see how it works in practice, then hone it more. Hopefully this will reduce the overall bug count, and allow us to focus on _investigating_ and _fixing_ more bugs, and spend less time reading bug reports, many of which are bogus or useless.
Well, I think I've dumped my brain enough for one blog entry, so I'll give everyone's eyes a rest now. If I see enough interest expressed, I might post a future blog discussing my "4D decision making approach to handling bug reports", as I believe it may be very valuable for other projects out there, and individual developers. Anyone interested, can express their interest by emailing me at my personal email address (if they can figure out what that is). ;o)
Very much looking forward to the future modularization of X.Org, which will hopefully happen for the next release of X.Org X11 after 6.8.0. I'm modularizing a small amount of our monolithic xorg-x11 rpms for Fedora Core 3, and will hopefully have some of this into rawhide by tomorrow or so.
This will give many benefits to our users in terms of downloading stuff like fonts over and over and over again, as well as when security updates are released. It will also benefit myself and others on our X Development Team, as it will ease maintenance burdens, and also allow us to divide up workloads, bug report handling, and other tasks more easily. It also will make it easier to let the various subcomponents be updated on their own release cycles rather than an artificial release cycle mandated by the release of the X server. This will also hopefully improve the interoperability of different components, and improve the robustness of the whole X11 user experience.
Now where did my bingo card go ...?
I'm currently investigating http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=115769 as more and more people are reporting this issue, and the trend is likely to start increasing. 4.3.0-57 will have a debugging patch included which will hopefully shed some light on the problems.
Lately I've been overly-irritated by angry Radeon users who have UNSUPPORTED video cards [Radeon 9[268]00(SE|XT)]. People just want their hardware to work, without any problems, end of story. They don't want "half works", or "kindof works". I had added "partial experimental support" to our radeon driver a while back in hopes that it would work fork people, however I didn't have hardware at the time to test it. Since the new code didn't affect support of cards that were already supported, it seemed to me at the time to be a good idea because it was low-risk for supported hardware, however it would allow some people to use the driver on "some" newer radeons.
Bad idea.
Bugzilla is flooded with bug reports from irate users who can't install Fedora Core on their UNSUPPORTED video card, who think that because the installer can DETECT the card, that the card is supported. They have a demand list longer than my arm. <sigh> So much for my volunteered time to help people. The negativity I've received from people gives me no incentive to prioritize fixing the driver, at least not on MY PERSONAL TIME. So the driver probably wont get any love until it gets allocated _work_ time, which wont be any time soon likely because I have tonnes of higher priority tasks. Oh well.
On the bright side, I had bacon wrapped filet mignon tonight which was amazing. I feel much better now.
Thank God that most people, black hats included are frightened at X source code and generally wont touch it with a 10 foot pole.
Once all mainstream distributions begin shipping the modularized and autotooled freedesktop.org xlibs packages, the amount of pain which occurs when security issues are found will hopefully subside. The entire source tree really needs a heavy security audit, as well as a general cleanup. There's a lot of cruft there that hasn't been seen by human eyes for over 10 or more years.
On a far less serious note however ... *cough*
Observations: People who come from a software development background mostly based in the closed cathedral style of "design by voting commitee" software development, just don't "get" open source, or fit into our open source culture very well. It's very much like trying to fit a square peg into a round hole.
One of us open source developers would perhaps discuss a given problem in IRC or a quick email to a developmental mailing list, and probably receive back a useful "well, square pegs don't fit into round holes, try this instead <hands round peg>" type of response, then check some code into CVS to work around the problem.
These people however, tend to over-study the problem at hand, like it was rocket-science, congregate in groups to talk for hours on teleconfrences and bored meetings (oops, did I mean board there?), have a meeting about the meeting, turn to each other and say "I concur" a few times, then finally after maybe 3 months, or 4 years as the case may be, perhaps come up with a conclusion and draw up a positional draft entitled "Square pegs, and their alleged incompatibility with round holes".
After many more conference calls to debate the exact wording and various issues as to the size of the pegs, and whatnot, it is then of course time for a vote on the proposal after a few more conference calls.
My theory however, is that while they (in a theoretical sense that is) are voting on trive, the open source community walks right past with code in hand, and solutions present themselves. Did someone say something about the meek?
So what is my point? Simple really, I just wonder how much money is wasted every year in long distance telephone bills for largely wasted efforts, when "show me the code" reigns supreme. Makes me want to purchase stock in a long distance carrier.
I'm glad now however, that I did actually post it here on advogato, rather than yelling it into the pillow. While I knew before that I wasn't the only one to share these opinions and ideas, I did not know just how many people out there shared my viewpoint. By being open about it, others have come forward as well and shared their opinions. Not just a bunch of "me too" people, but other developers whom have felt XFree86 development to be lacking a community spirit and wider participation. It is clear to me now that there are a significant amount of people out there who are interested in XFree86 development or in contributing to the project in one way or another, that we do have the possibility of building up a community around XFree86 if we really want to.
Several people have felt that there was no way to get involved because they had to "become a member" first, which created a chicken vs. egg scenario for them. I've tried in the past to show people that the membership thing wasn't really necessary for them to get involved, and I've helped people debug/troubleshoot and hack on problems themselves, however with XFree86 lacking in the community spirit department, only a few people have actually stayed interested. Some of them stick around on the #xfree86, #xfree86-devel, #dri, #dri-devel channels on irc.freenode.net which are all open public forums.
Since my diary entry, the XFree86.org private development list devel@xfree86.org has now been made open to the general public. Prior to this, people were urged to use the xpert@xfree86.org mailing list for development, including core team members themselves. Some discussions occured on xpert list that were development related, but this one-list-fits-all approach fails miserably in my opinion when developers and potential developers are mixed into a giant list along with hundreds of people whom are just trying to get configuration help, report problems, and whatnot.
I've watched people ask development questions on xpert list, which I wasn't personally able to help them with, but which I knew another developer was present who could have easily helped. More likely than not, the other developer just wasn't paying attention to the medium to high volume xpert list, as it is mostly saturated with newbie questions and cries for help. It was a list that was often too easy to just tune out of from a developmental standpoint, and so many did from time to time, myself included.
The XFree86 xpert list has now been decommissioned by XFree86.org, and the previously private bug report address 'xfree86@xfree86.org' has been now turned into a public list that combines the bug reports and what was previously the xpert list, newbie list, and others. Different people have different opinions on wether or not this was a good decision, however I am of the opinion that it is for the good. Just the name of the "xpert" list alone frightened away many people who didn't want to bother some expert with their small troubles. Others thought the list was only for experts, and similarly did not want to infringe upon the experts. Just a bad mailing list name for the most part. Having the new list named "xfree86", while just a cosmetic change for the most part, I believe takes down one barrier for XFree86 users.
Having the bug reports fed directly into the mix also allows users to help users solve their problems, as a large number of the incoming bug reports have traditionally been simply end user configuration problems, or people reporting hardware not working which isn't actually supported anyway. Now these users are starting to get responses to their submissions by other list members, rather than have their problems fall into the void. I've even seen an ATI employee offer advice to an Nvidia user. Now isn't that the community spirit? I definitely see what appears to be more of a community being formed, and I'm very glad to see these changes.
With the devel@xfree86.org list now public, I'm hoping that we will all see the developer community around XFree86 increase, and to see developers work more closely together, and with the core team as well. With newbies and end users directed towards the xfree86@xfree86.org mailing list now, hopefully the devel list will have more developmental content, and perhaps lure more developers as well, while remaining free of people asking newbie/help oriented questions and the like.
Definitely some progress happening.
I was quite surprised that every single one of the emails was completely encouraging and positive. Not one single negative comment in over 100 emails! Mind boggling. I guess I just happened to put into words what a rather large number of people have been thinking for quite some time now, at least according to the plethora of feedback I received.
One thing that is very very aparent from the feedback I got, is that the community very much wants to be able to communicate openly with X developers. People want to be heard, and they want to be able to learn more about X development in general. Many people expressed an interest in getting involved in XFree86 development but just had no idea where to start, and in their prior efforts at trying to learn something, or to hack on a given part of the X server, or somesuch, were extremely disappointed at being unable to get anyone to respond on public forums.
Another overwhelming thing people commented on, is that they have reported bugs to the xfree86@xfree86.org list before, which were well detailed, and not gotten any response back. They were not acknowledged, they had no idea if anyone ever looked at the problem, and they had no idea if the problem was fixed already, or if it would ever be fixed, or how to tell in either case. This drives home the need for bugzilla.
Some of the XFree86.org developers are of the mindset that a bug tracking database will do nothing more than consume their time, time they could be spending working on X. Some believe that such a bug tracking database would only make things easier for distribution vendors, as people would file bug reports to XFree86.org instead of their distribution vendor. And they forsee bug database maintenance being an overwhelming task that just wouldn't work.
Well, considering the extreme majority of bugs that get reported are not distribution specific, and ARE generic XFree86 bugs, why SHOULDN'T they be reported to a central bug tracking database viewable by all? Bugs can be easily cross referenced to distro specific bug databases as is done in many other projects that are successfully running right now like GNOME and KDE for example. GNOME, KDE, and other large projects have very well ran bug tracking databases, and the developers would not dream of giving up this wonderful tool.
Tell me - how many projects the size of XFree86 do not have some form of bug tracking database? How many projects the size of XFree86 do not have some kind of project management? How many projects the size of XFree86 do not have development communities built up around them where the developers work with other people in the community, and try to harness new developers to contribute to the code?
GNOME is huge. KDE is huge. Both of these projects could probably not function well at all if all they had was an incoming email address that wasn't publically viewable where bug reports were submitted.
Also, I don't know how accurate this is, but last night, I was told that these projects have several *hundred* developers whom have CVS write access. The general rule is that you follow the CVS rules, and if you screw up, your CVS write priveledges are suspended temporarily for a period of time. If someone screws up in a huge big bad way, then their priveledges are suspended for longer. It was news to me, but it certainly seems to work for them.
One way or another, X11 needs a publically available, real world bug tracking database that the community can use, even if zero of the developers use it who think it is a useless waste of time. Lord knows how that will happen, but it is IMHO something dreadfully needed.
Another thing that annoys the hell out of me, is there always seems to be an aire of politicalness that lays over X development. Instead of being a contributing X developer, I sometimes feel like I'm treated negatively by some people simply because I work at Red Hat, like that somehow should make any difference.
I sometimes wonder if I were to create a bob23424@yahoo.com address and communicate with that if I'd get treated differently by some people. Mind you, this doesn't occur all the time, and certainly not by everyone. It does happen occasionally though, and when it does, it frustrates the hell out of me. I look at X development completely outside the walls of my position at Red Hat. I think of myself as mharris the person interested in doing open source X development, and not as mharris the person forced to work on X development as an employee of Red Hat as a work duty.
I work on open source software because I enjoy working on open source software, and I always will. I work on X for the same reason. It's just my employment at Red Hat that got me started working on X in the first place. I like my job greatly, because it lets me do something I enjoy doing, not because I HAVE to do it. Why must people put up these walls and boundaries? It's senseless. We're all open source developers, and I at least, for one, don't care if another developer is using Red Hat Linux, Debian, Mandrake, FreeBSD, or whatever. Who cares? Why create barriers and refuse to help someone or to listen to them because of their distro of choice, or their employer? Childish.
Anyway, the good side of things is that I see some of this wall coming down now, and have talked with several other developers of whom use other OSs or work for some other company, or whatever, and there seems to be a consensus that people want to work together.
I think in our quest to produce open source software, we have more in common than we have in difference, and that the more we communicate _with_ each other, the more that can be accomplished.
In closing, I'd like to make one more comment. Aparently, after my last entry, people discussed it on the xpert/xfree86 mailing list. Despite being subscribed to the list on 2 separate email addresses, I haven't received mail from xpert list since Dec 17, 2002, and so I completely didn't see any messages on the topic posted there, and so I haven't been able to respond to whatever discussions occured. I have subscribed to the list under a third address now, and am receiving xfree86@xfree86.org once again now. I certainly look forward to hopefully positive minded discussions about these topics on the list, and towards future open source / free software collaboration with everyone in the community who feels like doing so. And of course, regardless of what OS or distribution they use.
And finally - thanks to the hundreds of people for all of your supportive emails! It's very encouraging!
Been spending a lot of time lately split between reading tonnes of video hardware documentation, standards, and other technical documentation, as well as the usual XFree86 hacking.
I am considering lately releasing my own Radeon driver that is separate from the XFree86 project's driver. There is just way too much time that lapses between the time someone submits a patch to XFree86.org and the time that it gets reviewed by someone and applied to XFree86 source code base - a priveledge that only around 14 people have, and of which about 6-8 use on any regular or semi-regular basis.
ATI submitted open source patches for the Radeon 9500 hardware about the time it was released. That was many many months ago. That patch still hasn't been integrated into XFree86 CVS, and as time goes on, I am thinking it is very likely not going to get integrated into 4.3.0 either. Just yesterday, ATI sent me 2 or 3 more patches that build upon the last patch they submitted which wasn't applied. How long is ATI going to continue submitting patches to XFree86.org that take 9 months to get into CVS, and then perhaps another 4-6 months to be available in an OS distribution? Quite frankly, if I were ATI, and submitting patches as frequently as they do, and the patches just sat there, I might start thinking twice about bothering in the future.
So, where does this leave things? Well currently, it leaves the latest XFree86 CVS radeon driver missing Radeon 9500 support, and a few other bug fixes and enhancements from the first ATI patch. It also leaves it missing various other fixes, etc. from the newest 2-3 ATI patches that are supposed to apply on top of that. Also, I have written a patch to add support for some FireGL 8700/8800 hardware, and some Radeon 8500 hardware that was previously unsupported. I haven't sent my patch upstream yet, as it is obvious to me it won't get applied, at least not anytime soon, so why bother? Until they commit ATI's patches, there is no point submitting further patches. Also, there is a number of other Radeon driver features I would like to work on, and get them out into the hands of actual real users sooner rather than later. If you submit a patch though for something, and sit and wait 6-9 months before someone on the inner circle has time to volunteer to review it, commit it into the repository, or reject it with given reasons (or silently), you get discouraged rather easily.
After thinking about this for a while, and also thinking about our upcoming Red Hat Linux release, I decided that I don't NEED to wait for the various things to be incorporated into CVS. I receive a copy of ATI's patches as well, and am more than capable of integrating them myself into the driver base. And, as an added bonus, I *DO* have the time, or I will _make_ the time. I'll be doing this work this week hopefully, and will put up a new web page somewhere where people can download the driver source code and also binary drivers to try. I will do my best to support the drivers as well, and I plan on also trying to get the code into XFree86 sources upstream too over time, but that wont however be my priority.
When people bitch and whine on the XFree86 mailing lists, or email an XFree86 core team member to ask why some patch that was sent in has not gotten applied yet, the general response is that "There are only a certain number of developers available, of whom are working on XFree86 on a voluntary basis, and there is quite a bit of work to do. The patch will be reviewed in time, and probably applied before the next XFree86 release.", or something to that effect. In other words, there is a shortage of volunteer man hours in the project with which to get work done. However, on the same note, no new developers are ever given CVS write access to the CVS repository, not even a small portion, such as a single X driver. So basically patches sit there and rot until someone "gets around to it". XFree86 development is just not scalable.
What is worse (IMHO), is that recently, they have removed the link to their developer page from the website. At least I am unable to find the link to the developer page anywhere on their web site. If you type in the URL manually however, the page is still there. It can be seen at: http://www.xfree86.org/developer.html. You'll notice at the bottom of the page there is a section "How to become an XFree86 developer." and it says:
How to Become an XFree86 DeveloperWe are no longer accepting applications at this time.
I've no idea of their reasoning behind that decision, however it concerns me greatly. Granted, many people apply for XFree86.org membership, submit a simple patch, are accepted, and then do little or nothing after that, so from that angle, I can understand their position to not accept more developers. On the other hand, more developers are needed IMHO. Do people really need to be "accepted" as an X developer first in order to contribute? Well, it might help a bit, but it is not really necessary. Most of the NDA information available to member developers is largely out of date, and new NDA docs haven't been forthcoming for quite a while since companies that do give docs out, tend to do so on a person by person basis now rather than to a single entity like XFree86.org.
So, where does this leave everything? I really don't know. I would like nothing more than to see the XFree86 project thrive, and also to see more capable and willing people contributing to the code. I'd like to see a community of development around X11 like that of the Linux kernel, and I'd like to see people openly discussing developmental issues in an open forum not unlike the linux-kernel mailing list. That type of a developer community has just never formed around XFree86 however, and I have no idea why.
The XFree86 development team has a number of very talented people working on each release, mostly voluntarily, but like any developer, they can only accomplish so much work in a finite amount of time. XFree86 development doesn't really seem to scale to well. The more people who might try to contribute, might have to wait a long time for one of the core members to even look at their code, comment on it, accept/reject it, etc. I think most people just give up before getting too involved, simply due to the lack of close and open communication between all developers. I know many developers who have commented that to me at least.
I've talking a bit with a few other Linux and BSD distribution XFree86 maintainers lately, and they've shown an interest in collaborating on sharing patches more closely and directly, and on working closer together as well. I'm hoping we can form a closer development community that is very open and be able to help each other, and not duplicate efforts as much as is done now. XFree86 does NOT have a bugzilla or similar bug tracking database. Instead, bug reports are sent to an email address (xfree86@xfree86.org) which someone might or might not ever actually read. Many people think of this bug report address as a blackhole, or a /dev/null alias. In many cases, I believe they are right.
Since there is no way to "track" a bug through various stages, through resolution, many bugs, and likely many fixes also just fall through the cracks. Since XFree86.org doesn't really have any way for various distributions to share patches efficiently, or to keep track of the status of an issue, or to even find out if a bug has been fixed or not, and if so, if it is in CVS or not, distribution development tends to needlessly end up with each vendor/distro doing a lot of duplication.
I would like to bring all XFree86 package maintainers, and developers together collaboratively somehow in order to track issues, share patches, and reduce duplication of effort as much as possible, and also to feed things back to XFree86.org in a manner that is useful to them, and preferably also cuts down on the amount of work that they need to do in order to incorporate changes, bugfixes, etc.
Another thing that is a problem, is that once a new release of XFree86 comes out, the older releases are more or less completely abandoned after a short while. Even if you send in a bugfix patch for an older release, with patches for the various new releases as well, usually only the newest release or head of CVS gets the bug fix. In some cases someone will commit the fix to an older release as well, however that is not usually the case. It is understandable with a fixed number of vulunteer core developers, that maintaining code for several years for every release of the codebase is a huge task, however many distributions and vendors MUST support the XFree86 releases they've shipped for a certain amount of time, and in many cases can NOT upgrade the given distro release to a new XFree86 release on a whim, due to various customer commitments, or simply due to regression of certain features/drivers in newer releases. There needs to be a more long term way for others to get involved in the maintenance of XFree86 releases that the core team is not interested in maintaining any more, or simply do not have the volunteer time to spend doing so. I would like to discuss this issue with XFree86.org, and with other distribution maintainers also.
All in all, I would like to see XFree86 development increase rather than decrease, by helping other developers, and by getting more people interested in being developers in the first place. I'd also like to contribute a lot more to the project than I do now, and to help remove duplication of effort.
I also realize that XFree86.org does not have the manpower or volunteer time to necessarily contribute to these ideas as they already have their hands quite full. So by helping to organize external community projects that do not make demands upon XFree86.org members, I think many of us developers can accomplish a lot of work on our own, which we can then submit to XFree86.org at intervals that are more convenient for them, rather than things sitting in patch queues for months on end, awaiting volunteer time.
Anyhow, this is a rather long entry, comprised of various thoughts that have been on my mind for quite some time. Many thoughts of which people have ignored that I've mentioned things too, or have refused to comment on for whatever their reasons. Perhaps I should be more proactive with my ideas and see where it leads. I'll start with the Radeon driver and see where that goes...
I just got back from OLS a day or so ago. OLS was a blast. Don't be fooled! OLS is not a Linux Symposium, it is an alcoholic symposium! Sure, there were Linux presentations, BOF sessions, and other Linux related things going on, but that is just a front for the REAL reason OLS exists - to get piss drunk!
Alan and Telsa shared their hotel suite with me during the symposium, and we imbibed in many different spirits during the 6 day drink fest that endured. ;o) One thing that prevails in Ottawa, is bars and pubs. I've never seen a town anywhere that has street after street of fine drinking establishments. I now understand why the Symposium is held in Ottawa. ;o)
After Canada day, I visited an old drinking buddy Richard in Ottawa and partied for a few days with him and other friends. After Ottawa I went to Toronto for the remainder of the week. I walked around Toronto for several days until my feet fell off. I am still recovering from the blisters.
While in Ottawa and Toronto, I finally got a chance to work on my music collection of the bands "Annihilator", "Dream Theatre", "Symphony X", and "Rhapsody". I'm only missing one DT album now, one Annihilator, a few Symphony X CD's, and am just beginning my Rhapsody collection. $600 worth of music. I'm afraid to look at my VISA bill for the whole OLS/Ottawa/Toronto rendezvous as it will likely be quite scary. Oh well, I had a great vacation, so it was worth it. Might as well enjoy things while I'm here, as I can't take it with me when I go right? ;o)
I'll be posting my OLS pictures somewhere on http://people.redhat.com/mharris/ once I pick up a CF reader soon. Now that I'm back, it is time to catch up on email and bug reports.
12 Mar 2002 (updated 12 Mar 2002 at 07:38 UTC) »
You're in a twisty maze of deeply nested heirarchial directories. There are Imakefiles in all directions. Beware you do not become eaten by a grue.
FOAF updates: Trust rankings are now exported, making the data available to other users and websites. An external FOAF URI has been added, allowing users to link to an additional FOAF file.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!