Older blog entries for mjg59 (starting at number 341)

DMI checks are a last resort

Most x86 devices export various bits of system information via SMBIOS, including the system manufacturer, model and firmware version. This makes it possible for the kernel to alter its behaviour depending on the machine it's running on, usually referred to as "DMI quirking". This is a very attractive approach to handling machine-specific bugs - unfortunately it also means that we often end up working around symptoms with no understanding of the underlying issue.

Almost all x86 hardware is tested with Windows. For the most part vendors don't ship devices that don't work - if you install a stock copy of Windows on a system, you expect it to boot successfully and reboot properly. Basic ACPI functionality should be present and correct, including processor power-saving states. Time should pass at something approximating the real rate. And since stock Windows isn't updated with large numbers of DMI quirk entries, the hardware needs to do this with an operating system that's already shipped.

Which means that if Linux doesn't provide the same level of functionality, it means we're doing something different to Windows. Sometimes this is because we're doing something fundamentally different with the hardware - the HP NX6125, for example, resets its thermal trip points if the timer is set up in a different way to Windows. In this case we've decided that the additional functionality of doing things the Linux way is worth it, and we'll just blacklist the small set of machines that are broken by it.

But other times there's no need for the difference. For years we were triggering system reboots in a different way to Windows and then adding DMI workarounds for any systems that didn't work. The problem with this approach is that it's basically impossible to guarantee that you've found the full set of broken hardware. It's very easy to add another DMI quirk, but it doesn't solve the problem for anyone who bought a machine, tried Linux and gave up when they found reboot didn't work. More recently we've added a pile of quirks for Dells - turns out that in all cases we're working around a bug in their firmware that hangs if we're using VT-d.

We typically don't merge code that fixes a specific example of a problem without at least considering whether there's a wider class of similar problems that could all be solved at once. We should take the same attitude to DMI quirks. Sometimes they're the least harmful way of handling an issue, but most of the time they're just fixing one person's itch and leaving a larger number of people to continue cursing at Linux. There should at least be a demonstration of an attempt to understand the underlying problem before just adding another quirk, and every patch that permits the removal of some DMI checks should be greeted with great cheer.

comment count unavailable comments

Syndicated 2012-07-06 14:19:43 from Matthew Garrett

Fun with firmware ABI confusion

I spent a bunch of the past week trying to figure out why my UEFI bootloader code was crashing in bizarre ways. It turns out to be a known issue that various people have hit in the past. After a lot of staring (and swearing) I finally got to the point where I could run a debugger against a running UEFI binary under qemu and found that I was entering a function with valid arguments but executing it with invalid ones. After spending a while crying on IRC people suggested that maybe the red zone was getting corrupted, and it turned out that this was the problem.

The red zone is part of the AMD64 System V ABI and consists of 128 bytes at the bottom of the stack frame. This region is guaranteed to be left untouched by any interrupt or signal handlers, so it's available for functions to use as scratch space. For the code in question, gcc was copying the registers onto the stack before executing the function. Something then trod on those registers. This also explained why I wasn't seeing the problem when I added debug print statements - signal and interrupt handlers won't touch it, but other functions might, so it tends to only be used by functions that don't call any other functions. Adding a print statement meant that the compiler left the red zone alone.

But why was it getting corrupted? UEFI uses the Microsoft AMD64 ABI rather than the System V one, and the Microsoft ABI doesn't define a red zone. UEFI executables built on Linux tend to use System V because that means we can link in static libraries built for Linux, rather than needing an entirely separate toolchain and libraries to build UEFI executables. The problem arises when we run one of these executables and there's a UEFI interrupt handler still running. Take an interrupt, Microsoft ABI interrupt handler gets called and the red zone gets overwritten.

There's a simple fix - we can just tell gcc not to assume that the red zone is there by passing -mno-red-zone. I did that and suddenly everything was stable. Frustrating, but at least it works now.

comment count unavailable comments

Syndicated 2012-07-02 22:24:22 from Matthew Garrett

Ubuntu ODM UEFI requirements for secure boot

A couple of people have asked me about the Ubuntu ODM UEFI requirements, specifically the secure boot section. This is aimed at hardware vendors who explicitly want to support Ubuntu, so it's not necessarily the approach Canonical will be taking for installing Ubuntu on average consumer hardware. But it's still worth looking at.

In a nutshell, the requirements for secure boot are:

  • The system must have an Ubuntu key preinstalled in each of KEK and db
  • It must be possible to disable secure boot
  • It must be possible for the end user to reconfigure keys

It's basically the same set of requirements as Microsoft have, except with an Ubuntu key instead of a Microsoft one.

The significant difference between the Ubuntu approach and the Microsoft approach is that there's no indication that Canonical will be offering any kind of signing service. A system carrying only the Ubuntu signing key will conform to these requirements and may be certified by Canonical, but will not boot any OS other than Ubuntu unless the user disables secure boot or imports their own key database. That is, a certified Ubuntu system may be more locked down than a certified Windows 8 system.

(Practically speaking this probably isn't an issue for desktops, because you'll need to carry the Microsoft key in order to validate drivers on any PCI cards. But laptops are unlikely to run external option ROMs, so mobile hardware would be viable with only the Ubuntu key)

There's two obvious solutions for this:
  1. Canonical could offer a signing service. Expensive and awkward, but obviously achievable. However, this isn't a great solution. The Authenticode format used for secure boot signing only permits a single signature. Anything signed with the Ubuntu key cannot also be signed with any other key. So if, say, Fedora wanted to install on these systems without disabling secure boot first, you'd need to have two sets of install media - one signed with the Ubuntu key for Ubuntu hardware, one signed with the Microsoft key for Windows hardware.
  2. Require that ODMs include the Microsoft key as well as the Ubuntu key. This maintains compatibility with other operating systems.

This kind of problem is why we didn't argue for a Fedora-specific signing key. While it would have avoided a dependence on Microsoft, it would have created an entirely different kind of vendor lock-in.

comment count unavailable comments

Syndicated 2012-06-19 17:45:06 from Matthew Garrett

No, really, secure boot does add security

Carla Schroder wrote a piece for linux.com on secure boot, which gives a reasonable overview of the technology and some of the concerns. It unfortunately then encourages everyone to ignore those legitimate concerns by making the trivially false claim that secure boot is just security theatre.

Security isn't some magic binary state where you're either secure or insecure and you can know which. Right now I'd consider my web server secure. If someone finds an exploitable bug in Apache then that would obviously no longer be the case. I could make it more secure by ensuring that Apache runs in a sufficiently isolated chroot that there's no easy mechanism for anyone to break out, which would be fine up until there's also a kernel exploit that allows someone to escalate their privileges enough to change their process root. So it's entirely possible that someone could right now be breaking into my system through bugs I don't even know exist. Am I secure? I don't know. Does that mean I should just disable all security functionality and have an open root shell bound to a well known port? No. Obviously.

Secure boot depends on the correctness of the implementation and the security of the signing key. If the implementation is flawed or if control of the signing key is lost then it stops providing security, and understanding that is important in order to decide how much trust you place in the technology. But much the same is true of any security technique. Kernel flaws make it possible for an unprivileged user to run with arbitrary privileges. Is user/admin separation security theatre? SSL certificate authorities have leaked keys. Is it security theatre for your bank to insist that you use SSL when logging in?

Secure boot doesn't instantly turn an insecure system into a secure one. It's one more technology that makes it more difficult for attackers to take control of your system. It will be broken and it will be fixed, just like almost any other security. If it's security theatre, so is your doorlock.

Why is this important? Because if you tell anyone that understands the technology that secure boot adds no security, they'll just assume that you're equally uninformed about everything else you're saying. It's a perfect excuse for them to just ignore discussion of market restrictions and user freedoms. We don't get anywhere by arguing against reality. Facts are important.

comment count unavailable comments

Syndicated 2012-06-15 01:00:33 from Matthew Garrett

The security of Secure Boot

The other complaint about UEFI Secure Boot is that it doesn't add any security. There's two aspects to this - people either think it'll be quickly broken, or people think that the public availability of signing services will render it useless.

Will Secure Boot be broken?

Almost certainly, yes. I'd put a significant amount of money on it. The most obvious avenue of attack is probably a malformed FAT on a USB stick, or alternatively a programmable USB device that can trigger undesired behaviour in the USB stack. If you're really lucky then you might be able to corrupt the filesystem on the EFI system partition in such a way that this happens on every boot, even if there's no USB hardware connected. It's possible that there'll be a bug in the binary validation process and a cleverly designed binary may be able to validate even though it contains unsigned code. Various vendors will probably manage to produce signed option ROMs that rely on unsigned data in such a way that they can be forced to misbehave. But most of these attacks will either involve physical access or depend on the target machine having a specific piece of hardware installed, and I'd also put money on each of them being quickly fixed once they're found. And, eventually, people will run out of new attacks.

People desperately want to believe that the Secure Boot implementation is fundamentally broken, and that's just not true. It's based on well-tested encryption technology. You calculate a SHA256 of the binary you want to sign. You encrypt that hash with the private half of a 2048-bit RSA key. You embed the encrypted hash in the binary. When someone tries to run the binary you decrypt the hash with the public key in your firmware. You then calculate the hash of the binary and compare the two. If they match then you know that (a) whoever signed the binary had access to the private half of a key you trust, and (b) the binary hasn't been modified since then.

The difference between Secure Boot and things like CSS or WEP is that Secure Boot uses existing cryptographic techniques and has been designed by people who actually understand how they work. Precisely the same technology is used in Microsoft's driver signing, and people decided it was easier to steal a key from Realtek than it was to break the underlying technology. Unless there's a crippling bug in the implementation that nobody's noticed yet, the only real way to attack this is to either force a SHA256 collision (achievable with current technology, as long as you're willing to spend many orders of magnitude more time than the universe has existed) or break RSA (which would also mean that SSL is broken). If anyone can do either of these things then we are so unbelievably fucked that Secure Boot should not be the thing you're most worried about.

So, in summary: Yes, Secure Boot will be broken. And then it will be fixed. And after this happens a few times, there will be no further breaks.

Everyone can sign, so what's the point

Anyone can pay $99 and get their binaries signed. So why won't malware authors just do that? For starters, you'll need to provide some form of plausible ID for Verisign to authenticate you and hand over access. So, sure, you provide some sort of fake ID. That's the kind of thing that tends to irritate local authorities - not only are you talking about breaking into a bunch of computers, you're talking about committing the sort of crime that results in genuinely bad things happening to you if you get caught. And secondly, the only way you get access to the system is using some physical smartcards that Verisign send you. So you've also had to hand over an actual physical address, and even if you've used some sort of redirection service that's still going to make it easier to track you down. You're taking some fairly significant real-world risks.

But ok, you're willing to do that and you obtain a signing key and you sign some malware. You'll infect some number of machines. But eventually you'll be noticed, and then Microsoft produce a key revocation blob and vendors push it out via their OS update mechanisms. If your malware was basically harmless then that's probably the end of it, but if you were using it as the basis for an attack on people's banking details then a lot of people are suddenly going to be very interested in the person who bought that key, and also your malware isn't going to be able to infect any more machines.

To put this in perspective - Windows is the dominant desktop operating system. If you want to run a botnet or steal people's bank details then Windows is the obvious thing to attack. Windows security has now improved to the point where it's not massively straightforward to do that from userspace, so the logical thing to do would be to run your malware in the kernel. The easiest way to do that would be to load a driver, but Windows won't load unsigned drivers. So you'd need a signed driver. How do you get one? With the same $99 access to the Microsoft signing service that you'd use for signing an EFI binary.

Anyone could already use the signing service to attack Windows. But people haven't. The Stuxnet authors thought it was easier to steal a key from a real hardware vendor than it was to buy their own key. So, yes, in theory an attacker could obtain signing services. But the real world evidence suggests that that's not how they behave.

Summary

Secure Boot is unlikely to be broken via fundamental flaws in its implementation, and there's no evidence that public availability of the signing service will result in an explosion of boot level malware.

comment count unavailable comments

Syndicated 2012-06-08 03:56:15 from Matthew Garrett

"Why not just use Coreboot?"

Why not just avoid the entire Secure Boot problem by using Coreboot? Because the reason we have the Secure Boot problem is because Microsoft's Windows 8 certification requirements mean vendors have to ship a UEFI implementation with Secure Boot. You could satisfy that by using Coreboot with a Tiano payload, but it'll still have Secure Boot enabled so you still have the same set of problems. But maybe you could just reflash your system with Coreboot? No, because another part of the requirements states that all firmware updates have to be cryptographically signed now. The only way to reflash will be to attach a flash programmer directly to your motherboard.

So why not just use Coreboot? Because it doesn't help solve this problem in any way.

comment count unavailable comments

Syndicated 2012-06-06 14:36:17 from Matthew Garrett

Implementing UEFI Secure Boot in Fedora

(Brief disclaimer - while I work for Red Hat, I'm only going to be talking about Fedora here. Anything written below represents only my opinions and my work on Fedora, not Red Hat's opinions or future plans)

Fedora 17 was released this week. It's both useful and free, and serves as a welcome addition to any family gathering. Do give it a go. But it's also noteworthy for another reason - it's the last Fedora release in the pre-UEFI secure boot era. Fedora 18 will be released at around the same time as Windows 8, and as previously discussed all Windows 8 hardware will be shipping with secure boot enabled by default. While Microsoft have modified their original position and all x86 Windows machines will be required to have a firmware option to disable this or to permit users to enrol their own keys, it's not really an option to force all our users to play with hard to find firmware settings before they can run Fedora. We've been working on a plan for dealing with this. It's not ideal, but of all the approaches we've examined we feel that this one offers the best balance between letting users install Fedora while still permitting user freedom.

Getting the machine booted


Most hardware you'll be able to buy towards the end of the year will be Windows 8 certified. That means that it'll be carrying a set of secure boot keys, and if it comes with Windows 8 pre-installed then secure boot will be enabled by default. This set of keys isn't absolutely fixed and will probably vary between manufacturers, but anything with a Windows logo will carry the Microsoft key[1].

We explored the possibility of producing a Fedora key and encouraging hardware vendors to incorporate it, but turned it down for a couple of reasons. First, while we had a surprisingly positive response from the vendors, there was no realistic chance that we could get all of them to carry it. That would mean going back to the bad old days of scouring compatibility lists before buying hardware, and that's fundamentally user-hostile. Secondly, it would put Fedora in a privileged position. As one of the larger distributions, we have more opportunity to talk to hardware manufacturers than most distributions do. Systems with a Fedora key would boot Fedora fine, but would they boot Mandriva? Arch? Mint? Mepis? Adopting a distribution-specific key and encouraging hardware companies to adopt it would have been hostile to other distributions. We want to compete on merit, not because we have better links to OEMs.

An alternative was producing some sort of overall Linux key. It turns out that this is also difficult, since it would mean finding an entity who was willing to take responsibility for managing signing or key distribution. That means having the ability to keep the root key absolutely secure and perform adequate validation of people asking for signing. That's expensive. Like millions of dollars expensive. It would also take a lot of time to set up, and that's not really time we had. And, finally, nobody was jumping at the opportunity to volunteer. So no generic Linux key.

The last option wasn't hugely attractive, but is probably the least worst. Microsoft will be offering signing services through their sysdev portal. It's not entirely free (there's a one-off $99 fee to gain access), but it's cheaper than any realistic alternative would have been. It ensures compatibility with as wide a range of hardware as possible and it avoids Fedora having any special privileges over other Linux distributions. If there are better options then we haven't found them. So, in all probability, this is the approach we'll take. Our first stage bootloader will be signed with a Microsoft key.

Bootloaders


We've decided to take a multi-layer approach to our signing for a fairly simple reason. Signing through the Microsoft signing service is a manual process, and that's a pain. We don't want to have bootloader updates delayed because someone needs to find a copy of Internet Explorer and a smartcard and build packages by hand. Instead we're writing a very simple bootloader[2]. This will do nothing other than load a real bootloader (grub 2), validate that it's signed with a Fedora signing key and then execute it. Using the Fedora signing key there means that we can build grub updates in our existing build infrastructure and sign them ourselves. The first stage bootloader should change very rarely, and we don't envisage updating it more than once per release cycle. It shouldn't be much of a burden on release management.

What about grub? We've already switched Fedora 18 over to using grub 2 by default on EFI systems, but it still needs some work before it's ready for secure boot. The first thing is that we'll be disabling the module loading. Right now you can load arbitrary code into grub 2 at runtime, and that defeats the point of secure boot. So that'll be disabled. Next we'll be adding support for verifying that the kernel it's about to boot is signed with a trusted key. And finally we'll be sanitising the kernel command line to avoid certain bits of functionality that would permit an attacker to cause even a signed kernel to launch arbitrary code. These restrictions will all vanish if secure boot is disabled.

Kernel


Secure boot is built on the idea that all code that can touch the hardware directly is trusted, and any untrusted code must go through the trusted code. This can be circumvented if users can execute arbitrary code in the kernel. So, we'll be moving to requiring signed kernel modules and locking down certain aspects of kernel functionality. The most obvious example is that it won't be possible to access PCI regions directly from userspace, which means all graphics cards will need kernel drivers. Userspace modesetting will be a thing of the past. Again, disabling secure boot will disable these restrictions.

Signed modules are obviously troubling from a user perspective. We'll be signing all the drivers that we ship, but what about out of tree drivers? We don't have a good answer for that yet. As before, we don't want any kind of solution that works for us but doesn't work for other distributions. Fedora-only or Ubuntu-only drivers are the last thing anyone wants, and this really needs to be handled in a cross-distribution way.

Wait signed what


Secure boot is designed to protect against malware code running before the operating system. This isn't a hypothetical threat. Pre-boot malware exists in the wild, and some of it is nastier than you expect. So obviously bootloaders need to be signed, since otherwise you'd just replace the signed bootloader with an unsigned one that installed malware and booted your OS.

But what about the kernel? The kernel is just code. If I take a signed Linux bootloader and then use it to boot something that looks like an unsigned Linux kernel, I've instead potentially just booted a piece of malware. And if that malware can attack Windows then the signed Linux bootloader is no longer just a signed Linux bootloader, it's a signed Windows malware launcher and that's the kind of thing that results in that bootloader being added to the list of blacklisted binaries and suddenly your signed Linux bootloader isn't even a signed Linux bootloader. So kernels need to be signed.

And modules? Again, modules are just code. It's a little trickier, but if your signed kernel loads an unsigned module then that unsigned module can set up a fake UEFI environment and chain into a compromised OS bootloader. Now the attacker just has to include a signed kernel and a minimal initramfs that loads their malware module. It'd slow down boot by a couple of seconds, but other than that it'd be undetectable. X? If you can access the registers on a GPU then you can get the GPU to DMA over the kernel and execute arbitrary code. Trickier again, but still achievable - and if you've locked down every other avenue of attack, it's even attractive.

If we produce signed code that can be used to attack other operating systems then those other operating systems are justified in blacklisting us. That doesn't seem like a good outcome.

Customisation


A lot of our users want to build their own kernels. Some even want to build their own distributions. Signing our bootloader and kernel is an impediment to that. We'll be providing all the tools we use for signing our binaries, but for obvious reasons we can't hand out our keys. There's three approaches here. The first is for a user to generate their own key and enrol it in their system firmware. We'll trust anything that's signed with a key that's present in the firmware. The second is to rebuild the shim loader with their own key installed and then pay $99 and sign that with Microsoft. That means that they'll be able to give copies to anyone else and let them install it without any fiddling. The third is to just disable secure boot entirely, at which point the machine should return to granting the same set of freedoms as it currently does.

But I don't trust Microsoft


A system in custom mode should allow you to delete all existing keys and replace them with your own. After that it's just a matter of re-signing the Fedora bootloader (like I said, we'll be providing tools and documentation for that) and you'll have a computer that will boot Fedora but which will refuse to boot any Microsoft code. It may be a little more awkward for desktops because you may have to handle the Microsoft-signed UEFI drivers on your graphics and network cards, but this is also solvable. I'm looking at ways to implement a tool to allow you to automatically whitelist the installed drivers. Barring firmware backdoors, it's possible to configure secure boot such that your computer will only run software you trust. Freedom means being allowed to run the software you want to run, but it also means being able to choose the software you don't want to run.

You've sold out


We've been working on this for months. This isn't an attractive solution, but it is a workable one. We came to the conclusion that every other approach was unworkable. The cause of free software isn't furthered by making it difficult or impossible for unskilled users to run Linux, and while this approach does have its downsides it does also avoid us ending up where we were in the 90s. Users will retain the freedom to run modified software and we wouldn't have accepted any solution that made that impossible.

But is this a compromise? Of course. There's already inequalities between Fedora and users - trademarks prevent the distribution of the Fedora artwork with modified distributions, and much of the Fedora infrastructure is licensed such that some people have more power than others. This adds to that inequality. It's not the ideal outcome for anyone, and I'm genuinely sorry that we weren't able to come up with a solution that was better. This isn't as bad as I feared it would be, but nor is it as good as I hoped it would be.

What about ARM


Microsoft's certification requirements for ARM machines forbid vendors from offering the ability to disable secure boot or enrol user keys. While we could support secure boot in the same way as we plan to on x86, it would prevent users from running modified software unless they paid money for a signing key. We don't find that acceptable and so have no plans to support it.

Thankfully this shouldn't be anywhere near as much of a problem as it would be in the x86 world. Microsoft have far less influence over the ARM market, and the only machines affected by this will be the ones explicitly designed to support Windows. If you want to run Linux on ARM then there'll be no shortage of hardware available to you.

Is this all set in stone?


No. We've spent some time thinking about all of this and are happy that we can implement it in the Fedora 18 timescale, but there's always the possibility that we've missed something or that a new idea will come up. If we can increase user freedom without making awful compromises somewhere else then we'll do it.

[1] In fact, chances are that everything will carry the Microsoft key. Secure boot requires that UEFI drivers also be signed. The signing format only permits a single signature per binary. For compatibility, approximately all add-on hardware shipped will be signed with Microsoft's key, and that means that all system vendors have to recognise Microsoft's key in order to permit that hardware to run on their systems.
[2] Current source is here. It relies on a port of the UEFI crypto library and OpenSSL that I have building with some handholding, and which I'll upload as soon as possible.

comment count unavailable comments

Syndicated 2012-05-31 05:41:15 from Matthew Garrett

Fedora 17 and Mac support

Fedora 17 will be shipping next week. It's got a bunch of new features, none of which I contributed to in the slightest. What I did work on was improving our support for installation on x86 Apple hardware. There's still a few shortcomings in this so it's not an announced or supported feature, but it's sufficient progress that it's worth writing about.

There's a few ways that Apple platforms differs from normal PC hardware. The first is that Apples are very much intended to be EFI first and BIOS second, while that's still not really true for commodity PC hardware. Apple launched Boot Camp shortly after they started shipping x86 Macs (and shortly after I'd got Ubuntu running on one for the first time[1]) but the focus of Boot Camp has always been to be good enough to boot Windows and not much else[2]. The other is the integration with the boot environment. You can boot Linux easily enough by setting the standard EFI boot variables, but if you ever boot back into OS X it's easy to trigger deletion of those. There's then no straightforward way of resetting them, and you're left having to recover your installation.

The other difficult bit has been actually starting the installer at all. If you're happy to use BIOS emulation than this can be done without too much misery, but you're then left with dealing with Apple's custom synchronised GPT/MBR. More modern Macs will boot via the standard EFI mechanisms, but there's a range of problems that can be caused that way. Fedora 17 is the first Linux distribution to ship install media that will EFI boot a Mac when either written to optical media or a USB stick[3]. Put the install media in your system and then either select it in the OS X Startup Disk preferences menu and reboot, or reboot and hold down the alt key. In the latter case there'll be a nice Fedora logo. Click on it and it'll boot.

The biggest difference between Apple hardware and anyone else is that we'd normally put the bootloader in a FAT partition that's shared with any other installed operating systems. That does actually work on Apples, but then you run into the earlier point I mentioned. The Apple startup picker that you get by holding down the alt key doesn't pay any attention to the standard EFI boot variables, so it won't show a FAT partition merely because it's got an EFI bootloader on it. The same is true of the OS X Startup Disk preferences. The best way to get a drive to appear in the menu is to make it look like an OS X drive.

This was problematic in a few ways, due to the requirement for an OS X drive to be HFS+. The first was that we didn't have any support for that in our installer, so work had to be done there. The second was that the state of HFS+ was woefully poor in Linux[4]. Linux will refuse to write to an HFS+ filesystem if it hasn't been cleanly unmounted or fscked, and since we can't rely on everyone always shutting their machine down cleanly that means a working fsck.hfsplus. In theory we were shipping one of those. In practice, it reliably segfaulted on 64-bit systems[5]. So I had to package the latest version of Apple's code, and doing that involved dealing with the fact that Apple now use blocks almost as much as Grub 2 uses nested functions, and that meant using Clang and that meant dealing with a bug where Clang segfaulted if it had been built with gcc 4.7[6], but finally once all of those yaks had been shaven we could trust our filesystem.

Of course, more problems existed. You can't just drop a bootloader onto an HFS+ partition and expect it to work. You need to write its inode value into the superblock. Trivial, except that if you do this from userspace then the kernel kindly overwrites your update when you unmount the disk and it updates the superblock. That required a mechanism for updating that data via the kernel, which meant adding an ioctl. This would have been fine, except the OS X Startup Disk preference looks for a bootloader in a location other than where Fedora installs it. Changing our bootloader install path wasn't a great option, so the easiest thing to do was just to link them together. Except HFS+ doesn't really support hardlinks. When you hardlink something the original file gets moved into a magic directory in the filesystem root and the links are special files that refer to it. And it turns out that it's not actually the inode that the firmware wants when finding the bootloader, it's the catalogue ID, and these aren't the same when hardlinks are involved. So, another kernel patch.

All that was left for the boot partition then was adding an icon and a drive label. We already had code for generating the icon, and the drive label wasn't that difficult. Hurrah!

Ok, so we can construct a filesystem that (a) appears in the bootpicker, and (b) appears in the OS X Startup Disk preferences. We can even put a bootloader on it. This led to the next problem. The bootloader started without any trouble. But the bootloader couldn't read any files off the boot partition, including its configuration. This should have been obvious with hindsight - the problem was simply that grub legacy[7] doesn't support HFS+. Writing an HFS+ driver seemed like an excessive amount of work, especially since the firmware already supported reading HFS+, so instead I wrote a simple grub filesystem driver that uses the UEFI interfaces to read files.

A working bootloader! One more problem there. grub looks in its startup directory to find its configuration file. We have two "copies" of grub hardlinked to each other, each looking in a different directory for configuration. Hardlinking the config files together worked fine, except that most tools that update config files have this irritating habit of writing to a temporary file and moving that over the original, thus breaking the hardlink. Easily solved by using a symlink instead, except that the UEFI filesystem semantics don't really support symlinks because UEFI only really envisaged using FAT. If you use Apple's UEFI HFS+ driver to open a symlink, you get a short file containing the target path of the symlink. Rather less than ideal, and a rather gross hack to cope with it.

All set now, other than a bug in some older Mac firmware where read would return BUFFER_TOO_SMALL without returning the real buffersize and the odd way that Apple treat CDs. Oh, and a bug in Apple's Broadcom wifi driver that could overwrite the kernel, and case-sensitivity, something that obviously isn't something you often hit on UEFI when all you usually see is FAT.

Kernel-side, we had a couple of things. Obviously there the bugs I'd mentioned in the past, but those had already been dealt with. New issues included the fact that we were frequently picking the wrong framebuffer address, especially on machines with multiple GPUs. Fixed now, along with another issue where we'd sometimes get confused by models with different GPU configurations. And at that point, most of the implementation work was done.

Of course, there was also the work of integrating this into the tools that we use to generate our install media, and I'd like to thank Will Woods, Brian Lane and Dennis Gilmore for the work they put into that, along with anyone I've forgotten. Tom Calloway handled getting the device label graphics generated right before deadline. A cast of thousands helped with testing.

So why did I mention shortcomings? First, this only works on machines with 64-bit firmware. Early 64-bit Apples still had 32-bit firmware. In theory we can support that case - in practice it's a moderate amount of writing, including some mildly awkward kernel code. But anything later than mid-2007 should have 64-bit firmware, so it's not a massive concern. Second, we seem to have fairly significant trouble with Radeon hardware on a lot of Macs. We fixed one bug there, but something's still going wrong in setup and you'll frequently end up with a black screen. And thirdly, I'm sure there's some other machines that still don't work for (as yet) undetermined reasons.

With luck we'll get these things dealt with by Fedora 18. But even with these flaws, Fedora 17 will work fine on a lot of Apple hardware, and testing it is as simple as downloading and booting a live image. Bugs go in the usual place.

[1] Note my charming belief that the Ubuntu installer would fully support this hardware via EFI in the near future. 6 years ago. It still doesn't.
[2] This can be fairly easily seen from little things like the name "Windows" being hardcoded into every UI that sees a Boot Camp drive.
[3] I cover this in more depth here
[4] To be fair, it mostly still is - there's plenty of work to be done there.
[5] Approximately nobody had noticed this, which is all kinds of unreassuring.
[6] Thanks to Kalev Lember for dealing with that
[7] Still the EFI bootloader in Fedora 17. We've already swapped Fedora 18 over to grub 2.

comment count unavailable comments

Syndicated 2012-05-25 03:42:53 from Matthew Garrett

I've been a terrible person (and so have most of you)

John Scalzi recently wrote a piece on straight white male privilege. If you haven't read it already, go and do so. No rush. I'll wait.

So. Some facts:

  • Women are underrepresented in free software development
  • Those women who are involved in free software development are overwhelmingly more likely to have been subject to sexual harassment, belittling commentary or just plain ignored because of their gender
  • When asked, women tend to believe that these two facts are fairly strongly related

(If you disagree with any of these then that's absolutely your right. You're wrong, but that's ok. But please do me a favour and stop reading here. Otherwise you'll just get angry and then you'll write something ill-tempered and still wrong in the comments and then I'll have to delete it and why not just save everybody the time and effort and go and eat ice cream or something instead)

I know I've said this before, but inappropriate and marginalising behaviour is rife in our community, and at all levels of our community. There's the time an open source evangelist just flat out told a woman that her experiences didn't match his so she must be an outlier. There's the time a leading kernel developer said that most rape statistics were basically made up. There's the time that I said the most useful thing Debian could do with its money would be to buy prostitutes for its developers, simultaneously sexualising the discussion, implying that Debian developers were all straight men and casting sex workers as property. These aren't the exceptions. It's endemic. Almost all of us have been part of the problem, and in doing so we've contributed to an environment that has at best driven away capable contributors. You probably don't want to know what it's done at worst.

But what people have done in the past isn't important. What's important is how we behave in the future. If you're not angry about social injustice like this then you're doing it wrong. If you're reading this then there's a pretty high probability that you're a white male. So, it's great that you're angry. You should be! As a straight white male born into a fairly well-off family, a native English speaker in an English speaking country, I have plenty of time to be angry before going back to my nice apartment and living my almost entirely discrimination-free life. So if it makes me angry, I have absolutely no way of comprehending how angry it must make the people who actually have to live with this shit on a daily basis.


(Were tampon mouse able to form and express coherent thoughts, tampon mouse would not put up with this shit)

The point isn't to be smugly self aware of our own shortcomings and the shortcomings of others. The point is to actually do something about it. If you're not already devoting some amount of your resources to improving fairness in the world, then why not? It doesn't have to be about women in technology - if you're already donating to charity or helping out at schools or engaging in local politics or any of the countless other ways an individual can help make the world a better place, large or small, then keep on doing that. But do consider that many of us have done things in the past that contributed to the alienation of an astounding number of potential community members, and if you can then please do do something to make up for it. It might be donating to groups like The Ada Initiative. It might be mentoring students for projects like the GNOME Outreach Program for Women, or working to create similar programs. Even just making our communities less toxic by pointing out unacceptable behaviour when you see it makes a huge difference.

But most importantly, be aware that it was people like me who were responsible for this problem in the first place and people like me who need to take responsibility for solving it. We can't demand the victims do that for us.

comment count unavailable comments

Syndicated 2012-05-22 01:57:11 from Matthew Garrett

Fixing Asus UX21e unexpected power off

The Asus UX21e (and maybe the UX31e?) has the irritating misfeature that it reloads the CPU thermal tables when you unplug the power. One consequence of this is that it'll automatically throttle itself much more aggressively on battery (reducing performance) but the more serious one is that the new critical power off temperature may then be lower than the temperature the CPU is currently operating at, resulting in the machine turning itself off. As far as I can tell from debugging, this is completely OS-independent - it still happens even if I stub out all the ACPI code for power supply events and there are reports of the same thing occurring on Windows. The good news is that it seems to be fixed in newer firmware versions. The even better news is that you can flash it without Windows. Just download the BIOS image from the Asus website, copy it onto a FAT formatted USB stick, insert that, go into the firmware (hit F2 on the splash screen) and start the flash program from there.

comment count unavailable comments

Syndicated 2012-05-19 23:26:29 from Matthew Garrett

332 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!