Older blog entries for mjg59 (starting at number 317)

Making timeouts work with suspend

A reasonably common design for applications that want to run code at a specific time is something like:

time_t wakeup_time = get_next_event_time();
time_t now = time(NULL);
sleep(wakeup_time-now);

This works absolutely fine, except that sleep() ignores time spent with a suspended system. If you sleep(3600) and then immediately suspend for 45 minutes you'll wake up after 105 minutes, not 60. Which probably isn't what you want. If you want a timer that'll expire at a specific time (or immediately after resume if that time passed during suspend), use the POSIX timer family (timer_create, timer_settime and friends) with CLOCK_REALTIME. It's a signal-driven interface rather than a blocking one, so implementation may be a little more complicated, but it has the advantage of actually working.

comment count unavailable comments

Syndicated 2011-11-17 13:51:47 from Matthew Garrett

Properly booting a Mac

This is mostly for my own reference, but since it might be useful to others:

By "Properly booting" I mean "Integrating into the boot system as well as Mac OS X does". The device should be visible from the boot picker menu and should be selectable as a startup disk. For this to happen the boot should be in HFS+ format and have the following files:

  • /mach_kernel (can be empty)
  • /System/Library/CoreServices/boot.efi (may be booted, if so should be a symlink to the actual bootloader)
  • /System/Library/CoreServices/SystemVersion.plist which should look something like
    <xml version="1.0" encoding="UTF-8"?>
    <plist version="1.0">
    <dict>
            <key>ProductBuildVersion</key>
            <string></string>
            <key>ProductName</key>
            <string>Linux</string>
            <key>ProductVersion</key>
            <string>Fedora 16</string>
    </dict>
    </plist>
That's enough to get it to appear in the startup disk boot pane. Getting it in the boot picker requires it to be blessed. You probably also want a .VolumeIcon.icns in / in order to get an appropriate icon.

Now all I need is an aesthetically appealing boot loader.

comment count unavailable comments

Syndicated 2011-11-09 22:06:07 from Matthew Garrett

Understanding the current state of UEFI

This story has been floating around for a week or so. The summary is that someone bought a system that has UEFI and is having trouble installing Linux on it. In itself, not a problem. But various people have either conflated this with the secure boot issue or suggested that UEFI is a fundamentally anti-Linux technology.

Right now there are no machines shipping to the public with secure boot enabled. None at all. If you're having problems installing Linux on a machine with UEFI then it's not because of secure boot. So what is actually causing the problem?

UEFI is a complicated specification, with 2.3.1A being 2214 pages long. It's a large body of code. There's a lot of subtleties. It's very easy for people to get things wrong. For example, we've seen issues where calling SetVirtualAddressMap() resulted in the firmware referencing boot services code, a clear violation of the spec on the firmware authors' part. We've also found machines that failed to boot because grub wasn't aligning its stack properly, a clear violation of the spec on our part.

Software is difficult. People make mistakes. When something mysteriously fails to work the immediate assumption should be that you've found a bug, not a conspiracy. Over time we'll find those bugs and fix them, but until then just treat UEFI boot failures like any other bug - annoying, but not malicious.

comment count unavailable comments

Syndicated 2011-11-03 17:47:36 from Matthew Garrett

UEFI secure boot white paper

As people have probably noticed, last week we published a white paper on the UEFI secure boot issue. This was written in collaboration with Canonical and wouldn't have been possible without the tireless work of Jeremy Kerr. It's the kind of problem that affects the entire Linux community, and I'm glad that we've both demonstrated that being competitors is less important than working together for the benefit of everyone.

comment count unavailable comments

Syndicated 2011-11-03 17:47:12 from Matthew Garrett

Feeding the trolls

A few years ago I got up on stage and briefly talked about how the Linux community contained far too many people who were willing to engage in entirely inappropriate behaviour, how this discouraged people from getting involved and how we weren't very good at standing up against that sort of behaviour. Despite doing this in front of several hundred people, and despite the video of me doing so then being uploaded to the internet, this got me a sum total of:

  • No death threats
  • No discussion about any of my physical attributes or lack thereof
  • No stalkers
  • No accusations that I was selling out the Linux community
  • No accusations that I was a traitor to my gender
  • No real negative feedback at all[1]

Which is, really, what you'd expect, right? The internet seems intent on telling me otherwise:

Well, she didn't do herself any favors by talking at conferences about women in tech, or setting up a feminist movement. If you wanted to attract abuse, that's a good way to go about it. It should be expected.
(Source)

MikeeUSA is a troll. He has no means to actually harm anyone, and he does it purely for the lulz.

Thus, MikeeUSA trolled a woman, and she took the bait. I just don't get why this is news, I've been trolled before, I don't get a news story.

(Source)

I was going to start a rant about how this behavior is encouraged by the macho men online, but this was just one guy harassing her. "Due to harassment" reads as due to harassment from the community, but she gave in to one idiot. She let him win.
(Source)

The full comment thread has rather more examples. If you stand up and say anything controversial, you should expect abuse. And if you let that abuse change your behaviour in any way, you've let the trolls win.

These attitudes are problematic.

The immediate assumption underlying such advice is that the degree of abuse is related to what you've said, not who you are. I'm reasonably visible in the geek world. I've said a few controversial things. The worst thing that's happened to me has been Ryan Farmer deciding to subscribe me to several thousand mailing lists. Inconvenient, but not really threatening. I haven't, for instance, been sent death threats. Nobody has threatened to rape me. And even if they had done, I wouldn't need to worry too much - there's a rather stronger track record of violent antifeminism being targeted at women than men.

I don't have to worry about this kind of thing. That means I don't get to tell other people that they should have expected it. Nor do I get to tell them that they should ignore it, or that if they don't call the police then they have no grounds to complain. And nor does anyone else.

The trolls don't win because someone decides that getting out of the tech business is more appealing than continuing to face abuse. The trolls win because we consider their behaviour acceptable and then blame the victim for inviting them in the first place. That needs to change.

[1] It was justifiably pointed out that saying all this while standing on stage next to a mostly naked guy wearing a loincloth with a raccoon tail covering his penis may have weakened my message somewhat.

comment count unavailable comments

Syndicated 2011-10-28 07:29:01 from Matthew Garrett

Management of UEFI secure booting

This post is made in a personal capacity. While it represents the outcome of multiple discussions on the topic with a variety of people, nothing written below should be construed as Red Hat's corporate position

The FSF have released a statement on UEFI secure boot. It explains the fundamental issue here, which isn't something as simple as "will OEMs let me install Linux". It's "Does the end user have the ability to manage their own keys".

Secure boot is a valuable feature. It does neatly deal with the growing threat of pre-OS malware. There is an incentive for it to be supported under Linux. I discussed the technical aspects of implementing support for it here - it's not a huge deal of work, and it is being worked on. So let's not worry about that side of things. The problem is with the keys.

Secure boot is implemented in a straightforward way. Each section of a PE-COFF file is added together and a hash taken[1]. This hash is signed with the private half of a signing key and embedded into the binary. When you attempt to execute a file under UEFI, the firmware attempts to decrypt the embedded hash. This requires that the firmware have a either a copy of the public half of the signing key in its key database, or for their to be a chain of trust from the signing key to a key in its key database. Once it has the decrypted hash, it generates its own hash of the binary and compares them. If they match, the binary is executed.

What happens if it doesn't match? Per the UEFI specification, the firmware can then prompt the user and ask if they want to execute it anyway. If the user accepts then the hash of the binary is remembered[4] and can continue to be executed in future. This is similar to what you get when you visit a self-signed https site, or when you connect to an ssh server for the first time - the user must explicitly state that they trust the software that is being booted.

This is a little awkward for a couple of reasons. First, it means that any updates to the bootloader would require the user to manually accept the binary again. Second, as we've seen with https, there's the risk of uesrs just blindly accepting things. If it's acceptable to do this off internal media[5] then malware could just rely on some number of users saying "yes". And thirdly, we've been told that Microsoft's Windows logo requirements forbid this option.

The first is problematic but manageable with appropriate documentation. The second is a real risk that makes this approach far less compelling. And the third is probably a showstopper.

So signed but untrusted binaries are probably out, on technological, UI and policy grounds. What else?

We can sign binaries with a trusted key. This gets us interoperability, but comes at a cost. The first issue here is that we still don't know what the process for getting a trusted key is going to be. One option is for something like a generic Linux key that's provided by all firmware vendors. As long as the OEM doesn't remove it, then that Linux key could be used to sign other keys that are used by individual Linux vendors. The main difficulty involved in this is having the Linux key be managed by an authority that OEMs can trust to only provide keys to trustworthy organisations and maintain control of the private half of the key. Remember, if any malware is signed with any of these keys, it'll boot on any machine carrying this key. It's a lot of trust. It's not obvious that there's an organisation with sufficient trust at the moment.

The second approach is for each Linux vendor to generate their own key and get every OEM to carry it. Let's rule this out on the basis that it's an approach that doesn't scale at all. Red Hat could probably get its key into a bunch of OEM systems. Canonical too. Oracle? Probably for the vendors they care about. It's likely to be much harder for everyone else, and you still end up with the risk that you've just bought a computer that'll boot Red Hat but not Ubuntu.

The third approach is for each Linux vendor to obtain a key from someone who's already trusted. The most obvious authority here is Microsoft. Ignoring the PR unpleasantness involved with Linux vendors having to do business with Microsoft to get signed, so far Microsoft haven't given us any indication that this will be possible.

So every approach that involves us signing with a trusted key comes with associated problems. There's also a common problem with all of them - you only get to play if you have your own trusted key. Corporate Linux vendors can probably find a way to manage this. Everyone else (volunteer or hobbyist vendors, end users who want to run their own OS, anyone who wants to use a modified bootloader or kernel) is shut out.

The obvious workaround is for them to just turn off secure boot. Ignoring the arguments over whether or not OEMs will provide that option[6], it benefits nobody for Linux installation to require disabling a legitimate security feature. It's also not likely to be in a standard location on all systems and may have different naming. It's a support nightmare. Let's focus on trying to find a solution that provides the security and doesn't have obvious scaling issues.

There's a pretty simple one. If the problem centres around installing signing keys, why not have a standardised way of installing signing keys?

What we've proposed for this is an extension of the current UEFI standard for booting off removable media. There's a standardised location for putting the bootloader for USB sticks or optical media. For 32-bit PCs, it's EFI/BOOT/BOOTIA32.EFI. For 64-bit, it's EFI/BOOT/BOOTX64.EFI. Our proposal is to specify a path for key storage on the media. If the user attempts to boot off an item of removable media and finds a key file, we would like the firmware to prompt the user to install the key. Once the key is installed, the firmware will attempt to boot the bootloader on the removable media.

How does this avoid the problems associated with prompting the user to boot untrusted binaries? The first is that there's no problem with updates. Because a key has been imported, as long as future bootloader updates are signed with the same key, they'll boot without prompting the user. The second is that it can be limited to removable media. If malware infects the system and installs itself onto the hard drive, the firmware won't prompt for key installation. It'll just refuse to boot and fall back on whatever recovery procedures the OEM has implemented. The only way it could get on the system would involve the user explicitly booting off removable media, which would be a significant hurdle. If you're at that stage then you can also convince the user to disable secure boot entirely.

This proposal has been brought up with the UEFI standards working group, and with the UEFI plugfest taking place next week we're hoping it'll be discussed. It seems to satisfy the dual requirements of maintaining system security and ensuring that users have the flexibility to choose what they run on their system. But as with all technical proposals, feedback is entirely welcome.

[1] Hilariously, Microsoft's signing tool gets this wrong by also adding the contents of gaps between sections in direct contravention of their own specification. This is fine for binaries generated by Microsoft's toolchain because they don't have any gaps, but since our binaries do contain gaps[2] and since the standard firmware implementation[3] does implement the specification correctly, any Linux-generated binaries signed with the Microsoft tool fail validation. Go team.
[2] Something that is, as far as we can tell, permitted by the PE-COFF specification
[3] Written by Intel, not Microsoft
[4] Note that this means that only the specific binary is considered trusted. No key is imported, and any other binaries signed with the same key will also need to be manually trusted in the same way.
[5] As would be required if you ever want to be able to update your bootloader
[6] And to forestall panic, at this point we expect that most OEMs will provide this option on most hardware, if only because customers will still want to boot Windows 7. We do know that some hardware will ship without it. It's not implausible that some OEMs will remove it in order to reduce their support burden. But for the foreseeable future, you'll be able to buy hardware that runs Linux.

comment count unavailable comments

Syndicated 2011-10-19 17:58:19 from Matthew Garrett

Margaret Dayhoff

It's become kind of a cliché for me to claim that the reason I'm happy working on ACPI and UEFI and similarly arcane pieces of convoluted functionality is that no matter how bad things are there's at least some form of documentation and there's a well-understood language at the heart of them. My PhD was in biology, working on fruitflies. They're a poorly documented set of layering violations which only work because of side-effects at the quantum level, and they tend to die at inconvenient times. They're made up of 165 million bases of a byte code language that's almost impossible to bootstrap[1] and which passes through an intermediate representations before it does anything useful[2]. It's an awful field to try to do rigorous work in because your attempts to impose any kind of meaningful order on what you're looking at are pretty much guaranteed to be sufficiently naive that your results bear a resemblance to reality more by accident than design.

The field of bioinformatics is a fairly young one, and because of that it's very easy to be ignorant of its history. Crick and Watson (and those other people) determined the structure of DNA. Sanger worked out how to sequence proteins and nucleic acids. Some other people made all of these things faster and better and now we have huge sequence databases that mean we can get hold of an intractable quantity of data faster than we could ever plausibly need to, and what else is there to know?

Margaret Dayhoff graduated with a PhD in quantum chemistry from Columbia, where she'd performed computational analysis of various molecules to calculate their resonance energies[3]. The next few years involved plenty of worthwhile research that aren't relevant to the story, so we'll (entirely unfairly) skip forward to the early 60s and the problem of turning a set of sequence fragments into a single sequence. Dayhoff worked on a suite of applications called "Comprotein". The original paper can be downloaded here, and it's a charming look back at a rigorous analysis of a problem that anyone in the field would take for granted these days. Modern fragment assembly involves taking millions of DNA sequence reads and assembling them into an entire genome. In 1960, we were still at the point where it was only just getting impractical to do everything by hand.

This single piece of software was arguably the birth of modern bioinformatics, the creation of a computational method for taking sequence data and turning it into something more useful. But Dayhoff didn't stop there. The 60s brought a growing realisation that small sequence differences between the same protein in related species could give insight into their evolutionary past. In 1965 Dayhoff released the first edition of the Atlas of Protein Sequence and Structure, containing all 65 protein sequences that had been determined by then. Around the same time she developed computational methods for analysing the evolutionary relationship of these sequences, helping produce the first computationally generated phylogenetic tree. Her single-letter representation of amino acids was born of necessity[4] but remains the standard for protein sequences. And the atlas of 65 protein sequences developed into the Protein Information Resource, a dial-up database that allowed researchers to download the sequences they were interested in. It's now part of UniProt, the world's largest protein database.

Her contributions to the field were immense. Every aspect of her work on bioinformatics is present in the modern day — larger, faster and more capable, but still very much tied to the techniques and concepts she pioneered. And so it still puzzles me that I only heard of her for the first time when I went back to write the introduction to my thesis. She's remembered today in the form of the Margaret Oakley Dayhoff award for women showing high promise in biophysics, having died of a heart attack at only 57.

I don't work on fruitflies any more, and to be honest I'm not terribly upset by that. But it's still somewhat disconcerting that I spent almost 10 years working in a field so defined by one person that I knew so little about. So my contribution to Ada Lovelace day is to highlight a pivotal woman in science who heavily influenced my life without me even knowing.

[1] You think it's difficult bringing up a compiler on a new architecture? Try bringing up a fruitfly from scratch.
[2] Except for the cases where the low-level language itself is functionally significant, and the cases where the intermediate representation is functionally significant.
[3] Something that seems to have involved a lot of putting punch cards through a set of machines, getting new cards out, and repeating. I'm glad I live in the future.
[4] The three-letter representation took up too much space on punch cards

comment count unavailable comments

Syndicated 2011-10-07 15:47:48 from Matthew Garrett

Supporting UEFI secure boot on Linux: the details

An obvious question is why Linux doesn't support UEFI secure booting. Let's ignore the issues of key distribution and the GPL and all of those things, and instead just focus on what would be required. There's two components - the signed binary and the authenticated variables.

The UEFI 2.3.1 spec describes the modification to the binary format required to produce a signed binary. It's not especially difficult - you add an extra entry to the image directory, generate a hash of the entire binary other than the checksum, the certificate directory entry and the signatures themselves, encrypt that hash with your key and embed the encrypted hash in the binary. The problem has been that there was a disagreement between Microsoft and Intel over whether this signature was supposed to include the PKCS header or not, and until earlier this week the only widely available developer firmware (Intel's) was incompatible with the only widely available signed OS (Microsoft's). There's further hilarity in that the specification lists six supported hash algorithms, but the implementations will only accept two. So pretty normal, really. Developing towards a poorly defined target is a pain. Now that there's more clarity we'll probably have a signing tool before too long.

Authenticated variables are the other part of the puzzle. If a variable requires authentication, the operating system's attempt to write it will fail unless the new data is appropriately signed. The key databases (white and blacklists) are examples of authenticated variables. The signing actually takes place in userspace, and the handoff between the kernel and firmware is identical for both this case and the unauthenticated case. The only problem in Linux's support here is that our EFI variable support was written to a pre-1.0 version of the EFI specification which stated that variables had a maximum size of 1024 bytes, and this limitation ended up exposed to userspace. So all we really need to do there is add a new interface to let arbitrary sized variables be written.

Summary: We don't really support secure boot right now, but that's ok because you can't buy any hardware that supports it yet. Adding support is probably about a week's worth of effort at most.

comment count unavailable comments

Syndicated 2011-09-23 15:48:20 from Matthew Garrett

UEFI secure booting (part 2)

Microsoft have responded to suggestions that Windows 8 may make it difficult to boot alternative operating systems. What's interesting is that at no point do they contradict anything I've said. As things stand, Windows 8 certified systems will make it either more difficult or impossible to install alternative operating systems. But let's have some more background.

We became aware of this issue in early August. Since then, we at Red Hat have been discussing the problem with other Linux vendors, hardware vendors and BIOS vendors. We've been making sure that we understood the ramifications of the policy in order to avoid saying anything that wasn't backed up by facts. These are the facts:


  • Windows 8 certification requires that hardware ship with UEFI secure boot enabled.
  • Windows 8 certification does not require that the user be able to disable UEFI secure boot, and we've already been informed by hardware vendors that some hardware will not have this option.
  • Windows 8 certification does not require that the system ship with any keys other than Microsoft's.
  • A system that ships with UEFI secure boot enabled and only includes Microsoft's signing keys will only securely boot Microsoft operating systems.

Microsoft have a dominant position in the desktop operating system market. Despite Apple's huge comeback over the past decade, their worldwide share of the desktop market is below 5%. Linux is far below that. Microsoft own well over 90% of the market. Competition in that market is tough, and vendors will take every break they can get. That includes the Windows logo program, in which Microsoft give incentives to vendors to sell hardware that meets their certification requirements. Vendors who choose not to follow the certification requirements will be at a disadvantage in the marketplace. So while it's up to vendors to choose whether or not to follow the certification requirements, Microsoft's dominant position means that they'd be losing sales by doing so.

Why is this a problem? Because there's no central certification authority for UEFI signing keys. Microsoft can require that hardware vendors include their keys. Their competition can't. A system that ships with Microsoft's signing keys and no others will be unable to perform secure boot of any operating system other than Microsoft's. No other vendor has the same position of power over the hardware vendors. Red Hat is unable to ensure that every OEM carries their signing key. Nor is Canonical. Nor is Nvidia, or AMD or any other PC component manufacturer. Microsoft's influence here is greater than even Intel's.

What does this mean for the end user? Microsoft claim that the customer is in control of their PC. That's true, if by "customer" they mean "hardware manufacturer". The end user is not guaranteed the ability to install extra signing keys in order to securely boot the operating system of their choice. The end user is not guaranteed the ability to disable this functionality. The end user is not guaranteed that their system will include the signing keys that would be required for them to swap their graphics card for one from another vendor, or replace their network card and still be able to netboot, or install a newer SATA controller and have it recognise their hard drive in the firmware. The end user is no longer in control of their PC.

If Microsoft were serious about giving the end user control, they'd be mandating that systems ship without any keys installed. The user would then have the ability to make an informed and conscious decision to limit the flexibility of their system and install the keys. The user would be told what they'd be gaining and what they'd be giving up.

The final irony? If the user has no control over the installed keys, the user has no way to indicate that they don't trust Microsoft products. They can prevent their system booting malware. They can prevent their system booting Red Hat, Ubuntu, FreeBSD, OS X or any other operating system. But they can't prevent their system from running Windows 8.

Microsoft's rebuttal is entirely factually accurate. But it's also misleading. The truth is that Microsoft's move removes control from the end user and places it in the hands of Microsoft and the hardware vendors. The truth is that it makes it more difficult to run anything other than Windows. The truth is that UEFI secure boot is a valuable and worthwhile feature that Microsoft are misusing to gain tighter control over the market. And the truth is that Microsoft haven't even attempted to argue otherwise.

comment count unavailable comments

Syndicated 2011-09-23 13:01:12 from Matthew Garrett

UEFI secure booting

Since there are probably going to be some questions about this in the near future:

The UEFI secure boot protocol is part of recent UEFI specification releases. It permits one or more signing keys to be installed into a system firmware. Once enabled, secure boot prevents executables or drivers from being loaded unless they're signed by one of these keys. Another set of keys (Pkek) permits communication between an OS and the firmware. An OS with a Pkek matching that installed in the firmware may add additional keys to the whitelist. Alternatively, it may add keys to a blacklist. Binaries signed with a blacklisted key will not load.

There is no centralised signing authority for these UEFI keys. If a vendor key is installed on a machine, the only way to get code signed with that key is to get the vendor to perform the signing. A machine may have several keys installed, but if you are unable to get any of them to sign your binary then it won't be installable.

This impacts both software and hardware vendors. An OS vendor cannot boot their software on a system unless it's signed with a key that's included in the system firmware. A hardware vendor cannot run their hardware inside the EFI environment unless their drivers are signed with a key that's included in the system firmware. If you install a new graphics card that either has unsigned drivers, or drivers that are signed with a key that's not in your system firmware, you'll get no graphics support in the firmware.

Microsoft requires that machines conforming to the Windows 8 logo program and running a client version of Windows 8 ship with secure boot enabled. The two alternatives here are for Windows to be signed with a Microsoft key and for the public part of that key to be included with all systems, or alternatively for each OEM to include their own key and sign the pre-installed versions of Windows. The second approach would make it impossible to run boxed copies of Windows on Windows logo hardware, and also impossible to install new versions of Windows unless your OEM provided a new signed copy. The former seems more likely.

A system that ships with only OEM and Microsoft keys will not boot a generic copy of Linux.

Now, obviously, we could provide signed versions of Linux. This poses several problems. Firstly, we'd need a non-GPL bootloader. Grub 2 is released under the GPLv3, which explicitly requires that we provide the signing keys. Grub is under GPLv2 which lacks the explicit requirement for keys, but it could be argued that the requirement for the scripts used to control compilation includes that. It's a grey area, and exploiting it would be a pretty good show of bad faith. Secondly, in the near future the design of the kernel will mean that the kernel itself is part of the bootloader. This means that kernels will also have to be signed. Making it impossible for users or developers to build their own kernels is not practical. Finally, if we self-sign, it's still necessary to get our keys included by ever OEM.

There's no indication that Microsoft will prevent vendors from providing firmware support for disabling this feature and running unsigned code. However, experience indicates that many firmware vendors and OEMs are interested in providing only the minimum of firmware functionality required for their market. It's almost certainly the case that some systems will ship with the option of disabling this. Equally, it's almost certainly the case that some systems won't.

It's probably not worth panicking yet. But it is worth being concerned.

comment count unavailable comments

Syndicated 2011-09-20 18:23:20 from Matthew Garrett

308 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!