Older blog entries for mjg59 (starting at number 393)

Hacker News is a social echo chamber

Hacker News is a fairly influential link aggregation site, with stories submitted and voted on by users. As explained in the FAQ, the ranking of stories is roughly determined by the number of votes divided by a function of the time since submission. It's not a huge traffic driver (my personal experience of stories on the front page is on the order of 30,000 hits), but it's notable because the demographic tends to include a more technically literate and influential set of readers than most other sites. The discussion that ensues from technical posts often includes meaningful feedback from domain experts. Stories that appear there are likely to be noted by technology workers, especially in the Silicon Valley startup field[1].

That rather specific demographic appears to correlate with other traits. There's a rather more techno-libertarian bias on Hacker News than on most general discussion forums, which is consistent with the startup-oriented culture that it springs from - the desire to provide disruptive solutions to real world problems tends to collide with existing regulatory frameworks, so it's unsurprising that a belief in individual rights and small government would overlap with US startup culture. There's a leaning towards the use of web technologies rather than traditional client applications, which matches what people are doing in the rest of the world. And there's more enthusiasm for liberal open-source licenses over Copyleft licenses, which makes sense in a web-focused environment (as I wrote about here).

Now, personally I'm a big-government, client-app, Copyleft kind of person, but for the most part I don't think the above is actively dangerous. It's inevitable that political views will vary, we'll probably continue to cycle between thick and thin clients for generations and nobody's ever going to demonstrably prove that one licensing model deserves to win over another. But what is important is that the ongoing debates between these opinions be driven by facts, and that it remain obvious that these disagreements exist. As far as technical (and even political) discussion goes, Hacker News doesn't seem to
have a problem with that. Disagreeing with the orthodoxy is tolerated.


This seems less true when it comes to social issues. When a posting discussing the myth of the natural born programmer[2] hit the front page, the top rated comment is Paul Graham[3] off-handedly discounting the conclusions drawn. The original story linked to a review of peer-reviewed scientific research. Graham simply discounts it on the basis of his preconceptions. Shortly afterwards, the story plummeted off the front page, now surrounded by stories posted around the same time but with much lower scores.

How does this happen? There's two publicised methods which can result in stories dropping down the order. Users with high karma scores (either via submitting popular stories or writing popular comments) are able to flag submissions, and if enough do so then a negative weighting is applied to the story. There's also a flamewar detector, a heuristic that attempts to detect contentious subjects and pushes them off the front page.

The effect of both is to enforce the status quo of social beliefs. Stories that appear to challenge the narrative that good programmers are just naturally talented tend to vanish. Stories that discuss the difficulties faced by minorities in our field are summarily disappeared. There are no social problems in the technology industry. We have always been at war with Eastasia.

This isn't healthy. We don't improve the state of the software industry by hiding stories that expose conflicts. Flamewars don't solve problems, but without them we'd be entirely unaware of how much of a victim blaming mentality exists amongst our peers. It's true that conflict may reinforce preconceptions, causing people to dig in as they defend their beliefs. However, the absence of conflict does nothing to counteract that. If you're never exposed to opinions you disagree with, you'll never question your existing beliefs.

Hacker News is a privately run site and nobody's under any obligation to change how they choose to run it. But the focus on avoiding conflict to such an extent that controversial stories receive less exposure than ones that fit people's existing beliefs doesn't enhance our community. If we want to be able to use technology as an instrument of beneficial change to society as a whole, we benefit from building a diverse and welcoming community and questioning our preconceptions. Building a social echo chamber risks marginalising us from the rest of society, gradually becoming ignored and irrelevant as our self-reinforcing opinions drift ever further away from the mainstream. It doesn't help anybody.

[1] During the batch of interviews I did last year, two separate interviewers both mentioned a story they'd read on Hacker News that turned out to have been written by me. I'm not saying that that's what determined a hire/don't hire decision, but it seems likely that it helped.
[2] The article in question discusses the pervasive idea that some people are inherently good programmers. It turns out that perpetuating the idea that some people are just born good at a particular skill actually discourages others from even trying to learn it, even those who are just as capable.
[3] One of the co-founders of Y-Combinator and creator of Hacker News.

(Edited to fix a footnote reference)

comment count unavailable comments

Syndicated 2013-10-15 14:11:21 from Matthew Garrett

The state of XMir

XMir's been delayed from Ubuntu 13.10. The stated reason is that multi-monitor support isn't sufficiently reliable. That's true, but it's far from the only problem that XMir still has:


  • It's still broken on some single-monitor systems

  • Some machines have display corruption. There are writes to freed memory. To be fair, some of the behaviour that's been seen has been down to underlying bugs in the Xorg drivers that were never triggered under normal use but are hit by XMir. Others are down to implicit assumptions made in the drivers that XMir happens to violate. The problem is that there doesn't appear to have been enough room in the schedule to deal with these interactions, presumably because nobody accounted for the inevitable "This thing we thought would be easy turns out to be difficult" part of the project.

  • The input driver bug still isn't really fixed

  • I've mentioned this bug before. It's now marked "Fix released" which is kind of true but not really. Mir now tells XMir that the VT is changing before the VT is changed, but it doesn't wait for that to complete before switching the VTs. Until XMir is scheduled and runs, it's still receiving input events. In most circumstances that window is small, so there's no real risk of triggering it accidentally.

    There's one corner case where it might still cause problems. Simply running isn't enough - XMir has to make progress through its event loop. That'll only happen if the X server is processing its wakeup events, and it's possible to effectively stall that by submitting a sufficiently awkward drawing request to the server. The X server will appear to freeze, and if the user then attempts to work out what's going on by switching to a VT and logging in, those input events will still be going to XMir. It's left as an exercise to the reader to construct ways to take advantage of this.

    This can't happen in Xorg because the VT switch is blocked until the input devices have been released. Mir could have done the same, but doesn't because of a conscious design decision - in the Ubuntu Phone world, clients stop doing things when they're told to. Ubuntu Desktop is expected to behave the same way.

    This is an unfortunate situation to be in. Ubuntu Desktop was told that they were switching to XMir, but Mir development seems to be driven primarily by the needs of Ubuntu Phone. XMir has to do things that no other Mir client will ever cope with, and unless Mir development takes that into account it's inevitably going to suffer breakage like this. Canonical management needs to resolve this if there's any hope of ever shipping XMir as the default desktop environment.

  • It's still missing features

  • XMir doesn't support colour profiles. XRandR properties aren't exposed, so there's no way to control TV output encoding or overscan. There's still no hardware cursor support. Switching to XMir now would reduce functionality without providing any user-visible gain.


It's clear that XMir has turned into a larger project than Canonical had originally anticipated, but that's hardly surprising. There's only one developer with previous X experience working on it full-time, and the announced schedule provided no opportunity to deal with unexpected problems. As if that weren't enough, there's obvious conflict between Ubuntu Desktop and Ubuntu Phone when it comes to developer time and required functionality. The one hardware vendor who's willing to take a public stand has demonstrated that they have no interest in supporting XMir, despite Canonical assuring people that they were already engaging in productive negotiations with GPU manufacturers.

Multiple monitor support may be the single most obvious feature where XMir is lacking, but it's not the only one. Technically and politically, this code is a long way from being ready for prime time. Without significant community contribution, Canonical need to focus on prioritising it appropriately rather than expecting a tiny number of developers to perform miracles. Or, alternatively, they could just drop XMir entirely and focus on Unity 8.

comment count unavailable comments

Syndicated 2013-10-02 21:31:51 from Matthew Garrett

Implementing UEFI Boot to Zork

One of the earlier examples of students using MIT computer resources to lay the groundwork for a later commercial endeavour, Zork was originally written in a LISP derivative called MDL. This was later tuned into the Zork Implementation Language, a domain specific language that was compiled to target the Z-machine rather than a specific piece of hardware. Combined with machine-specific Z-machine interpreters, this allowed rapid porting of games to a wide range of platforms - the only thing that needed to be rewritten was the interpreter, and that could be reused for any future games running on the same hardware.

Infocom were eventually acquired and killed off, but fan interest in their games continued. New Z-machine interpreters were written in order to allow their games (including Zork) to be run on platforms that Infocom had never targetted. One of the best known is Frotz. This has the advantage of being (a) portable and (b) including a "dumb" UI that makes no assumptions about the availablity of any vaguely useful functionality. Like, say, a Curses library.

So, Frotz seemed like the natural choice when this happened. But despite having a set of functionality that makes it look much more like an OS than a boot environment, UEFI doesn't actually expose a standard C library. The EFI Application Development Kit solves this particular design decision. Porting Frotz ended up involving far more fixing up of Frotz bugs that tripped up -Werror than anything else. One note, though - make sure you include DevShell in the list of required packages at build time, otherwise file i/o will mysteriously fail.

The tying of file i/o to the shell protocol unfortunately means that Frotz can't be directly launched by the firmware. The Boot to Zork images therefore contain a UEFI shell in the standard boot location (\EFI\BOOT\BOOTX64.EFI) which is executed when the firmware attempts to boot the device. The shell then looks for a file called "startup.nsh" in the root directory of the boot device and executes it. Unfortunately this doesn't actually set the shell equivalent of the current device, and so just launching Frotz from startup.nsh fails when Frotz can't open the Zork data file. The solution for this is simple, if ugly - the script walks through the list of devices, looking for one that contains ZORK1.DAT in the root directory. It then changes to that device and launches Frotz. If Frotz exits, it then resets the system.

This could be avoided by doing some more work and turning Frotz into a more UEFI-native application. Teaching Frotz to make native UEFI calls would avoid the requirement for the shell protocols, and the firmware provides a mechanism to obtain the path of the currently running executable which would avoid the need to explicitly locate the device. But I'm lazy and this was a "I'm spending the day on a plane" project initially inspired by a Sazerac-fuelled conversation during the UEFI plugfest, not a demonstration of UEFI best practices.

UEFI Boot To Zork and the source code to the modified Frotz can be downloaded here.

comment count unavailable comments

Syndicated 2013-09-22 18:17:46 from Matthew Garrett

Re: Default offerings, target audiences, and the future of Fedora

Eric (a fellow Fedora board member) has a post describing his vision for what Fedora as an end goal should look like. It's essentially an assertion that since we have no idea who our users are or what they want, we should offer them everything on an equal footing.

Shockingly enough, I disagree.

At the most basic level, the output of different Special Interest Groups is not all equal. We've had issues over the past few releases where various spins have shipped in a broken state, because the SIG responsible for producing them doesn't have the resources to actually test them. We're potentially going to end up shipping F20 with old Bluetooth code because the smaller desktops aren't able to port to the new API in time[1]. Promoting these equally implies that they're equal, and doing so when we know it isn't the case is a disservice to our users.

But it's not just about our users. Before I joined the Fedora project, I'd worked on both Debian and Ubuntu. Debian is broadly similar to the current state of Fedora - no strong idea about what is actually being produced, and a desire among many developers to cater to every user's requirements. Ubuntu's pretty much the direct opposite, with a strongly defined goal and a willingness to sacrifice some use cases in order to achieve that goal.

This leads to an interestingly different social dynamic. Ubuntu contributors know what they're working on. If a change furthers the well-defined aim of the project, that change happens. Moving from Ubuntu to Fedora was a shock to me - there were several rough edges in Fedora that simply couldn't be smoothed out because fixing them for one use case would compromise another use case, and nobody could decide which was more important[2]. It's basically unthinkable that such a situation could arise in Ubuntu, not just because there was a self appointed dictator but because there was an explicit goal and people could prioritise based on that[3].

Bluntly, if you have a well-defined goal, people are more likely to either work towards that goal or go and do something else. If you don't, people will just do whatever they want. The risk of defining that goal is that you'll lose some of your existing contributors, but the benefit is that the existing contributors will be more likely to work together rather than heading off in several different directions.

But perhaps more importantly, having a goal can attract people. Ubuntu's Bug #1 was a solid statement of intent. Being freer than Microsoft wasn't enough. Ubuntu had to be better than Microsoft products on every axis, and joining Ubuntu meant that you were going to be part of that. Now it's been closed and Ubuntu's wandered off into convergence land, and signing up to spend your free time on producing something to help someone sell phones is much less compelling than doing it to produce a product you can give to your friends.

Fedora should be the obvious replacement, but it's not because it's unclear to a casual observer what Fedora actually is. The website proudly leads with a description of Fedora as a fast, stable and powerful operating system, but it's obvious that many of the community don't think of Fedora that way - instead it's a playground to produce a range of niche derivatives, with little consideration as to whether contributing to Fedora in that way benefits the project as a whole. Codifying that would actively harm our ability to produce a compelling product, and in turn reduce our ability to attract new contributors even further.

Which is why I think the current proposal to produce three first-class products is exciting. Offering several different desktops on the download page is confusing. Offering distinct desktop, server and cloud products isn't. It makes it clear to our users what we care about, and in turn that makes it easier for users to be excited about contributing to Fedora. Let's not make the mistake of trying to be all things to all people.

[1] Although clearly in this case the absence of a stable ABI in BlueZ despite it having had a dbus interface for the best part of a decade is a pretty fundamental problem.
[2] See the multi-year argument over default firewall rules and the resulting lack of working SMB browsing or mDNS resolving
[3] To be fair, one of the reasons I was happy to jump ship was because of the increasingly autocratic way Ubuntu was being run. By the end of my involvement, significant technical decisions were being made in internal IRC channels - despite being on the project's Technical Board, I had no idea how or why some significant technical changes were being made. I don't think this is a fundamental outcome of having a well-defined goal, though. A goal defined by the community (or their elected representatives) should function just as well.

comment count unavailable comments

Syndicated 2013-08-22 15:37:56 from Matthew Garrett

If you ever use text VTs, don't run XMir right now

It'd be easy to assume that in a Mir-based world, the Mir server receives input events and hands them over to Mir clients. In fact, as I described here, XMir uses standard Xorg input drivers and so receives all input events directly. This led to issues like the duplicate mouse pointer seen in earlier versions of XMir - as well as the pointer being drawn by XMir, Mir was drawing its own pointer.

But there's also some more subtle issues. Mir recently gained a fairly simple implementation of VT switching, simply listening for input events where a function key is hit while the ctrl and alt modifiers are set[1]. It then performs the appropriate ioctl on /dev/console and the kernel switches the VT. The problem here is that Mir doesn't tell XMir that this has happened, and so XMir still has all of its input devices open and still pays attention to any input events.

This is pretty easy to demonstrate. Open a terminal or text editor under Xmir and make sure it has focus. Hit ctrl+alt+f1 and log in. Hit ctlr+alt+f7 again. Your username and password will be sitting in the window.

This is Launchpad bug 1192843, filed on the 20th of June. A month and a half later, Mir was added to the main Ubuntu repositories. Towards the bottom, there's a note saying "XMir always listening to keyboard, passwords may appear in other X sessions". This is pretty misleading, since "other X sessions" implies that it's only going to happen if you run multiple X sessions. Regardless, it's a known bug that can potentially leak user passwords.

So it's kind of odd that that's the only mention of it, hidden in a disused toilet behind a "Doesn't work on VESA" sign. If you follow the link to installation instructions you get this page which doesn't mention the problem at all. Now, to be fair, it doesn't mention any of the other problems with Mir either, but the other problems merely result in things not working rather than your password ending up in IRC.

This being developmental software isn't an excuse. There's been plenty of Canonical-led publicity about Mir and people are inevitably going to test it out. The lack of clear and explicit warnings is utterly inexcusable, and these packages shouldn't have landed in the archive until the issue was fixed. This is brutally irresponsible behaviour on the part of Canonical.

So, if you ever switch to a text VT, either make sure you're not running XMir at the moment or make sure that you never leave any kind of network client focused when you switch away from X. And you might want to check IRC and IM logs to make sure you haven't made a mistake already.

[1] One lesser-known feature of X is that the VT switching events are actually configured in the keymap. ctrl+alt+f1 defaults to switching to VT1, but you can remap any key combination to any VT switch event. Except, of course, this is broken in XMir because Mir catches the keystroke and handles it anyway.

comment count unavailable comments

Syndicated 2013-08-22 06:36:17 from Matthew Garrett

When to add a platform-specific quirk to a Linux driver

If you're tempted to add a platform-specific quirk to a Linux driver, pause and do the following:


  1. Check whether the platform works correctly with the generic Windows driver for the hardware in question. If it requires a platform-specific driver rather than the generic one, adding a quirk is probably ok.
  2. If the generic Windows driver works, check whether there's any evidence of platform-specific code in the Windows driver. This will typically be in the .inf file, but occasionally you'll want to run strings against the Windows driver and see whether any functions or strings match the platform in question. If there's evidence of special-casing in the generic Windows driver, adding a quirk is probably ok
  3. If the generic Windows driver works and doesn't appear to have any platform-specific special casing, don't add a quirk. You'll plausibly fix the machine you care about, but you won't fix any others that have the same behaviour. Even worse, if someone does eventually fix the problem properly, there's a risk that your special-casing will now break your system.

The moral to this story is: if you think adding a quirk is the right solution, you're almost certainly wrong.

comment count unavailable comments

Syndicated 2013-08-10 04:30:08 from Matthew Garrett

Don't ship 32-bit UEFI firmware on x86

Shipping a UEFI-bootable Linux distribution is a touch awkward[1], with the main sticking point being the necessity to produce boot media with multiple El-Torito images. An El-Torito image is either an image of a floppy disk or a small hard drive, embedded into the ISO. This allows BIOS systems to look for an El Torito image, hook some interrupts and then boot without the BIOS having to care about the fact that the image is embedded in an ISO-9660 filesystem. UEFI systems will look for an El-Torito image with a special tag - if they find it they'll mount it as a FAT filesystem and look for a bootloader, and if not they'll fall back to BIOS compatibility.

So, if you want a CD that'll boot on both BIOS and UEFI systems, you need two El-Torito images - one for BIOS, one for UEFI. Unfortunately, various BIOSes deal exceptionally badly with CD images that contain more than one El-Torito image. The most common failure case is to print a menu asking you to choose an option without labelling the options, but some will just fail outright. Thankfully, this is pretty much exclusively limited to 32-bit systems.

Things get irritatingly more complicated due to a quirk of UEFI. UEFI is based on executing code in native mode. That means that 32-bit UEFI systems can't execute 64-bit code in firmware, even if the CPU is capable of it. A 64-bit OS can only boot on 32-bit UEFI if it has very ugly compatibility hacks, including having to rewrite structures and register state every time it makes a UEFI call. The only OS I'm aware of that implements this is MacOS X. Having looked into what it'd take to implement it in Linux, I decided that hammering rusty nails through my feet would be a preferable use of time. Thankfully, I went drinking instead.

So distributions have a choice. They can either produce UEFI-bootable CD images for 32-bit x86 and risk failures on actual 32-bit systems, or they can ignore UEFI entirely on 32-bit and succeed in booting on all the hardware that people actually own[2]. Unsurprisingly, they tend to choose the second option.

So, if you're building an x86 hardware platform, don't ship with 32-bit UEFI. If you're stuck with a 32-bit CPU then just ship BIOS. If you have a 64-bit CPU then ship a 64-bit UEFI. If you ship with 32-bit UEFI, no significant existing Linux distribution will support you, and you'll face an uphill battle to convince them to do so.

32-bit UEFI. Just say what on earth were you thinking, please, no, can't you find a solution that doesn't involve me getting tetanus jabs.

(If you're worried about the extra memory consumption of 64-bit OSes, just encourage 32-bit userspace on a 64-bit kernel. Or just boot via BIOS)

[1] See the number that still don't manage it despite having had several years to adapt
[2] Until recently, the only vendor to ship 32-bit UEFI firmware on volume hardware was Apple. This was fine on their 32-bit systems, but on their 64-bit systems with 32-bit UEFI booting a 64-bit version of Windows would result in boot failure. Apple rectified this by stating that 64-bit Windows wasn't supported on platforms with 32-bit UEFI, which is a neat trick if you can manage it. 32-bit Windows (and Linux) was fine because it didn't include a UEFI boot image and so didn't trigger the bug.

comment count unavailable comments

Syndicated 2013-08-01 05:30:30 from Matthew Garrett

ARM and firmware specifications

Jon Masters, Chief ARM Architect at Red Hat, recently posted a description of his expectations for baseline arm64 servers. The quick summary is that systems should implement UEFI and ACPI, and any more traditional ARM boot mechanisms should be ignored. This is an interesting departure from the status quo in the ARM world, and it's worth thinking about the benefits and drawbacks of this approach.

It's very easy to build a generic kernel for most x86 systems, since the PC platform is fairly well defined even if not terribly well specified. Where system hardware does vary, it's almost always exposed on an enumerable bus (such as PCI or USB) which allows the OS to bind appropriate drivers. Things are different in the ARM world. Even once you're past the point of different SoC vendors requiring different kernel setup code and drivers, you still have to cope with the fact that system vendors can wire these SoCs up very differently. Hardware is often attached via GPIO lines without any means to enumerate them. The end result is that you've traditionally needed a different kernel for every ARM board. This is viable if you're selling the OS and hardware as a single product, but less viable if there's any desire to run a generic OS on the hardware.

The solution that's been adopted for this in the Linux world is called Device Tree. Device Tree actually has significant history, having been used as the device descriptor format in Open Firmware. Since there was already support for it in the Linux kernel, adapting it for use in ARM devices was straightforward. Device Tree aware devices can pass a descriptor blob to the kernel at startup[1], and devices without that knowledge can have a blob build into the kernel.

So, if this problem is already solved, why the push to move to UEFI and ACPI? This push didn't actually originate in the Linux world - Microsoft mandate that Windows RT devices implement UEFI and ACPI, and were they to launch a Windows ARM server product would probably carry that over. That makes sense for Microsoft, since recent versions of Windows have been x86 only and so have grown all kinds of support for ACPI and UEFI. Supporting Device Tree would require Microsoft to rewrite large parts of Windows, whereas mandating UEFI and ACPI allowed them to reuse most of their existing Windows boot and driver code. As a result, largely at Microsoft's behest, ACPI 5 has grown a range of additional features for describing things like GPIO pinouts and I2C connections. Whatever your weird device layout, you can probably express it via ACPI.

This argument works less well for Linux. Linux already supports Device Tree, whereas it currently doesn't support ACPI or UEFI on ARM[2]. Hardware vendors are already used to working with Device Tree. Moving to UEFI and ACPI has the potential to uncover a range of exciting new kernel issues and vendor bugs. It's not obviously an engineering win.

So how about users? There's an argument that since server vendors are now mostly shipping ACPI and UEFI systems, having ARM support these technologies makes it easier for customers to replace x86 systems with ARM systems. This really doesn't fly for ACPI, which is entirely invisible to the user. There are no standard ACPI entry points for system configuration, and the bits of ACPI that are generically useful (such as configuring system wakeup times) are already abstracted away to a standard interface by the kernel. It's somewhat more compelling for UEFI. UEFI supports a platform-independent bytecode language (EFI Byte Code, or EBC), which means that customers can write their own system management utilities, build them for EBC and then deploy them to their servers without caring about whether they're x86 or ARM. Want a bootloader that'll hit an internal HTTP server in order to determine which system image to deploy, and which works on both x86 and ARM? Straightforward.

Arnd Bergmann has a interesting counterargument. In a nutshell, ARM servers aren't currently aiming for the same market as x86 servers, and as a result customers are unlikely to gain any significant benefit from shared functionality between the two.

So if there's no real benefit to users, and if there's no benefit to kernel developers, what's the point? The main one that springs to mind is that there is a benefit to distributions. Moving to UEFI means that there's a standard mechanism for distributions to interact with the firmware and configure the bootloader. The traditional ARM approach has been for vendors to ship their own version of u-boot. If that's in flash then it's not much of a problem[3], but if it's on disk then you have to ship a range of different bootloaders and know which one to install (and let's not even talk about initial bootstrapping).

This seems like the most compelling argument. UEFI provides a genuine benefit for distributions, and long term it probably provides some benefit to customers. The question is whether that benefit is worth the flux. The same distribution benefit could be gained by simply mandating a minimum set of u-boot functionality, which would seem much more straightforward. The customer benefit is currently unclear.

In the end it'll probably be a market decision. If Red Hat produce an ARM product that has these requirements, and if Suse produce an ARM product that will work with u-boot and Device Tree, it'll be up to vendors to decide whether the additional work to support UEFI/ACPI is worth it in order to be able to sell to customers who want Red Hat. I expect that large vendors like HP and Dell will probably do it, but the smaller ones may not. The customer demand issue is also going to be unclear until we learn whether using UEFI is something that customers actually care about, rather than a theoretical benefit.

Overall, I'm on the fence as to whether a UEFI requirement is going to stick, and I suspect that the ACPI requirement is tilting at windmills. There's nothing stopping vendors from providing a Device Tree blob from UEFI, and I can't think of any benefits they gain from using ACPI instead. Vendor interest in the generic parts of the ACPI spec has been tepid even in the x86 world (the vast majority of ACPI spec updates come from Microsoft and Intel, not any of the system vendors), and I don't see that changing with the introduction of a range of ARM vendors who are already happy with Device Tree.

We'll see. Linux is going to need to gain the support for UEFI and ACPI on ARM in any case, since there's already hardware shipping in that configuration. But with ARM vendors still getting to grips with Device Tree, forcing them to go through another change in how they do things is going to be hard work. Red Hat may be successful in enforcing these requirements at the cost of some vendor unhappiness, or Red Hat may find that their product doesn't boot on most of the available hardware. It's an aggressive gamble, and while it'll be interesting to see how it plays out, I'm not that optimistic.

[1] The blob could be pulled from the firmware, but it's not uncommon for it to be built into u-boot instead. This does mean that you have a device-specific u-boot even if you have a generic kernel, but that's typically true anyway.
[2] Patches have been posted for ARM UEFI support. They're not mergeable in their current form, but they should be in the near future. ACPI support is in development.
[3] Although not all u-boots are created equal - some vendors ship versions that will only boot off FAT, some vendors ship versions that will only boot off ext2. Having to special case this stuff in your installer is a pain.

comment count unavailable comments

Syndicated 2013-07-22 17:17:17 from Matthew Garrett

How XMir and Mir fit together

Mir is Canonical's new display server. It fulfils a broadly similar role to Wayland and Android's Surfaceflinger, in that it takes final responsibility for getting pixels onto the screen. XMir is an X server that runs on top of Mir. It permits applications that know how to speak the X protocol but don't know how to speak to Mir (ie, approximately all of them at present) to run in a Mir-based environment.

For Ubuntu 13.10, Canonical are proposing to use Mir by default. This doesn't mean that most applications will be using Mir, though - instead, the default session will run XMir as a full-screen client and a normal X environment will be run on that. This lets Canonical deploy Mir without forcing anyone to update their applications, allowing them to take a gradual approach. By 14.10, Canonical expect the default Unity session to be a Mir client rather than an X client. In theory it will then be possible to run an Ubuntu system without any X applications at all, leaving XMir to do nothing other than run legacy applications.

Of course, this requires a certain amount of replumbing. X would normally be responsible for doing things like setting up the screen and pushing the pixels out to the hardware, but this is now handled by Mir instead. Where a native X server would allocate a framebuffer in video memory and render into it, XMir asks Mir for a window corresponding to the size of the screen and renders into that and then simply asks Mir to display it. This step is actually more interesting than it sounds.

Unless you're willing to throw lots of CPU at them, unaccelerated graphics are slow. Even if you are, you're going to end up consuming more power for the same performance, so XMir would be impractical if it didn't provide access to accelerated hardware graphics functions. It makes use of the existing Xorg accelerated X drivers to do this, which is as simple as telling the drivers to render into the window that XMir requested from Mir rather than into the video framebuffer directly. In other words, when displaying through XMir, you're using exactly the same display driver stack as you would be if you were using Xorg. In theory you'd expect identical performance - in practice there's a 10-20% performance hit right now, but that's being actively worked on. Fullscreen 3D apps will also currently take a hit due to there being no support for skipping compositing, which is being fixed. XMir should certainly be capable of performing around as well as native X, but there's no reason for it to be any faster.

The output drivers aren't the only part of the stack. XMir still needs some way to get input events. You might naively expect that Mir would forward input events to XMir, but the current state of affairs is that XMir loads the existing X input drivers which in turn open the input devices themselves. This explains why XMir currently shows two cursors[1] - the first cursor is the Mir cursor that's being driven by the input events Mir is receiving, while the second is the cursor being drawn by XMir in response to the input events that it's receiving[2].

The final main piece of duplicated functionality is output management. In the case where X is its own display server, it can simply ask the kernel to program the outputs. XMir can't do that, so would have to ask Mir to do it instead. Sadly, that functionality's not implemented yet. Running xrandr on Xmir gives:

Screen 0: minimum 320 x 320, current 1366 x 768, maximum 8192 x 8192
XMIR-1 connected primary 1366x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
   XMIR mode of death[3]   60.0*+
which translates as "I have a screen that can be any size from 320x320 to 8192x8192, and have one output of unknown physical size that can only display 1366x768 at 60Hz". Other than the 1366x768, the result will be identical no matter how many outputs you have - XMir currently has no support for configuring multiple displays, and will always report 60Hz. It also has no support for placing monitors into any kind of DPMS state.

Obviously, this is still software under active development. There's three months from now until until Ubuntu is due to release with this functionality enabled by default, and the only significant missing features are xrandr and DPMS support. As long as Mir exposes interfaces for controlling those, they shouldn't be a problem to implement. To most users, the transition from native Xorg to XMir should be completely invisible.

Which is kind of the problem. What benefits does the user gain from using XMir on Mir rather than native Xorg?

Features? XMir is a total of around 1000 lines of code on top of Xorg. It implements no functionality that isn't present in Xorg.

Performance? XMir effectively is Xorg - it's the same code running the same drivers. It's not going to be any faster. At the moment it's actually slower, but that's probably fixable.

Security? Again, XMir is effectively Xorg. It still runs as root. It has the same privileges. There's no security benefit to using XMir instead of Xorg.

Like I said, the transition should be completely invisible - no drawbacks, but no benefits. What it does do is get Mir deployed on a larger number of systems, which means Canonical can both get wider testing of the basic Mir functionality and argue that the number of deployed Mir systems means hardware vendors should support it. Users won't see any of the benefits until Unity transitions to being a native Mir client, which is slated for 14.10 next year.

As a PR move, it's pretty great. The public perception is that Mir has gone from not existing to being sufficient to run a full desktop environment in very little time, and it's natural to compare it to Wayland which has been in development for longer and still isn't running a fully featured desktop session. The problem with that perception is that Mir is doing very little at the moment. It's setting an output mode (by asking the kernel to), allocating a window for XMir to draw into and then pushing that window onto the screen whenever XMir tells it that it's changed. This isn't advanced display server functionality, it's the bare minimum that a display server could possibly do[4]. Which is a shame, because Mir is significantly more functional than that, it's just not being used to do anything we couldn't already do.

In summary: XMir on Mir in Ubuntu provides no user benefits and isn't a compelling technology demo. Mir itself will permit a range of additional features, but isn't slated to be running a user session itself until 14.10. The only obvious benefit to Canonical in shipping XMir on Mir is to gain additional testing, which makes using it in 14.04, a supposedly stable and long term release, a somewhat surprising choice.

[1] Although this is in the process of being fixed
[2] This also explains why, on systems with a trackpoint and a touchpad, only the trackpoint moves the Mir cursor while both move the XMir cursor. The X touchpad driver has put the touchpad in absolute mode, which means it's now reporting events that Mir currently ignores.
[3] "XMIR mode of death" is hardcoded into XMir. I'm really hoping it's a Regular Show reference.
[4] So why hasn't anyone done this with Wayland? Mostly because, as described above, there's no technical benefit in doing so. Wayland does have an X server for compatibility purposes, and if you wanted you could run it as a full screen client and run an entire session underneath it. But you'd gain nothing by doing so. The Wayland X server is intended for running individual clients rather than an entire X session. Run an X client under Wayland and it'll pop up in its own individual window and managed by your Wayland session's window manager, just like it would under X. XMir currently has no support for this "rootless" mode - right now if you want to run X apps under Mir, you'll need to launch an entire X session with its own window manager.

comment count unavailable comments

Syndicated 2013-07-15 16:10:30 from Matthew Garrett

Explaining Intel Rapid Start Technology

Recent Intel-based systems often implement something called Intel Rapid Start Technology. Like many things with the word "Technology" in the name, there's a large part of this that's marketing. The relatively small amount of technical documentation available implies that it's tied to your motherboard chipset and CPU, but as far as I can tell it's entirely implemented in firmware and could work just as well on, say, a Cyrix on a circa 1996 SIS-based motherboard if someone wrote the BIOS code[1]. But since nobody has, we're stuck with the vendors who've met Intel's requirements and licensed the code.

The concept of IRST is pretty simple. There's a firmware mechanism for setting a sleep timeout. If you suspend your computer and this timeout expires, it'll resume. However, instead of handing control back to the OS, the firmware just copies the entire contents of RAM to a special partition and turns the computer off. Next time you hit the power button, the firmware dumps the partition contents back into RAM and resumes as if nothing had changed. This takes a few seconds longer than resume from S3 but is far faster than resume from hibernation since it starts the moment the system gets power.

At a more technical level, it's a little more complicated. The first thing to know about this feature is that it's entirely invisible unless your hard drive is set up correctly. There needs to be a partition that's at least the size of your system's physical RAM. For GPT systems, this needs to have a type GUID of D3BFE2DE-3DAF-11DF-BA-40-E3A556D89593. For MBR systems, you need a partition type of 0x84[2]. If the firmware doesn't find an appropriate partition then the OS will get no indication that the firmware supports it. Boo.

(The second thing is that it seems like it really does have to be on an SSD, and if you try to do this on spinning media your firmware will ignore it anyway)

If all the prerequisites are in place, an ACPI device with an HID of INT3392 will exist. It has four methods associated with it: GFFS, SFFS, GFTV and SFTV. GFFS returns an integer representing the events that will cause the system to wake up from S3 and suspend to SSD. The system will wake after the timeout expires if bit 0 is set, and will wake when the battery becomes critically low if bit 1 is set. The other bits appear to be unused at the moment[3]. SFFS sets the wakeup events, using the same bit values as GFFS. GFTV returns an integer containing the current wakeup timeout in minutes. SFTV sets it. Values above 1440 (ie, 24 hours) seem to be considered invalid - if I set them the value instead ends up as 10 and the timeout flag gets cleared from the wakeup events field.

I've submitted a patch that adds a sysfs interface for setting these values, and unless anyone objects it'll probably end up in 3.11. There's still the remaining question of how userspace should make use of these, and also how installers should behave when it comes to systems that support IRST. As previously mentioned, there's no obvious indication to the OS that the feature is supported unless the appropriate partition already exists. The easiest way to deal with this is for installers to default to retaining any partitions with the magic IDs, but I'm still looking into whether it's possible to get the firmware to cough up some more information so it can be created automatically even if the drive's entirely blank.

And now, having got this working on a test machine, I just need to split my Thinkpad's swap partition in half and make sure it works here as well. Woo.

[1] Note: I am not going to do this.
[2] Conveniently, the same as the partition type that APM systems used for suspend to disk back when dubstep hadn't been invented yet
[3] At least, if you attempt to set them they get ignored.

comment count unavailable comments

Syndicated 2013-07-03 16:50:49 from Matthew Garrett

384 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!