Maintaining Screen Output

In my post about getting started with KVM I noted the fact that I had problems keeping screen output after the program exits [1].

The following snippet of shell code demonstrates the solution I’ve discovered for this problem. It determines whether SCREEN is the parent process of the shell script and if so it sleeps for 60 seconds before exiting so I can see the KVM error messages. The other option is for the script to call “exec bash” to give me a new shell in the same window. Note that if I start a screen session and then run my KVM script I don’t want it to do anything special on exit as I will return to the command-line in the same window. If I run “exec kvm-unstable” or have a system boot script run “start-stop-daemon -S -c USER --exec /usr/bin/screen -- -S kvm-unstable -d -m /usr/local/bin/kvm-unstable” then on exit I will be able to see what happened.

#!/bin/bash
set -e
kvm ETC || echo “KVM gave an error return code”
COUNT=$(ps aux|grep $PPID|grep SCREEN|wc -l)
if [ "$COUNT" = "1" ]; then
  echo "screen is the parent"
  sleep 60
else
  echo no screen
fi

Update: Thanks to John for the Slee for suggesting the following:
#!/bin/bash
set -e
kvm ETC || echo “KVM gave an error return code”
if grep -q SCREEN /proc/$PPID/cmdline ; then
  echo "screen is the parent"
  sleep 60
else
  echo no screen
fi

• [1] http://etbe.coker.com.au/2010/03/17/starting-with-kvm/

Syndicated 2010-03-18 09:00:08 from etbe - Russell Coker

Maintaining Screen Output

In my post about getting started with KVM I noted the fact that I had problems keeping screen output after the program exits [1].

The following snippet of shell code demonstrates the solution I’ve discovered for this problem. It determines whether SCREEN is the parent process of the shell script and if so it sleeps for 60 seconds before exiting so I can see the KVM error messages. The other option is for the script to call “exec bash” to give me a new shell in the same window. Note that if I start a screen session and then run my KVM script I don’t want it to do anything special on exit as I will return to the command-line in the same window. If I run “exec kvm-unstable” or have a system boot script run “start-stop-daemon -S -c USER --exec /usr/bin/screen -- -S kvm-unstable -d -m /usr/local/bin/kvm-unstable” then on exit I will be able to see what happened.

#!/bin/bash
set -e
kvm ETC || echo “KVM gave an error return code”
COUNT=$(ps aux|grep $PPID|grep SCREEN|wc -l)
if [ "$COUNT" = "1" ]; then
  echo "screen is the parent"
  sleep 60
else
  echo no screen
fi

• [1] http://etbe.coker.com.au/2010/03/17/starting-with-kvm/

Syndicated 2010-03-18 01:38:08 from etbe - Russell Coker

Starting with KVM

I’ve just bought a new Thinkpad that has hardware virtualisation support and I’ve got KVM running. <h3>HugePages</h3>

The Linux-KVM site has some information on using hugetlbfs to allow the use of 2MB pages for KVM [1]. I put “vm.nr_hugepages = 1024” in /etc/sysctl.conf to reserve 2G of RAM for KVM use. The web page notes that it may be impossible to allocate enough pages if you set it some time after boot (the kernel can allocate memory that can’t be paged and it’s possible for RAM to become too fragmented to allow allocation). As a test I reduced my allocation to 296 pages and then increased it again to 1024, I was surprised to note that my system ran extremely slow while reserving the pages – it seems that allocating such pages is efficient when done at boot time but not so efficient when done later.

hugetlbfs /hugepages hugetlbfs mode=1770,gid=121 0 0

I put the above line in /etc/fstab to mount the hugetlbfs filesystem. The mode of 1770 allows anyone in the group to create files but not unlink or rename each other’s files. The gid of 121 is for the kvm group.

I’m not sure how hugepages are used, they aren’t used in the most obvious way. I expected that allocating 1024 huge pages would allow allocating 2G of RAM to the virtual machine, that’s not the case as “-m 2048” caused kvm to fail. I also expected that the number of HugePages free according to /proc/meminfo would reliably drop by an amount that approximately matches the size of the virtual machine – which doesn’t seem to be the case.

I have no idea why KVM with Hugepages would be significantly slower for user and system CPU time but still slightly faster for the overall build time (see the performance section below). I’ve been unable to find any documents explaining in which situations huge pages provide advantages and disadvantages or how they work with KVM virtualisation – the virtual machine allocates memory in 4K pages so how does that work with 2M pages provided to it by the OS?

But Hugepages does provide a slight benefit in performance and if you have plenty of RAM (I have 5G and can afford to buy more if I need it) you should just install it as soon as you start.

I have filed Debian bug report #574073 about KVM displaying an error you normally can’t see when it can’t access the hugepages filesystem [6]. <h3>Permissions</h3>

open /dev/kvm: Permission denied
Could not initialize KVM, will disable KVM support

One thing that annoyed me about KVM is that the Debian/Lenny version will run QEMU instead if it can’t run KVM. I discovered this when a routine rebuild of the SE Linux Policy packages in a Debian/Unstable virtual machine took an unreasonable amount of time. When I halted the virtual machine I noticed that it had displayed the above message on stderr before changing into curses mode (I’m not sure the correct term for this) such that the message was obscured until the xterm was returned to the non-curses mode at program exit. I had to add the user in question to the kvm group. I’ve filed Debian bug report #574063 about this [2].

Performance

Below is a table showing the time taken for building the SE Linux reference policy on Debian/Unstable. It compares running QEMU emulation (using the kvm command but without permission to access /dev/kvm), KVM with and without hugepages, Xen, and a chroot. Xen is run on an Opteron 1212 Dell server system with 2*1TB SATA disks in a RAID-1 while the KVM/QEMU tests are on an Intel T7500 CPU in a Thinkpad T61 with a 100G SATA disk [4]. All virtual machines had 512M of RAM and 2 CPU cores. The Opteron 1212 system is running Debian/Lenny and the Thinkpad is running Debian/Lenny with a 2.6.32 kernel from Testing.

	Elapsed	User	System
QEMU on Opteron 1212 with Xen installed	126m54	39m36	8m1
QEMU on T7500	95m42	42m57	8m29
KVM on Opteron 1212	7m54	4m47	2m26
Xen on Opteron 1212	6m54	3m5	1m5
KVM on T7500	6m3	2m3	1m9
KVM Hugepages on T7500 with NCurses console	5m58	3m32	2m16
KVM Hugepages on T7500	5m50	3m31	1m54
KVM Hugepages on T7500 with 1800M of RAM	5m39	3m30	1m48
KVM Hugepages on T7500 with 1800M and file output	5m7	3m28	1m38
Chroot on T7500	3m43	3m11	29

I was surprised to see how inefficient it is when compared with a chroot on the same hardware. It seems that the system time is the issue. Most of the tests were done with 512M of RAM for the virtual machine, I tried 1800M which improved performance slightly (less IO means less context switches to access the real block device) and redirecting the output of dpkg-buildpackage to /tmp/out and /tmp/err reduced the built time by 32 seconds – it seems that the context switches for networking or console output really hurt performance. But for the default build it seems that it will take about 50% longer in a virtual machine than in a chroot, this is bearable for the things I do (of which building the SE Linux policy is the most time consuming), but if I was to start compiling KDE then I would be compelled to use a chroot.

I was also surprised to see how slow it was when compared to Xen, for the tests on the Opteron 1212 system I used a later version of KVM (qemu-kvm 0.11.0+dfsg-1~bpo50+1 from Debian/Unstable) but could only use 2.6.26 as the virtualised kernel (the Debian 2.6.32 kernels gave a kernel Oops on boot). I doubt that the lower kernel version is responsible for any significant portion of the extra minute of build time. <h3>Storage</h3>

One way of managing storage for a virtual machine is to use files on a large filesystem for it’s block devices, this can work OK if you use a filesystem that is well designed for large files (such as XKS). I prefer to use LVM, one thing I have not yet discovered is how to make udev assign the KVM group to all devices that match /dev/V0/kvm-*. <h3>Startup</h3>

KVM seems to be basically designed to run from a session, unlike Xen which can be started with “xm create” and then run in the background until you feel like running “xm console” to gain access to the console. One way of dealing with this is to use screen. The command “screen -S kvm-foo -d -m kvm WHATEVER” will start a screen session named kvm-foo that will be detached and will start by running kvm with “WHATEVER” as the command-line options. When screen is used for managing virtual machines you can use the command “screen -ls” to list the running sessions and then commands such as “screen -r kvm-unstable” to reattach to screen sessions. To detach from a running screen session you type ^A^D.

The problem with this is that screen will exit when the process ends and that loses the shutdown messages from the virtual machine. To solve this you can put “exec bash” or “sleep 200” at the end of the script that runs kvm.

start-stop-daemon -S -c USERNAME --exec /usr/bin/screen -- -S kvm-unstable -d -m /usr/local/sbin/kvm-unstable

On a Debian system the above command in a system boot script (maybe /etc/rc.local) could be used to start a KVM virtual machine on boot. In this example USERNAME would be replaced by the name of the account used to run kvm, and /usr/local/sbin/kvm-unstable is a shell script to run kvm with the correct parameters. Then as user USERNAME you can attach to the session later with the command “screen -x kvm-unstable“. Thanks to Jason White for the tip on using screen.

I’ve filed Debian bug report #574069 [3] requesting that kvm change it’s argv[0] so that top(1) and similar programs can be used to distinguish different virtual machines. Currently when you have a few entries named kvm in top’s output it is annoying to match the CPU hogging process to the virtual machine it’s running.

It is possible to use KVM with X or VNC for a graphical display by the virtual machine. I don’t like these options, I believe that Xephyr provides better isolation, I’ve previously documented how to use Xephyr [5].

kvm -kernel /boot/vmlinuz-2.6.32-2-amd64 -initrd /boot/initrd.img-2.6.32-2-amd64 -hda /dev/V0/unstable -hdb /dev/V0/unstable-swap -m 512 -mem-path /hugepages -append "selinux=1 audit=1 root=/dev/hda ro rootfstype=ext4" -smp 2 -curses -redir tcp:2022::22

The above is the current kvm command-line that I’m using for my Debian/Unstable test environment. <h3>Networking</h3>

I’m using KVM options such as “-redir tcp:2022::22” to redirect unprivileged ports (in this case 2022) to the ssh port. This works for a basic test virtual machine but is not suitable for production use. I want to run virtual machines with minimal access to the environment, this means not starting them as root.

One thing I haven’t yet investigated is the vde2 networking system which allows a private virtual network over multiple physical hosts and which should allow kvm to be run without root privs. It seems that all the other networking options for kvm which have appealing feature sets require that the kvm process be started with root privs. <h3>Is KVM worth using?</h3>

It seems that KVM is significantly slower than a chroot, so for a basic build environment a secure chroot environment would probably be a better option. I had hoped that KVM would be more reliable than Xen which would offset the performance loss – however as KVM and Debian kernel 2.6.32 don’t work together on my Opteron system it seems that I will have some reliability issues with KVM that compare with the Xen issues. There are currently no Xen kernels in Debian/Testing so KVM is usable now with the latest bleeding edge stuff (on my Thinkpad at least) while Xen isn’t.

Qemu is really slow, so Xen is the only option for 32bit hardware. Therefore all my 32bit Xen servers need to keep running Xen.

I don’t plan to switch my 64bit production servers to KVM any time soon. When Debian/Squeeze is released I will consider whether to use KVM or Xen after upgrading my 64bit Debian server. I probably won’t upgrade my 64bit RHEL-5 server any time soon – maybe when RHEL-7 is released. My 64bit Debian test and development server will probably end up running KVM very soon, I need to upgrade the kernel for Ext4 support and that makes KVM more desirable.

So it seems that for me KVM is only going to be seriously used on my laptop for a while.

Generally I am disappointed with KVM. I had hoped that it would give almost the performance of Xen (admittedly it was only 14.5% slower). I had also hoped that it would be really reliable and work with the latest kernels (unlike Xen) but it is giving me problems with 2.6.32 on Opteron. Also it has some new issues such as deciding to quietly do something I don’t want when it’s unable to do what I want it to do.

• [1] http://www.linux-kvm.com/content/get-performance-boost-backing-your-kvm-guest-hugetlbfs
• [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=574063
• [3] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=574069
• [4] http://etbe.coker.com.au/2010/03/16/thinkpad-t61/
• [5] http://etbe.coker.com.au/2007/01/07/xephyr/
• [6] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=574073

Syndicated 2010-03-16 20:42:31 from etbe - Russell Coker

Thinkpad T61

I’ve now had my new Thinkpad T61 [1] for almost a month. The letters on the keyboard are not even starting to wear off which is unusual, either this Thinkpad is built with harder plastic than the older ones or I’m typing more softly. <h3>Memory</h3>

The first thing I did after receiving it was to arrange a RAM upgrade. It shipped with two 1GB DDR2 666MHz PC2-5300 SODIMM modules and as I want to run KVM I obviously need a lot more than that. The Intel Chipset page on Wikipedia is one of the resources that documents the Intel GM965 chipset as supporting up to 8G of RAM. Getting 4G in two 2G modules seemed like a bad idea as that would limit future expansion options and also result in two spare modules. So I decided to get a 4G module for a total of 5G of RAM. I’ve updated my RAM speed page with the test results of this system [2], I get 2,823MB/s with a matched pair of DIMMs and 2,023MB/s with a single DIMM. But strangely with a pair of unmatched DIMMs Memtest86+ reported 2,823MB/s – I wonder whether the first 2G of address space is interleaved for best performance and the last 3G runs at 2,023MB/s. In any case I think that losing 29% of the maximum RAM speed would be an acceptable trade-off for saving some money and I can always buy another 4G DIMM later. I had to order a DDR2-800MHz PC2-6400 module because they are cheaper than the PC2-5300 modules and my Thinkpad works equally well with either speed. I have used the spare 1G SODIMM in my EeePC701 which takes the same RAM – presumably because the EeePC designers found PC2-5300 modules to be cheaper than slower modules (I think that the 701 was at the time it was released the slowest PC compatible system that was selling in quantity). The EeePC gets only 798MB/s out of the same memory. My document about Memtest86+ results has these results and more [2].

I noticed that if I run Memtest86+ booted from a USB flash device then inserting or removing a USB device can cause memory errors, but if I boot memtest86+ from a CD it seems to work correctly. So it seems that Memtest86+ doesn’t disable some aspect of USB hardware, this might be considered a bug – or it might just be a “don’t do that” issue. <h3>Misc</h3>

To get the hardware virtualisation working (needed to load the kvm_intel kernel module) I had to enable it in the BIOS and then do a hard reset (power off). Telling the BIOS to save and reboot was not adequate. This would be a BIOS bug, it knew that I had changed the virtualisation setting so it should have either triggered a hard reset or instructed me to do so.

The default configuration of Debian/Lenny results in sound not working, I had to run alsaconf as suggested on the Debian Etch on Thinkpad T61 howto [3] which solved it.

Generally I’m happy with this system, the screen resolution is 1680*1050 which has 20% more pixels than the 1400*1050 screen on my Thinkpad T41p, it’s a lot faster for CPU operations and should be a lot faster for video when I get the drivers sorted out (currently it’s a lot slower), and I have virtualisation working again. But when you buy a system that’s much like the last one but 6 years newer you expect it to be better.

Generally the amount of effort involved in the process of buying a new system, upgrading the RAM to the desired specs, installing Linux and tweaking all the options is enough to make me want to wait at least another 6 years before buying another. Part of the reason for this difficulty is that I want to get so much functionality from the machine, a machine with more modest goals (such as a Netbook) takes a lot less time to configure. <h3>Problems</h3>

There is Bluetooth hardware which is apparently enabled by default. But a quick search didn’t turn up any information on how to do the basic functions, I would like to just transfer files from my mobile phone in the same way that I transfer files between phones.

The video card is a nVidia Corporation Quadro NVS 140M (rev a1). 3D games seem slow but glxgears reports 300fps. It doesn’t have Xvideo support which appears to be the reason why mplayer won’t allow resizing it’s display area unless run with the -zoom option, and it’s also got performance problems such that switching between virtual desktops will interrupt the sound on a movie that mplayer is playing – although when alsaplayer is playing music the sound isn’t interrupted. Also when I play a Youtube video at twice the horizontal and vertical resolution it takes half of one CPU core. It’s a pity that I didn’t get an Intel video controller.

It seems that Debian is soon going to get the Nouveau NVidia drivers so hopefully video performance will improve significantly when I get them [4].

The next thing I have to do is to get the sound controls working. The older Thinkpads that I used had hardware controls, the T41p that was my previous system had buttons for increasing and decreasing the volume and a mute button that interacted directly with the hardware. The down-side of this was that there was no way for the standard software to know what the hardware was going to do, the up-side was that I could press the mute button and know that it would be silent regardless of what the software wants. Now I have the same buttons on my T61 but they don’t do anything directly, they just provide key-press events. According to showkeys the mute key gives “0×71 0xf1“, the volume down button gives “0×72 0xf2“, and the volume up button gives “0×73 0xf3“. Daniel Pittman has made some suggestions to help me get the keyboard events mapped to actions that can change the sound via software [5] – which I haven’t yet had time to investigate. I wonder if it will ever be possible to change the volume of the system beep.

The system has an SD card slot, but that doesn’t seem to work. I’m not really worried at the moment but in the future I will probably try and get it going. It has a 100G disk which isn’t that big, adding a 32G SD card at some future time might be the easiest way to upgrade the storage – copying 100G of data is going to be painful and usually a small increment in storage capacity can keep a system viable for a while.

Any advice on getting sound, the SD card, and Bluetooth working would be appreciated. I’ll probably upgrade to Debian/Testing in the near future so suggestions that require testing features won’t be ruled out.

• [1] http://etbe.coker.com.au/2010/02/15/new-thinkpad-lenovo-web-site-sucks/
• [2] http://doc.coker.com.au/computers/ram-speed-memtest86/
• [3] http://www.klabs.be/~fpiat/linux/debian/Etch_on_Thinkpad_T61.html
• [4] http://bgoglin.livejournal.com/19624.html
• [5] http://lists.luv.asn.au/wws/arc/luv-main/2010-03/msg00123.html

Syndicated 2010-03-16 02:32:59 from etbe - Russell Coker

The Yubikey

Some time ago Yubico were kind enough to send me an evaluation copy of their Yubikey device. I’ve finally got around to reviewing it and making deployment plans for buying some more. Above is a picture of my Yubikey on the keyboard of my Thinkpad T61 for scale. The newer keys apparently have a different color in the center of the circular press area and also can be purchased in white plastic.

The Yubikey is a USB security token from Yubico [1]. It is a use-based token that connects via the USB keyboard interface (see my previous post for a description of the various types of token [2]). The Yubikey is the only device I know of which uses the USB keyboard interface, it seems that this is an innovation that they invented. You can see in the above picture that the Yubikey skips the metal that is used to surround most USB devices, this probably fails to meet some part of the USB specification but does allow them to make the key less than half as thick as it might otherwise be. Mechanically it seems quite solid.

The Yubikey is affordable, unlike some token vendors who don’t even advertise prices (if you need to ask then you can’t afford it) and they have an online sales site. $US25 for a single key and discounts start when you buy 10. As it seems quite likely that someone who wants such a token will want at least two of them for different authentication domains, different users in one home, or as a backup in case one is lost or broken (although my experiments have shown that Yubikeys are very hardy and will not break easily). The discount rate of $20 will apply if you can find four friends who want to use them (assuming two each), or if you support several relatives (as I do). The next discount rate of $15 applies when you order 100 units, and they advise that customers contact their sales department directly if purchasing more than 500 units – so it seems likely that a further discount could be arranged when buying more than 500 units. They accept payment via Paypal as well as credit cards. It seems to me that any Linux Users Group could easily arrange an order for 100 units (that would be 10 people with similar needs to me) and a larger LUG could possibly arrange an order of more than 500 units for a better discount. If an order of 500 can’t be arranged then an order of 200 would be a good thing to get half black keys and half white ones – you can only buy a pack of 100 in a single color.

There is a Wordpress plugin to use Yubikey authentication [3]. It works, but I would be happier if it had an option to accept a Yubikey OR a password (currently it demands both a Yubikey AND a password). I know that this is less secure, but I believe that it’s adequate for an account that doesn’t have administrative rights.

To operate the Yubikey you just insert it into a USB slot and press the button to have it enter the pass code via the USB keyboard interface. The pass code has a prefix that can be used to identify the user so it can replace both the user-name and password fields – of course it is technically possible to use one Yubikey for authentication with multiple accounts in which case a user-name would be required. Pressing the Yubikey button causes the pass code to be inserted along with the ENTER key, this can take a little getting used to as a slow web site combined with a habit of pressing ENTER can result in a failed login (at least this has happened to me with Konqueror).

As the Yubikey is use-based, it needs a server to track the usage count of each key. Yubico provides source to the server software as well as having their own server available on the net – obviously it might be a bad idea to use the Yubico server for remote root access to a server, but for blog posting that is a viable option and saves some effort.

If you have multiple sites that may be disconnected then you will either need multiple Yubikeys (at a cost of $20 or $15 each) or you will need to have one Yubikey work with multiple servers. Supporting a single key with multiple authentication servers means that MITM attacks become possible.

The full source to the Yubikey utilities is available under the new BSD license. In Debian the base functionality of talking to the Yubikey is packaged as libyubikey0 and libyubikey-dev, the server (for validating Yubi requests via HTTP) is packaged as yubikey-server-c, and the utility for changing the AES key to use your own authentication server is packaged as yubikey-personalization – thanks Tollef Fog Heen for packaging all this!

The YubiPAM project (a PAM module for Yubikey) is licensed under the GPL [4]. It would be good if this could be packaged for Debian (unfortunately I don’t have time to adopt more packages at the moment).

There is a new model of Yubikey that has RFID support. They suggest using it for public transport systems where RFID could be used for boarding and the core Yubikey OTP functionality could be used for purchasing tickets. I don’t think it’s very interesting for typical hobbyist and sysadmin work, but RFID experts such as Jonathan Oxer might disagree with me on this issue.

• [1] http://www.yubico.com/
• [2] http://etbe.coker.com.au/2010/03/15/security-tokens/
• [3] http://henrik.schack.dk/yubikey-plugin/
• [4] http://www.securixlive.com/yubikey/index.php

Syndicated 2010-03-15 12:16:25 from etbe - Russell Coker