Fun with Ubuntu Lucid, KVM and iSCSI

---

(Note: I wrote this after doing this install and haven't walked through it and checked it, so there maybe thinkos)

I run Ubuntu Lucid on my desktop and wanted to play about with server stuff, and I've already got the KVM setup running with other guests running in it.

Here I show how I've set up a pair of Ubuntu Lucid server KVM guests, where one iSCSI serves the root to the other, with iSCSI booting using gPXE.

I won't claim this is efficient, but it's been interesting to setup - bits of this setup are probably useful in other scenarious, e.g. I think you should be able to iSCSI boot a KVM guest off an external SAN. There may even be more sane ways of doing this insane combination!

1) The base

For my base system I have Ubuntu Lucid (64bit) in my case on an i7-860 with 8GB of RAM. On it I have the kvm, qemu-common, qemu- kvm and kvm-pxe packages installed, together with libvirt- bin, libvirt0 and virt-manager.

I've got some of my disc space in a LVM volume group.

I configured the dhcp range for the default network in KVM downto only part of the range and setup a netboot like this ( /var/lib/libvirt/network/default.xml )

 <network>
  <name>default</name>
  <uuid>e72880a1-3159-cb84-4314-ee938ebb954b</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <tftp root='/var/lib/libvirt/tftp' />
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.127' />
      <host mac='52:54:00:41:73:fb' name='client1' 
ip='192.168.122.130' />
      <bootp file='netboot' />
    </dhcp>
  </ip>
</network>

 

NOTE: To restart the libvirt dhcp server to use the config file you'll need to:

   stop libvirt-bin
   killall dnsmasq
   ifconfig virbr0 down
   brctl delbr
   start libvirt-bin

Create /var/lib/libvirt/tftp

2) The server guest

I used lvm to create an logical volume called server1a (which is now /dev/main/server1a on the host), and then created a guest using virt-manager and gave it that volume as a virtio disk. (The ability to create a new lv within virt-manager doesn't seem to work for me).

I then booted this off a Lucid server CD and installed it, creating it's own LVM hierarchy on it's disk and giving it a fixed IP (192.168.122.129). For convenience I then added an entry to the host machines /etc/hosts to point to that.

Within the server guest I created a new logical volume to be the root disk of the client, so from within the server this is /dev/server1/export.

We now have:

   Host physical Disk (/dev/sda on the host)
   fdisk partition for LVM (/dev/sda3 on the host)
   LVM volume group (main on the host)
   LVM logical volume for the server guest 
(/dev/main/server1a)

    Guest virtual Disk (/dev/vda on the server1 guest, 
/dev/main/server1a on host)
   fdisk partition for LVM (/dev/vda5 on server1 guest)
   LVM volume group (server1 on the server1 guest)
   LVM logical volume for the client guest 
(/dev/server1/export)

Now we can get iscsi serving going. On the server1 guest I installed the iscsitarget package, made sure that /etc/default/iscsitarget enabled it and wrote an /etc/ietd.conf with just the lines:

Target iqn.2010-05.org.treblig:server1.export.a.b
       Lun 0 Path=/dev/mapper/server1-export,Type=blockio

I then made sure this would run at boot by using

update-rc.d iscsitarget defaults

3) The client

I created another guest with virt-manager with no disk and booted it using the Lucid server disk. Note that I have a fixed IP specified in the DHCP config for libvirt for the MAC address of the client.

When it realises it hasn't got a disk it should give you the option of using iscsi, and giving it the IP of the server it should then list the iscsi target name from the ietd.conf.

I then did a normal lucid server install, and then shut it down.

4) Booting the client

This is somewhat trickier than it should be; the PXE netboot ROM in Qemu/KVM on Lucid doesn't have iSCSI booting in it. There are various ways to fix this, you could PXE boot Grub2 and then the kernel/initrd copied out of the client, but that wouldn't get updated by any kernel updates in the client. You can also replace the ROM in the Qemu, but I didn't want to change any of the packaged stuff.

My solution was to chain boot a gPXE with iSCSI support; I went to :

rom-o-matic

and selected .kpxe format, virtio-net for the NIC type, TFTP download protocol, SANBOOT_PROTO_ISCSI, AUTOBOOT_CMD, SANBOOT_CMD and most of the other defaults and then added the following embedded script:

ifopen net0
dhcp net0
chain /${ip}

rom-o-matic gives back an image file that I copied into /var/lib/libvirt/tftp on the host as a file called netboot. When the client netboots it will load this new instance of gPXE that will then try and load a file based on it's IP, in my case I've got a file called 192.168.122.130 in /var/lib/libvirt/tftp on the host with the following contents:

#!gpxe
sanboot iscsi:192.168.122.129::::iqn.2010-
05.org.treblig:server1.export.a.b

where the first part is the IP of the iSCSI server partition, and the second part is the identifier from the ietd.conf on the server.

Remember to set the Boot Options on the client to Network.

Now when I boot the client VM I boot all the way into Lucid client!

The boot process of the client is:

   DHCP, PXE boot, tftp fetch the new gPXE
   DHCP, fetch the /192.168.122.130 script
   iSCSI boot off the remote disk

Notes: On the client there is a lot of iscsi config in /etc, and /etc/iscsi in particular, if you move the server after installation you'll have to fix up all of that and then fix up the initramfs by regenerating it - it'll pick up the /etc/iscsi/iscsi.initramfs