penguin42 is currently certified at Journeyer level.

Name: Dave Gilbert
Member since: 2000-02-26 17:27:50
Last Login: 2010-05-28 13:06:29

FOAF RDF Share This

Homepage: http://www.treblig.org

Notes:

Linux fiddler, author of a couple of emulators (beebem, arcem) and generally help the free source cause.

Articles Posted by penguin42

Recent blog entries by penguin42

Syndication: RSS 2.0

Fun with Ubuntu Lucid, KVM and iSCSI


(Note: I wrote this after doing this install and haven't walked through it and checked it, so there maybe thinkos)

I run Ubuntu Lucid on my desktop and wanted to play about with server stuff, and I've already got the KVM setup running with other guests running in it.

Here I show how I've set up a pair of Ubuntu Lucid server KVM guests, where one iSCSI serves the root to the other, with iSCSI booting using gPXE.

I won't claim this is efficient, but it's been interesting to setup - bits of this setup are probably useful in other scenarious, e.g. I think you should be able to iSCSI boot a KVM guest off an external SAN. There may even be more sane ways of doing this insane combination!

1) The base

For my base system I have Ubuntu Lucid (64bit) in my case on an i7-860 with 8GB of RAM. On it I have the kvm, qemu-common, qemu- kvm and kvm-pxe packages installed, together with libvirt- bin, libvirt0 and virt-manager.

I've got some of my disc space in a LVM volume group.

I configured the dhcp range for the default network in KVM downto only part of the range and setup a netboot like this ( /var/lib/libvirt/network/default.xml )




<network> <name>default</name> <uuid>e72880a1-3159-cb84-4314-ee938ebb954b</uuid> <forward mode='nat'/> <bridge name='virbr0' stp='on' delay='0' /> <ip address='192.168.122.1' netmask='255.255.255.0'> <tftp root='/var/lib/libvirt/tftp' /> <dhcp> <range start='192.168.122.2' end='192.168.122.127' /> <host mac='52:54:00:41:73:fb' name='client1' ip='192.168.122.130' /> <bootp file='netboot' /> </dhcp> </ip> </network>

NOTE: To restart the libvirt dhcp server to use the config file you'll need to:


   stop libvirt-bin
   killall dnsmasq
   ifconfig virbr0 down
   brctl delbr
   start libvirt-bin

Create /var/lib/libvirt/tftp


2) The server guest

I used lvm to create an logical volume called server1a (which is now /dev/main/server1a on the host), and then created a guest using virt-manager and gave it that volume as a virtio disk. (The ability to create a new lv within virt-manager doesn't seem to work for me).

I then booted this off a Lucid server CD and installed it, creating it's own LVM hierarchy on it's disk and giving it a fixed IP (192.168.122.129). For convenience I then added an entry to the host machines /etc/hosts to point to that.

Within the server guest I created a new logical volume to be the root disk of the client, so from within the server this is /dev/server1/export.

We now have:


   Host physical Disk (/dev/sda on the host)
   fdisk partition for LVM (/dev/sda3 on the host)
   LVM volume group (main on the host)
   LVM logical volume for the server guest 
(/dev/main/server1a)


Guest virtual Disk (/dev/vda on the server1 guest, /dev/main/server1a on host) fdisk partition for LVM (/dev/vda5 on server1 guest) LVM volume group (server1 on the server1 guest) LVM logical volume for the client guest (/dev/server1/export)

Now we can get iscsi serving going. On the server1 guest I installed the iscsitarget package, made sure that /etc/default/iscsitarget enabled it and wrote an /etc/ietd.conf with just the lines:


Target iqn.2010-05.org.treblig:server1.export.a.b
       Lun 0 Path=/dev/mapper/server1-export,Type=blockio

I then made sure this would run at boot by using

update-rc.d iscsitarget defaults


3) The client

I created another guest with virt-manager with no disk and booted it using the Lucid server disk. Note that I have a fixed IP specified in the DHCP config for libvirt for the MAC address of the client.

When it realises it hasn't got a disk it should give you the option of using iscsi, and giving it the IP of the server it should then list the iscsi target name from the ietd.conf.

I then did a normal lucid server install, and then shut it down.


4) Booting the client

This is somewhat trickier than it should be; the PXE netboot ROM in Qemu/KVM on Lucid doesn't have iSCSI booting in it. There are various ways to fix this, you could PXE boot Grub2 and then the kernel/initrd copied out of the client, but that wouldn't get updated by any kernel updates in the client. You can also replace the ROM in the Qemu, but I didn't want to change any of the packaged stuff.

My solution was to chain boot a gPXE with iSCSI support; I went to :

rom-o-matic

and selected .kpxe format, virtio-net for the NIC type, TFTP download protocol, SANBOOT_PROTO_ISCSI, AUTOBOOT_CMD, SANBOOT_CMD and most of the other defaults and then added the following embedded script:


ifopen net0
dhcp net0
chain /${ip}

rom-o-matic gives back an image file that I copied into /var/lib/libvirt/tftp on the host as a file called netboot. When the client netboots it will load this new instance of gPXE that will then try and load a file based on it's IP, in my case I've got a file called 192.168.122.130 in /var/lib/libvirt/tftp on the host with the following contents:


#!gpxe
sanboot iscsi:192.168.122.129::::iqn.2010-
05.org.treblig:server1.export.a.b

where the first part is the IP of the iSCSI server partition, and the second part is the identifier from the ietd.conf on the server.

Remember to set the Boot Options on the client to Network.

Now when I boot the client VM I boot all the way into Lucid client!

The boot process of the client is:


   DHCP, PXE boot, tftp fetch the new gPXE
   DHCP, fetch the /192.168.122.130 script
   iSCSI boot off the remote disk


Notes: On the client there is a lot of iscsi config in /etc, and /etc/iscsi in particular, if you move the server after installation you'll have to fix up all of that and then fix up the initramfs by regenerating it - it'll pick up the /etc/iscsi/iscsi.initramfs

Ho hum; fancied posting a diary.

Top one of the weekend; don't try and build a cross compiler from Alpha to 68k using an old host compiler; egcs-1.1.2 screws up some floating point stuff - that took me ages to find.

Now I'm just stuck with binutils fighting when it tries to link.

Fighting autoconf and its mates trying to conditionally remove the win32 support in xine so I cna build it on Alpha.

Hacked up one of he LCD panels fully off an old VGA card - working quite nicely; although the inverter circuit we cobbled together is a bit touchy. (Anyone with ideas for a nice, safe, stable, cheap inverter for cold cathode backlights please tell me!).

A kludge with SVGAtextmode gave me something almost right - although I face the option of loosing about 2 pixels off either edge (I can choose, but I always loose a couple of pixels...).

Fighting Linux/ARM on the old platforms; these machines have unreasonably large 32K pages (and not many of them) - there are some very odd things happening in some places. (Like a divide by 0 I got last week in fork_init).

Interesting weekend; Thursday saw the arrival of 20 640x480 10.4" LCD panels (sans light inverters and VGA interface and case).

Before the ManLUG talk on Saturday I managed to get cabling from Maplin and other bits off a friend. Saturday evening/Sunday morning was crimping duty - without a crimp tool. Hellishly small crimps.

Early on Sunday I managed to get the panel running off a VGA feature connector (with just one colour wired) - looks promising.

Now if only I could array them.....

Spent the rest of time fighting trying to get a modern ARM cross compiler going,

Spent Saturday fighting with my Sparc/IPX to try and install RedHat 6.2 on it. It already had 5.2 on another disc but since that disc was getting iffy I wanted a clean install.

Unfortunatly Disk Druid managed to create a partition with duff entries which seemed to indicate that it was around 1000 times larger than it really was. It took me some hours to discover this, and thus explain why mke2fs was taking many hours and consuming about 180M of RAM.

I selected 'server install' - and then spent a while fighting with RPM to get all the GNOME stuff in afterwards. Hmm GNOME is not that responsive on an IPX with 32MB of RAM.

I left it building XFree 4 over night; 14 hours later and it had built and I managed to get it not to start; ho hum - another X bug reported.

The interesting revelation of the day was bought to me by Owen Taylor. I had found that large selections just didn't work for me on Gnome on my Alpha - this turned out to be a brokedness in Gtk which was hitting a rather nasty 64 bit brokedness in X. Yuch!

6 older entries...

 

penguin42 certified others as follows:

  • penguin42 certified rmk as Master
  • penguin42 certified hp as Master

Others have certified penguin42 as follows:

  • rmk certified penguin42 as Journeyer
  • lmb certified penguin42 as Journeyer
  • gbritton certified penguin42 as Journeyer
  • Svartalf certified penguin42 as Journeyer
  • mstevens certified penguin42 as Journeyer

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page