20 Feb 2007 logic   » (Journeyer)

Linux on a BladeRunner system

So, you bought one of those shiny new Penguin Computing BladeRunner systems, and were thinking to yourself: "I wish there were a good guide to getting Linux to do what it should on these things". Well, I'm going to try and cover two of the basics here: serial console configuration, and interface bonding. Everything else is pretty much stock stuff, but I had a heck of a time figuring out configurations that worked here. The discussion below assumes you're running RHEL or Fedora, but the idea should be fairly clear.

First step: learn about some of the remote management features of the BladeRunner. You have a couple of configuration commands that are useful here, related to console management and power. Log into the chassis, and type conf; for some reason, they put this under "configuration" rather than "management" or some other similar tree of the command set. Now, you can control power to individual blades with server-blade power <num> [cycle|forced-off|off|on] (where <num> is the number of the blade you're managing). on means exactly what it sounds like; power on the blade if it's currently off, just like it would if you hit the power button on the front. off sends an ACPI power vevent to the blade, advising it to shut down; it doesn't actually force power to be pulled; that's what forced-off does. cycle removes power to the blade, and adds it again; off the top of my head, I don't recall if it does so gracefully or not. Check the documentation, it might be there (but probably not).

The next command that's interesting here is server-blade console-redirect <num>: that grabs the serial console for that particular blade, and displays it to your current session. The serial console configuration appears to be set in stone, from what I can see: 57600 bps, 8 data bits, no parity, 1 stop bit, with hardware (RTS/CTS) flow control. If you power up a blade fresh from the factory, you'll notice that the BIOS is already set up to redirect output to both the serial port and the built-in KVM in the chassis.

That's about all I'll say about the chassis management side of things, since they actually do cover this stuff fairly well in the documentation. The problem I found with the documentation was that it didn't cover host-side configuration of certain things, specifically interface bonding and serial console configuration. Maybe they thought it was out-of-scope for their documentation; I hate to be the bearer of bad news, but the chassis is pretty useless without systems to run on it. ;-) (Here endeth the snide remarks, hopefully.)

Next up is host-side serial console configuration. This is fairly standard stuff, once you know what the settings need to be, but I'll spell it out here too. First is the bootloader; GRUB, in my case. You'll want to add a couple of lines to the GRUB configuration file (in the main stanza):

  serial --unit=0 --speed=57600 --word=8 --parity=no --stop=1
  terminal --timeout=10 serial console

Optionally, you can also add hiddenmenu to the options; that reduces the amount of text being displayed at boot time to a minumum, unless you need it. Set the timeout on the terminal line to whatever is appropriate for you; 10 seconds before you're kicked to the primary display was good enough for my needs, since I didn't want the boot process to take much longer than it already did.

Next up, also in the grub configuration file (on the command line of each kernel you want to boot), are the kernel parameters that get your boot output sent to the right place. Adding console=tty0 console=ttyS0,57600n8r to the kernel command line seemed to do the job for me (display on /dev/ttyS0 at 57000 bps, no parity, 8 data bits, hardware flow control). If this is a stock Red Hat/Fedora installation, I suggest getting rid of the rhgb and quiet entries on that line as well; for server use, you really want to see that console output, so that when you inevitably get a kernel that has a bad day, you don't have to monkey with grub configuration via a 57600bps serial line to capture the panic.

The final step is making it possible to log in on the console. Adding the following line to /etc/inittab worked great for my needs:

  co:2345:respawn:/sbin/agetty -h ttyS0 57600 vt100

The co is an name for the entry, but it's become a defacto standard for the inittab name for the serial console entry. 2345 should be obvious (runlevels to operate in), respawn means run it again after you log out, not just once. The agetty command line just specifies hardware flow control (-h), linux device (ttyS0), speed (57600), and terminal type (vt100); change terminal type to anything appropriate for your environment, but vt100 is pretty safe for most folks.

That covers serial console configuration. For more details on that piece of the puzzle, I strongly recommend reading over the Remote Serial Console HOWTO; it discusses some of the ins-and-outs of other boot loaders, some other configurations you might be interested in trying, and some general advice for this kind of setup.

Next up is interface bonding; every blade has two Broadcom NetXtreme BCM5780S Gigabit Ethernet NICs built in, each connected to one of the two management blades through an internal backplane. I've been using the tg3 drivers for these quite successfully; the bcm5820 module is also available as a third-party add-on, but appears to be deprecated at this point. So, your /etc/modules.conf or /etc/modprobe.conf (depending on version) will likely have a couple of lines in it like alias eth0 tg3 for each interface. Next up is bonding configuration: add the following lines:

  alias bond0 bonding
  options bonding mode=2 arp_interval=500 arp_ip_target=10.0.0.1

Change 10.0.0.1 to whatever your default gateway is. You might be asking yourself why you shouldn't use MII instead of ARP on these systems, and it's a good question. The failure mode of the chassis internal network precludes it; you can end up with one management switch down, but the MII status of the ethernet interface still shows it being up. You can test this out by setting your bonding configuration to use MII instead, then yanking out one of the chassis management blades.

The next piece is very specific to RHEL or Fedora; please refer to your Linux vendor's documentation for details on how to do it elsewhere. In /etc/sysconfig/network-scripts, create a file called ifcfg-bond0 that looks exactly like a normal ifcfg-* file (in fact, you might want to just move your existing ifcfg-eth0 or similar file there); specify how you want this combined interface to behave. Then, for each of ifcfg-eth0 and ifcfg-eth1, add a file that looks like:

  DEVICE=eth0
  TYPE=Ethernet
  BOOTPROTO=none
  ONBOOT=yes
  MASTER=bond0
  SLAVE=yes

(Change DEVICE= to the appropriate line for each device.) More information on interface bonding under linux is available from the OSDL Linux networking site.

That's the main stuff. Obviously, test all of these changes with a complete boot to make sure console redirection is happening the way you want it, and to be sure that your bonded interfaces are showing up as a single bond0 interface (and that you have connectivity to and from them). Good luck. :-)

Syndicated 2006-08-26 10:23:00 from esm

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!