Linux on a BladeRunner system
So, you bought one of those shiny new
Penguin Computing
BladeRunner systems, and were thinking to yourself: "I wish there were
a good guide to getting Linux to do what it should on these things".
Well, I'm going to try and cover two of the basics here: serial console
configuration, and interface bonding. Everything else is pretty much
stock stuff, but I had a heck of a time figuring out configurations that
worked here. The discussion below assumes you're running
RHEL or
Fedora, but the idea should
be fairly clear.
First step: learn about some of the remote management features of the
BladeRunner. You have a couple of configuration commands that are useful
here, related to console management and power. Log into the chassis, and
type conf; for some reason, they put this under "configuration"
rather than "management" or some other similar tree of the command set.
Now, you can control power to individual blades with server-blade
power <num> [cycle|forced-off|off|on] (where <num>
is the number of the blade you're managing). on means exactly what
it sounds like; power on the blade if it's currently off, just like it would
if you hit the power button on the front. off sends an ACPI
power vevent to the blade, advising it to shut down; it doesn't actually
force power to be pulled; that's what forced-off does.
cycle removes power to the blade, and adds it again; off the top
of my head, I don't recall if it does so gracefully or not. Check the
documentation, it might be there (but probably not).
The next command that's interesting here is server-blade console-redirect
<num>: that grabs the serial console for that particular blade,
and displays it to your current session. The serial console configuration
appears to be set in stone, from what I can see: 57600 bps, 8 data bits,
no parity, 1 stop bit, with hardware (RTS/CTS) flow control. If you power
up a blade fresh from the factory, you'll notice that the BIOS is already
set up to redirect output to both the serial port and the built-in KVM in
the chassis.
That's about all I'll say about the chassis management side of things, since
they actually do cover this stuff fairly well in the documentation. The
problem I found with the documentation was that it didn't cover host-side
configuration of certain things, specifically interface bonding and serial
console configuration. Maybe they thought it was out-of-scope for their
documentation; I hate to be the bearer of bad news, but the chassis is
pretty useless without systems to run on it. (Here endeth the snide
remarks, hopefully.)
Next up is host-side serial console configuration. This is fairly standard
stuff, once you know what the settings need to be, but I'll spell it out
here too. First is the bootloader; GRUB, in my case. You'll want to add a
couple of lines to the GRUB configuration file (in the main stanza):
serial --unit=0 --speed=57600 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console
Optionally, you can also add hiddenmenu to the options; that
reduces the amount of text being displayed at boot time to a minumum,
unless you need it. Set the timeout on the terminal line to
whatever is appropriate for you; 10 seconds before you're kicked to the
primary display was good enough for my needs, since I didn't want the
boot process to take much longer than it already did.
Next up, also in the grub configuration file (on the command line of each
kernel you want to boot), are the kernel parameters that get your boot
output sent to the right place. Adding console=tty0
console=ttyS0,57600n8r to the kernel command line seemed to do the
job for me (display on /dev/ttyS0 at 57000 bps, no parity, 8 data bits,
hardware flow control). If this is a stock Red Hat/Fedora installation,
I suggest getting rid of the rhgb and quiet entries on
that line as well; for server use, you really want to see that console
output, so that when you inevitably get a kernel that has a bad day, you
don't have to monkey with grub configuration via a 57600bps serial line
to capture the panic.
The final step is making it possible to log in on the console. Adding the
following line to /etc/inittab worked great for my needs:
co:2345:respawn:/sbin/agetty -h ttyS0 57600 vt100
The co is an name for the entry, but it's become a defacto standard
for the inittab name for the serial console entry. 2345 should be
obvious (runlevels to operate in), respawn means run it again after
you log out, not just once. The agetty command line just specifies
hardware flow control (-h), linux device (ttyS0), speed (57600), and
terminal type (vt100); change terminal type to anything appropriate for
your environment, but vt100 is pretty safe for most folks.
That covers serial console configuration. For more details on that piece of
the puzzle, I strongly recommend reading over the
Remote
Serial Console HOWTO; it discusses some of the ins-and-outs of other
boot loaders, some other configurations you might be interested in trying,
and some general advice for this kind of setup.
Next up is interface bonding; every blade has two Broadcom NetXtreme
BCM5780S Gigabit Ethernet NICs built in, each connected to one of the
two management blades through an internal backplane. I've been using the
tg3 drivers for these quite successfully; the bcm5820
module is also available as a third-party add-on, but appears to be
deprecated at this point. So, your /etc/modules.conf or
/etc/modprobe.conf (depending on version) will likely have a
couple of lines in it like alias eth0 tg3 for each interface.
Next up is bonding configuration: add the following lines:
alias bond0 bonding
options bonding mode=2 arp_interval=500 arp_ip_target=10.0.0.1
Change 10.0.0.1 to whatever your default gateway is. You might
be asking yourself why you shouldn't use MII instead of ARP on these
systems, and it's a good question. The failure mode of the chassis internal
network precludes it; you can end up with one management switch down, but
the MII status of the ethernet interface still shows it being up. You can
test this out by setting your bonding configuration to use MII instead,
then yanking out one of the chassis management blades.
The next piece is very specific to RHEL or Fedora; please refer to your
Linux vendor's documentation for details on how to do it elsewhere. In
/etc/sysconfig/network-scripts, create a file called
ifcfg-bond0 that looks exactly like a normal ifcfg-*
file (in fact, you might want to just move your existing ifcfg-eth0
or similar file there); specify how you want this combined interface to
behave. Then, for each of ifcfg-eth0 and ifcfg-eth1,
add a file that looks like:
DEVICE=eth0
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
(Change DEVICE= to the appropriate line for each device.) More
information on interface bonding under linux is available from the
OSDL
Linux networking
site.
That's the main stuff. Obviously, test all of these changes with a complete
boot to make sure console redirection is happening the way you want it, and
to be sure that your bonded interfaces are showing up as a single
bond0 interface (and that you have connectivity to and from them).
Good luck.
Syndicated 2006-08-26 10:23:00 from esm