LVM mirroring: the right way

LVM now supports mirroring inside of LVM, rather than requiring that you put mirrors underneath LVM physical volumnes. This provides much more flexibility, and some volumnes can be mirrored, some not (such as swap partitions), and different RAID algorithms can be used. LVM uses the same underlying mechanisms as Linux RAID system (mdadm) to do the RAID operations, so there is no change in overall performance.

Lucas and I learnt on the Hydra project that creating a mirror as follows:

lvconvert -m 1 --corelog /dev/nv0/time1root

or at lvcreate time:

lvcreate -L 4G --name time1root -m 1 --corelog --nosync /dev/nv0

while it works, produces a mirror that keeps certain meta-info in memory only. Should the machine reboot in an uncontrolled way, the mirror will be marked as bad and rebuilt in order to validate the meta-data.

On a machine with with VMs running (nvxen-0, crtlXX) after a reboot it can take hours for the mirror to rebuild. The correct answer it turns out is to use --mirrorlog mirrored, and an option to put the mirror logs anywhere.

lvconvert -m 1 --mirrorlog mirrored --alloc anywhere /dev/nv0/time1root The allocation policy of "anywhere" permits the two 4M mirror logs (4M is the minimum allocation that LVM can do) to be kept on the same disks as the data they are mirroring. Otherwise, if you have only two physical volumnes, you can not put the log anywhere and the default policy (which I think is wrong) is to insist that the mirrorlogs go on different volumnes than the data. (I don't know why this necessary)

Converting between is a pain: the only way I found to do this is to remove the mirroring and then re-create it.

ionice -c3 lvconvert -m 0 /dev/nv0/time1root ionice -c3 lvconvert -m 1 --mirrorlog mirrored --alloc anywhere /dev/nv0/time1root

I wrote a script to process the output of lvs and do this. The ionice keeps the process in the background, not chewing up I/O.

On the fresh boot after the crash however, you may find your system is almost completely unresponsive as it tries to resync dozens of mirrors. On that, /dev/md0-style raid devices get it right. How to fix: find the kcopyd kernel processes and run ionice on them:

ps ax | grep kcopyd | awk '{print $1}' | while read pid; do sudo ionice -i3 -p$pid; done

once you have done this, you can then get in long enough to run the lvconvert. I suggest you remove all the mirrors first (-m 0) as that stops the resync operation from getting in the way of the resync you will have to anyway.

Syndicated 2012-02-05 12:25:00 from Michael's musings

Active Scaffold obscures internal errors

In a newly scaffold'ed model and controller, created with ActiveScaffold 3.0.5, on rails 3.0.9, I was getting errors from the default created rspec code that I could not diagnose:

  1) Admin::ConnectionsController POST create with valid params creates a new Connection
     Failure/Error: post :create, :connection => valid_attributes
     NoMethodError:
       You have a nil object when you didn't expect it!
       You might have expected an instance of Array.
       The error occurred while evaluating nil.each
     # ./spec/controllers/admin/connections_controller_spec.rb:54

Worse, these things were working just fine in RAILS_ENV=development.

Well, of course, it is failing on the line where the :create is invoked. But, where is the nil.each occuring?

I ran things with:

bundle exec rspec -d spec/controllers/admin/connections_controller_spec.rb \
   -e "POST create with valid params creates a new Connection"

after putting "debugger" in before the test case:

  describe "POST create" do
    describe "with valid params" do
      it "creates a new Connection" do
        # expect {
          debugger
          post :create, :connection => valid_attributes
        #}.to change(Connection, :count).by(1)
      end

(I'm still looking for a good ruby-debug mode that works like gdb-mode in Emacs works, that can show me the code around where I am...)

One winds up in the rescue in:

/var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/rescue.rb

on line 19.

So, stick a breakpoint on the super there:

break /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/rescue.rb:17

This lets you see the exception:

(rdb:1) p exception
#<NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.each>

The annoying part is that the action is invoked at /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/instrumentation.rb:29

ActiveSupport::Notifications.instrument("process_action.action_controller", raw_payload) do |payload|

so, it evaluates code, and there are in fact one block passed to another block, and it seems really hard (a major ruby-debug limitation), that I can not put a breakpoint easily into the beginning of a block passed in.

I had to resort to editing that file, and sticking "debugger" in there!

Finally, one gets to:

/var/lib/gems/1.8/gems/actionpack-3.0.9/lib/abstract_controller/base.rb:150
send_action(method_name, *args)

In the debugger, the right thing to do is:

catch NoMethodError

This finally shows me that the failure is at:

/corp/projects/credil/hydra/t3041/vendor/plugins/active_scaffold/lib/active_scaffold/attribute_params.rb:42

Why? Because attributes is nil.

Why, because the generated controllers spec file says:

    describe "with valid params" do
      it "creates a new Connection" do
        expect {
          post :create, :connection => valid_attributes
        }.to change(Connection, :count).by(1)
      end

should have been generated as:

    describe "with valid params" do
      it "creates a new Connection" do
        expect {
          post :create, :record => valid_attributes
        }.to change(Connection, :count).by(1)
      end

Syndicated 2011-12-01 11:50:00 (Updated 2012-02-05 17:28:21) from Michael's musings

Domain Squatter Avoidance tool

Here is a nice use for a distributed hash table, backed by the new IETF REPUTE work.

I just typed "antipope.net" rather than antipope.org to get to Charlies Stross' web site. A squatter offered to sell me the domain. Some of the squatters do it solely for ad revenue, and I'd rather not arrange for them to get a dime.

I want a button for my browser (Chromium) which logs that name into a reputation database indicating that these guys are squatters, and letting me (once I know the correct name) enter the proper name. The same plugin will consult that database if I type something wrong, and suggest an alternative.

Syndicated 2011-08-08 09:31:00 (Updated 2011-12-01 17:05:26) from Michael's musings

Eclipse and Android SDK never ran

I've had a problem getting Eclipse, and specifically the Android SDK to run on my Debian laptop for over a year now. I've generally just VNC'ed to a more powerful box and ran it there.

The problem I had was that most network operations in eclipse would fail with network unreachable. Not a big deal for day to day things, but you need the network to install the Android SDK kits and install Eclipse plugins.

I had been trying to strace things to figure out what it was, and finally found it:

connect(26, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "::ffff:74.125.95.91", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)

Huh, it's doing IPv6 connections. GOOD. But, it hasn't set the right IOCTL on the socket to permit IPv4 mapped connections to work, and on Debian, the bindv6only is now not set.

See: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560056

Syndicated 2011-07-19 00:51:00 from Michael's musings

Eclipse and Android SDK never ran

I've had a problem getting Eclipse, and specifically the Android SDK to run on my Debian laptop for over a year now. I've generally just VNC'ed to a more powerful box and ran it there.

The problem I had was that most network operations in eclipse would fail with network unreachable. Not a big deal for day to day things, but you need the network to install the Android SDK kits and install Eclipse plugins.

I had been trying to strace things to figure out what it was, and finally found it:

connect(26, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "::ffff:74.125.95.91", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)

Huh, it's doing IPv6 connections. GOOD. But, it hasn't set the right IOCTL on the socket to permit IPv4 mapped connections to work, and on Debian, the bindv6only is now not set.

See: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560056

Syndicated 2011-07-18 20:51:00 from Michael's musings