Older blog entries for johnnyb (starting at number 180)

Everything Going Wrong at Once
also known as
Shared memory problems on Postgres Bootup

Okay, so here's the deal. I am back in school getting a Master's in Theology. So, on Tuesdays I am at the seminary. I have one class that I need to be at the seminary during the day for - it's from 12:10PM to 1:15PM. So, I just bring all my work there and just work from the seminary for the day, and go to the class at lunch.

Today, at 11:30, my crew told me that one of our servers was very slow. At 11:50, the wireless internet at the seminary went out. At noon, I decided to use the bluetooth networking on my phone. I discover, to my horror, that the machine is completely down. I can't ssh in or anything. I call the other guys in the department --- oops, making a phone call disconnects my Internet access. We figured out it was a memory overcommit problem. I called our ISP and had them reboot the box. It comes up to a shell, and I can log in remotely. However, every page is erroring out because of the database.

And then...

The database just won't start. I try it again and again. The error log says:

FATAL: pre-existing shared memory block (key 5432001, ID 7667712) is still in use
HINT: If you're sure there are no old server processes still running, remove the shared memory block with the command "ipcclean", "ipcrm", or just delete the file "postmaster.pid".

I look using ipcs, and postgres doesn't have any shared memory blocks (and it shouldn't -- we just restarted). I tried upping shmmax. No go. I tried lowering shared memory buffer usage. No go. What is wrong with the system?

It turns out, Apache was using one of Postgresql's shared memory blocks. ipcs just showed Apache. So, I turned off Apache, and then turned on Postgresql. Viola! It worked! Then apache started just fine. The only thing I can think of is that Apache chooses its block randomly, and it just happened to hit Postgres's this time.

So, by the end of all of this, it's 12:45 (too late to go to class -- I've missed the entire reason for being at seminary during the day), and then 5 minutes later the wireless comes back on.

Whew! What an hour!

11 Sep 2007 (updated 11 Sep 2007 at 21:47 UTC) »
warbler is awesome -- it allows you to _easily_ make Ruby- on-Rails applications Jruby-ish, and war'd up to deploy in a container. Here are the instructions for setting this up under JBoss, but except for the last step, should run under any standard Java web container.

Here's the steps:

1) install jruby 2) install warbler

jruby --command gem install warbler

3) Do config

jruby --command warble config

This creates a file called config/warble.rb which can be used to add gems and other configuration goodness.

4) If you're going to use ActiveRecord-jdbc, then following these instructions:

Download ActiveRecord/JDBC.

Install ActiveRecord-jdbc:


jruby --command gem install /path/to/ActiveRecord-jdbc-0.5.gem

Add this gem to warble.rb:


config.gems << 'ActiveRecord-JDBC'
config.gems << 'jruby-openssl'     #not required makes other things 
easier

Add the following lines to environment.rb:


if RUBY_PLATFORM =~ /java/
  require 'rubygems'
  RAILS_CONNECTION_ADAPTERS = %w(jdbc)
end   

If you are connecting straight to a PostgreSQL database, you can just include the postgres-pr gem in your warble.rb file, and keep your database.yml file normal. If you are using JDBC, following the latest tips for that (it seems to change) for configuring database.yml. If you are using a JNDI connection, then specify it like this:


production:
  adapter: jdbc
  jndi: java:putyourjndipathhere

Configuring your datasource in JNDI is up to you :)

5) Warble it

jruby --command warble

5) Deploy it

cp yourfile.war /path/to/jboss/deploy

The only problem I have right now is that on every request it generates a warning message about having to close my connection for me. I'm not sure if that's a warble config thing or a Postgres thing or a JRuby thing or a JDBC thing.

I hope I didn't leave anything out :)</pre>

Been experimenting with evidence-based medical search and building medical systematic-reviews. Playing with a format for entering and viewing stuff. The first thing I did was put in some of the data for a systematic review of Surgery as a therapy for sciatica, which includes an evidence summary and a pubmed query.

In case your curious, there is evidently no evidence that surgery does better than conservative therapy.

Another review I put in was one for saw palmetto as a therapy for BPH.

Rather than finding no evidence for effectiveness, they found evidence for ineffectiveness of saw palmetto.

This had one decent study out of this list of studies

24 Aug 2007 (updated 24 Aug 2007 at 12:10 UTC) »

Been working on some stuff for Evidence-based health search. It's still under development.

Announcing: ActiveShipping for Ruby and/or Rails

I just got the initial version of ActiveShipping ready to go. ActiveShipping is a new shipping API for ruby based on projects that my company is developing internally.

Google Code Page:

http://code.google.com/p/activeshipping/

ActiveShipping Documentation:

http://dev.newmedio.com/activeshippingdocs/

How to Launch a Website

I have been frustrated at the several website launches (heavy on the technology - not just static stuff) that my company has performed. I think this is primarily because customers don't understand technology or project management. So here are several launch tips and why I think they are important:

1) Do not wait until a site is perfect to launch.

This is important, because a site will never be perfect. But what's great is that a website can always be changed. If you wait until a site is perfect to launch it, then it will be forever delayed, the developers will be frustrated, the customer will be frustrated (why haven't you launched my site? Because you're being unreasonably picky!).

People often get the false impression that writing code is like writing a book. The editorial process should be fairly predictable and at the end everything should work. But this is a false analogy. If you were to liken a software launch to a book launch, and demanded the same standard of perfection, you have to remember that if your code is off just a little then the site doesn't work. That's like saying that a book must not only not contain any typographical errors, but that it must also not contain any factual or logical errors. In that case, you would need to vet each book by several practitioners, philosophers, and the like, and not release the book until the full content of the book was agreed by all that it is logically sound. As you can see, this could take a while, and there is no predictable way to tell when this process will be finished.

So what you _have_ to do, to finish at all, is decide that at a certain point below perfection, it is reasonable to launch. Most importantly, your real bugs will be found by your customers after you launch anyway. So trying to clean up something that you're going to have to rewrite later is just a waste of time.

Not only that - as you spend your time "perfecting", the client will spend their time thinking -- and coming up with new ways that the site is lacking, thereby postponing the launch almost indefinitely.

So the long and short of it is, LAUNCH. Even if small problems remain, launching is the only way to move forward. Without a strong ethic of launching, the site will sit forever in purgatory.

2) Launch only when people will be around to watch it break. That means that you should NEVER launch on Friday, and NEVER launch in the evening.

The problem is that #1 and #2 are in constant conflict. Clients usually have to be "ready" to launch. They don't get "ready" to launch until the evening. I have countless experiences with this. If they are "ready" to launch in the evening and you wait a day, then in the morning they will no longer be "ready" to launch. What's worst is if they are ready to launch thursday evening, because then, if you delay it, you wind up on friday, which you should NEVER EVER launch on. Then, if you delay until Monday, not only have you lost pretty much a whole week, but the client has 4 DAYS to think about new ways in which they don't like the website! This will take at least until Thursday to implement, at which point you are in the same boat.

How to Solve It?

I'm really not sure. The only thing I can think of is to stress to the client (a) the importance of launching, (b) that just because you launch doesn't mean that it can't change or be fixed, and (c) that sometimes you just need to get something out the door for real-world feedback, and that endless speculation does not help to improve a site, it only delays its launch. Also, I should add (d) that you should launch early in the day because you the client should expect some initial issues. I've never done a large-scale web technology launch that didn't need some first-day post-launch fixes.

Having said that, I must admit that I will launch a site in the evening and on a friday, for the simple fact that if I don't I know that the project will never really end. A project _can_ be successful if it has starting bumps. A project can _never_ be successful if it is never launched.

Everyone should check out Hanson's new website. Yes, that Hanson. They've grown up a lot since the old days, and their current music is fantastic. You can listen to their whole album from their website - the music player is on the top right of the page. Check out "Great Divide" especially.

Converting a Base64-encoded SHA1 hash to a hex (hexadecimal) encoded SHA1 hash

We are getting user accounts from a new company. Thankfully, they do SHA1 encoding on their passwords like we do. However, depending on which subsystem they are in, some of them are encoded in BASE64 and some are encoded as HEX. Ruby, as far as I'm aware, only handles hex digests for SHA1. Therefore, I had to figure out a way to convert these two formats. I'm sure someone else has already figured this out and there's an easy tool for this. But, if not, here's my ruby code:


sha1_hash_base64 = "nSOeWicjE1xBYo9b7fB3e9XLYhQ="
sha1_hash_hex = 
   sha1_hash_base64.
   unpack("m")[0].
   unpack("aaaaaaaaaaaaaaaaaaaa").
   map {|x| sprintf("%x", x[0])}.
   join

The output is 9d239e5a2723135c41628f5bedf0777bd5cb6214 which I believe to be correct (I don't have my data in front of me, so I can't be sure). At minimum, it is the right length :)

So now, I can use this value to compare against the output of Digest::SHA1.hexdigest.

Hideous annoyances with IVR (voice and touch-tone interactive) systems:

  • If an IVR doesn't have a specific option to speak to a human, the IVR should be deactivated permanently, because the owners are too stupid to use one.
  • Unless there is a REALLY good reason, IVRs should make it _easy_ to do everything with the dial pad. Are you really doing secure banking over the phone if you have to speak your password or even your account number or your SSN? It is just easier for the entire interaction if you just push numbers.
  • Don't sound like you are being conversational. IVR systems ARE NOT conversational, so pretending to be so is just stupid and annoying. "Let me see" is a colloquialism of a human speaking. If you are not a human, you shouldn't be speaking this way. I hate using the phone anyway, and if you make me listen to idiocy while I'm on the phone I'm going to hate your company for all eternity.

If I'm going to bother to interact with an IVR, then I want it to be:

  • Serious. I'm making a serious call. If I wanted to talk to someone about a non-serious subject, I'd call my 4-year-old.
  • VERY CLEAR. Most IVR calls I make are about money or service. I want the communication to be clear and exact. If an IVR says "Ummm, okay", it is being personable. If they want the pretext of being personable, then GIVE ME A FREAKIN PERSON! Likewise for dialing versus speaking. Dialing is unambiguous. If I dial the wrong thing, it's my fault. But speaking is a different story. Speaking can be very ambiguous. So while I am not a big fan of any IVR, I _prefer_ those that can be dialed to those that you have to speak to. Both sides of the communication must be clear and exact. Don't BS around.
  • Optional. Computers are stupid, and computer systems are quirky. I don't know all of the quirks of your system, but I can bet that an operator does. So if there is some question, it should ALWAYS be an option to go to a real human being.
Simple Tabbed Pane/Panel using Rails

app/helpers/application.rb:


def tabbed_panel(base_id, &block)
	panels = []


concat("<div class='tabbed_panel' id='" + base_id + "'>\n", block.binding); concat("<ul class='tabselector' id='" + base_id + "_tabs'>\n", block.binding);

yield panels panels.each do |panel| concat("<li class='pane_tab tab_unselected' id='" + panel[2] + "_tab'>" + link_to_function(panel[0], "select_panel('" + base_id + "', '" + panel[2] + "')") + "</li>", block.binding) end

concat("</ul>\n", block.binding); concat("<ul class='tabpanes' id='" + base_id + "_panels'>\n", block.binding); panels.each do |panel| concat("<li class='panel_panel panel_unselected' id='" + panel[2] + "_panel'>" + panel[1] + "</li>\n", block.binding) end concat("</ul>\n", block.binding); concat("</div>\n", block.binding); end

public/javascripts/panel.js:


function select_panel(baseid, panel_id_base) {
	var tabs = $(baseid + "_tabs").childNodes;
	var tab_id = panel_id_base + "_tab";
	var panel_id = panel_id_base + "_panel";
	$A(tabs).each( 
		function(the_tab) {
			if(the_tab.nodeName == "LI") {
				the_tab.removeClassName("tab_unselected");
				the_tab.removeClassName("tab_selected");
				if(the_tab.id == tab_id) {
					the_tab.addClassName("tab_selected");
				} else {
					the_tab.addClassName("tab_unselected");
				}
			}
		}
	);
	var panels = $(baseid + "_panels").childNodes;
	$A(panels).each(
		function(the_panel) {
			if(the_panel.nodeName == "LI") {
				the_panel.removeClassName("panel_unselected");
				the_panel.removeClassName("panel_selected");
				if(the_panel.id == panel_id) {
					the_panel.addClassName("panel_selected");
				} else {
					the_panel.addClassName("panel_unselected");
				}
			}
		}
	);
}

public/stylesheets/panel.css:


/* Clearly you need better CSS, but this is a decent start */
.panel_selected { display: block; }
.panel_unselected { display: none; }
.tab_selected { background-color: gray; }
.tab_unselected { background-color: white; }

Then, to use it in your rails app, just put this in the view:


<% 
#3-panels to choose from in the following example:
tabbed_panel 'all_props' do |panels| 
	panels << ["Form Properties", render(:partial =>
"basic_info"), "fprops"]
	panels << ["Toolbox", render(:partial =>
"toolbox"), "tbox"]
	panels << ["Element Properties", render(:partial =>
"properties"), "eprops"]
end 
%>

171 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!