Older blog entries for mcr (starting at number 84)

1 Dec 2011 (updated 5 Feb 2012 at 17:28 UTC) »

Active Scaffold obscures internal errors

In a newly scaffold'ed model and controller, created with ActiveScaffold 3.0.5, on rails 3.0.9, I was getting errors from the default created rspec code that I could not diagnose:

  1) Admin::ConnectionsController POST create with valid params creates a new Connection
     Failure/Error: post :create, :connection => valid_attributes
     NoMethodError:
       You have a nil object when you didn't expect it!
       You might have expected an instance of Array.
       The error occurred while evaluating nil.each
     # ./spec/controllers/admin/connections_controller_spec.rb:54

Worse, these things were working just fine in RAILS_ENV=development.

Well, of course, it is failing on the line where the :create is invoked. But, where is the nil.each occuring?

I ran things with:

bundle exec rspec -d spec/controllers/admin/connections_controller_spec.rb \
   -e "POST create with valid params creates a new Connection"

after putting "debugger" in before the test case:

  describe "POST create" do
    describe "with valid params" do
      it "creates a new Connection" do
        # expect {
          debugger
          post :create, :connection => valid_attributes
        #}.to change(Connection, :count).by(1)
      end

(I'm still looking for a good ruby-debug mode that works like gdb-mode in Emacs works, that can show me the code around where I am...)

One winds up in the rescue in:

/var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/rescue.rb

on line 19.

So, stick a breakpoint on the super there:

break /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/rescue.rb:17

This lets you see the exception:

(rdb:1) p exception
#<NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.each>

The annoying part is that the action is invoked at /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/instrumentation.rb:29

ActiveSupport::Notifications.instrument("process_action.action_controller", raw_payload) do |payload|

so, it evaluates code, and there are in fact one block passed to another block, and it seems really hard (a major ruby-debug limitation), that I can not put a breakpoint easily into the beginning of a block passed in.

I had to resort to editing that file, and sticking "debugger" in there!

Finally, one gets to:

/var/lib/gems/1.8/gems/actionpack-3.0.9/lib/abstract_controller/base.rb:150
send_action(method_name, *args)

In the debugger, the right thing to do is:

catch NoMethodError

This finally shows me that the failure is at:

/corp/projects/credil/hydra/t3041/vendor/plugins/active_scaffold/lib/active_scaffold/attribute_params.rb:42

Why? Because attributes is nil.

Why, because the generated controllers spec file says:

    describe "with valid params" do
      it "creates a new Connection" do
        expect {
          post :create, :connection => valid_attributes
        }.to change(Connection, :count).by(1)
      end

should have been generated as:

    describe "with valid params" do
      it "creates a new Connection" do
        expect {
          post :create, :record => valid_attributes
        }.to change(Connection, :count).by(1)
      end

Syndicated 2011-12-01 11:50:00 (Updated 2012-02-05 17:28:21) from Michael's musings

8 Aug 2011 (updated 1 Dec 2011 at 17:05 UTC) »

Domain Squatter Avoidance tool

Here is a nice use for a distributed hash table, backed by the new IETF REPUTE work.

I just typed "antipope.net" rather than antipope.org to get to Charlies Stross' web site. A squatter offered to sell me the domain. Some of the squatters do it solely for ad revenue, and I'd rather not arrange for them to get a dime.

I want a button for my browser (Chromium) which logs that name into a reputation database indicating that these guys are squatters, and letting me (once I know the correct name) enter the proper name. The same plugin will consult that database if I type something wrong, and suggest an alternative.

Syndicated 2011-08-08 09:31:00 (Updated 2011-12-01 17:05:26) from Michael's musings

Eclipse and Android SDK never ran

I've had a problem getting Eclipse, and specifically the Android SDK to run on my Debian laptop for over a year now. I've generally just VNC'ed to a more powerful box and ran it there.

The problem I had was that most network operations in eclipse would fail with network unreachable. Not a big deal for day to day things, but you need the network to install the Android SDK kits and install Eclipse plugins.

I had been trying to strace things to figure out what it was, and finally found it:

connect(26, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "::ffff:74.125.95.91", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)

Huh, it's doing IPv6 connections. GOOD. But, it hasn't set the right IOCTL on the socket to permit IPv4 mapped connections to work, and on Debian, the bindv6only is now not set.

See: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560056

Syndicated 2011-07-19 00:51:00 from Michael's musings

Eclipse and Android SDK never ran

I've had a problem getting Eclipse, and specifically the Android SDK to run on my Debian laptop for over a year now. I've generally just VNC'ed to a more powerful box and ran it there.

The problem I had was that most network operations in eclipse would fail with network unreachable. Not a big deal for day to day things, but you need the network to install the Android SDK kits and install Eclipse plugins.

I had been trying to strace things to figure out what it was, and finally found it:

connect(26, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "::ffff:74.125.95.91", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)

Huh, it's doing IPv6 connections. GOOD. But, it hasn't set the right IOCTL on the socket to permit IPv4 mapped connections to work, and on Debian, the bindv6only is now not set.

See: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560056

Syndicated 2011-07-18 20:51:00 from Michael's musings

3 Jul 2011 (updated 19 Jul 2011 at 01:05 UTC) »

"Over The Top" Television

re: http://crtc.gc.ca/eng/archive/2011/2011-344.htm

In Broadcasting Regulatory Policy 2009-329, the Commission set out the results of its review of broadcasting in new media. This was followed by Broadcasting Order 2009-660, which amended, clarified and affirmed the continued appropriateness of the New Media Exemption Order applied to new media broadcasting undertakings. Since then, there has been an acceleration of technological, market and consumer behaviour trends that may influence the Canadian broadcasting system's ability to achieve the policy objectives of the Broadcasting Act. Increasingly, programming is being provided by entities on multiple platforms and separate from the physical infrastructure over which it is delivered. These "over-the-top" entities are both foreign and domestic.

1. My name is Michael Richardson. I am chief scientist of Sandelman Software Works. I am writing today about your consultation about "Over-The-Top" television, such as "netflix" and others like this. I am a pioneer of the Internet, my use of it dates back to 1987. I am active participant in the Internet Engineering Task Force, and I've authored a number of RFCs in the security field.

2. I find the entire question about "over-the-top" to itself be indicative of a bias to begin with. My question was, over top of what? I get as much television "over" Canada Post as I do "over-the-air".

3. The Internet does not run on top of other things, more and more, other things run on top of the Internet. Neither incumbent cable or telephone companies have been competent enough to supply my home office for internet. My family that has tried them for Internet has found their service to be lacking, and have gone to reliable Internet suppliers, ones that are not vertically integrated and therefore do not have a bias against other things.

4. Since 1995, I have not subscribed to "cable" TV. I tried microwave (LOOK), but when I moved it was not available, and then I went to satellite (Star Choice, now Shaw). Since it became Shaw, my level of service has steadily declined, while my rates have gone up. My family uses the satellite TV less and less (we are now on the lowest tier subscription, primary for US Network channels) and relies on DVD delivery from ZIP and netflix over my bridged-DSL connection with Storm Internet.

5. Netflix has reported "problems" with Canadian residential internet connections. I have none. I do not use an incumbent telco with a competing service as my supplier. Please connect the dots.

6. I do not use "HD" services at this time, as I have no TVs like that. I consider current HD TV systems to be too inflexible and yet too complicated for my use. When the time comes, I will replace the "screens" in my home with dumb computer-grade displays, connected to media boxes running open standard systems.

7. The available content on Netflix leaves a lot to be desired. The amount in Canada, I'm told is much less than in the US due to licensing problems. This upsets me greatly: I would like to see a mandatory licensing regime that seperated who I choose to deliver the content I want, from what content is available.

8. Netflix offers a service that apparently permits some Apple and some Microsoft users to watch television their computers. This system uses a proprietary copyright infringing system to display the content. I say that it infringes the copyright laws because it appears that this "Digital Rights Management" system in fact denies me rights that I would have on other systems. This system is incompatible with non-Microsoft systems (tied selling) such as Ubuntu Linux that runs at my house.

9. We happen to have a Nintendo WII game console that has a netflix system for it, and I'm told that the Netflix application for it may also contain DRM. However, the output of my WII is a DRM-free analogue signal, and therefore my rights are identical with this system as they would be with broadcast television.

10. I am preparing myself for ATSC. I intend to put an antenna on my roof to receive US Network Channels from Rochester NY, and along with an ATSC tuner on each of my three TVs, I should be able to get Ottawa broadcast channels from Camp Fortune. At that point I will stop subscribing to satellite service: they have provided me with essentially no value.

11. At this point, what I would like is the ability to pay for the content that I want. I would like to be able to vote with my wallet, rather than have the CRTC tell me. I expect some service (such as Netflix, or a competitor) to offer to intermediate my transactions, reducing the cost of the transaction, and dealing the production studios directly.

12. I would like to:

a) provide a tip of approximately 0.25 for a show that I like. This would be voluntary by me. I would do this because I want them to produce more like it. I want to do this even for shows that might have been out of "print" for a long time, for instance Threes Company, or old episodes of Sesame Street, which continue to have significant value. Right now, at most, I can provide a "star" rating.

b) provide a bond (a promise) that I would tip for more episodes of a series that I like. This removes the role of the executives of i) the incument cable/satellite companies, ii) the specialty channels. who it seems continue to be reluctant to take risks, and have significantly disrupted shows with significant fan bases with very good writing. If this scares these companies, tough. The CRTC has no mandate to protect companies with out-dated business plans.

c) provide a tip to a "network" such as CBCKids who might provide me with a playlist of shows to watch and timely interactive ways to engage kids. Note I would be tipping for the playlist (a list of recommendations) not for the shows themselves.

13. This is particularly important to me for children's shows, as I will only let me child watch the TV stations that do not feature advertising.

End of Document

Syndicated 2011-07-03 18:16:00 (Updated 2011-07-19 01:05:15) from Michael's musings

9 May 2011 (updated 9 May 2011 at 20:11 UTC) »

Problems (insecurities) in ActiveResource

I have an application that talks to Redmine/Chiliproject using its API with results in JSON. I use ActiveResource to make these calls, and it suddendly started failing after an upgrade from redmine to chiliproject:

test_retrieve_the_thomas_watson_project_by_id(ProjectTest):
ActiveRecord::UnknownAttributeError: unknown attribute: created_on

The fact that I was getting an error from ActiveRecord and not ActiveResource was puzzling. My ActiveResource class was called ProjectResource. The thing that I was retrieving was a "project", and yes, I happened to have a model called "Project", which was a subclass of ActiveRecord.

Looking at the JSON results using curl:

marajade-[~/C/dracula/hourbank3] mcr 10293 %curl 'http://localhost:3100/projects/show/16?format=json&key=abcdAPIKEY09123456789'
{"project":{"description":"Voice and Video softphone system for Android, with SIP support.","updated_on":"2010/10/08 10:10:24-0400","identifier":"thomas-watson","homepage":"","name":"Thomas-Watson","created_on":"2009/08/23 12:21:38 -0400","id":16}}

and also in the debugger, at

(rdb:1) c
Breakpoint 1 at /var/lib/gems/1.8/gems/activeresource-3.0.4/lib/active_resource/base.rb:889
/var/lib/gems/1.8/gems/activeresource-3.0.4/lib/active_resource/base.rb:889
new(record).tap do |resource|
(rdb:1) p record
{"project"=>{"name"=>"Thomas-Watson", "created_on"=>"2009/08/23 12:21:38 -0400", "id"=>16, "updated_on"=>"2010/10/08 10:10:24 -0400", "homepage"=>"", "description"=>"Voice and Video softphone system for Android, with SIP support.", "identifier"=>"thomas-watson"}}

what happens next is that the word "project" is passed to

find_or_create_resource_for(key)

and this finds and returns the "Project" class which is in my model. My model does not have a field created_on, thus the error.

So there three problems with this behaviour:

additions to the API should not break my old code, I should just ignore them.

there is no guarantee that the class that was found, "Project" has any of the behaviour that I need in the thing returned from ActiveResource.

worst, since the word "project" came from the remote system, the remote system could pick any class it wanted and invoke code on it. It's a reverse attack by a server on a client, but it's wrong to assume that the server is fully trusted by the client.

I'm not sure what the easiest way to fix this, but it's certainly wrong, and it's been there awhile in ActiveResource.

Syndicated 2011-05-09 15:07:00 (Updated 2011-05-09 20:11:02) from Michael's musings

24 Apr 2011 (updated 9 May 2011 at 19:11 UTC) »

A novel way to do PBX extensions

At CREDIL we are expanding our Asterisk out to service the entire floor. We didn't do our extensions particularly efficiently (numberwise), and I was thinking about ways to do them.

A really (math) geeky way occured to me: give employee number n the n+2'th prime (1-first prime, 2-second prime, first employee gets extension 3).

Then, if you need to have a conference call with employees number 4, 6 and 9, then you need to dial their product. Primes are 1,2,3,5,7,11,13,17,19,23,29,31,.. 4+2 = 6th prime is 11, 6+2=8th prime is 17, and 9+2=11th prime is 29. So dial 11*17*29 = 5423.

Primes are still in the 4 digits for the first 1000.

http://primes.utm.edu/lists/small/1000.txt

All multiples of your extension are yours to do anything you want with, and since the multiples times powers of 2 are never conference bridges, you have a lot of bits you can use to encode useful things. Want to call me and avoid ringing me? Okay, set bit number 2. Want to call me and never go to voice mail? Okay, set bit number 3... etc.

Syndicated 2011-04-24 15:15:00 (Updated 2011-05-09 19:11:44) from Michael's musings

21 Apr 2011 (updated 24 Apr 2011 at 20:10 UTC) »

Time for a new Monarch

To Her Majesty Her Majesty Elizabeth the Second,

by the Grace of God, of Great Britain, Ireland and the British Dominions beyond the Seas Queen, Defender of the Faith, Duchess of Edinburgh, Countess of Merioneth, Baroness Greenwich, Duke of Lancaster, Lord of Mann, Duke of Normandy, Sovereign of the Most Honourable Order of the Garter, Sovereign of the Most Honourable Order of the Bath, Sovereign of the Most Ancient and Most Noble Order of the Thistle, Sovereign of the Most Illustrious Order of Saint Patrick, Sovereign of the Most Distinguished Order of Saint Michael and Saint George, Sovereign of the Most Excellent Order of the British Empire, Sovereign of the Distinguished Service Order, Sovereign of the Imperial Service Order, Sovereign of the Most Exalted Order of the Star of India, Sovereign of the Most Eminent Order of the Indian Empire, Sovereign of the Order of British India, Sovereign of the Indian Order of Merit, Sovereign of the Order of Burma, Sovereign of the Royal Order of Victoria and Albert, Sovereign of the Royal Family Order of King Edward VII, Sovereign of the Order of Merit, Sovereign of the Order of the Companions of Honour, Sovereign of the Royal Victorian Order, Sovereign of the Most Venerable Order of the Hospital of St John of Jerusalem

(see http://en.wikipedia.org/wiki/List_of_titles_and_honours_of_Queen_Elizabeth_II )

On this, Our Birthday, where I turn 40, and you are still more than twice my age, and likely four times my wisdom, I wanted to share some thoughts I have had over the last few years.

I am your direct subject, having been born in London, as as well as your loyal subject in the "British Dominions beyond the Seas". I'm actually a fan of having a monarch, which is rather unpopular these days. I even met Your Highness once when you visited Fredericton, but I actually too little to know enough to be impressed.

First, congradulations on celebrating the marriage of your grandson. I know that things will go well next week, and we look forward his visit to Ottawa this summer.

I am sure that you have given a lot of thought to succession. I wondered if you had considered that Prince William would very nice King. A very nice Young King, one who could rally the youth of today, and bring a unity that politicians yearn for, but have seldom delivered.

Does Prince Charles actually want to be King? Perhaps after a brief Honeymoon, you and Prince Charles might consider abdicating in favour of Prince William.

I suggest sometime in 2012, maybe Feb. 29 would auspicious, or maybe April 21, 2012. I don't know: I am sure you will come up with something sensible.

Syndicated 2011-04-21 10:39:00 (Updated 2011-04-24 20:10:45) from Michael's musings

17 Mar 2011 (updated 21 Apr 2011 at 15:11 UTC) »

Dreamhost SSL certificates --- insecure

Dreamhost sells third-level GeoTrust SSL security certificates for $15/year. (You have to be an existing customer).

It seems however, they do not give you the chance to upload a CSR file. Instead, you are expected to fill out the DN information online, and then they generate a private key for you. And they keep the private key around in their database.

It also winds up in your browser cache, and if you have kind of a "trusted" SSL proxy between you and the Internet (like half of corporate users have), then it's gonna be in the cache of that device too.

This is a FAIL. Not only is your private key subject to whatever insecurity their might have, but it's total FBI Patriot Act fodder.

(If there is some place to upload a CSR, we couldn't find it)

Syndicated 2011-03-17 13:13:00 (Updated 2011-04-21 15:11:53) from Michael's musings

2 Mar 2011 (updated 17 Mar 2011 at 18:14 UTC) »

Deploying Django applications with Capistrano

Yesterday, I cooked up a deploy.rb so that Capistrano can deploy a Django application. While there is a Python app called http://docs.fabfile.org/0.9.0/ from what I could tell, it was very general to running commands on multiple servers, and not really specific to checking out a web framework and deploying it to one or more servers.

First, my deploy.rb, and then my notes about how I used it. I have changed only one or two things from my real code. My application is called "clientportal" and the host running it is called "clientportal.isp.example.net". On the server, it runs as a user called "clientportal".

This code does not yet invoke the Django database migrations, which it ought to, and I'll do another blog post once I figure out that part.

set :application, "clientportal"
set :me, "#{ENV['LOGNAME']}"
set :repository,  "git+ssh://#{me}@code.credil.org/git/path/to/repo/clientportal"

set :scm, :git
set :user, :clientportal

set :ssh_options, { :forward_agent => true }
set :use_sudo, false
set :git_enable_submodules, true
set :deploy_to, "/home/#{user}/#{application}"

role :web, "clientportal.isp.example.net"     # Your HTTP server, Apache/etc
role :app, "clientportal.isp.example.net"

# This is where Rails migrations will run
role :db,  "clientdb.isp.example.net", :primary => true

namespace :deploy do
  task :start do ; end
  task :stop do ; end

  # this overrides a rails specific thing.
  task :finalize_update do ; end
  task :migrate         do ; end

  task :restart, :roles => :app, :except => { :no_release => true } do
    # something to restart django.
    run "sudo /usr/sbin/apache2ctl graceful"

  end
  task :update_database_yml, :roles => [:app,:web] do
    db_config = "/home/#{user}/settings.py"
    run "cp #{db_config}   #{release_path}/settings.py"
    run "ln -f -s #{release_path} /home/clientportal/clientportal/clientportal"
    puts "Ran update database settings"
  end

end

after "deploy:update_code", "deploy:update_database_yml"

Some details. First, I put my settings.py file into my /home/clientportal directory. I do not check this file into my repo, because it always specific to the installation (it's different on your laptop than on the devel server or the production server). Also see my:

Like http://blog.perplexedlabs.com/2010/02/08/deployment-using-capistrano-and-webistrano-via-rails-and-phusion-passenger/ I had to adjust my django.wsgi file as well. I wound up with:

import site
site.addsitedir('/usr/local/pythonenv/CLIENTPORTAL/lib/python2.5/site-packages')

import os, sys

sys.path.append('/home/clientportal/clientportal')
sys.path.append('/home/clientportal/clientportal/current')
os.environ['DJANGO_SETTINGS_MODULE'] = 'clientportal.settings'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

The important changes were to the path that was added. It used to add $HOME/clientportal and $HOME to the path, but now it is one directory deeper, and you will notice above in the update_database_yml task that it creates a symlink in $HOME/clientportal with the name "clientportal" that is essentially the same as "current".

This is necessary because the settings are loaded as "clientportal.settings", and python basically turns the . into a / when looking for the file. I could have just changed the name of the settings file, but we had other modules that were loaded using the clientportal. namespace.

Note that the server already had it's apache configured to do what was needed. I would normally package these config files up into a .deb file, but I haven't done that yet for this project, it being my first django project.

I am not sure if I actually have to restart apache. I added that for good luck, and and I added:

clientportal ALL=NOPASSWD: /usr/sbin/apache2ctl graceful
to sudoers.

My apache config looks like:

<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        ServerName clientportal.isp.example.net
        ServerAlias portal1.isp.example.net
        ServerAlias portal.example.net

        DocumentRoot /home/clientportal/clientportal/current
        <Directory "/home/clientportal/clientportal/current">
                Options Indexes FollowSymLinks
                Options -MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

        ErrorLog /var/log/apache2/error.log

        Alias /media/ /home/clientportal/clientportal/current/media/
        WSGIScriptAlias / /home/clientportal/clientportal/current/wsgi/django.wsgi
        <Directory /home/clientportal/clientportal/current/apache/>
                Order allow,deny
                Allow from all
        </Directory>
...

Some other links I found, but I didn't use much: http://groups.google.com/group/django-developers/browse_thread/thread/f34e59275e04f9c5?pli=1 http://gnuvince.wordpress.com/2008/01/10/deploying-django/

Syndicated 2011-03-02 12:06:00 (Updated 2011-03-17 18:14:24) from Michael's musings

75 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!