Advogato blog for Pizza

5 Feb 2012

Sun, 5 Feb 2012 01:25:00 GMT

Good lord, can we get new account creations turned off? The spambots are creating accounts almost as fast as I can flag them as spam...

The (continuing) joy of photo printers (and free software)

Sat, 9 Jul 2011 05:09:40 GMT

Way back in 2007 I picked up a Canon SELPHY ES1 compact photo printer. After some gnashing of teeth, I hacked support into gutenprint, and all was good. Canon meanwhile announded three successive generations of their ES-series of printers, and I just assumed they worked the same way, since the CP-series were all compatible with each other.

It turned out the ES-series weren't. In fact, every generation was subtly incompatible. (Indeed, the latest generation of CP-series is also incompatible with everything else!). But thanks to a motivatd soul, I was supplied with printer dumps for nearly every dyesub printer Canon created. It piqued my interest, and I started generating dumps for the rest of the ES-series, filling in the support matrix for all paper types. The code was committed to guetenprint about a week ago, but remained untested.

I decided to trolling eBay for the second and third-generation printers (ES2/20 and ES3/30, respectively), and much to my surprise, my lowball bids on two of them won. My "new" ES2 turned up a couple of days ago, and the ES30 should be here tomorrow. I spent late last night and part of tonight rewriting the smart print spooler the ES1 needed, and about 30 minutes ago I finally succeeded with successful prints with the ES1 and ES2.

The ES2 is slightly larger than the ES1, has a better UI and a more efficient paper path. It also adds Memory Stick support. Otherwise, it's identical under the hood, with the same print speed and output quality as the ES1. We'll see what the ES3/30 brings to the table when I eventually end up with one.

But in keeping with tradition, here's the first picture I (successfully) printed with the ES2:

This was shot outside my office a couple of years ago.

I'd like to find a colorimeter so I can get a calibrated printer profile, but as usual, there's an endless line of more important stuff I'd like to get first -- like some grids for my studio flashes. Damnit, photography's an expensive thing to dabble in..

dmraid sucks. mdraid sucks less. 3Ware FTW.

Mon, 7 Feb 2011 19:13:17 GMT

About two hours ago I was happily puttering around in a shell on the server that hosts most of my digital existence. Until, after a routine software update, I was suddenly presented with some disturbing sights:

[pizza@stuffed] ~]$ yum  
Bus error
[pizza@stuffed] ~]$ dmesg
-bash: /bin/dmesg: input/output error

It seems that something had gone very, very wrong. I trumped home to find the console full of disk error messages. But that shouldn't have brought the system down; The OS was installed on a pair of drives set up in a RAID1 mirror. There's no excuse for a read failure -- If one drive failed, the other should have picked up the slack and kept the system working.

Well, *should*. I made the mistake of setting up the RAID1 mirror using the "dmraid" tools, which piggyback on the motherboard's "fakeraid" metadata. Unfortunately, it seems that this mode of operation doesn't handle failures worth a damn. And that there's no easy way to migrate from the "dmraid" stuff to the very mature and robust native Linux "dmraid" tools.

The dmraid tools have one major disadvantage though -- they are much more dificult to boot from, and from the motherbord's perspective, if the primary drive fails, the array becomes unbootable. So by using dmraid intead of mdraid, I ended up trading one (minor) failure case for a much more serious one.

How does one work around this quandry? By going for a real hardware RAID controller. The ones with onboard pocessors that do all the heavy lifting. The ones that hide the messy details from the OS. The ones that are designed to JustWork(tm) and NotFail(tm).

The ironic thing is that this server already has a hardware RAID controller in it, a 3Ware 9550SXU-8LP, but it's maxed out with two 4-drive RAID5 arrays totalling about 7TB. This server's predecessors both had 3Ware RAID controllers -- and in my decade-long experience with 3Ware controllers, I've not had a single controller-related failure, ever. They JustWork(tm), handling too-numerous-to-count drive failures cleanly and transparently.

So, I just ordered another 3Ware 9550SXU-4LP controller to plug my OS array into. It's used so I got it for cheapcheapcheap, and like the one I already have it's two generations older than their current top-line models, but it'll be more than good enough for a simple RAID1 array. Migrating the dmraid array over to the new controller will be an interesting proposition even without the flaky drive complicating things, but since downtime is unavoidable thanks to dmraid's failings, I might as well make sure things are done right.

If I hadn't picked up another hardware controller, I'd have migrated to mdraid, and booted off of an USB stick. That solution has worked great at work, and those USB sticks are trivial to back up and replicate in case of failures.

In other news, the tally from this weekend is 5581 RAW photos taking up some 62Gigs of space. Been to busy to post anything, but as my backlog empties out, you'll be seeing more here again.

On 802.11

Tue, 9 Nov 2010 18:09:48 GMT

On 802.11:

apenwarr waxed poetic about various aspects of the 802.11 standard, and as someone who knows far more about the specs than any supposedly-sane person ever should, I think I can clarify a few things.

First, if you're trying to get up to speed with it, I suggest you start with the original 802.11-1999 spec. Everything else is built on top of this, and the current 1200 pages of so of the (incomplete!) rollup spec is due to the numerous amendments that generally just add complexity and confusion when you don't know what the core protocol is about.

Second, you have to remember that 802.11 is a *radio* protocol, not a wired protocol, and subsequently the fundamental Physical (PHY) layer is completely different than a wired layer because of this. That said, your two basic questions are actually due to the 802.2/3 heritage that 802.11 builds on. But first your bit about antennae.

Radio waves aren't constrained to nice neat cables. They radiate outwards in all directions. They bounce off of things, and these bounced signals may make it to the receiver too. Since these bounced signals have to travel further, they arrive later than the original signal, causing ghosts some of you may remember from the analog TV days. This phenomena is called multipath interference, and there's a ton of complexity in the receivers to detect and deal with this.

802.11n gets its speedups over 802.11g from three things: Better modulation (65Mbps vs 54Mbps); Wider bandwidth (40MHz channels vs 20MHz, doubling throughput to 130Mbps), and finally supporting multiple simultaneous streams (which takes us up to 600Mbps with four streams, but nothing I've seen supports more than 300Mbps with two streams). The problem is that you can't transmit multiple streams from the same antenna; they'll interfere with each other. That same multipath mess that causes problems before is instead deliberately harnessed -- but to do that, you need need multiple antennae, one for each spatial stream. Similarly, you'll need multiple antennae on the receiver, one more than the number of streams.

Meanwhile. Wired ethernet is not considered "reliable" but compared to wireless, it is bulletproof. Wired ethernet can detect collisions as they happen due to every transmitter sharing the same wire (CSMA/CD), but Wireless transmitters have no way of knowing if the receiver was being locally interfered with or not. This is the reason for adding a positive acknowledgement and retransmissions at such a low level.

Similarly, stations may (and often are) highly mobile, and may drop off of a network at any time. If the station connects to a different access point, how is the rest of the network to know that it's moved? By making association an explicit action, the AP knows to send a notification to the rest of the network to update their MAC address tables. Which brings us to "why can't we join networks simultaneously?" Fundamentally, it's because each radio only has one MAC address. If the station supported using multiple MAC addresses, then it could join multiple networks. There are other factors in play (mainly synchronization/timing; a STA is slaved to the AP's clocks), but that's one of the big ones. Oh, and the 802.11 spec can't just assume everyone's using IP, and there are 802.11 chipsets that support multiple MAC addresses.

Disassociate messages are there to explicitly tell the AP to free up the resources that the STA is using. It's not strictly necessary, but instead a highly useful optimization when you consider the bigger picture of multiple APs servicing the same logical network and that a single AP can only handle so many STAs before they interfere themselves into oblivion or simply run out of resources. (Anyone who's been to tradeshows with public wifi has seen this for themselves). Also keep in mind that the AP can also send out disassociations to force the STA to hand off to a different AP or if the AP has to go away for some reason (such as switching channels due to radar interference). Explicit notifications are always preferable to implicit ones, especially on a highly unreliable medium.

QoS stuff is (unfortunately) here to stay, and provides tangible throughput improvements by adding additional mechanisms to reduce collisions and minimize round trips (and their latencies, the real throughput killer), something that "over-provisioning" simply can't deal with -- remember, you can always add more wires bonded together ad nauseum, but you can't just add more RF spectrum to achieve the same thing. Again, radio, being a shared medium, is completely different. The nodes all have to be smart to not step on each other's toes or the whole house of cards collapses.

802.11 as it stands now is actually pretty well designed; it's complicated because it is trying to solve some very complicated problems. Trying to grok the whole thing at once is migraine-inducing, but if you start from the original 802.11-1999 spec, work your way through the amendments chronologically, and keep in mind its ethernet heritage (and the fundamental differences RF brings over a hardline connection) it'll make more sense more quickly.

Anyway, I'll shut back up now..

Asus G50Vt quirks with Linux

Sat, 17 Apr 2010 05:11:19 GMT

Over the months I've owned this laptop, I've run into quite a few little quirks, and have been slowly knocking them out one by one. Tonight I managed to get the last of 'em quashed, and I thoght I'd write everything up in case anyone else finds this stuff useful.

Among the quirks: (and note these are current as of 2.6.32)

Buggy IOMMU. If you enable virtualization in the BIOS, the kernel spews out boatloads of iommu warnings/errors due to some kind of glitch. It's presumably a kernel bug, but until it's fixed, you gotta add 'intel_iommu=off' to the kernel command line.

Keyboard lagging This was particularly annoying, but adding 'i8042.noloop i8042.nopnp' to the kernel command line made quite a difference. It ain't perfect, but it's at least no longer infuriating. It's worth noting that the laggy keyboard also afflicts operation under Vista too -- only I can't do anything about that!

ExpressCard slot not working post-suspend. Due to a kernel bug, the PCIExpress hotplug driver wasn't working unless the ' pciehp.pciehp_force=1' option is added to the kernel command line. Now I can hotplug ExpressCards left and right, yay.

Post-suspend, the ethernet interface not coming up with GigE speeds. Apparently the network chipset somehow decides to stop advertising 1000Mbps support to the switch, so it's not autonegotiated properly. The workaround is pretty simple: As root, run:

'ethtool -s eth0 speed 1000M'

and the hardware re-enables 1000M operation and negotiates with the switch properly.

Headphone jack not working. When something's plugged into the headphone jack, the speakers are muted -- and so is the headphone jack. if you sorta plug it in loosely, you hear both the speakers and the headphone simultaneously. Something's screwy with the HDA Codec routings! This was a particularly annoying fix to solve, and has been broken since the 2.6.28 kernel. There's been a patch rotting in ALSA's bugtracker for more than a year now.

Tonight I got sick of the headphone jack screwiness, and started poking around and discovered the 'hda analyzer' tool; and when poking around with the various routing lines referenced by the patch, I suddenly start hearing output from the headphones. Plugging and unplugging them JustWorks(tm). Apparently the default routing for the headphone jack is incorrect. A bit more digging led me to the hda-verb tool, which can then be invoked with:

hda-verb /dev/snd/hwC0D0 0x21 SET_CONNECT_SEL 0x0d

And voila, everything JustWorks(tm). Yay!

I added a rule to the pm-utils scripts that kicks both ethtool and hda-verb to fix things up after a suspend, and I'm as happy as a clam now.

When combined with the 'asusg50oled' app that does something useful with the little OLED display above the keyboard, this laptop is actually more functional in Linux than it is under Vista -- It won't come out of a suspend properly there -- and I can even directly control the LEDs around the touchpad. oooo.

Other joy I encountered included a slightly buggy ExpressCard CompactFlash adapter. Modern CF cards, basically being full PATA implementations, can operate at UDMA speeds -- current high-end cards can sustain 90MB/s throughput, well in excess of the ~25-ish that the best USB-based readers can accomplish. I picked up a 'SIIG ExpressCard/54 R/W' adapter, which promises to let me run all-out. So, once I got the hotplug problem fixed up, I slapped in my UDMA-capable CF card.. and.. barely managed 20MB/s, actually worsethan my USB reader.

Further investigation showed that the kernel PATA later was saying that the '80-wire detection' was failing, forcing the card to revert to at most UDMA-33 speeds. Apparently the CF adapter wasn't reporting the proper cable status -- CF cards by definition are 40-pin, but the cables are nonexistent so they can operate at full UDMA/133 speeds if they support it. Unfortutaely, SIIG used the same PCI subvendor/model ID as the reference design for the PATA chipset... so there was no way to hack in a kernel quirk to work around this buggy hardware. But not all was lost!

There was a way to force this to be overridden, but only on a per-adapter basis -- and since the ExpressCard is removable, each time a hotplug event happens it's assigned a new adapterid. I was able to work around this by forcing the kernel to default to a "short 40-pin cable" mode, and explicitly setting the hard drive and DVD drive to full sata operation -- adding 'libata.force=short40c,1:sata,2:sata' to my kerel command line. And my "300X" CF cards sustain 40MB/s. Whee!

So what's my cmdline after all of this?

"ro root=/dev/sda7 rhgb pciehp.pciehp_force=1 intel_iommu=off SYSFONT=ter-u16b LANG=en_US.UTF-8 KEYTABLE=us libata.force=short40c,1:sata,2:sata i8042.noloop i8042.nopnp"

Yeah, it's a mouthful..

To PostGIS Or Not To PostGIS, that is the question...

Tue, 17 Nov 2009 16:10:33 GMT

Development of Photo Organizer has slowed down lately, in part thanks to RealLife(tm) getting in the way, but mostly due to the remaining feature requests becoming increasingly more invasive. This isn't to say that these features aren't a good idea, but rather that due to PO's craptastic code structure, a seemingly "minor feature" would require a major internal overhaul.

Features like replacing the internal permission model with a finer-grained group-based model. Moving to a real templating engine. Better "social" features. Adding an external RPC API. Adding some sort of caching of search results or other complex queries that involve permission tests. And so on.

One deceptively simple feature request is to integrate PostGIS support. While PO currently extracts GPS data out of images and stores it in the database, it doesn't really do anything useful with that data. Integrating PostGIS support would instantly give PO access to a very powerful geospatial backend that can tie in to all sorts of other spatially-aware systems. There is a near-endless list of upsides, even if PO never uses anything more advanced than spatially-aware searching.

The downsides, however, are doozeys -- From an administration perspective rather than from a code perspecitve. First, due to the level of effort it would take to make PostGIS support optional, we'd have to require it across the board. PostGIS is not part of the standard PostgreSQL distribution, and would consequently make setting up a PO installation more difficult. It would greatly complicate upgrading an existing PO installation to a newer version of PostgreSQL and/or PostGIS, and upgrading to newer PO releases could also get more complex.

So all of that said, PostGIS support would be interesting and cool, but is it necessarily the right direction to take? I know PO is already used by at least one municipality to hold photos relating to their tax rolls, but without a better idea of real-world workflows, I don't know what PO can do to better tie in to the rest of their (or anyone else's) systems.

Meanwhile, regardless of PO's support for PostGIS, more user-visible features like "pull up a google map with locations of this set of photos marked" can be implemented, and now that I have a GPS widget for my camera, I'm actually interested in such things. :)

I get nearly no feedback from PO users; indeed aside from the freshmeat subscriber stats I really have no idea how many folks actually use PO. My best efforts with Google show a few dozen public PO installations, including at least two which the admins have independently translated into Russian. Come on folks, send me patches so all users can benefit from this work!

So, peanut gallery, any thoughts?

To PostGIS Or Not To PostGIS, that is the question...

Mon, 13 Oct 2008 16:09:15 GMT

So, peanut gallery, any thoughts?

Photo Organizer 2.36 is (finally) out

Mon, 18 Aug 2008 01:07:40 GMT

It's been stuck in -rc status for four months. Much less feedback this time around, which can be attributed to less interest, or perhaps the code's been more robust this time around. We'll see.

There are many more user-visible changes than usual this time around, ihcluding a nice dark theme, pretty URLs, and per-folder/album thumbnails. Oh, and a 40x speed improvement on a hot-path sql query. Yikes.

Each release has made PO's internals less obnoxious and easier to change, but I've hit another brick wall and the next set of internal improvements will be pretty invasive, with no real user-visible benefit.

Unfortunately, development has slowed down considerably lately, in part due to RealLife(tm).. but as always, it's nice to get feedback.

I also just switched PO over to using git. Due to differences in the usage model (from svn), there was no easy way to migrate the old history in the same repo and still continue using git's best pracices. C'est la vie.

Photo Organizer 2.35

Wed, 20 Feb 2008 03:11:12 GMT

Yeah, Photo Organizer 2.35 came out two weeks ago, but I'd figure I should toot my own horn a little bit.

A lot of work went into making client/event management more, well, manageable. Multi-day events and the ability to directly tie clients to events tie into date-based searching to make it easy to find out just what you took for any given point in time.

Also new is pluggable authentication, two-step registration, sortable folder/album listings, much (much) faster exporting, plus a large pile of under-the-hood changes to facilitate future features. Oh, and an Italian translation.

v2.35a will probably be released this week with a small pile of bugfixes. Most of these bugs were found while testing out changes made to the development trunk.

On that note, there are a lot of cool things in the pipeline for v2.36; the most visible of which is a new theme! Rickard Olsson got the ball rolling and contributed a dark theme, which I then mangled a bit and committed. When combined with pretty URLs and per-folder thumbnails, things look pretty slick. It's funny how sometimes just how effective superficial changes can be.

More ES1 gutenprint goodness

Sat, 24 Nov 2007 01:12:03 GMT

Gutenprint has accepted my second patch, so it now has a working Selphy ES1 raster driver. Unfortunately, it still requires a custom print spooler, but I'm now one step closer.

es_print_assist.c is now updated to properly poll the printer status, so it can now take the raw dump from gutenprint and shove it out to the printer with minimal delay.

The third step will be to rework it so that it can deal with an arbitrary file on stdin, properly parsing the dumpfile to determine length and paper type.. and for step four, adapting it into a proper CUPS backend. Yay.