Older blog entries for apenwarr (starting at number 574)

9 Nov 2010 (updated 9 Nov 2010 at 07:09 UTC) »

More on 802.11 wireless

As part of my top secret plans to try to make a space-age new wireless router, I've decided to wade through the official IEEE 802.11 specification.

Now okay, I decided that before I actually found out the thing is 1233 pages long, so I might yet change my mind. And let me tell you, IEEE reading isn't quite as captivating as IETF reading. There are at least a couple dozen pages of definitions, like "STA" as the abbreviation for "station," because there is apparently a worldwide shortage of lowercase letters.

Word to the wise: if you're interested in this spec, you might want to start at section 5, which actually gives a decent architectural overview. I actually read the entire definitions section before I got there, which was confusing and maybe non-optimal, but I do feel like I recognize a lot more of the unnecessary acronyms now.

My major goal when I started reading the spec was to find the answers to these two questions:

  • Is there any reason I can't join more than one wireless network at once?
  • If I forward packets from one network to another, will it cause interference and packet loss? And if so, can that be avoided somehow?
I'm only at page 42, and I don't have firm answers on these yet. But I feel like I'm getting there.

Before you ask, the answer to the first question is definitely not "you can join as many networks as you have antennas." I know enough electrical engineering to know why that's nonsense, since I was somehow granted a Bachelor's degree in such things; just enough knowledge to be dangerous. But even if the details are fuzzy, let's try this thought experiment:

Back in the before-times, people used to have these analog powered whatzits called "televisions" which would receive "signals" from the "airwaves" using "antennas." Some of these antennas had cutesy sounding names like "rabbit ears," presumably so that people would be allowed to bring them in the house and ruin the careful job the Local Interior Design Authority had done arranging the furniture.

But if you really got fancy, you could get a big fancy antenna and mount it outside somewhere. Then you could put a splitter on the wire from that antenna, and run its signal to more than one TV at a time. Or to your "VCR," which could record one channel even while you watched a totally different one!!! All with a single antenna!!!

I know, it sounds like science fiction, but bear with me, because I clearly remember it, in an abstract sort of way, from my childhood.

(If you used a splitter, you ended up getting less signal strength to each of your receiving devices. But let's ignore that factor here; that's just an analog artifact, similar to what happens when you copy from one "video tape" to another (another item of science fiction; someday, we'll uninvent DRM and get this sort of stuff back). If the antenna is connected to just one digital signal processor, we should be able to mangle it a million different ways and not worry about analog losses.)

So anyway, as much as technology changes, it still mostly seems to stay the same. 802.11 channels are a lot like TV channels; each one gets its own little band of the frequency spectrum. (I was a little surprised that such a recent technology didn't bother using spread spectrum / frequency hopping stuff, but that's how it is.) Thus, just like with your old TV network, you should be able to use a single antenna and receive as many channels as you want.

Relatedly, it seems that 802.11n gains most of its speed by using multiple channels at once. I haven't gotten to that part of the spec yet; I read it elsewhere. But I notice from my online browsing that there are 802.11n "lite" routers with only one antenna, and 802.11n "real" routers with two or three. I think this is pretty theoretically bogus - one antenna ought to be enough for anyone - but probably practically does make a difference.

Why? Because I have a feeling the chipset manufacturers are still in the past. The problem is, sending/receiving on multiple channels at once is kind of hard to do, even if you're working in a purely digital world. At the very least, you need a much higher clock frequency on your DSP to handle multiple full-rate baseband signals simultaneously. But worse, I don't know how much of this stuff is purely digital; they're probably still using analog modulators/demodulators and whatnot. If so, it's probably hard to modulate/demodulate multiple channels at once without using an analog splitter and multiple analog modulators... which would degrade the signal, just like it did with your old TV antenna.

It sounds to me like a solvable problem, but without having yet looked at the hardware/software that implements this stuff, I'm guessing it hasn't been solved yet. This is some pretty leading-edge signal processing stuff, and cheapskates like you are only willing to pay $50-$75 for it, which makes it extra hard. So it was probably just easier to mount multiple antennas and include multiple DSP cores and modulators - in fact, maybe just throw in the same Broadcom chip more than once on the motherboard - and just run them simultaneously. Not optimal, but easier, which means they got to market faster. Expect single-antenna, full rate 802.11n boxes eventually.

So from the above reasoning - all unconfirmed for now - I conclude that, even still, you ought to be able to send/receive on as many channels as you have antennas. And if there's more than one wireless network (SSID) on a single channel, you should be able to join all those wireless networks at once using only one antenna.

As it happens, already by page 42 of the spec I've read the part where it says you absolutely must not join more than one network (literally, "associate with more than one AP") at a time. Party poopers.

But why? The stated reason for the rule is that otherwise, alas, the poor helpless network infrastructure won't know which AP to route through when it looks for your MAC address and multiple APs respond that they're connected to it. But that actually can't be true, because shortly after, they say that you "must attempt to send a disassociate message" when leaving an AP, while admitting that's kind of impossible to do that reliably, since the reason you're leaving might be that you went out of signal range, and how would you know that in advance? Thus, if you're carrying your laptop around and you move out of range of one AP and into range of another and you don't get to disassociate from the first one, the network must be able to handle it, and therefore by extension, it can handle it if you deliberately join more than one network, since the network won't know the difference.

Apparently the guys down at the IEEE 802.11 working group have never heard of crash-only programming; there never should have been a disassociate command in the first place, just like having a DHCP "release my IP address" command was a stupid idea.

Anyway, question #1 looks promising; it looks like a software hack could let us join multiple networks at once. And systems with multiple antennas could even join multiple networks on multiple channels, perhaps.

For my second question, about forwarding packets from one network to another, things are much more screwy. I suspect that forwarding packets between two networks on the same channel will be a problem unless you're careful (ie. receive packet on A, send it out on B, but someone sends the next packet on A while you're sending on B and they interfere), because the APs on the two networks can't easily coordinate any collision control. On separate non-interfering channels it should be okay, of course. But I'll need to read much more before I can conclude anything here.

Interestingly, the standard has accrued a whole bunch of QoS stuff, supposedly designed for real-time audio and video. I doubt that will go anywhere, because overprovisioning is much simpler, especially on a LAN. But the otherwise-probably-pointless QoS stuff includes some interesting timeslot-oriented transmit algorithms (don't expect the 802.11 guys to ever say "token ring") that might be fudgeable for this kind of forwarding. We could just reserve alternate timeslots on alternate networks, thus avoiding overlap. Maybe.

I bet nobody implements the QoS stuff correctly, though, which is why every router I've seen lets you turn it off.

Other interesting things about 802.11

You might know that WEP stands for "wired equivalent privacy." After reading the spec - which mentions in a few places that WEP is deprecated, by the way, which is wise since it was hacked long ago - I think I see where they got that strange name. See, they correctly noted that all IEEE 802 networks (like ethernet) are pretty insecure; if you can plug in, you can see packets that aren't yours. And the world gets along even so; that's why they invented ssh, which is why I invented sshuttle, and so on. You don't need ethernet-layer security to have application-layer security.

However, they didn't want to make it even worse. The theory at the time they were inventing 802.11 must have been this: the security requirement that "they must be able to physically plug in a wire" isn't very strong, but it's strong enough; it means someone has to physically access our office. By the time they can do that, they can steal paper files too. So most people are happy with wired-level security. With wireless, it goes one step too far; someone standing outside our locked office door could join our office network. That's not good enough, so we have to improve it.

And they decided to improve it: exactly to the same level (they thought) as a wired network. Which is to say, pretty crappy, but not as crappy.

From what I can see, WEP is simply this: everybody on your network takes the same preshared key to encrypt and decrypt all the packets; thus everybody on the network can see everybody else's packets; thus it's exactly as good as (and no better than) a wire. Knowing the digital key is equivalent to having the physical key to the office door, which would let you plug stuff in.

And actually that would have been fine. Wired-equivalent security really is good enough, mostly, on a private network. (If you're in an internet cafe, well, mere wires wouldn't save you, and neither will WEP or WPA2. Imagine that someone has hacked the router.) Unfortunately WEP ended up having some bugs (aka "guess we should have hired a better security consultant") that made it not as good as wired. Reading between the lines of the spec, I gather that one major flaw in WEP is replay attacks: even if someone doesn't have the key, they can replay old packets, which can trick hosts into doing various things even if you yourself can't read the packet contents. You can't do that on a wired network, and therefore WEP isn't "wired-equivalent privacy" at all.

So anyway, all that was interesting because I hadn't realized that WEP wasn't even supposed to be good. The only problem was it was even worse than it was supposed to be, which put it over the edge. The result was the massive overcorrection that became WPA, which as far as I can tell ends up being overkill and horrendously complex, reminiscent of IPsec.

Admittedly I haven't read all the way ahead to WPA though, and the fact that lots of people have implemented it successfully (and interoperably!) kind of implies that it's a better standard than IPsec. (Still: see my previous post for an example of how either dd-wrt or Apple Airport Express apparently still *doesn't* implement it correctly.)


The WEP thing is also a good example of a general trend I'm observing while reading the spec: 802.11 does a lot of stuff that really doesn't belong at the low-level network layer. Now, the original "OSI protocol stack" has long been discredited - despite still being taught in my horrible university courses in 2001 and maybe beyond - but the overall idea of your network stack being a "stack" is still reasonable. The whole debate about network stacks comes down to this: higher layers always end up needing to assume things about lower layers, and those assumptions always end up causing your "stack" to become more of a "mishmash."

Without necessarily realizing it, this happened with the world's most common network stack: ethernet + IP + TCP.

First, people have been assuming that ethernet is "pretty secure" (ie. if you're on a LAN, encryption isn't needed). Second, TCP implicitly assumes that ethernet has very low packet loss - packet loss is assumed to mean Internet congestion, which is not true on a wireless network. And third, most IP setups assume that a given ethernet address will always be on the same physical LAN segment, which is how we should route to a particular IP address.

The 802.11 guys - probably correctly - decided that it's way too late to fix those assumptions; they're embedded in pretty much every network and every application on the Internet. So instead, they hacked up the 802.11 standard to make wireless networks act like ethernet. That means wired-equivalent (and with WPA, better-than-wired-equivalent) encryption to bring back the security; device-level retransmits before TCP ever sees a lost packet; association/disassociation madness to let your MAC address hop around, carrying its IP address with it.

It's kind of sad, really, because it means my network now has two retransmit layers, two encryption layers, and two routing layers. All three of those decrease debuggability, increase complexity (and thus the chance of bugs), increase the minimum code size for any router, and increase the amount of jitter that might be seen by my application for a random packet.

Would the world be a better place if we turned off all this link-layer stuff and just reimagined TCP and other protocols based on the new assumptions? I don't know. I suppose it doesn't matter, since I'm pretty sure we're stuck with it at this point.


Oh, there was one bit of good news too: 802.11 looks like it's designed well enough to be used for all sorts of different physical wireless transports. That is, it looks like they can switch frequencies, increase bandwidth, reduce power usage, etc. without major changes to the standard, in the same way that ethernet standards have been recycled (with changes, but surprisingly small ones) up to a gigabit (with and without optical fibre) and beyond.

So all this time developers have spent getting their 802.11 software stacks working properly? It won't be wasted next time we upgrade. 802.11 is going to be around for a long, long time.

Update 2010/11/09: Note that a perfectly legitimate reason to have more than one antenna is to improve signal reception. I don't know if that's what routers are actually doing - I half suspect that the venerable WRT54G, for example, just has them to give the impression of better reception - but it's at least possible. The idea of multiple antennas to allow subtracting out the noise goes all the way back to the old days of TV rabbit ears, which generally had two separate antenna arms. Or ears, I guess. The math is a bit beyond me, but I can believe it works. My point was that you shouldn't, in theory, need multiple antennas to use multiple channels.

Syndicated 2010-11-07 07:29:14 (Updated 2010-11-09 07:09:17) from apenwarr - Business is Programming

Getting an Apple Airport Express to join a DD-WRT wireless network

...should be easy. But it's not, unless you pick "WPA2 Personal Mixed" and "TKIP" instead of "WPA2 Personal" and "TKIP." What's the difference? Who knows! Probably the "mixed" setting is somehow insecure in random unadvertised ways. But at least my music plays through the wireless speaker connection now.

Also, while DD-WRT claims to support WDS, as does the Airport Express, I am not nearly smart enough to figure out how to set it up. Just pick "join an existing wireless network" instead of "extend my wireless network" in the Airport setup tool.

(I'm writing this mostly so that other people who Google for these terms might have some chance of coming up with a useful answer. Naturally all the random sites Google popped up for me were totally useless, and trial-and-error won the day.)

Syndicated 2010-11-07 05:43:36 from apenwarr - Business is Programming

5 Nov 2010 (updated 7 Nov 2010 at 06:04 UTC) »

World Tour Reflections

For some reason, I always imagined that park rangers would travel through the forest either on foot (big hiking boots) or on horseback. In retrospect, that makes no sense; of course they drive. In North and South Dakota, they drive pickup trucks. (Including, not surprisingly, the Dodge Dakota.) In California, they drive cars, which is actually more sensible, but somehow not as romantic.

The midwest is amazingly huge and empty except for farmland and/or ranches. Rumour has it that it takes about an acre of farmland to feed an average American. (If you think about it, this unit doesn't need a timespan portion; an acre of farmland per year feeds an American for a year, so you can cancel out the "per year" part.) This sounds kind of crazy, until you actually drive through it and realize that most of the land area of the U.S. is occupied as farmland, which is what allows the existence of such big, dense cities. Which is not a problem at all; there's plenty of room for everybody. As a person who's mostly ever just flown into the big cities before, I didn't realize just how much of the place is huge and empty. Canada definitely does not have a monopoly on huge and empty.

Yellowstone park is unbelievably fascinating if you care about geology, or if you've ever heard the term "fire and brimstone." They actually have fire and brimstone, and yes, I mean that literally. (Brimstone is a fancy word for sulphur or something like that.) Geysers are great. Old Faithful, while famous, is not the biggest, best, nor most reliable of the geysers, though, so don't limit yourself if you visit.

The Grand Canyon, while indeed very grand, is a lot like the terrain for miles and miles in any direction. There's lots of trees, rocks, and grass, and the mountains are made of sandstone. The main difference is that the height of the grand canyon is negative, while the height of the nearby mountains is positive. But the sandstone looks about the same. Apparently it's tremendously unsafe to try to hike to the bottom and back in one day; however, it looks like it would be a very fun hike to go all the way down, across, and up the other side. Reputedly this takes 3-4 days, but it would be quite the adventure.

The game F-Zero for Gamecube appears to have had its levels modeled after Utah (level 1.3), Las Vegas (all the casino levels), and Redwoods National Forest in California (the foresty/pipe level). The game's landscapes are in fact remarkably lifelike, although in real life the road goes upside-down less often.

The other really interesting thing about Utah is that so much of it is so desolate, but not a desert. It's more of a tundra or something; hard to explain. It's neat because unlike, say, Northern California or Oregon or British Columbia, where the nature constantly reminds you how valuable it is and how careful you need to be about it, you get the feeling that you could build anything in Utah and not do any significant damage. It's kind of an engineer's dream: infinite possibility, minimal consequences. Thrilling, in an empty sort of way.

While in Utah, we were sure to visit the old Wordperfect office (in Orem, UT). Of course, Wordperfect isn't there anymore; it was acquired years ago by Canadian company Corel, which proceeded to more or less rip it to shreds and utterly fail to capitalize on it. Nevertheless, its former building still stands proudly in the middle of its industrial park, which is a pretty nice industrial park really. The building has been taken over by a cooking school. In honour of the building's heritage, I was happy to see that they have named their cafeteria "Wordperfect Cafeteria." Of course, this is a bit ironic, since as it's a cooking school, I expect the food is neither perfect (probably prepared by students) nor word-related. We wanted to go eat there for historical reasons, but unfortunately time did not permit.

The Novell office in Provo, UT was a disappointment. It's just a really boring mass of cubicles. Or at least, that's how it seems from peering through the windows and trying not to get caught by a security guard. It's not even the head office anymore, apparently, since Novell's head office was moved to Boston, I suppose due to Ximian-related activity. Bleh.

We went through the Nevada desert, which was, frankly, a disappointment because it doesn't look anything like the Sahara desert does in movies. In retrospect, I shouldn't have expected it to look like some other place that it isn't, in the same way that I shouldn't have expected the food in Spain to taste like Mexican food. But it didn't, and it didn't, and I was disappointed both times. Oh well. The Nevada desert has a lot of vegetation. As far as I knew, the lack of vegetation is rather the definition of a desert. Apparently not; it's just that deserts have less than a certain threshold of annual rainfall. This saddens me somewhat, for the same reason that classifying Vancouver as a rainforest doesn't turn out to make it warm or filled with giant man-eating spiders.

Even more saddening was that it rained in Las Vegas the first night we were there. Desert, my bum. The rest of Las Vegas was as advertised. One interesting observation: "the strip" (the new area) is exactly as it's portrayed in the movies; we leaned on the balustrades of The Bellagio's giant desert fountain, just like they did in Ocean's 11 (or was it 12? or 13?). Meanwhile, the rest of Las Vegas is just like it's portrayed in books; kind of dirty and drug-infested and poor and with lots of pawn shops. It had never occurred to me that movies and stories tend to portray Vegas in such completely opposite ways. Even less did it occur to me that Vegas could actually be both ways at once.

Southern California was fascinating and unique in the sense that it's probably the only place in the world that is portrayed completely accurately by Hollywood movies, on account of it being the only place in the world that Hollywood people actually know well.

San Diego's beach district is exactly like "surfer movies" depicted it in the 1960's through 1980's. I thought that culture must have died out by now; no. They just stopped making movies about it. Even the hairstyles are the same (maybe with slightly more advanced hair gels now). Although I will never really fit in with surfer culture myself, I was gratified that San Diego was much more like its stereotype than I ever would have guessed, and it's a pretty positive stereotype.

Los Angeles is far scarier than expected, but its mood exactly matches the tone of DJ Tiesto's In Search of Sunrise 5: Los Angeles. It's such a perfect soundtrack for the place that I was honestly a little bit shocked. This makes me want even more to someday visit Latin America, since I really liked In Search of Sunrise 4.

Northern California (highways 1 and 101) are scary and the passenger in your car will probably get carsick, but it's just as worth driving as everyone says. The views are amazing.

Oregon is way prettier than advertised, and lots of people have been known to advertise its prettiness. Many of the campgrounds allow you to rent a pre-installed yurt, which sounds pretty great. Unfortunately I didn't know this before the reservation deadline, so I couldn't get one. There are also sand dunes in Dune City, which were very cool; much more like the Sahara than Nevada is, if you ask me. Portland is also a great city; excellent public transit, sensible weather, a giant bookstore, and excellent people.

As I write this, I'm in Seattle. All three of my friends that I met up with here work at Amazon.com, and they don't know each other, which tells me two things. First, apparently there are surprisingly few tech companies in Seattle (Redmond, home of Microsoft, is near here, but I somehow don't know anyone who works there anymore). Second, Amazon is obviously really big. Apparently in the past 9 or 10 years their revenues have expanded insanely, but you don't hear that much about them. Moreover, their hosting services, which you hear all about (S3, EC2, etc) form only a tiny fraction of their business. The real Amazon.com - the one that sells tangible and intangible things to customers in exchange for money - is massive, very profitable, and rapidly growing. And they don't use Amazon Web Services for all their web services.

Seattle, like Las Vegas, had rain. As a whole this trip has been pretty non-rainy. We'll see what happens for the last leg of the journey.

Summary: the U.S. is a pretty good place all around, and has lots of very interesting diversity, despite rumours to the contrary. For politics, I still greatly prefer how we do things in Canada, but as countries go, you can see why so many people like it here.

Update 2010/11/06: Erin tells me that there are, in fact, giant man-eating spiders in Vancouver, so I no longer have to be disappointed about that part. Now if it only it were warm.

Syndicated 2010-11-05 22:25:37 (Updated 2010-11-07 06:04:00) from apenwarr - Business is Programming

Employee Cults

    The way you know your processes are working is when you overhear employee number 5 saying, "thank God I got a job early on in this company, because there's no way I could get one now."

    -- blognewcomb

I remember hearing that a few times at NITI. It seems like good advice. So is the rest of the article.

Syndicated 2010-10-17 00:47:51 from apenwarr - Business is Programming

Avery World Tour 2010

Although the tour has already begun, I suppose some of my faithful (ha!) readers may be interested to know where this trip has yet to go:

  • Theodore Roosevelt Park, ND ("I grow very fond of this place, and it certainly has a desolate, grim beauty of its own, that has a curious fascination for me." - T.R.)
  • Black Hills Park, SD (including Mount Rushmore, apparently)
  • Yellowstone Park (home of Old Faithful)
  • Boulder
  • Denver
  • Salt Lake City
  • Provo and Orem, UT (founding places of Novell and Wordperfect)
  • Bonneville Salt Flats (where the land vehicle world speed record is held: faster than Mach 1!)
  • Grand Canyon, AZ
  • Las Vegas
  • Los Angeles
  • San Diego
  • Somewhere in Mexico (maybe)
  • Mountain View (for GitTogether 2010)
  • San Francisco
  • Portland
  • Seattle
  • Victoria and/or Vancouver
  • Saltspring Island
If you are in one of those places and you think you want to see me while I'm there, send me an email. Even better if you have an extra couch to crash on.

Also if you want a postcard, let me know. No guarantee about which of the above places it will come from. I'm not that good.

Syndicated 2010-10-11 14:15:20 from apenwarr - Business is Programming

1 Oct 2010 (updated 1 Oct 2010 at 18:04 UTC) »

sshuttle now works on MacOS

I know there are quite a few people who will be as happy to know this as I was. Thanks to a helpful contributor known only as dkf, I've now added the magic incantation (namely, sysctl -w net.inet.ip.scopedroute=0) that's needed to make ipfw transparent proxying work on MacOS 10.6 Snow Leopard. I don't know what it does, and I don't care! All I know is that sshuttle restores it on exit to whatever its previous value was :)

This means you, as well as I, can now use a Macbook to VPN into any network that gives you ssh access. That's a lot more helpful than always having to do it from a Linux VM.

Why sshuttle is awesome

For those who are just joining us, sshuttle provides VPN-like connectivity using a plain ssh connection, without needing root access (and thus the admin's permission) on the server side. Basically, it makes a transparent proxy server on your client that automatically forwards through an ssh tunnel.

Compared to ssh's built-in port forwarding and tunneling and socks mode, sshuttle is better because:

  • it auto-discovers hostnames from the server side and puts them in your /etc/hosts while the tunnel is running.
  • it auto-discovers network routes from the server so you don't have to specify them on the client.
  • it works even if you don't know how to configure socks, and with programs that don't support socks, and you don't have to remember to turn socks on and off when your tunnel comes and goes.
  • you can easily run more than one tunnel at once to more than one remote server (as long as the remote subnets use different IP addresses, of course).
  • the remote admin can't disable it in the sshd configuration.
  • there's a workaround for ssh's tendency to use megabyte-sized tx/rx socket buffers, which result in horrible latency. So you can download a large file *and* have interactive traffic on the same tunnel, and performance doesn't suck.
  • it doesn't have the random freeze-up bugs that ssh port forwarding does. (Though maybe that's specific to Debian's ssh, and maybe it only happens to me. Nobody has ever corroborated my story that ssh port forwarding freezes ALL THE TIME.)
And although ssh forwarding also has the following, not every VPN package does. It's worth pointing out that sshuttle is awesome because:

  • you get exactly the same level of transport security and key management as ssh (ie. a lot), because it uses ssh.
  • you don't have to install anything on the server (as long as you have sshd, a shell account, and python). It installs a temporary copy of itself on the server ("internet worm technology"), whenever you connect. There's never a client/server version mismatch.
  • it's astonishingly easy to configure; you don't even need to 'make install'.
  • it elegantly avoids the infamous tcp-over-tcp problem that most tcp-based SSL VPNs have, while still being able to use chained ciphers that udp-based VPNs can't (at least not without incurring a lot of overhead).
  • almost all of it does not run as root (including 100% of the server side).
You can download sshuttle 0.40 from github. Enjoy!

Syndicated 2010-10-01 08:15:34 (Updated 2010-10-01 18:04:10) from apenwarr - Business is Programming

25 Sep 2010 (updated 25 Sep 2010 at 09:03 UTC) »

Traffic shaping on a sheevaplug

In March 2009 I ordered two $100 Sheevaplugs (sometimes misspelled as Shivaplugs, hi Google). These are ARM-based Linux-capable "plug computers." Where by "plug computer" we mean that it's about the same size as a wall wart, and includes the AC/DC converter internally, so you don't need a separate stupid power adapter unit.

The devices are quite amazing for the price: gigabit ethernet (I don't know if they ever actually saturate it, though), 512 MB of flash, 512 MB of RAM, 1.2 GHz ARM processor, USB host and target ports, and an SD card slot. With Ubuntu pre-installed. And although you're definitely not getting as much "per GHz" as you would on a modern x86 processor, they're still surprisingly fast. Seems like Pentium-level performance, at least.

I ordered two of them because I figured I was going to do some experimenting, and since there was (is?) a 6-week lead time on ordering them, I figured it would be good to have a second one around in case I toasted the first one.

As it turned out, it's pretty hard to permanently break things; the only real way I can find to do it is to corrupt U-Boot, which in itself is kind of hard to do and I haven't managed it yet. And even if you do, the USB target port can be used as a serial console and as a JTAG port, which means you can reprogram the boot flash from any USB host computer, like, say, my laptop.

I haven't tried the JTAG yet, because I haven't needed to. But I believe the program necessary is called openocd. See this Sheevaplug-specific openocd tutorial. There is also something called a "sheevaplug installer," which purports to automate this process (it seems to use openocd internally), but such platform-specific tools are probably a bad idea. Hardware people seem to have a nasty propensity to make one-off solutions to their problems - after all, every circuit layout is unavoidably a one-off solution, so you get into the habit. Someday, there will be a post-Sheeva-plug, and we don't want to start a whole new software project with an installer for that. Do we? Well, someone will probably do it anyway. In any case, if you want to maximize your benefit-to-effort ratio, learn how to use openocd and forget about the "Sheevaplug installer."

If you haven't guessed yet, this article mostly exists to appease Google with a bunch of useful links, plus to write down the current overflowing contents of my web browser tabs in case I need the information again in the future, which I intend to do.

By the way, you can't order a Sheevaplug directly from Marvell (the chipset maker and the designer of the Sheevaplug reference design). The supplier I used was Globalscale Technologies, which seems to be rather clearly based in China, and has the customer service to prove it. There's probably a business model out there just for a trivial company who will stock these things and make a non-ridiculously-horrible web order form so it doesn't take 6 weeks to order one.

Now, the original Sheevaplug (still the only model of Sheevaplug, as far as I know, as of this writing) is a bit limited. The most terrible limitation is that it only has one network port and no wireless, which means, essentially, that the only thing you can possibly do with it is make a fileserver, because routing is impossible. Not surprisingly, the first and only Sheeva-based products (technically, that means using the Sheevaplug reference design, which uses the Marvell Fereoceon CPU core, part of the Marvell "Orion" (or is it "Kirkwood"?) product line) were all crappy fileservers. Essentially, Ubuntu + samba + terrible web UI + no attention to backups + low performance. Uninspiring.

Meanwhile, these systems are physically way more powerful than the original Weaver we made in 1999, and are also 1/10th the price, albeit software-wise much more crude than good old Weaver. The same old story: hardware improves, software backslides to keep up. But I digress.

Now, this Sheevaplug model was designed by Marvell to inspire developers (and hit the magic $99 price point!), and it's quite nicely done. Then companies like Globalscale manufacture it so Marvell doesn't have to. But interestingly, Globalscale has also done their own bit of innovation on top of the reference design. For example, their Guruplug Plus, which is essentially a Sheevaplug with 802.11g, Bluetooth, 2 x Gigabit ethernet, 2 x USB, an eSATA port, and a somewhat higher price tag ($129 instead of $99 USD). They also make a more expensive model with video out. Oh, and also it reputedly has terrible heat dissipation problems.

It's not surprising that the Guruplug has heat problems: while the Sheevaplug was designed with Western Hardware Designer Principles (make it look decent, not kill people, and largely useless), the Guruplug seems to have been designed using Chinese Hardware Designer Principles (make it amazingly cheap, flashy, potentially useful, and don't worry about those pesky safety underwriters). I'm not sure which I'd rather have. FWIW, I've run my "original" Sheevaplug full blast for long periods of time and it never gets any more than slightly warm, so any heat problems of the Guruplug are definitely not shared by the Sheeva design.

Nevertheless, everything factual (as opposed to opinionated) below applies to the Sheevaplug and maybe the Guruplug, but maybe not. I don't have a Guruplug.

Connecting to the serial console

Before you start any serious hacking on a Sheevaplug, you really want to get the serial console working. That's the only way to get to talk to U-boot, which you will want to do if you need to replace the kernel, boot from an external disk, etc.

The serial console uses the USB "target" port on the Sheevaplug. That is, you plug it into your USB "host" (eg. a laptop) and the Sheevaplug acts like a USB device, in this case a serial port or modem. (There's also a USB "host" port that works the other way around.) It even comes with a convenient cable. So you plug one end into your laptop, and the other end into the Sheevaplug.

In my case, I have a Macbook, which of course doesn't have the right driver for screwing around with Sheevaplugs, because such screwage is obviously not Apple Approved. Luckily though, I do all my real work in a Linux virtual machine running under Parallels (4.0.3848, in case you care).

One of the miracles of USB is that it's so simple - it's just a really fast serial port, after all - that it's trivially easy to forward it from your physical machine to your virtual one. So it's very easy to control the Sheevaplug from my virtual Linux box as if it was physically attached, and sure enough, it works perfectly.

The driver you need is called ftdi_sio. I had a pretty old kernel installed on my laptop (2.6.24), so the ftdi_sio driver didn't know the Sheevaplug's device ids. Luckily, Linux is designed for people like me, so it was easy to force the issue:

    modprobe ftdi_sio vendor=0x9e88 product=0x9e8f

(I got the above magic incantation from Nozell.com.)

If you have a 2.6.24 kernel like me, you can then talk to /dev/ttyUSB1, 8N1, 115200. /dev/ttyUSB0 also exists, but that seems to be a bug since it doesn't do anything. I used minicom at first, but then got tired of it and wrote port.py, a totally trivial terminal connector that lets your terminal emulate VT100 the way God intended, rather than faking it up (badly) by hand. To use it, you'll also need a copy of the awesome options.py from bup, downloadable at that link for your convenience. You just run it like this:

    port.py /dev/ttyUSB1 115200

Since my next step was to download and compile a new kernel for my Sheevaplug, I figured I might as well upgrade my laptop's Linux kernel at the same time. I picked Linus's standard, unadulterated 2.6.34 kernel. Two items of good news with this version: you no longer need the special vendor/product codes for the ftdi driver, and the mysterious broken /dev/ttyUSB0 no longer appears. (Which means the working /dev/ttyUSB1 is now renamed to /dev/ttyUSB0, so don't be surprised.)

Using a Sheevaplug as a router/firewall/shaper

I said earlier that a Sheevaplug only has one ethernet port, which makes it useless for anything but a fileserver. That was a slight lie; in fact, because it has a USB port, you can add whatever you want. I added an external ethernet device, the Trendnet TU-ET100C. It cost me about $40 at the overpriced local computer store on an island in the middle of nowhere, which isn't bad. It works with the Linux-USB "pegasus" ethernet driver, and it's fine.

Interestingly, either the hardware or the driver for the Trendnet turns out not to support automatic crossover cable detection/emulation. It's been so long since I ran into this problem that I'd forgotten what a hard-to-diagnose pain in the neck it is when you run into it. Someone put a lot of work into fixing that problem, and we as a society have already forgotten all about it.

Let's all just have a moment of silence to celebrate automatic ethernet port sensing.


Right, so, anyway, this USB device doesn't have automatic port sensing. Which is really not a big deal, once you know about it, but it sure confused me for about half an hour because I didn't have a crossover cable. In case you're wondering, the onboard ethernet on the Sheevaplug does do port sensing. Nice job, guys. So I just swapped ports and my problems were solved.

Okay, so, after all that, we now have a $140 device ($100 box + $40 ethernet) that can do routing. Right? Well, no, because the kernel is missing all the happy fun bits like iptables and iproute2. We need to upgrade the kernel.

Building the new kernel

Ah, but *which* kernel? On my x86 laptop, I used Linus's latest stable Linux kernel (2.6.34), but I gather that's not the best thing to do for ARM platforms right now. All the latest Marvell patches have been merged into Linus's 2.6.35-rc1, but I know better than to use a Linus -rc1 release. Instead, I tracked down Marvell's repo and used their "stable-2.6.31" branch, which corresponds to git commit 17e9ccfb3d083d9ad566f867d47de4cf37511cce, aka v2.6.31.1-277-g17e9ccf. It seems to work fine. By the way, here's my kernel .config file. It probably has some extra stuff enabled that's not necessary, but it works for me, and it works with my USB ethernet adapter.

For reference, the interesting git repositories are:

  • Linus: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
  • Marvell: git://git.marvell.com/orion.git
Supposedly you can build the kernel with a cross-compiler. I didn't bother trying to do this. I just installed gcc and friends on the provided Ubuntu distribution and built it there. Warning: it took a long time. This processor isn't exactly running at warp speed, but worse, I was building it all on the cheapest 8GB SD card I could get at Future Shop a year and a half ago. Result: slow disk writes. *Very* slow. But I couldn't use a USB disk because my USB port was taken up by the USB ethernet, and I didn't want to blow *more* money on a USB hub.

plugcomputer.org has a very nice article with all you need to know to build a kernel for the Sheevaplug. Most of what you need to know is what I just wrote, plus that you need to 'make uImage' to get a U-Boot image, and the 'make uImage' process requires you to have a 'mkimage' program that comes with U-Boot. They have links in that article, but the particular one I used (a pre-compiled binary for x86) was one I downloaded from a random place on the net. I can't find it anymore, so in the interest of sharing the love, I have mirrored mkimage.gz here. gunzip it, and rename it to /usr/local/bin/mkimage and you'll be fine.

Fixing broken Ubuntu

Now that you've upgraded a bunch of packages in order to install the compiler, you'll probably notice that your system is completely scrambled and won't let you log in anymore. The bad news is, this sucks. The good news is, maybe you read this far in advance so you're still logged in so you can fix it. The magic incantation to fix it is:

    passwd -d root
    passwd -d root
    passwd root

Somehow, the passwords that got written into /etc/shadow didn't get encrypted properly, and some kind of Ubuntu upgrade "fixes" it, but makes future password checks fail. It's also suspicious that you have to delete the password twice, and that only changing the password doesn't work. I don't want to know. The /etc/shadow file has always been a mystery to me. Thanks, anyway, Ubuntu.

In case you managed to log out and now can't back log in, all is not lost. You need to run this series of commands in U-Boot to get into an emergency recovery mode:

    setenv bootargs console=ttyS0,115200
       rw root=/dev/mtdblock1 init=/bin/bash

(You have to use init=/bin/bash. Using the normal sysvinit "emergency" command doesn't work, because it makes you enter your password, which has always been a pretty stupid idea. Because if you can type "emergency" you can also type "init=/bin/bash", so there's no added security, just more hassle. It used to not be stupid and just happily logged you in. But I digress.)

And then run the 'passwd' commands above, sync, and reboot.

Booting the new kernel

You can't, because you have to upgrade U-Boot first. Something about the bootloader operation in the more recent kernels is different from the experimental bootloader operation in the experimental Sheevaplug development kit, so it won't work by default.

Upgrading U-Boot

Luckily this is pretty easy. I found some nice instructions for upgrading U-Boot as part of a document about installing Debian on a Sheevaplug.

The easiest way to do this part is to use a TFTP server. Any TFTP server will do fine here, since you'll only do it once. However, WvTFTPd (wvtftp) is still the world's fastest TFTP server, and by comparison all the other ones are... "retarded," in the literal sense of not using negative latency.

WvTFTP really is at least 3-5x faster (and even more on a network with significant packet drops) than a typical TFTP server. Seriously. If you're netbooting embedded devices and you're still using a TFTP server that sucks, you're wasting your time. Stop doing that.

Okay, now we really boot the new kernel

Just follow the instructions and you'll be fine. However, I'm terrible at following instructions without knowing what I'm doing, so I didn't follow them and got myself into a bit of a mess which required a certain amount of Googling to get myself out of.

Here are the really important points that you shouldn't forget:

First, the new version of U-Boot supports booting both old-style kernels (like the one it came with) and new-style ones (like the current release kernels). Strangely, it defaults to the former, not the latter. To switch it into a mode capable of booting new-style kernels, you have to do (from the U-boot serial console) setenv mainlineLinux yes and saveenv and then reboot.

Note note note! This may sound obvious in retrospect, but once you've done this, booting your old kernel will stop working, because it's not a mainline kernel. Right? Well, it's obvious if you remember, hours of panic later, that you've run this command in the first place. Ha ha. Yeah. Anyway, if you want to boot your old kernel again, just run setenv mainlineLinux no and saveenv and then reboot.

The other thing you need to do is tell Linux what kind of hardware you're running on. Once upon a time, before the invention of so called plug-and-play, Linux used to "probe" for hardware devices automatically, so you didn't have to do such silly things. Linux had the very best hardware probing anywhere I've ever seen. I personally spent hours optimizing my Linux ISA arcnet driver's probing routines, and I personally solved the mystery of the "IRQ autoprobing in kernel modules causes a crash" in Linux 1.1.x or 1.2.x or so. But those days are long gone. Nowadays, we're back to specifying the hardware platform by hand, as if we were running Windows 3.0, which by the way is at least 10x smaller than the Sheevaplug's default Ubuntu install and it has a GUI. Hardware improves, software backslides. But I digress.

If you have an original Sheevaplug like mine, the magic incantation (at the U-boot console) is:

    setenv arcNumber 2097

Yes, you do have to saveenv. You might also have to reset. Don't be like me, and think you can "try a temporary experiment" by setting the environment variable and *not* saving it in the hopes it won't screw up. It will fail.

Note note note! Unlike the mainlineLinux option, setting the arcNumber option does not mess up your old kernel. That's because mainlineLinux is interpreted by U-Boot (I assume), which does something different depending if it's a new or old kernel. But arcNumber is interpreted by Linux, and the old Linux didn't know to look for it. (This also explains why you have to saveenv; Linux won't be able to find it if it's not saved.)

Once again, I refer you to the Debian Sheevaplug instructions for how to get your kernel booted, depending on what device you want to boot from.

Note note note! I don't recommend installing the new kernel into the "boot flash" memory segment until you're really sure it works, because that will overwrite the old kernel, and then you might be stuck. For experimenting, boot from an SD card or TFTP instead.

I personally recommend starting your experiments with TFTP, since that's the quickest way to change out your kernel while you're experimenting. I went from there to using an SD card, which gave me more disk space for messing around. Both boot methods are described at the Debian Sheevaplug link above.

Recovering after you've mangled all your U-Boot settings

If you're like me, you'll inevitably end up getting into a situation where you really wish you could just get back to your old kernel and tweak some more stuff from a nice, safe, userland Linux process. Here's the magic U-Boot incantation that works for me:

    setenv bootargs console=ttyS0,115200
        mtdparts=nand_mtd:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs) rw
    setenv bootargs_root root=/dev/mtdblock2 ro
    setenv bootcmd 'nand read.e 0x800000 0x100000 0x400000; bootm 0x800000'
    setenv console console=ttyS0,115200
    setenv mainlineLinux no
    setenv run_diag no

Not so shockingly, the "resetenv" command resets the environment to "factory defaults," but apparently the guys down at the factory do not like to have their Sheevaplugs in a state that is actually bootable. Ha ha. Sucker.

Actually routing those packets

After all that, you're finally ready to set up your Sheevaplug to be a router. The good news: now that you have the @$!@#!@ thing booted, it's just like any old Linux box. iptables and iproute2 work exactly like they would on any Ubuntu system. Which is to say, they're amazingly obtuse and complex, and in the case of iproute2, the documentation *outright lies*.

I do have it working now, if that's any consolation.

Some other time, I'll write about my amazing adventures with token buckets, stochastic fair queuing, traffic shaping vs. policing, random early detection, explicit congestion notification (oh man, ECN performance actually is amazing when you have a router that uses it!) and how it really is possible to have a "usable" 256k Internet link even when you're downloading 10 files simultaneously and your provider thought a 5 second downstream DSL queue length was a good idea and you technically can't even shape downstream traffic.

(I put "usable" in quotes because 256k is getting to be pretty unusable nowadays even when it's not loaded down. Over time, you lose sight of the fact that that the 6 megabit link back at home really *is* 23 times faster than a "perfectly acceptable" 256 kbit link used to be. But the speed mostly doesn't bother me; it's the terrible queuing. Example: before I installed the shaping/policing, leaving Google Reader open in the background totally killed all other traffic. And it's mostly just downloading text!)

Too bad Linux's rather awesome RED traffic shaper doesn't work on downstream traffic, which means you can't use ECN on downstream traffic, so you end up dropping lots of packets all over the place. And too bad that if you use the "intermediate device" method (basically, egress queuing on the packets going to your internal network) you add tons of latency - of course.

But I think I know how to fix it. More later.

Syndicated 2010-09-25 02:45:10 (Updated 2010-09-25 09:03:48) from apenwarr - Business is Programming

Poetry Adaptation

A dark cave. In the middle, a caldron boiling. Thunder.

Enter the three Witches.

    1 WITCH. Thrice the fishy fish hath twined.
    2 WITCH. Thrice and once, the fam'ly dined.
    3 WITCH. Mother cries: 'tis time! 'tis time!
    1 WITCH. Round about the kitchen go;
       In the poison'd cookies throw.
       Snails, who without a thought,
       Captured slime that hell hath wrought;
       Swelter'd steam go up the flue,
       Boil thou first the Jana goo!
    ALL. Weather, weather, boils and feather;
       Conjure Jana from the nether!
Witches ad-lib more two more verses with imaginary spell ingredients. Man, that's a lot of work.

MacFluffin enters, visibly upset.

    MacFluffin. Ay! For towhich thou hast wrought herminity upon thy shoustest?

Witches cackle, then disappear in a puff of smoke, which goes up the flue.

    MacFluffin. Aught! Wherefore this always happens?

Syndicated 2010-09-16 07:01:25 from apenwarr - Business is Programming

9 Sep 2010 (updated 10 Sep 2010 at 01:07 UTC) »

Browsing Google Groups Now Requires You to Log In

Someone emailed me to point out that the bup-list archive link in my previous post doesn't work unless you're logged in with a Google Account.

I know this used to work, once upon a time; it's not a restricted mailing list. And the Do I need a Google Account help page is the same as ever, ie. it says you can search, browse, and read public groups without an account. There are also instructions for how to join one of their lists via email without using the web interface, and I'm pretty sure that still works.

But if you go to groups.google.com (even the home page) and you're not logged in, you get bounced to the login screen.

But it's even more insidious...

Testing this was harder than I expected. In Google Chrome, the archive page above still worked after I logged out!! It even worked when I opened a "New Incognito Window," which supposedly eats all your cookies and guarantees you some form of privacy.

But when I got to the same page in Firefox or Safari - in which I haven't logged into Google for a long time - I do indeed get the login page.

So now we have two bugs:

  • Google Groups isn't as open as (I assume) they actually intended to be because they didn't test carefully enough;
  • Incognito Mode in Google Chrome (6.0.472.55/MacOS) doesn't successfully hide my identity.
And maybe a third, if you're generously assuming it's classified as a bug down at Google:

  • Logging out of Google doesn't remove all my cookies, so they can still associate me with my account after I log out.

I didn't bother posting this to Google's support mailing lists, because their support is so terrible that everyone knows it's a waste of time. This posting probably won't accomplish anything either, but at least I'll feel like someone heard me.

I did google around a bit to see if anyone had mentioned the problem, but I didn't see any matching results. Please let me know if I'm totally crazy and this problem doesn't really exist; but test carefully, as leftover cookies might be letting you see the site anyway.

Update 2010/09/09: Aha, not quite as scary as I was worried it might be: when I clear my Firefox cookies, it lets me access Google Groups without signing in. That suggests that it's actually old Google login credentials that are the problem; they cause it to jump to the login screen instead of just letting you through. Still a bug in Google Groups, but at least it means Chrome's private browsing seems to be okay after all.

Syndicated 2010-09-09 21:04:10 (Updated 2010-09-10 01:07:28) from apenwarr - Business is Programming

bup backups are 2.7 times smaller than rsnapshot

Zoran has been doing some interesting tests of bup space efficiency and import performance vs. good old rsnapshot.

Lately, he's been working on an rsnapshot-to-bup importer tool. According to his most recent message, after importing the incremental backups from two servers, the bup repo was 4.6GB vs. the original rsnapshot disk usage of 12.6GB. That's 2.7 times smaller.

YMMV. Results may not be typical. Always consult a physician.

And Also...

As of last night, you can also restore your bup backups. Yeah, I know. It's that awesome.

Syndicated 2010-09-08 20:30:56 from apenwarr - Business is Programming

565 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!