Urbone - Introduction

Posted 18 Jan 2001 at 23:30 UTC by Radagast Share This

This article introduces ideas for Urbone, a peer to peer collaborative massively distributed metropolitan area wireless network. It is the result of some rather intense discussion between myself and Federico Mena over the last month or so, which in turn built on a wireless networking project I did in my previous job, and as such owes a lot to different people who gave input on the ideas as they evolved. Of course,this is still very preliminary, but I think it's right to get some sortof peer review of the basic ideas, to confirm that they are indeed sound and implementable, as well as to find other people who might be interested in lending their expertise to the project.

Background

The Internet today has several characteristics that are contrary to the original design and intent, and more importantly, contrary to a effective, robust and democratic allocation of resources. The ability of the Internet to dynamically reroute traffic around points of failure was one of the earliest features (made to survive nuclear war, as the saying goes), and is still touted as extremely valuable.

In reality, though, dynamic rerouting requires a highly interconnected, redundant network, and due to market conditions, this is not an accurate description of the Internet today, especially not on local and regional levels. On these levels, the network topology closely resembles several plain trees, where an end user is connected to a provider, who is in turn connected to an upstream provider, and so on for a few levels. Interconnection and redundancy rarely happens, other than on the very highest level, where large ISPs/backbone providers have peering agreements to exchange traffic, and in interconnection points such as MAE East and West in the US, and miscellaneous national-level exchanges in Europe. Since only large ISPs can enter into peering agreements, this effectively controls and monopolizes the backbones.

The direct and most noticable result for end users is the extremely high number of hops and latency to get to hosts that reside on other backbone providers' networks. The situation in Mexico City, my current place of residence, is an excellent example. There are three or four large backbone providers active in Mexico City, and all of them maintain separate fiber lines to the US. There are no national level peering agreements, so if you want to reach a host down the street that happens to be on a different provider's network, your traffic goes to Texas and back.

Federico gives me the following example:

On more than one occasion, I have tried to get my mail. It is hosted on the Ximian servers, in Cambridge. I am in Mexico City. For some reason the (apparently only) route to that machine is being unusably slow, and downloading my mail just stalls. So I ssh to one of the gnu.org machines, hosted in downtown Boston. Physically there are not more than 5 kilometers between the Ximian box and the GNU box. My connection to it is reasonably fast most of the time; fortunately this is the case this time as well. Copying my mail from ximian.com to gnu.org is really fast. And copying it from gnu.org to where I am in Mexico City, somewhere inside unam.mx, takes a reasonably small amount of time, i.e. no more than what it would take from ximian.com on a good network day. This is ridiculous. The Ximian and the GNU boxes are practically next to each other physically, and yet the routes from UNAM in Mexico City are extremely different performance- and reliability-wise. If the network was not a lame tree, things would be fine and redundant and happy.

Additionally, this makes the entire network infrastructure extremely vulnerable to failure. Basically, you can consider yourself singly connected to the world. Any piece of hardware in your upstream is a potential single point of failure, making the failure frequency rise exponentially with the number of hops.

Available technology that might solve the problem

There are several low-cost, fairly long range point to point wireless networking solutions available. All are line of sight, using directional antennas, and cost in the range of 600-1500 USD per end (possibly less for homebrew solutions using, for instance, amplified wavelan). The range is typically 3-10 kilometers, and the bandwidth is in the low megabits (1-10Mbps).

These devices are perfectly suited to setting up a network across a fairly large metropolitan area, literally from rooftop to rooftop. It's important to achieve critical mass, which suggests high population density in the area would be preferable. We believe we have a perfect testing ground available: Mexico City. With its 22 millon (low estimate) people, booming economy, and roughly 40 by 60 kilometers of mostly flat ground, it should have more than enough potential participants within radio range of any point in the city to make it easy to reach critical mass quickly.

Such a network could be densely interconnected, with each node connected to at least three others, and thus allowing dynamic rerouting around faults and efficient bandwidth sharing.

The practical and social implications

Urbone aims to replace profit-driven, singly connected, ineffective network structures with collaborative, non-profit, redundant ones, in which all users are connected to several of their nearest neighbours. This has advantages both on the social, economical and technical level.

On the social level, this sort of structure encourages cooperation, partnership and sharing of duties between entities on the network. As the network grows, the participants will take advantage of the possibilities of unhindered dataflow to and from the other participants, hopefully helping to effectivize work processes and streamline the way people work and cooperate, making vital information disseminate wider and more quickly. This would make an excellent incentive for a company to recruit its providers and customers as participants in the network. A good example is ad agencies, newspapers, and offset/printing houses, which often send large amounts of data back and forth between them, currently often on removable media, by courier. There are doubtlessly many more groups of entities who would benefit from speedy digital data interchange in the same way. The lower price also enables more egalitarian access to the network, wiring individuals and organizations that were previously unable to afford high-speed access.

On the economical level, the utility of the network grows exponentially with the number of nodes, and the cost per participant decreases accordingly. A network that's maintained by a non-profit entity will be able to outperform any commercial provider both on price and availability. Everyone pays the same price for the same service, and the network as an entity can buy bandwidth from several external providers, sharing the cost equally.

On the technical level, the network will be massively interconnected, facilitating on the fly rerouting to avoid network congestion and bottlenecks. In an egalitarian, non-profit network, the traffic can flow where it's most effective, utilizing the network optimally, without regard for who paid for that particular piece of pipe, or who's paying for the bandwidth used. Additionally, the network will be extremely fault-tolerant, and will allow a fairly large proportion of the nodes to go offline without splitting the graph in two.

Future possibilities

When the Urbone grows to a certain size, there are a number of possibilities we'd like to look at pursuing. They vary in difficulty, desirability and plausibility, but all of them are practically possible.

Voice over IP
As VoIP becomes more plausible, it would be natural for the Urbone to also carry voice traffic between the nodes. This would allow the participants to place free, high-quality phone calls to other participants. The Urbone could also potentially be hooked up to a phone switch and collaboratively rent a number of outgoing phone lines at bulk prices, allowing participants to cut costs.

Streaming media/multicast
If the Urbone reaches a certain (rather large) size, streaming media over the network would be quite possible and desirable. Since the gateways on the network are centrally controlled, it would be one of the first places it would be possible to deploy IP multicast outside of an experimental setting. One possible use would be for "Urbone Radio" or "Urbone TV", although several are possible. High-quality sound only requires a fraction of the bandwidth on a given link, and with multicast, the traffic could reach all interested nodes while using a minimum of bandwidth.

Mobile networking/ubiquitous access
Given an even geographical distribution of Urbone nodes, it would be quite plausible to outfit each one with a transciever for non-line of sight mobile networking, along the lines of wavelan. This would allow any Urbone participant to register his wavelan card's MAC address centrally, and have access to wireless networking almost anywhere in the Urbone area. This could be done with or without roaming support (roaming being somewhat more tricky to implement, although Mobile IP is a fairly simple standard, especially if one actually controls the nodes of the network). This would make ubiquitious, low-cost wireless networking a reality for a wide range of portable devices, from laptops to handhelds. At the very minimum, this would allow you to bring a laptop anywhere in the Urbone coverage area, run pump, and go to work, regardless of whether or not you're in a park, a café, visiting a friend, or in your home or office.

Mobile Voice over IP
Given the implementation of both VoIP and mobile IP in the Urbone network, a natural continuation would be to cobine the two, turning every portable device into a potential cellphone equivalent.

InterUrbone
We're hoping that if the Urbone becomes successful in our initial deployment area, people will start similar efforts in other cities. As the citywide networks grow, it would be natural to start considering possibilities for interconnecting them. Depending on geographical distance, several possibilities exist (for relatively short ranges, there's microwave, for longer distances, leased fiber).

Aid efforts
Extending Urbone to cover non-urban areas is a distinct possibility. Mexico, for instance, has a vast number of small villages which are sadly underpriviledged when it comes to communication infrastructure (and mostly everything else). The Red Escolar project aims to equip all Mexican schools with computer labs of older computers running free operating systems. Wireless connections from school to school across the country would make a perfect compliment to this effort. Corporate efforts to sponsor schools are currently very popular in Mexico, with entities such as the Coca-Cola Company donating large amounts to schools. It seems plausible to get similar corporate entities to sponsor network access for poor communities.

Implementation details

Each node would have a small router (probably a small Intel PC running Linux) and radio links to at least two other nodes. It should regularly broadcast network status information, such as link saturation and latency, to allow rerouting of traffic to utilize the network optimally. Exactly what protocol would be used to transmit this information and the corresponding routing configuration information is an open question at the moment. Both traditional routing and network information protocols like BGP and SNMP, as well as more high-level solutions with custom protocols (over TCP/IP) would be possible options. We're looking for feedback from people more knowledgable about routing than us who can advise us on what the best strategy would be. It's essential that the routers be possible to upgrade and reconfigure remotely.

It could be useful for the nodes to run very modern networking protocols, such as IPv6. After all, when deploying a completely homogenous network where there's complete control of the standards, it would be a shame to not utilize the progress made since IPv4 was finalized. Cryptography (IPSec) might be particularly useful, given that the data itself is easily interceptable.

Optionally, each node could also run miscellaneous high-level services to lighten the load on the networking infrastructure, such as web caches, caching DNS servers, etc. It would presumably be fairly trivial to tune standard software to function optimally in a network where each node requests the information from its nearest neighbour. This could also dramatically reduce the load on the network's connections to the world.

The embryonic Urbone wouldn't require more than one connection to the world, although two or perhaps three would be ideal. We presume that once the network reaches a certain size, ISPs, particularly smaller ones, would see the benefits of hooking up to the network and allocate some of their own connections to the world to Urbone traffic, since it would result in a net reduction in traffic on connections they actually pay for. In this way, Urbone will also function as an ad-hoc interchange between Mexico City ISPs, reducing the amount of traffic that has to go to the US and back.

What would it cost? Well, if each point is connected to three others, the point cost will be a couple of thousand dollars, plus the router/server box, for a total of about 3000 dollars per node. This is a high estimate, it might be possible to bring it as far down as half of that, or even lower.

2 Mbps connectivity costs about 5000 dollars/month in Mexico today (possibly less), in addition to installation costs. 512kbps is about half that price, with upgrades from 512kpbs to 2Mbps being very cheap. It seems clear that the network doesn't need many nodes before the cost becomes substantially lower than what any telco can offer. At about 10-15 nodes, the savings become extreme, at about 100 nodes they become negligible for companies and institutional users, and at about 200 it gets interesting to compete with dialup internet access for single users, by wiring up every apartment building with a node. Beyond that, well, that's when it starts getting really interesting.

Philosophical implications

Competing with telcos is interesting from a social and philosophical perspective. Communications companies are transnational entities, created by a long series of mega-mergers, in which the commercial interests controlling the infrastructure intermingles with similar commercial interests producing the content. Many people, including myself, find this to be a reason for concern. It's a socially correct thing to offer up a grassroots alternative to infrastructure near-monopolies, much like the free software movement has become a grassroots alternative to the software giants who seemed poised to completely monopolize the market just five to ten years ago.

In the interest of thinking big, it would not be impossible to envision a fully digital wireless network covering and interconnecting large urban areas completely in 5-10 years, all financed and built by the users, in an egalitarian, fair manner, delivering data networking, voice, broadcast media, and other services.

Where we are, and where we're going

We're currently very early in this process. We're trying to map out our options for technologies and structures to use, preparing for an initial deployment of a handful of nodes some time in the first half of 2001. We're looking for people with knowledge in a wide range of networking-related fields to join the discussion at the Urbone mailing list. Areas of knowledge that are relevant include wireless and general networking (theory and/or practice), routing, IPv6, Mobile IP, and related areas. We'd love to hear general comments about the concept here on Advogato, and to take detailed technical discussion to the mailing list.


Related projects, posted 18 Jan 2001 at 23:43 UTC by egnor » (Journeyer)

I can't believe you posted that entire thing without reference to the many very similar projects going on around the world today.

Consume in London is probably the most well-known. Others include SeattleWireless in Seattle and BAWUG in the Bay Area. SeattleWireless has a more comprehensive list of related community wireless networking projects.

Few of your ideas are not shared by at least some of these projects. You would do well to distinguish yourself; if nothing else, it would save you from having to repeat a description of concepts other people have already explained in depth.

I know that there were similar projects, posted 19 Jan 2001 at 00:07 UTC by Radagast » (Journeyer)

But the only concrete one I knew of was Consume. The list of similar projects at Seattle Wireless is definitely interesting, and hearing from others who have been involved in similar projects was one of the reasons I wanted to post this.

Additionally, I wanted to air the ideas on Advogato, since it has some of the smartest people around. Consume, for instance, seems to have a very practical, whatever-works type approach, while I'd like to hear about available technologies, protocols, and so on from people in the know, as well as seeing a discussion of the social and economical aspects of it.

Additionally, Consume doesn't seem to be very much further along than Urbone is, from what I've been able to tell from the website.

So basically, while there are similar projects, I'd like to encourage discussion of all aspects of building such networks, without the ballast of any decisions already having been made. We also claim to have a somewhat unique situation in wanting to use Mexico City as a testing ground, given the extreme population density.

Grid and addresses, posted 19 Jan 2001 at 03:48 UTC by wmf » (Master)

Grid is my favorite wireless routing protocol; it is much more efficient than competing protocols, but the cost is that every node must know its geographical location. Grid assumes omnidirectional antennas, which I would reccomend anyway in areas where you can use them (if there are serveral neighboring routers within a mile, you could reach them all with only one antenna). Of course this gives less bandwidth, but if you need more you can probably "stack" multiple radios on different channels.

Are you planning to provide Internet transit to wireless nodes in the network? If so, where will the IP addresses come from? This is not as much of a problem with IPv6 (because you can get a /48 for free using 6to4). How do you plan to handle multihoming (multiple gateways between the Urbone and the rest of the Net)?

And BTW, the name is really cool.

guerilla.net, posted 19 Jan 2001 at 07:42 UTC by voltron » (Journeyer)

although i don't have anything constructive to add, i'm surprised no one has yet mentioned guerilla.net.

redundancy, posted 19 Jan 2001 at 12:09 UTC by lkcl » (Master)

summary references by Steve Gribble and Armando Fox

funny, but a scenario in the book i am writing: i am _counting_ on the vulnerability of the internet and the lack of redundancy and the potential ease with which it can be destroyed [i'm also counting on the open source community amongst others to rapidly come up with solutions to the problem!!!]

the questions i have are, which i think must be seriously and carefully considered:

is wireless enough, and are the protocols good enough, to not be totally dependent on land-based communications any more?

what's the threshold (user-base) above which total [independent] wireless coverage becomes possible, 24x7 (is napster - a file-based community - a good indication of how this might work?)

[slight-tangent-reference], posted 19 Jan 2001 at 12:13 UTC by lkcl » (Master)

< a href="http://www.cs.uoregon.edu/research/wearables/">wearable computing</a>

The bad news, posted 19 Jan 2001 at 17:41 UTC by jmason » (Master)

Here's what we found with our current efforts (the wan-dublin list):

  1. You must have line-of-sight for it to work well. "Just 5 feet short of LoS" doesn't cut it, and neither does "LoS with a tree in the way" ;)

  2. The power output of those wireless LAN cards is quite high, at least for the current Irish regulations. :(

Go for it though, and good luck!

BTW I'd recommend looking at consume.net, or the earlier Aussie efforts in Canberra. They're both well up to speed (well, the Aussies are far ahead AFAICS).

Routing is hard, posted 21 Jan 2001 at 19:23 UTC by apenwarr » (Master)

Mesh networks are all very nice philosophically, but the deadly problem will be actually figuring out which routes to take at which times. Traditional Internet routing algorithms like BGP basically just cheat, and store the list of all possible routes in memory. This works okay for people with just two links, or for big backbone providers with _really_ expensive routers, but it won't work for the whole internet.

The Grid project that someone mentioned above looks interesting, because they forget about trying to be too clever with routing, and just resort to selection based on geographical location. For a wireless system, that makes sense. But if you and your next-door neighbour are both on the network, how will you differentiate their positions when they're so close together? (GPS would work, I guess.)

The other problem comes down to the reason we charge money for bandwidth in the first place. Say you spent extra money on your transceiver and you have a high-speed point-to-point link with someone 5km North. If your routing algorithm is any good, then if your next-door neighbour (who didn't pay extra for the faster link) needs to reach someone 6km away, the software will choose to send it through your single, fast link rather than a bunch of slower, 250m links.

In fact, everyone south of you might want to use your link to reach up north, rather than the slower links. So now you're sharing with all those people and it's slow anyway -- how do you benefit from the extra money you invested in your fast link? Or, put another way, what incentive do you have to spend the extra money building a fast link?

Mesh networks are great, but there are lots of hard, unsolved problems still. Don't seriously expect to improve your network price/performance ratio too much without finding solutions to some of them.

Grid, tunnels, and economics, posted 21 Jan 2001 at 23:40 UTC by wmf » (Master)

Yes, tunnels are something that none of the ad-hoc wireless routing protocols I've seen try to solve. I imagine the simplest solution is for each end of a tunnel (of length L) to announce its existence (i.e. the location of both ends) to all nodes within a distance of L/2. A slight modification of the geographical forwarding algorithm should take care of the rest.

I wonder if it makes sense to use free-space lasers for point-to-point links; depending on conditions (pollution?) it should be possible to get 1 Gbps.

As for economics, I would definitely look at the Digital Silk Road. We can imagine a wireless network where packet forwarding is paid for using some arbitrary currency, so if you route traffic for your neighbors, they'll be willing to route traffic for you. This opens up lots of questions, though.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page