8 Sep 2009 rcaden   » (Journeyer)

There's a Reason RSSCloud Failed to Catch On

RSS iconWordPress and Dave Winer are working together to bring real-time, Twitter-style updates to RSS feeds using the cloud element and the accompanying RSSCloud Interface. Yesterday, WordPress added RSS cloud support to "all 7.5 million blogs on WordPress.com." Winer's documenting the ongoing work at RSSCloud.org.

Although some tech sites are reporting this as a new initiative, cloud has been around since RSS 0.92 in December 2000. I was getting real-time RSS updates as a Radio UserLand blogger back then, and it was a great feature.

However, there's a reason that UserLand turned off cloud support in its products several years ago and shut down all of its cloud notification servers. The approach has massive scaling and firewall issues.

To explain why, it's worth looking at an example. I publish the Drudge Retort, which has around 16,000 subscribers, including 1,000 who get the feeds using desktop software on their home computers. If I add cloud support and all of my subscribers have cloud-enabled readers, each time I update the Retort, my cloud update server will be sending around 1,050 notifications to computers running RSS readers -- 1,000 to individuals and 50 to web-based readers.

That's just for one update. The Retort updates around 20 times a day, so that requires 21,000 notifications sent using XML-RPC, SOAP or REST.

On Internet servers it's extremely expensive to request data from clients, in terms of CPU time and networking resources. You have to make a connection to the computer, wait for a response and deal with timeouts from servers that are unavailable or blocked by a firewall. Every time I've tried to do something like this, it ends up being a huge server bottleneck and I remove the functionality. Last year, I crashed my servers for several days, and the cause was outgoing network connections that supported trackback. I no longer support trackback.

RSSCloud also requires that all desktop software receiving cloud notifications functions as a web server. So if an RSS reader like BottomFeeder or FeedDemon adds cloud support, it must show its users how to turn off firewall ports to accept these incoming requests and possibly turn them off in their router as well. UserLand's attempt to put web servers on user desktops failed because it was too cumbersome to support. Back when I was writing the book Radio UserLand Kick Start and working closely with UserLand developers, their biggest customer service issue was helping users open up their firewalls so that Radio UserLand could act as a web server.

I don't mean to be a dark cloud, because this functionality could be a nice improvement for web-based RSS readers, letting services like Google Reader and Bloglines receive much quicker updates than they get from hourly polling.

But if the effort to make RSS real time extends to desktop software and mobile clients, cloud won't work. I think that RSS update notification would require peer-to-peer technology and something like XMPP, the protocol that powers Jabber instant messaging.

Syndicated 2009-09-08 16:25:42 from Workbench

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!