Older blog entries for raz (starting at number 1)

23 Nov 2012 (updated 23 Nov 2012 at 05:18 UTC) »
Towards ‘serverless’ social-networking

The rise of ‘cloud’ services and the rapid uptake of smartphones has created an unexplored – and perhaps quite large – niche for social software outside the control of advertiser-funded social network services (Facebook et al). While smartphone power and connectivity constraints make pure peer-to-peer social software on smartphones impractical, it is possible to construct a hybrid approach which moves much of the heavy lifting to undifferentiated/non-sticky services in the cloud while retaining owner/user control.

By contrast:


  • Many people, perhaps a majority, are perfectly happy to depend
    upon advertiser-funded social network services.


  • A visible majority is not and is therefore putting effort into personal server projects like FreedomBox to run a server in their own home which stores/shares/controls their own data and perhaps some of that of their friends. This approach avoids the power and connectivity barriers in smartphones, but requires the purchase, installation, connection, maintenance and physical securing of a device at the owner/user’s home and requires some technical expertise in dealing with the maintenance
    of the server operating system and software. Even if backups (and restores!) and upgrades are fully automated, diagnosing and correcting failures requires specialist expertise – and the
    time to use it – that the vast majority of people don’t have. This latter piece is a major part of the value that SaaS-providers generally – and social network services in particular – provide.


  • For people not concerned about governmental/law-enforcement interference, a [virtual] personal server in a data centre provides all of the other relevant benefits of a personal server and eliminates all of the physical aspects, but still requires specialist expertise in diagnosing and correcting failures.


  • For people who aren’t willing to run a server – whether virtual or real – but are willing to have their data in the hands of someone who isn’t selling advertising to fund their service and are willing to incur a small cost in time, money, inconvenience, etc., a variety of approaches are being explored. Notable amongst these are distributed/federated social networking software (e.g. Diaspora) and paid-subscription-only services (e.g. App.net).


  • Another group of people – myself included – would prefer not to run a server if possible – or are unable to – but would very much prefer that their data was under their own control. This is the unexplored niche.

The options are:




























Will purchase, install, connect, maintain and physically secure
device at residence. Will maintain server software.
Will maintain server software. Won’t maintain server software. Willing to
pay $/time/inconvenience for increased freedom.
Will only use phone.
Concerned about governmental/law enforcement interference. FreedomBox
Concerned about control of data by others. FreedomBox on a virtual server. P2P app with non-sticky service help.
Concerned about advertising-funded sites skewed incentives
and/or constant unpleasant changing of the rules.
Diaspora on a friend’s server.

App.net.

Not concerned. Facebook

To understand where the additional niche exists, imagine that smartphones generally had:


  • effectively unlimited battery capacity (comparable to that of a PC plugged into a national grid)

  • effectively unlimited CPU capacity (smartphones are now so powerful that this is rarely a constraint, but it would be nice if a photo/video that the owner/user shared suddenly going viral didn’t make it impossible to use the phone for several hours)

  • effectively unlimited network capacity (enough that authorised people browsing the owner/user’s photos could be loading them directly from owner/user’s the phone as they viewed
    them)

  • a fixed IP address and no NAT between it and the public Internet (so it could serve data without help from hosted services)

In this environment, it would be possible to produce social network software that ran only on phones and talked only to peers on other phones. Unfortunately on current and likely future mobile phone networks, three of those things are always false and the fourth is usually false. It is possible, however, to use a certain class of network-hosted service as force-multipliers for an app running on a phone to give it capabilities [almost] as good as those four things, and to do so without giving control away:



  • The simplest approach uses an object-storage service (Amazon S3, Rackspace CloudFiles, OpenStack Swift, … possibly enhanced with a CDN for even better speed) to share objects (files, possibly encrypted and/or subject to access control) to make things that have been shared available to others. For asynchronous browsing by others of things that the user has shared, this immediately provides all four capabilities described above. Importantly it is possible to share to multiple services of this type at the same time and to add and remove services at will, meaning that the user is never tied to one provider.


  • To add timely notification (which improves interactivity and reduces polling workload), any of a number of IM services (notably IRC and XMPP/Jabber) can be used to deliver short ‘message available at https://storageservice/objectid’ notifications between apps in near-real-time. This is not ideal as (a) such services are not currently available on a pay-per-use IaaS/PaaS basis, meaning that the user is dependent upon the willingness of someone else to carry their traffic free of charge and (b) this use (machine-to-machine) may be outside the intended use of such services, meaning that this use may not be as reliable as typical IM use. To the extent that this use is possible, parallel use of multiple services is also possible because when the traffic is
    machine-to-machine, the difficulties of untangling multiple streams of messages can be resolved by automated means, meaning again that services can be added and removed at will and the user is never tied to one provider. (Note also that there are several other approaches to the timely notification problem, some of which may be considerably better options; IM services are simply the most obvious example.)

This is not strictly ‘serverless’, but it introduces the use of hosted services in a way which (a) doesn’t cede control to an advertiser-funded social network service and (b) doesn’t require that the owner/user be willing/able to take on the administration of a virtual/real server.

An important objection in both cases is that identifiers in domains controlled by others are still required (host names for the storage-services’ web-servers in the first case,
nicknames/usernames in the second case), however it is not necessary for any of these to take the traditional role of an email address as a personal identifier known to the user’s contacts, they are merely communication endpoints and if the means of stating which ones to use is automated, use of multiple endpoints of multiple types can be sustained. This does require a less obvious means of representing identity, but note for comparison that until recently Facebook users used nothing analogous to an email address, users were located by their name and their proximity to others in the social graph. Each user has a unique identification number, but in general only developers need to know this. The recent addition of email addresses doesn’t materially change the means of locating people, it simply happens that Facebook has added email support. The same identifier-independence is true for the scheme proposed here: the use and propagation of multiple communication endpoints can happen out of the sight of owners/users.

Another important concern is that if too many people start using this approach, IM networks are more likely to start blocking this kind of use. I’d suggest – as a hypothetical example – that FreedomBox-like projects may provide a way to address this: in many cases someone owning a FreedomBox is likely to be willing to have their friends use the device to deal with real-time notification needs. The FreedomBox XMPP/Jabber server could perhaps be enhanced to allow the option for certificate-based authentication by any of the owner’s friends without requiring registration formalities, meaning that this approach could extend non-advertiser-controlled social networking software to a much, much larger audience than those who are willing to run a [virtual] FreedomBox themselves. Not everyone knows someone who’s willing to run their own server, but the pool of people who do know such a person is dozens or hundreds of times as large as the pool of people who are able to do so themselves, meaning that if this approach is of interest to FreedomBox-like projects then there may be an opportunity here to reach a much larger audience much sooner.

This post is not yet a call to action, more a partial statement of vision, I intend to write several more posts over the next few weeks/months fleshing this idea out.

(permalink at rolandturner.com)

Prefixing stdout and stderr with helpful markers

I’m testing a piece of logging code, so I care a lot about what goes where. I figured that there had to be a shell one-liner to allow me to mark stdout/stderr without any setup or code changes. Here it is:

(( some_command | sed '-es/^/stdout: /' >&3 ) 2>&1 | sed '-es/^/stderr: /') 3>&1

So, for a trivial example:

$ ((( echo out ; echo err >&2; ) | sed '-es/^/stdout: /' >&3 ) 2>&1 | sed '-es/^/stderr: /') 3>&1
stderr: err
stdout: out
$

As the example shows, sequence between stdout and stderr may not be preserved.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!