Recurring outages on major networking sites such as Twitter and

LinkedIn, along with incidents where Twitter members were



mysteriously dropped for days at a time,

have led many people to challenge the centralized control exerted by

companies running social networks. Whether you’re a street

demonstrator or a business analyst, you may well have come to depend

on Twitter. We may have been willing to build our virtual houses on

shaky foundations might when they were temporary beach huts; but now

we need to examine the ground on which many are proposing to build our

virtual shopping malls and even our virtual federal offices.

Instead of the constant churning among the commercial sites du

jour (Friendster, MySpace, Facebook, Twitter), the next

generation of social networking increasingly appears to require a

decentralized, peer-to-peer infrastructure. This article looks at

available efforts in that space and suggests some principles to guide

its development.

Update: a few days ago, OpenID expert Chris Messina and

microblog developer Jyri Engeström published



an article with conclusions similar to mine;

clearly this is a felt need that’s spreading across the Net.

Interestingly, they approach the questions from a list of what

information needs to be shared and how it needs to be transmitted; I

come from the angle of what people want from each other and how their

needs can be met. The two approaches converge, though. See the

comments for other interesting related blogs.

The peer-to-peer concept

The Internet was originally a parliament convened among peers. Every

host was a server, almost always providing file downloads and usually

email as well. To this day, ISPs “peer” when they accept data from one

ISP’s customer and delivers it to the other ISP’s customer.

To peer doesn’t mean simply to be of equal status–in fact, that

notion could be misleading, because two systems with vastly different

roles and resources can peer. More importantly, to peer means to have

no intermediary.

When the architecture requires an intermediary, it should play as

unobtrusive and minimal role as possible. For instance, Napster and

Skype have central servers, but they are used just to sign up

participants and set up connections among them.

Napster’s and Skype’s partial decentralization won them a key benefit

of peer-to-peer networking that Twitter could well take note of: they

offload most traffic from their central servers to the users and the

ISPs that connect them.

But being partially centralized means the service can still be

disrupted as a whole. Napster was shut down by a court ruling; Skype

shut itself down once through a programming error that it never

clearly explained to the public.

The Internet itself quickly developed into this hybrid model as well.

Modems and terminals created a new layer of second-class citizens,

vastly expanded by the PC revolution. These Internet users were tucked

away behind firewalls and blocked from using any services not approved

by system administrators.

By the year 2000, new companies springing up in the dot-com boom found

themselves frustrated by these restrictions, and designed their

innovative protocols to deliver data over port 80 because everybody

kept that open for Web traffic. When the practice started, traditional

Internet developers derided it as “port 80 pollution.” Now it’s called

Web Services.

As happens so often, the way forward proved to be the way

backward–that is, to restore the democracy of the early Internet–and

also predictably, was pioneered by outlier movements with dubious

legality, ethics, and financial viability. Napster made the first

impact on public consciousness, followed by services that rigorously

avoided any hint of central servers (see my 2000 article,



Gnutella and Freenet Represent True Technological Innovation).

By the end of 2000, the term peer-to-peer had become a

household word. But the movement quickly went into retreat, facing

difficult design problems that were already under discussion in the

O’Reilly book

Peer to Peer,

published in February 2001. I summarized the problems, which remain

ongoing, in the articles



From P2P to Web Services: Addressing and Coordination and



From P2P to Web Services: Trust.

The issue of addressing would arise right away for a social network

developed in a pure peer-to-peer fashion. How would you check whether

your old college buddy was on the network, if you couldn’t query a

central server? And how could you choose a unique name, without a

single place to register? Names would have to be qualified by domain

names or some other identifiers–which is actually a step forward

right there. It seems to me ridiculous that a company would plan to

provide a service to the whole world using a flat namespace. And while

we’re at it, you ought to be able to change your name and bring along

all your prior activity.

Trust would also become an issue in decentralized social networks. You

could ban a correspondent from your personal list, but you couldn’t

inform a central authority about abuse. And the problem Twitter has

recently started to tackle–preventing random users from impersonating

well-known people–would be a challenge.

But decentralization brings many benefits. A failure at one person’s

site, or even on a whole segment of the network, would have no effect

on the rest of the world. A misconfigured router in Pakistan could not

keep everyone from accessing the most popular video content on the

Internet. And because each peer would have to obey common, understood

protocols, a decentralized social network would be transparent and

support the use of free software; nobody would have to puzzle over

what algorithms were in use.

Visiting many different sites instead of central server to pull

together information on friends would increase network traffic, but

modern networks have enough bandwidth to stand up to the load. Even in

places with limited bandwidth, service would degrade gracefully

because messages would be small.

The

StatusNet

project, which underlies

identi.ca,

represents a half-way step toward the kind of full decentralization

illustrated by RSS. StatusNet can power a variety of microbloggin

services, each signing up any number of members. The services can

interchange data to tie the members together.

The rest of this article looks at two possible models for a

distributed social network (RSS and XMPP), followed by an examination

of the recurring problems of peer-to-peer in the social networking

context.

Possible models

Many examples can be found of filesystems, version control systems,

and other projects that lack central servers. But I’m just going to look

at two protocols that other people are considering for decentralized

social networking.

When thinking of decentralized systems for sending short messages, RSS

and Atom have to come to mind first. They’re universal and work well

on a large scale. And Dave Winer, the inventor of RSS, has created an

enhanced version called



rssCloud,

recently



incorporated into WordPress.

Given the first question I asked about decentralization–how do you

find the people you’re looking for?–the RSS answer is “by

serendipity.” Like everything else on the Internet, you could come

across new treasures in many ways: surfing, searching, friends, or

media outlets. Lots of bloggers provide links from their sites to

their own faves. And RSS has developed its own ecosystem, sprouting

plenty of aggregators that offer you views into new fields of

information.

rssCloud is meant to carry more frequent traffic and more content than

the original RSS and Atom. It maintains an XML format (making it

relatively verbose for SMS, although Winer tries to separate out the

rich, enhanced data). Perhaps because of the increased traffic it

would cause, it’s less decentralized than RSS, storing updates in

Amazon S2.

XMPP was invented about the same time as RSS by a programmer named

Jeremie Miller, who wanted a standard instant messaging protocol with

tags that could support semantics, and therefore powerful new

applications. Most important, his creation, Jabber, made it possible

for individual users to run their own servers instead of depending on

America Online or Yahoo!. Jabber had the potential to complement Tim

Berners-Lee’s idea of a Semantic Web.

Because Jabber used XML, it was seen as a bit heavyweight, and the

servers were reportedly hard to configure. But the possibilities were

too promising to pass up. So the IETF formalized it, gave it a clumsy

name suitable for a standard, and released a set of RFCs about it.

Unfortunately, XMPP languished until Google adopted it for their Talk

and Wave services. These high-profile applications suggest that it has

the scalability, flexibility, and robustness for social networking.

The P2P problems, in today’s context

Even if decentralized protocols and clients were invented, there will

be a long road to democratizing social networks. The messages are

expected to be lightweight, so photos and other large batches of

content would have to be stored somewhere outside the messages. Most

users wouldn’t trust their laptops (much less their mobile devices) to

store content and serve it up 24 hours a day, so they would need a

cloud service, which might or might not be distributed.

A backup service is also necessary in order to recover from a local

disk failure or other error that wipes out several years of your

accumulated identity.

Problems such as impersonation and unsolicited communications (spam)

are hard to solve in decentralized systems because trust is always a

hierarchical quality. This is true everywhere in life, beyond the

level of a family or neighborhood. We expect our professors to be good

because they were hired by the college, and expect the college to be

good because it was accredited by a centralized facility, whose

managers were in turn appointed by elected officials. This system can

and does break down regularly, so mechanisms for repair are always

built in.

Nobody can be banned from a decentralized social network because

there’s nothing to ban them from. But there are ways to re-introduce

enough centralization to validate credentials. For instance, the

American Bar Association could register lawyers in good standing, and

you could check whether someone claiming to be a lawyer in the US was

registered. But we wouldn’t want to take this process too far and

create a web of accreditations, because that would devalue people

whose skills and viewpoints lie outside the mainstream.

You could still check whether someone shares friends with you, because

one person’s claims of friendship could be traced back to the sites he

claims to be friends with. Someone could game the system by setting up

fake sites claiming to be people you know and linking back to them,

but this is a huge amount of work and leaves the perpetrator open to

arrest for fraud. Free software developer Thomas Lord suggests that

identity could also be verified through “a fairly shallow and

decentralized hierarchy of authentication like the system of notary

publics in physical life.”

All in all, the problems of finding people and trusting people

suggests that there’s role for aggregators, just as in the case of

RSS. And these aggregators could also offer the kind of tracking

services (who talked about me today?) and statistical services (is

Michael Jackson’s death still a major topic of conversation?) that get

business people so excited about Twitter. A decentralized social

network could still be business-friendly, because traffic could be

analyzed in order to target ads more accurately–but hopefully,

because peering clients are under the users’ control, people who

didn’t want the ads could configure their systems to screen them out.

When you set up an account, you could register with aggregators of

your choice. And whenever you connected to someone, you could

automatically register his account with a list of your favorite

aggregators, in case he hadn’t registered himself. If people wanted

control over where they’re aggregated, I supposed something equivalent

to a robots.txt file could be invented. But it’s not sporting

to refuse to be counted. And there’s no point in invoking privacy

concerns–face it, if the NSA wants to read your tweets, they’ll find

a way.

So those are some of the areas where the problems of P2P and social

networking intersect. Let’s remember that current social networks are

far from solving problems of findability, trust, and persistence as

well. I don’t check how many followers I have on Twitter; I figure

most of them are spam bots. (Apologies to any of my followers who

actually are sufficiently embodied to be reading this.)

Could

OpenSocial

be used to implement a P2P social network? It’s based on a single

object that is expected to query and update a single server. But the

interface could probably be implemented to run on a single user’s

system, registering the users or aggregators with whom she

communicates and querying all those users and aggregators as

necessary.

Industry analysts have been questioning for years whether Twitter is

financially viable. Well, maybe it isn’t–maybe this particular kind

of Internet platform is not destined to be a business. Responsibility

for the platform can be distributed among millions of sites and

developers, while business opportunities can be built on top of the

platform as services in analytics, publicity, and so forth.

Like Google, Twitter and the other leading commercial Internet sites

have made tremendous contributions to the functionality of the

Internet and have earned both their popularity and (where it exists)

their revenue. But the end-to-end principle and the reliability of

distributed processing must have their day again, whenever some use of

the Internet becomes too important to leave up to any single entity.