My previous post was a response to Gartner’s prediction last month that social networking would replace email as the “primary vehicle for interpersonal communications for 20 percent of business users.” In it, I named some properties that would need to be held by any social networking system that would successfully replace email.

• Ease of use
• Ubiquity across devices
• Platform, service and infrastructure independence

My argument boiled down to the following statement:

Email has succeeded because it’s open, standard and decentralized; for social networks to replace it, they must also be open, standard and decentralized.

Email is useful because just about everybody has an email address. I can get in touch with my clients in London, my friends here in Oxford or my grandfather in Austin, Texas, with equal ease, even though all of them are using different infrastructure and software provided by different companies. I use Gmail, but there doesn’t need to be any kind of formal agreement between Google and whoever’s providing my grandfather’s email, say. It just works; nobody owns email as a communications method, and anyone can set up an email server. The same is true with websites: anyone can set one up, and nobody owns the web.

For social communications to be as popular and ubiquitous as email, there must be one social web, and it must be owned by nobody. That means that each socially-aware site or application must implement the same social communication standards.

The best standards aren’t dictated: they evolve through common usage. If you look at HTTP (the protocol that the web relies on), SMTP (one of the protocols behind email) and file formats like RSS and HTML, the common thread behind them is that they’re simple. It turns out that through excellent work at companies like Google, Plaxo, SixApart, Twitter, JanRain and – perhaps incredibly – JPMorgan Chase & co, we already have a number of technologies that collectively embody the properties I listed above.

Notes and server architecture for one possible social web

These are my ideas about how these standards might be used. These aren’t intended as replacements for existing social networking platforms or services; rather, they could easily be added as additional features both to those and to many other types of application. The ability to share isn’t a uniquely required feature of social networking software – think about its usefulness in applications like Word or Google Docs, for example.

With email, you use a software client (Outlook, say, or the Gmail web interface) that speaks to an email server which does the hard business of sending and receiving messages to and from the wider Internet. Here, I will be describing a system where everyone has their own node on the social web, which effectively acts as a client and server. Mine might be here at benwerd.com, for example. It’s my website – my profile on the social web – and it’s where I send social communications. That’s the server side. However, it also acts as the client when I’m accessing resources stored on other peoples’ servers.

Establishing connections and granting permissions

Let’s say I want to make a resource available to my clients. With email, I’d send them each a separate copy. This is both insecure and inefficient: I have no control over what happens to that copy, and each time I send it I create a new version. With some back-and-forth, there could easily be ten or twenty individual copies of a document floating around. (I often bounce software specifications – typically Word documents – around with my clients, and this is something that happens to me regularly. Google Docs is probably a better solution, but not everybody has a Google account.)

With the social web, only one version needs to exist, which I own. If my clients have established a connection with me, I can restrict that resource so that only they may see it. The tricky bit is that in order to know if it’s really them, they must be authenticated in some way.

In monolithic systems like Facebook, where everyone uses the same website, that’s easy: my client must be logged in, and we must have established a friend connection. In a decentralized system, that’s a much harder problem, but not insurmountable. Two technologies will help us:

• OpenID: the open, decentralized authentication standard, which currently uses a website address as a kind of universal username
• OAuth: an open protocol that “allows users to share their private resources (e.g. photos, videos, contact lists) stored on one site with another site without having to hand out their username and password.” OAuth provides a secret token to applications that they can use to access authenticated services and resources behind the scenes

Specifically, we’ll need OpenID Connect (or, until that’s up and running, the OpenID / OAuth hybrid protocol), because we’ll be using OpenID to authenticate, OAuth to power our decentralized access permissions, and a number of other protocols and endpoints along the way. It’s much neater if these are all established at once.

Making friends and getting updates

The process would work in the following way. Let’s say I want to make a connection with my friend Marcus Povey.

1. I visit his site, and see that he is displaying a “connect to me” icon, indicating that it is a node on the social web. Later on, perhaps my browser would detect that this was a social web node in the same way that most browsers detect RSS feeds today, and light up an icon. Chris Messina has started a five part series on the browser as a social agent, which is worth a read.
2. Either way, I click on “connect to me”. Marcus’s site prompts me for the address of my profile, which I enter. (Later on, my browser does this bit for me.)
3. My profile address is an OpenID, and through the authentication process my social web node receives an OAuth token from him. No further authentication is required.
4. On his social web node dashboard, Marcus sees that I’ve established a connection with him. He can ignore it, in which case nothing happens, or he can mark me as a friend (or any other arbitrary designation, which could be unique to the software he’s using).
5. My social web node periodically checks for activity updates from Marcus’s, signing each request with that OAuth token so it knows who I am. This may be at my direct request; through repeated polling, RSS-style; or the update may be pushed to me through a PubSubHubbub ping.
6. Depending on the assignation he’s given me, Marcus’s node either responds with just a feed of public activity (if he’s ignored the request), or with additional activity he’s allowed me to see, in Activity Streams format.
7. Marcus can change my assignation or withdraw my OAuth token at any time from his dashboard. (Of course, throughout all this, the OAuth token mechanism is invisible to both users: it’s simply presented as a social connection.)

Embedded content and interacting directly on other social web nodes

Activity Streams is based on Atom, so content for items like blog posts (and resources like photos, using Atom Media) can be embedded directly in the activity feed. (Rob Dolin from Windows Live has some great examples.)

However, not all content is standard enough to be embeddable. In those cases, I can simply click through from Marcus’s activity update to his site, possibly log in again using OpenID, and interact with the content there. Additionally, by allowing users to log directly into his site via OpenID, Marcus can show selected people restricted content even if they don’t have the full range of social web software.

Friends lists and commenting

Further standards help us add extra functionality. If Marcus gives me permission, I might be able to download his contacts via Portable Contacts. Salmon is a protocol for commenting on distributed resources and allowing those comments to find their way upstream to the original, which is compatible with Activity Streams. Using this, I might be able to comment on Marcus’s activity items from within my dashboard and have them show up in his. Through this mechanism, all his friends could have a conversation on his activity stream items.

Reliability

So far, so good: we have a simple technological basis for permissive social communications. But if the social web is really going to replace email, we have to address one of the most important features for enterprise users: reliability. Businesses will not accept their critical communications being subject to fail whales.

In my next post, then, I’ll discuss person-to-person messaging and the thorny issue of guaranteed delivery.

The analysis firm Gartner just released five key predictions for social software:

1. By 2014, social networking services will replace e-mail as the primary vehicle for interpersonal communications for 20 percent of business users.
2. By 2012, over 50 percent of enterprises will use activity streams that include microblogging, but stand-alone enterprise microblogging will have less than 5 percent penetration.
3. Through 2012, over 70 percent of IT-dominated social media initiatives will fail.
4. Within five years, 70 percent of collaboration and communications applications designed on PCs will be modeled after user experience lessons from smartphone collaboration applications.
5. Through 2015, only 25 percent of enterprises will routinely utilize social network analysis to improve performance and productivity.

Social networks replacing email. Really?

I broadly agree with all of these, but that first prediction needs a little more analysis. Let’s think about why email has succeeded:

• Ease of use
• Ubiquity across devices
• Platform, service and infrastructure independence

I access email from my Dell PC, my iPhone, and have in the past used Blackberry phones, Macs, Linux boxes, etc, all the way down to Windows 3.1, using a combination of software that’s included Eudora, Thunderbird, Phoenix, Turnpike, and many more. Right now I use a combination of GMail, Google Apps and self-hosted email addresses; in the past I’ve used Microsoft Exchange in various guises, Yahoo Mail, and so on. No matter which provider or hardware I used, I could email anyone else with an email address, no matter which provider or hardware they used. Email is a completely open, interoperable standard.

Social networking is anything but an open, interoperable standard. If you use Facebook, you can communicate with other people on Facebook, full stop. Even networks based on open source solutions like Elgg are essentially social islands.

What needs to be done?

I strongly believe that social messaging can be significantly more useful to both enterprises and individuals than standard email. Proof-of-concept applications like Google Wave are beginning to show the way: you can make resources available to whoever needs to see them, rather than the current, inherently insecure practice of making copies and sending them out. Whereas email takes inspiration from letters and faxes, the social messaging paradigm is based more closely around conference calls and conversations.

Nonetheless, in a business situation, you need to be reasonably certain your message is going to reach the recipient, and the current platform constraints – only being able to message someone using the same site as you – are untenable. Let’s look again at those email success factors:

• Ease of use
• Ubiquity across devices
• Platform, service and infrastructure independence

Social networks do currently have ease of use. They may approach near-ubiquity across devices only if they create a developer ecosystem around their proprietary APIs, as Twitter has done, but this requires a lot of faith in a single third-party service.

No, I think it comes down to one principle:

Email has succeeded because it’s open, standard and decentralized; for social networks to replace it, they must also be open, standard and decentralized.

Next: real world, technical approaches to this that can be implemented today.

Going out and checking in

I’ve been using Foursquare quite a bit lately (here’s my profile). There’s a lot to be impressed by: not least the level of mobile integration. Foursquare doesn’t make much sense if you’re sitting at your desk, so it’s far easier to see where your friends are and check into a new location from the mobile app.

The way it promotes participation is even more interesting. The designers decided that just seeing where your friends were, and getting personalized travel tips, weren’t enough. You gain points – the same kind you get in Sonic the Hedgehog, say – for checking in at a location, exploring new places and telling the app about venues it’s never seen before. In turn, the points lead to badges, and there’s a weekly leaderboard for the top scorers among your friends. There’s no real tangible value to any of this, but you feel good about joining in. As a result, Foursquare is hugely addictive.

Open source participation

Over on Twitter, I asked:

Could the game dynamics used by apps like Foursquare be harnessed to make a more participative open source community?

Open source projects depend on contributions from their communities. Getting people to participate can be difficult; although many people will join in because it scratches some kind of itch, moral incentives like a place in the credits help. However, adding these kinds of game dynamics over the top could provide an extra push. Currently, the only quantifiable open source contributions are source code patches, and any software project has a lot more going on; this would provide an opportunity to quantify other, equally useful forms of participation.

Game dynamics in the enterprise

Graeme Hunter pointed out to me that this model wouldn’t solely be useful for open source. An internal project communications framework that also incorporated game dynamics could be a very interesting platform for ideas, solutions and internal innovation. He’s right; I think it’s an idea to keep in mind if you’re looking for software to use internally for your project.

There are also implications for online communities, where game dynamics are often already used (to rate individual contributions, for example). What if we used similar ideas for education? Or a community centering around journalism?

Photo by dpstyles, released under a Creative Commons license. It’s of a Target store in Milford, Massachusetts, where they use game dynamics to encourage faster checkout times.

Update: Graeme comments below with an exploration of what a participation framework using game dynamics might involve.

Google was an advertising company.

Back in 2005, Daring Fireball’s John Gruber described Google’s business as follows:

Judged by their profits, Google is an advertising company. They don’t profit from search, they don’t profit from software. They profit by selling ads. This isn’t to belittle them — I think Google is a terrific company, and they are profiting handsomely from ad revenue ($369 million last quarter). […] If Google has a platform, it’s an advertising platform, not a developer platform. I’m not even saying Google should have a developer platform — I’m just saying they don’t.

Fast forward to 2009, and Internet advertising is beginning to fail, declining slightly during the first half of the year. Sites like TechCrunch were quick to herald its demise with articles like Why Advertising Is Failing On The Internet, which declared:

My basic premise is that the internet is not replacing advertising but shattering it, and all the king’s horses, all the king’s men, and all the creative talent of Madison Avenue cannot put it together again.

It’s become clear that for a lot of purposes, advertising is not a viable or useful business model. Although it may still be suitable for very high-volume, mass-market sites and applications, it’s almost impossible to make money through advertising with niche or specialized content in most areas. (Some areas, like real estate, remain relatively lucrative.) Additionally, targeted ads require the advertising software to track your activity and store data about you, which more consumers are becoming concerned about. And perhaps most importantly of all, nobody actually wants to see ads – and advertisers are having to become more creative and invasive in order to compensate.

Similarly, if you want to make headway in the enterprise or educational spaces, targeted ads are inappropriate or impossible, for legal and policy reasons. For publicly-funded organizations like educational institutions, allowing commercial companies to track users is an ethical nightmare. For private enterprise, the data collection required for ad targeting is unacceptable, and the visual presence of advertising threatens their brands.

However, they are willing to pay for software, to the tune of $222.6 billion worldwide.

Boldly going to the enterprise & paid software.

The web is fast becoming a viable platform for applications: rather than visiting websites, we are increasingly using applications that happen to use the web as an interface. Google is at the forefront of this change.

On November 11, Google announced SPDY, an “embrace and extend” version of the HTTP protocol that underpins the web (it’s how browsers and web servers talk to each other). This new version has numerous tweaks that result in pages that load up to 55% faster – important if you’re trying to build responsive applications with web interfaces. Google have also been betting big on HTML 5, which extends the web’s UI infrastructure to provide support for a much richer experience without falling back on plugins like Flash. Two of the most important requirements for enterprise applications that use a web-based interface are offline capability (the ability to use the application with no Internet connection) and support for concurrent processes (allowing your web interface to perform more than one task at once). HTML 5 has both.

Google has evolved from a consumer search and advertising company, into one that provides enterprise infrastructure applications. Its plan is clearly to dominate Microsoft’s leadership and become a bona-fide software power. Recently, Microsoft has been playing catch-up, by including web-based versions of its applications in its enterprise Sharepoint intranet offering. It has also be moving against the tide by planning on offering advertising-supported versions.

Google’s CEO, Eric Schmidt, told the Garner Symposium last month why it was charging for their enterprise applications:

"Enterprise is a huge priority for the management team and me personally […] It’s the next big billion-dollar opportunity after our display (ad) business. […] We looked at ad-supported enterprise applications and decided most corporations would not be comfortable with random ads showing up on somebody’s desktop."

The web is moving away from advertising.

It’s not just Google that is moving away from a purely ad-supported, consumer strategy. Markus Witte, co-founder of the language learning portal Babbel, wrote on their blog about adjusting their business model:

Our plan, in fact, was to partially finance Babbel with advertising. We intended to provide a “freemium” product that would have a basic version that was public, while providing additional premium content for those who might want to dig deeper. But now we see this just doesn’t work. It simply is not possible to build a high-quality online learning environment while simultaneously selling ad space effectively. We tried to bring these two objectives together. But ultimately we had to accept that a business model appropriate for social networks and news services is plain wrong when applied to online education.

The numbers speak for themselves. The US paid e-learning market has been estimated to be worth $16.7 billion in 2009 and has a relatively small number of players; the US advertising revenues for the Internet as a whole were estimated to be $10.9 billion for the first half of 2009. (That’s $10.9 billion to the advertising companies, rather than the amount content and site owners see, which will be a subset of that amount.) When you run a startup company, you can either put your trust in display advertising and number of eyeballs looking at your site, or you can employ a sales team and ask for cash. Entranced by the model that Google originally promoted, Babbel tried the former, and discovered that it didn’t work; recognizing that they were a software company rather than a mass-media outlet, they then reverted to traditional business methods.

Using a centralized software service for non-core activities like language learning is probably fine. However, enterprise organizations can be uneasy about trusting software hosted by third parties (in what’s almost ubiquitously called “the Cloud”). Blog posts and photos are one thing, but it’s quite another to place your internal strategy documents, confidential discussions and financial data on servers owned by another firm with no real guarantee that they’ll remain unseen by prying eyes. It’s also insecure on a technical level: by using the Cloud, you’re outsourcing the fidelity and availability of your data. A much more preferential option would be to gain the ease of use of web applications, but store them securely on local infrastructure.

Open source software is commercial.

Later in Markus Witte’s post, he discusses some of the things that are successfully given away for free on the Internet; among them is open source.

In contrast to Open Source software and Creative Commons, where developers and authors often work for free, ad-sponsored services are designed to make money – and they do. […] But there is another, more insidious, drawback of ad-sponsoring that is less visible to the naked eye: the true customers of these ad-sponsored services are not the users but rather the advertisers. And as everywhere else, the Customer is King.

His remark about open source developers is a misconception: most open source development is done for profit. For example, over 70% of Linux kernel development is done by paid professionals, with a commercial goal in mind. This may be the basis of directly commercial activities like support; a market-based goal, for example to diminish Microsoft’s share; or it may be to ensure the longevity of the infrastructure that a company relies on. (More web servers are powered by open source than not; Netcraft reported this month that 55.33% of active websites are running Apache.) Make no mistake: open source is a business model – one that marries the free ethos of the Internet with paid commerce.

The most common open source business strategy is to use your “community edition” – the unadulterated open source software – as a loss leader that brings users to your commercial products and services. Releasing your software under an open source license theoretically means you gain a community of developers; if your software doesn’t work in a particular set of circumstances, they will often contribute back a fix for the problem. They may also contribute plugins and extra code that extends the functionality of your product. They get software that works for them (and the security that they can always use and modify the code to fit their needs); you get a wider market that you can sell commercial services to, using a wider, more solid set of functionality. Whereas, as Markus points out, the advertiser is king in ad-supported software, in open source the user is king.

Here are some examples you’ve probably heard of:

• The database software MySQL is released for free under the GNU Public License. Unusually, you’re allowed to mix and match it with software released under other open source licenses (but not closed-source software): they really want their product to spread. This is because they’ve got commercial options based on training, certification, partner agreements and consultancy services, as well as extra features for power users that aren’t available in the community product. (See the article MySQL’s Quid Pro Quo.)
• Ubuntu is a version of Linux designed with ease of use in mind; it riffs on the interfaces of operating systems like Microsoft Windows and Mac OS X. Canonical, the company behind it, make money through extensive commercial support and partner services. The partner ecosystem is their main bread and butter; the more companies pay, the better access they get to the core Ubuntu team and project strategy, marketing materials, rights to use Ubuntu branding and so on. In turn, those things help the partner companies earn more through their downstream Ubuntu services.
• Android is an open source operating system sponsored by Google. Although it’s mostly been used on mobile phones so far, it can actually run on a much wider range of devices; Android-powered netbooks are beginning to appear. This has the benefit of holding back Microsoft’s market share – Google is positioning its application suite, which is paid software, against Microsoft Office. (Windows 7 is said to run well on netbooks, and Google will soon have two open source netbook operating systems out: Android and Chrome OS.) There is also a directly commercial component: although Android is open source, it has direct links to Google’s consumer applications like Gmail and Calendar. Those applications, both within Android and on the web, are not open source, and must be licensed.

There are many more. Check out Network World’s list of 10 open source companies to watch, and note that one thing links them: they are all providing services aimed at the business market.

Charging for web-based software.

Google and Microsoft have both demonstrated that the market is ready for web-based business software: products that have the benefits of the web (you can access it from anywhere, on any compatible device) but are designed with the needs of enterprise organizations in mind. It must be secure, have the ability to be installed on an organization’s own infrastructure, and have a solid business model that ensures longevity of the product.

I also strongly believe that an open source development and licensing model, when coupled with a strong commercial strategy from the outset, is a great way to build a product’s feature set, userbase and reputation on the kinds of budgets that web startups are used to. It also makes it easily available to students, as well as a vast talent pool in places where buying software at western license prices is a trickier proposition; two groups that can be invaluable for promotion, feedback and involvement.

Finally, the commercial open source model for web-based applications allows you to easily create an ecosystem: if you create a compelling application that really does have a solid business model, other companies will be very interested in taking a cut. The more people who have an interest in your product succeeding, the better. If you give them a solid commercial reason to invest upstream, and create a great product that makes end-users’ lives easier, everyone wins.

I’ve spent the last week in a series of very interesting mind-sharing meetings. First, the American Association of Colleges & Universities flew me into Washington DC to discuss the future of assessment in higher education with education professionals as well as new techsphere friends like Silona Bonewald and Amber Case. Second, Michael Byrne from Harvard University’s Hauser Center for Nonprofit Organizations invited me to speak there about the open, social web, the notes for which I’ll write up and post here soon. A great big thank you to both organizations for inviting me; I’ve felt privileged to have such insightful and interesting conversations over the last few days.

It seems like the shift in innovation in social tools has gone from developing new and interesting technologies to developing interesting models that happen to use technologies. This is a big step, and in some ways represents the space coming of age. There’s still plenty of technological development and innovation to do, but the platform and concepts are at a point where they can be adapted into all manner of social collaborative spaces, business tools, social experiments, games, art projects, and combinations of those things. It’s becoming a very exciting field to work in.

That said, some adaptation needs to happen, and it’s important to realize that these ideas only work in an effective way when they’re made relevant to the outside world. The social web is extremely political: it imposes an opinion about how the world should be open and social, democratic and centered on individual preferences, but ironically doesn’t allow for differences in that point of view. That makes it very hard for late adopters, enterprises, governments and public organizations to feel the benefit.

Over on Persona Prime, Silona makes this point about technology-inspired government transparency:

Where is the change management?  We are doing some big stuff here and we are poised to make serious mistakes and I see no prelim work being done to prevent this.  Where are the best practices in open govt documents?  All I see are “I want” lists.  I have not seen us doing anything serious to ally [the fears of people who might be wary of transparency].

It would be cool if every Fortune 500 company wanted to be on Twitter, but the reality is that they don’t, and often for very legitimate reasons. If what we’re doing is establishing a new, global, decentralized way to create, share, disseminate and discover information, then we have to take into account the differences in all the decentralized nodes. Embracing different corporate cultures, and different opinions on how communication should be, is part of that. Compromising and addressing the fears of companies and late adopters will build a larger userbase for all our tools, and make the platform much more useful in the long run.