XMPP: powering the real-time, really live web

June 12, 2009 | 7 comments

When Google Wave was previewed last week, most people focused on its Gmail-like user interface and the slick way it handled collaborative spaces in real time. What was more exciting from a web technology point of view, although much less sexy, was the underlying protocol: Google Wave is built on top of an extended version of an open source real-time data standard called XMPP (aka Jabber when it’s used as an instant messaging context). This means that anyone can build servers and end-user applications that are compatible with Wave.

Yesterday, the BBC announced that they were using an open source framework called Hemlock in order to provide real-time web games on their Childrens’ site. Hemlock’s interfaces are built in Flash, but the real-time aspect is powered by XMPP. Developer Ron DeVera explained why over on the Hemlock blog:

XMPP is commonly used to enable chatrooms, like Gtalk or Facebook Chat. However, instead of just sending text, Hemlock lets you send all kinds of data at the speed of instant messaging. Even better, XMPP is designed to scale up and send data to many people at once, so you can push that data to multiple people in real-time.

In other words, it’s perfect for real-time applications on the web.

Traditionally, live web applications have used a development methodology called AJAX, which stands for Asynchronous Javascript And XML. In order to provide the illusion of a real-time application, the web browser needs to continue to ask the server if there’s any new data to ask, via a new AJAX request – which is a waste of bandwidth and resources both on the user and server side. XMPP, on the other hand, pings the client software whenever there’s new data.

Servers already need software to host and process dynamic web pages, as well as a database server to handle most types of user data. To use XMPP, you need to add an XMPP server to the mix – something that in the past was more fiddly than most people were willing to play with. However, times have changed, and ejabberd is free, open source and easy to install on Windows, Mac OS X and Linux servers. (I’m still at the early stages of playing with it; I’ll let you know how I get on.)

The XMPP community is still very small compared to the wider web, but developments like Wave and Hemlock should accelerate growth and interest in it. There are some interesting projects springing up, and lots of libraries available for XMPP development; XMPP programming is a skill developers should get in on now, before it really heats up.

Social networking: beyond the silo

June 8, 2009 | 1 comment

  1. The rise of social networking
  2. Monetization vs. collaboration
  3. The open web
  4. Fluid collaboration

The rise of social networking

Social forces have been the driving force behind application innovation on the web. Whereas previously we might have looked to advances in computer science for new directions, now some of the most dramatically impactful applications are lightweight, simple, and technologically unimpressive. The best new web applications have centered around collaboration, sharing and discovery with other people.

Correspondingly, enterprises have been relatively quick to pick up on this trend, and software vendors have been quick to grab the market. In an Intranet Journal article earlier this year, Kara Pernice, managing director at the Nielsen Normal Group, had this to say about the rise of social technology on the intranet:

"In the 9 years [the Intranet Design Annual, which highlights the ten best-designed intranets of the year] has been coming out (since 2001), I’ve never seen a change quite as great as this one."

On the Internet at large, social network use is growing at ten times the rate of other activities and now accounts for 10% of all online time, according to Nielsen Online in this March 2009 report (PDF), and is now more popular than email. Jerimiah Owyang has a list of more relevant statistics over on this digest blog post. Executive summary: social networks are big, transformative in terms of how we communicate and share information, and growing at an enormous rate.

Monetization vs. collaboration

Wikipedia defines a “walled garden”, in software terms, as being:

[..] A closed set or exclusive set of information services provided for users (a method of creating a monopoly or securing an information system).

In other words, a walled garden is a system where the data can not easily be imported or exported. These are often also called data silos, after the solid buildings used for secure storage.

Facebook, the #1 social networking site in most western countries, has over 200 million users, including over 30 million who update their profiles at least once a day. The network is free to use, yet their revenue for 2008 has been estimated at around $265 million, despite a decidedly “in progress” revenue strategy.

This has traditionally required a walled garden strategy: the content that users put into Facebook has not been easily removed for export or viewing in other interfaces, in order to preserve revenue from advertising (and – although this is a hunch – revenue from statistical analysis of users’ data). It’s only been in the light of some extremely negative publicity (for example this February 2008 New York Times article) that they have begun to relax this policy and embrace the open direction that much of the rest of the web is heading in.

Speaking personally, I get more enquiries from people wanting to build something “Facebook-like” than anything else, presumably because of its phenomenal popularity. However, this kind of walled garden approach is not conducive to true collaboration; generally people who ask for this are lacking a full understanding of the processes involved in social networking.

According to Nielsen, there are almost 1.6 billion people online. While Facebook’s 200 million sounds like a lot, it’s actually a drop in the digital ocean – so what happens if I want to share a Facebook conversation with someone who hasn’t signed up? The only way is currently to email them a link and force them to register for the service. Facebook would love me to do this, of course, because they get more eyeballs to view their ads and more people to fill in profiles. But what’s the point of even being on the web if you can’t make use of the decentralized communication features that form its backbone?

If I want to collaborate effectively online centering around a resource (which could be a file, a discussion or a pointer to something external), I need to be able to:

  • Share that resource with the people who need to see it
  • Grant access for them to edit it if required
  • Notify them that it’s been shared with them
  • Restrict access from everyone else

Furthermore, I need to do this with the lowest possible barrier to entry. My aim is to collaborate, not to get people to use a particular piece of software. By restricting this process, the Facebook model hinders collaboration.

The open web

The web was designed to be an open system, and adheres to principles (notably “every object addressable”, ensuring that every resource on the web has a unique reference address) set out by Doug Engelbart for open hypertext systems generally. Because web pages are interoperable, and all use the same basic standards, any page on the web is allowed to link to any other page on the web, no matter who wrote it or where it is hosted. In many ways that’s the key to why the platform is successful: despite being fragmented across millions of computers throughout the world, it navigates like a cohesive whole and can be viewed using a single piece of browsing software. (The downside to this is that the whole platform lives or dies depending on the capabilities of the browser you use: the sad fact is that Internet Explorer users, who often don’t have a choice because of policy decisions in their working environment, are at a disadvantage.)

While the original web was content-based, the social web is collaborative and centered around live data. However, because web applications are each developed separately using different sets of back-end infrastructure, their data does not adhere to the principle of interoperability – their user interfaces all use the same basic standards and can be viewed in a browser, but the underlying applications and data models tend to not work with each other. When social networks emerged, for example, there was no way to get Livejournal and Friendster, two of the pioneers in the space, to speak the same language; you still can’t add someone as a friend on one social network from another. More recently, this has become apparent in the walled garden approaches of Facebook and others.

Not only does this situation create a bottleneck for application design, and run contrary to the underlying principles that made the web a success, but it’s also a bottleneck to better collaboration. As Tim Berners-Lee, the web’s inventor, put it recently in this essential TED talk, data needs to be linked and interoperable in the same way pages are now. Beyond that, because walled garden services are making money out of the private information we’re loading onto them, there’s a human issue regarding the overall control of that data. Marc Canter, Joseph Smarr and others codified this into a Bill of Rights for users of the social web back in 2007. Though the issue has moved on since then, the underlying principles set out there are essential for open, collaborative, social tools on the web.

While the World Wide Web Consortium works on academically-developed standards for linked data in the form of the semantic web, developers have been getting their game on trying to solve the problems of interoperability between their applications and user control over their data. Application Programming Interfaces (APIs) – published sets of instructions for programmatically querying and extending web applications – have become popular, but in a very walled garden kind of way. Arguably the most successful has been Twitter’s API, which has led to a number of high profile third-party applications like TweetDeck and Tweetie that collectively eclipse Twitter’s own website interface in volume of usage. But these APIs are their own form of walled garden: an application written for Twitter will only work with Twitter, for example. The APIs are not generalized between applications, and as such are not truly open; in many ways they’re a way for services to get more functionality and reach for free.

One of the first attempts to publicize the benefits of truly open data was Marc Canter’s Data Sharing Summit, which I wrote about at the time for ZDNet. Chris Saad’s DataPortability.org attempted (largely successfully) to brand it, and latterly the Open Web Foundation has attracted some of the web’s leading lights in order to create a single organization to handle the creation of a set of open web application standards. Many of these comprise the Open Stack, which I’ve written about before; more generally, Chris Messina has written a very thoughtful overview on the topic.

Fluid collaboration

It used to be that to use the web, you would need to sit down at your computer and log on. Those days are over; the web is becoming more and more ubiquitous, thanks to devices like the iPhone. It’s also being integrated into software that wasn’t previously connected – it’s as easy, for example, to paste the URL of an image into the ‘Insert Image’ dialog box in most word processors as it is to pick an image from your own hard disk. The open, generalized API standards being created by groups like the Open Web Foundation bring us closer to enjoying that level of integration with collaborative social technologies.

The Internet is people, not technology: tools on the web (or anywhere else) facilitate social networks, but are not the network themselves. Currently they consist of destination sites, like Facebook, LinkedIn or Twitter – places that you explicitly have to visit in order to collaborate or share. This is the currently-fashionable model, but it’s a necessarily limited view of how collaboration can take place: all of these sites thrive on the walled garden model and are designed around keeping participation within their walls.

Not everything on the Internet works this way. Email, and increasingly Instant Messaging, are two technologies that generally do not: messages on email, Jabber and to a much lesser extent Skype are peer-to-peer and do not go through a central service:

  1. You select the people you wish to collaborate (in this case, email or chat) with. Nobody but the listed recipients will be able to see the content you share with them, and it doesn’t matter if they’re using the same service as you; you don’t have to invite them to join email in the same way you have to invite people to join Facebook.
  2. You write your content.
  3. You send it.
  4. They (hopefully) send content back.
  5. The collaborative exchange lasts only as long as it’s useful, and then disappears (but is archived for reference).

Recently, Google announced Wave, a decentralized pairing of protocol and open source web application that took email and IM as its inspirations to redefine how collaborative social technologies could work. Questions have been raised about how a decentralized tool like this can work with corporate data policies present in most large enterprises and public sector organizations, but in some ways they miss the point: Google Wave is best thought of as a proof of concept for how decentralized, transient communities can work in a standard way on the web. In short, websites are a kind of walled garden in themselves: what we will return to is the idea of the web as an open patchwork of people, data and information that links together to form a whole, much stronger than the sum of its parts.

Predicting the future of social networking on the web is hard. However, I believe that as general open social technologies develop and become more commonplace, the “social networking site” will shrink in importance – instead, social network facilitators will become more and more ingrained in all the software you use. This will dramatically increase the types of content and communication that can be used, and present opportunities for much wider, more fluid and – most importantly – more productive collaboration as a whole.

Google Wave is exciting and transformative

May 28, 2009 | 3 comments

For almost five years, I’ve had a dream of creating a decentralized social networking system with granular access permissions and a customizable workflow. It would be open source, with an underlying, decentralized open protocol based on XMPP that anyone could build on top of and extend. It would redefine the way we work on the web, and make social connections as much of a part of the Internet infrastructure as email is today.

Google just released it.


In all seriousness, Google Wave, and particularly the Wave Protocol, have the ability to completely change how we communicate on the Internet. That might sound a little over-enthusiastic, but so far the project seems to be getting everything right. It’s distributed, extensible, granular (as public or private as you want) and open. There’s been some talk about the interface for their sample client being a little cluttered, but the team are at pains to point out that it’s in the early stages – and this misses the wider implications of the technology.

I’m not the only one talking in superlatives. Tim O’Reilly points out:

Suddenly, familiar applications look as old-fashioned as DOS applications looked as the GUI era took flight. Now that the web is the platform, it’s time to take another look at every application we use today, and ask the same question [project founders] Lars and Jens asked themselves [with email]: "What would this look like if we invented it today instead of twenty-five years ago?"

It remains to be seen how the project will develop, but I’ll be paying very close attention.