My previous post was a response to Gartner’s prediction last month that social networking would replace email as the “primary vehicle for interpersonal communications for 20 percent of business users.” In it, I named some properties that would need to be held by any social networking system that would successfully replace email.

• Ease of use
• Ubiquity across devices
• Platform, service and infrastructure independence

My argument boiled down to the following statement:

Email has succeeded because it’s open, standard and decentralized; for social networks to replace it, they must also be open, standard and decentralized.

Email is useful because just about everybody has an email address. I can get in touch with my clients in London, my friends here in Oxford or my grandfather in Austin, Texas, with equal ease, even though all of them are using different infrastructure and software provided by different companies. I use Gmail, but there doesn’t need to be any kind of formal agreement between Google and whoever’s providing my grandfather’s email, say. It just works; nobody owns email as a communications method, and anyone can set up an email server. The same is true with websites: anyone can set one up, and nobody owns the web.

For social communications to be as popular and ubiquitous as email, there must be one social web, and it must be owned by nobody. That means that each socially-aware site or application must implement the same social communication standards.

The best standards aren’t dictated: they evolve through common usage. If you look at HTTP (the protocol that the web relies on), SMTP (one of the protocols behind email) and file formats like RSS and HTML, the common thread behind them is that they’re simple. It turns out that through excellent work at companies like Google, Plaxo, SixApart, Twitter, JanRain and – perhaps incredibly – JPMorgan Chase & co, we already have a number of technologies that collectively embody the properties I listed above.

Notes and server architecture for one possible social web

These are my ideas about how these standards might be used. These aren’t intended as replacements for existing social networking platforms or services; rather, they could easily be added as additional features both to those and to many other types of application. The ability to share isn’t a uniquely required feature of social networking software – think about its usefulness in applications like Word or Google Docs, for example.

With email, you use a software client (Outlook, say, or the Gmail web interface) that speaks to an email server which does the hard business of sending and receiving messages to and from the wider Internet. Here, I will be describing a system where everyone has their own node on the social web, which effectively acts as a client and server. Mine might be here at benwerd.com, for example. It’s my website – my profile on the social web – and it’s where I send social communications. That’s the server side. However, it also acts as the client when I’m accessing resources stored on other peoples’ servers.

Establishing connections and granting permissions

Let’s say I want to make a resource available to my clients. With email, I’d send them each a separate copy. This is both insecure and inefficient: I have no control over what happens to that copy, and each time I send it I create a new version. With some back-and-forth, there could easily be ten or twenty individual copies of a document floating around. (I often bounce software specifications – typically Word documents – around with my clients, and this is something that happens to me regularly. Google Docs is probably a better solution, but not everybody has a Google account.)

With the social web, only one version needs to exist, which I own. If my clients have established a connection with me, I can restrict that resource so that only they may see it. The tricky bit is that in order to know if it’s really them, they must be authenticated in some way.

In monolithic systems like Facebook, where everyone uses the same website, that’s easy: my client must be logged in, and we must have established a friend connection. In a decentralized system, that’s a much harder problem, but not insurmountable. Two technologies will help us:

• OpenID: the open, decentralized authentication standard, which currently uses a website address as a kind of universal username
• OAuth: an open protocol that “allows users to share their private resources (e.g. photos, videos, contact lists) stored on one site with another site without having to hand out their username and password.” OAuth provides a secret token to applications that they can use to access authenticated services and resources behind the scenes

Specifically, we’ll need OpenID Connect (or, until that’s up and running, the OpenID / OAuth hybrid protocol), because we’ll be using OpenID to authenticate, OAuth to power our decentralized access permissions, and a number of other protocols and endpoints along the way. It’s much neater if these are all established at once.

Making friends and getting updates

The process would work in the following way. Let’s say I want to make a connection with my friend Marcus Povey.

1. I visit his site, and see that he is displaying a “connect to me” icon, indicating that it is a node on the social web. Later on, perhaps my browser would detect that this was a social web node in the same way that most browsers detect RSS feeds today, and light up an icon. Chris Messina has started a five part series on the browser as a social agent, which is worth a read.
2. Either way, I click on “connect to me”. Marcus’s site prompts me for the address of my profile, which I enter. (Later on, my browser does this bit for me.)
3. My profile address is an OpenID, and through the authentication process my social web node receives an OAuth token from him. No further authentication is required.
4. On his social web node dashboard, Marcus sees that I’ve established a connection with him. He can ignore it, in which case nothing happens, or he can mark me as a friend (or any other arbitrary designation, which could be unique to the software he’s using).
5. My social web node periodically checks for activity updates from Marcus’s, signing each request with that OAuth token so it knows who I am. This may be at my direct request; through repeated polling, RSS-style; or the update may be pushed to me through a PubSubHubbub ping.
6. Depending on the assignation he’s given me, Marcus’s node either responds with just a feed of public activity (if he’s ignored the request), or with additional activity he’s allowed me to see, in Activity Streams format.
7. Marcus can change my assignation or withdraw my OAuth token at any time from his dashboard. (Of course, throughout all this, the OAuth token mechanism is invisible to both users: it’s simply presented as a social connection.)

Embedded content and interacting directly on other social web nodes

Activity Streams is based on Atom, so content for items like blog posts (and resources like photos, using Atom Media) can be embedded directly in the activity feed. (Rob Dolin from Windows Live has some great examples.)

However, not all content is standard enough to be embeddable. In those cases, I can simply click through from Marcus’s activity update to his site, possibly log in again using OpenID, and interact with the content there. Additionally, by allowing users to log directly into his site via OpenID, Marcus can show selected people restricted content even if they don’t have the full range of social web software.

Friends lists and commenting

Further standards help us add extra functionality. If Marcus gives me permission, I might be able to download his contacts via Portable Contacts. Salmon is a protocol for commenting on distributed resources and allowing those comments to find their way upstream to the original, which is compatible with Activity Streams. Using this, I might be able to comment on Marcus’s activity items from within my dashboard and have them show up in his. Through this mechanism, all his friends could have a conversation on his activity stream items.

Reliability

So far, so good: we have a simple technological basis for permissive social communications. But if the social web is really going to replace email, we have to address one of the most important features for enterprise users: reliability. Businesses will not accept their critical communications being subject to fail whales.

In my next post, then, I’ll discuss person-to-person messaging and the thorny issue of guaranteed delivery.

Related entries

• How social networks can replace email (11)
• The Open Stack and truly open APIs (0)
• Social networking: beyond the silo (1)

This is inspired by the iBooks launch, but it’s applicable to any ereader that uses the ePub format. (Or, indeed, it could use any ebook format – MobiPocket, Kindle, DAISY, etc.)

A podcast is just an RSS feed with a file enclosure – part of the RSS standard – that points to an MP3 file. Similarly, video podcasts point to video files. An obvious evolution, then, is the pubcast: periodical publications delivered through RSS feeds.

Free publication subscriptions

In the free case, a user would simply subscribe to a public pubcast feed with a compatible reader. The reader software would check regularly for updates, and new publications would be downloaded and fed into the user’s ereader software on release. Easy.

Paid publication subscriptions

In the case of paid publications, there are two options:

An authenticated pubcast feed. When you subscribe to a publication, you get an address to an RSS feed that requires a username and password to download content. (Gmail is an example of an application which already does this.) This authentication ensures that only paid subscribers can access the file, but you could go a step further and watermark the publications themselves.

Activation within the ebook file. The RSS feed itself is public, but each downloaded publication could require an access code to read. This would open the door for public feeds of paid journals, where users could buy each issue individually to read.

Making subscriptions an open standard

Either way, this approach would allow any ereader using any compatible software solution to subscribe to periodicals. It could be used for newspapers, magazines, journals, zines, or new kinds of periodical; they could be hosted anywhere and, in the case of paid content, use any payment provider. I love reading, but dislike monopolies, so this is something I’d like to see.

Related entries

• iBooks is a killer app for ebooks (1)
• Using game dynamics to drive participation (1)
• The death of newspapers, and why it matters (2)

If you pay any attention at all to the tech press, you’re probably sick to death of the iPad, Apple’s announced tablet device. I’m posting about it anyway, because there are two things that haven’t been discussed enough, which I think deserve a mention.

One: this isn’t a device for the tech community. I think Rafe Colburn hits it on the head:

It’s just an iPod Touch with a big screen, but that’s all that many people need from a computer. You can use it to surf the Web, read email, listen to music, watch video, or compose documents. That’s the personal computer use case for many people. And I think a lot of people are going to buy them.

He goes on to discuss the locked-down nature of the device, which I agree is a setback that may have a profound impact on the consumer computing industry. (On the other hand, as Yehuda Katz argues, this is a major win for standards-based web applications.)

Two: for me, the big news wasn’t the iPad at all. It was iBooks: Apple’s new iTunes-like store for ebooks. You may remember that iTunes pretty much revolutionized how we buy music, and this is the same; the books are stored in the open ePub standard, so they’ll play with other ereaders, and the experience is seamless. (You almost certainly won’t need an iPad to buy from iBooks.)

Mashable notes that some big players are on board:

iBooks is backed by big-time launch partners Penguin, Simon and Schuster, HarperCollins, Macmillan and Hachette, all publishing powerhouses in their own rights.

You can think about the iPad as a kind of $499 catwalk model, that other devices will slowly emulate over the next couple of years. But iBooks? That’s a store that anyone will be able to use right away, which just might change the publishing industry forever.

Photo by kennymatic, released under a Creative Commons license.

Related entries

• PubCasts: subscribe to publications through RSS (1)
• A projector inside your cellphone (0)

The British equivalent to Obama’s data.gov opened today. Over at ReadWriteWeb, Marshall Kirkpatrick points out the scale of the ambition involved:

At launch, Data.gov.uk has nearly 3,000 data sets available for developers to build mashups with. The U.S. site, Data.gov, has less than 1,000 data sets today.

[…][Unlike the US equivalent, the site] includes 22 military data sets at launch, including one called Suicide and Open Verdict Deaths in the U.K. Regular Armed Forces.

However, these are raw datasets. As Paul Clarke points out, the site only pays lip service to openness until someone comes along and turns these sets into useful reports and applications:

The only test of real success is: use. Not usefulness. Not theoretical use. Real use. Getting beyond the novelty application, the demonstrator, and the hobby lies at the heart of really untapping the potential of data.gov.uk.

Indeed, the figures that Techcrunch Europe report suggest that turning this data into something useful may be harder than it sounds:

So far over 2,400 developers have registered to test the site and provide feedback, [while] 10 applications have been created.

I left a comment on Paul Clarke’s post pointing out some potential pitfalls that may inhibit innovation, including the government’s insistence on licensing the data under Crown Copyright and their impartiality regarding Twitter. There’s also been some criticism around the lack of a common data format for each feed (although the RDF triple proudly displayed on the front page suggests this is likely to change).

Nonetheless, I believe this represents a huge step forward. Turning raw materials into useful, compelling applications that improve the users’ quality of life requires a huge amount of creativity and talent, and providing the data feeds in the first place is a crucial first step.

You can list all the available datasets here.

Related entries

• The mechanics of "open" (0)
• Public IT project hell: let’s make government work for us (1)
• Activity Streams and OAuth: a social web architecture (2)

Google was an advertising company.

Back in 2005, Daring Fireball’s John Gruber described Google’s business as follows:

Judged by their profits, Google is an advertising company. They don’t profit from search, they don’t profit from software. They profit by selling ads. This isn’t to belittle them — I think Google is a terrific company, and they are profiting handsomely from ad revenue ($369 million last quarter). […] If Google has a platform, it’s an advertising platform, not a developer platform. I’m not even saying Google should have a developer platform — I’m just saying they don’t.

Fast forward to 2009, and Internet advertising is beginning to fail, declining slightly during the first half of the year. Sites like TechCrunch were quick to herald its demise with articles like Why Advertising Is Failing On The Internet, which declared:

My basic premise is that the internet is not replacing advertising but shattering it, and all the king’s horses, all the king’s men, and all the creative talent of Madison Avenue cannot put it together again.

It’s become clear that for a lot of purposes, advertising is not a viable or useful business model. Although it may still be suitable for very high-volume, mass-market sites and applications, it’s almost impossible to make money through advertising with niche or specialized content in most areas. (Some areas, like real estate, remain relatively lucrative.) Additionally, targeted ads require the advertising software to track your activity and store data about you, which more consumers are becoming concerned about. And perhaps most importantly of all, nobody actually wants to see ads – and advertisers are having to become more creative and invasive in order to compensate.

Similarly, if you want to make headway in the enterprise or educational spaces, targeted ads are inappropriate or impossible, for legal and policy reasons. For publicly-funded organizations like educational institutions, allowing commercial companies to track users is an ethical nightmare. For private enterprise, the data collection required for ad targeting is unacceptable, and the visual presence of advertising threatens their brands.

However, they are willing to pay for software, to the tune of $222.6 billion worldwide.

Boldly going to the enterprise & paid software.

The web is fast becoming a viable platform for applications: rather than visiting websites, we are increasingly using applications that happen to use the web as an interface. Google is at the forefront of this change.

On November 11, Google announced SPDY, an “embrace and extend” version of the HTTP protocol that underpins the web (it’s how browsers and web servers talk to each other). This new version has numerous tweaks that result in pages that load up to 55% faster – important if you’re trying to build responsive applications with web interfaces. Google have also been betting big on HTML 5, which extends the web’s UI infrastructure to provide support for a much richer experience without falling back on plugins like Flash. Two of the most important requirements for enterprise applications that use a web-based interface are offline capability (the ability to use the application with no Internet connection) and support for concurrent processes (allowing your web interface to perform more than one task at once). HTML 5 has both.

Google has evolved from a consumer search and advertising company, into one that provides enterprise infrastructure applications. Its plan is clearly to dominate Microsoft’s leadership and become a bona-fide software power. Recently, Microsoft has been playing catch-up, by including web-based versions of its applications in its enterprise Sharepoint intranet offering. It has also be moving against the tide by planning on offering advertising-supported versions.

Google’s CEO, Eric Schmidt, told the Garner Symposium last month why it was charging for their enterprise applications:

"Enterprise is a huge priority for the management team and me personally […] It’s the next big billion-dollar opportunity after our display (ad) business. […] We looked at ad-supported enterprise applications and decided most corporations would not be comfortable with random ads showing up on somebody’s desktop."

The web is moving away from advertising.

It’s not just Google that is moving away from a purely ad-supported, consumer strategy. Markus Witte, co-founder of the language learning portal Babbel, wrote on their blog about adjusting their business model:

Our plan, in fact, was to partially finance Babbel with advertising. We intended to provide a “freemium” product that would have a basic version that was public, while providing additional premium content for those who might want to dig deeper. But now we see this just doesn’t work. It simply is not possible to build a high-quality online learning environment while simultaneously selling ad space effectively. We tried to bring these two objectives together. But ultimately we had to accept that a business model appropriate for social networks and news services is plain wrong when applied to online education.

The numbers speak for themselves. The US paid e-learning market has been estimated to be worth $16.7 billion in 2009 and has a relatively small number of players; the US advertising revenues for the Internet as a whole were estimated to be $10.9 billion for the first half of 2009. (That’s $10.9 billion to the advertising companies, rather than the amount content and site owners see, which will be a subset of that amount.) When you run a startup company, you can either put your trust in display advertising and number of eyeballs looking at your site, or you can employ a sales team and ask for cash. Entranced by the model that Google originally promoted, Babbel tried the former, and discovered that it didn’t work; recognizing that they were a software company rather than a mass-media outlet, they then reverted to traditional business methods.

Using a centralized software service for non-core activities like language learning is probably fine. However, enterprise organizations can be uneasy about trusting software hosted by third parties (in what’s almost ubiquitously called “the Cloud”). Blog posts and photos are one thing, but it’s quite another to place your internal strategy documents, confidential discussions and financial data on servers owned by another firm with no real guarantee that they’ll remain unseen by prying eyes. It’s also insecure on a technical level: by using the Cloud, you’re outsourcing the fidelity and availability of your data. A much more preferential option would be to gain the ease of use of web applications, but store them securely on local infrastructure.

Open source software is commercial.

Later in Markus Witte’s post, he discusses some of the things that are successfully given away for free on the Internet; among them is open source.

In contrast to Open Source software and Creative Commons, where developers and authors often work for free, ad-sponsored services are designed to make money – and they do. […] But there is another, more insidious, drawback of ad-sponsoring that is less visible to the naked eye: the true customers of these ad-sponsored services are not the users but rather the advertisers. And as everywhere else, the Customer is King.

His remark about open source developers is a misconception: most open source development is done for profit. For example, over 70% of Linux kernel development is done by paid professionals, with a commercial goal in mind. This may be the basis of directly commercial activities like support; a market-based goal, for example to diminish Microsoft’s share; or it may be to ensure the longevity of the infrastructure that a company relies on. (More web servers are powered by open source than not; Netcraft reported this month that 55.33% of active websites are running Apache.) Make no mistake: open source is a business model – one that marries the free ethos of the Internet with paid commerce.

The most common open source business strategy is to use your “community edition” – the unadulterated open source software – as a loss leader that brings users to your commercial products and services. Releasing your software under an open source license theoretically means you gain a community of developers; if your software doesn’t work in a particular set of circumstances, they will often contribute back a fix for the problem. They may also contribute plugins and extra code that extends the functionality of your product. They get software that works for them (and the security that they can always use and modify the code to fit their needs); you get a wider market that you can sell commercial services to, using a wider, more solid set of functionality. Whereas, as Markus points out, the advertiser is king in ad-supported software, in open source the user is king.

Here are some examples you’ve probably heard of:

• The database software MySQL is released for free under the GNU Public License. Unusually, you’re allowed to mix and match it with software released under other open source licenses (but not closed-source software): they really want their product to spread. This is because they’ve got commercial options based on training, certification, partner agreements and consultancy services, as well as extra features for power users that aren’t available in the community product. (See the article MySQL’s Quid Pro Quo.)
• Ubuntu is a version of Linux designed with ease of use in mind; it riffs on the interfaces of operating systems like Microsoft Windows and Mac OS X. Canonical, the company behind it, make money through extensive commercial support and partner services. The partner ecosystem is their main bread and butter; the more companies pay, the better access they get to the core Ubuntu team and project strategy, marketing materials, rights to use Ubuntu branding and so on. In turn, those things help the partner companies earn more through their downstream Ubuntu services.
• Android is an open source operating system sponsored by Google. Although it’s mostly been used on mobile phones so far, it can actually run on a much wider range of devices; Android-powered netbooks are beginning to appear. This has the benefit of holding back Microsoft’s market share – Google is positioning its application suite, which is paid software, against Microsoft Office. (Windows 7 is said to run well on netbooks, and Google will soon have two open source netbook operating systems out: Android and Chrome OS.) There is also a directly commercial component: although Android is open source, it has direct links to Google’s consumer applications like Gmail and Calendar. Those applications, both within Android and on the web, are not open source, and must be licensed.

There are many more. Check out Network World’s list of 10 open source companies to watch, and note that one thing links them: they are all providing services aimed at the business market.

Charging for web-based software.

Google and Microsoft have both demonstrated that the market is ready for web-based business software: products that have the benefits of the web (you can access it from anywhere, on any compatible device) but are designed with the needs of enterprise organizations in mind. It must be secure, have the ability to be installed on an organization’s own infrastructure, and have a solid business model that ensures longevity of the product.

I also strongly believe that an open source development and licensing model, when coupled with a strong commercial strategy from the outset, is a great way to build a product’s feature set, userbase and reputation on the kinds of budgets that web startups are used to. It also makes it easily available to students, as well as a vast talent pool in places where buying software at western license prices is a trickier proposition; two groups that can be invaluable for promotion, feedback and involvement.

Finally, the commercial open source model for web-based applications allows you to easily create an ecosystem: if you create a compelling application that really does have a solid business model, other companies will be very interested in taking a cut. The more people who have an interest in your product succeeding, the better. If you give them a solid commercial reason to invest upstream, and create a great product that makes end-users’ lives easier, everyone wins.

Related entries

• Using game dynamics to drive participation (1)
• Activity Streams and OAuth: a social web architecture (2)
• How social networks can replace email (11)

In response to recent events, I’d like to propose a different kind of web service that overcomes the privacy and reliability issues with cloud web applications, while providing a solid business model for both application developers and service providers, as well as a seamless, easy-to-use experience for end users.

The T-Mobile storm

Over the weekend there’s been a storm surrounding the T-Mobile Sidekick, which is produced by Microsoft’s Danger subsidiary. It turns out the device stores the primary copy of data like calendar and address book information in the cloud rather than on each device; perhaps a fair proposition if you knew you could trust Microsoft’s servers. Unfortunately, said servers went down last week, and Microsoft didn’t have a working backup. Sidekick users suddenly found themselves without their personal information.

Is cloud computing safe?

Understandably, this has sparked a wider conversation about computing in the cloud. AppleInsider summed it up:

More immediate types of cloud services take away users’ control in managing their own data.

While Ina Fried over at CNet noted:

The Danger outage comes just a month before Microsoft is expected to launch its operating system in the cloud–Windows Azure. That announcement is expected at November’s Professional Developer Conference. One of the characteristics of Azure is that programs written for it can be run only via Microsoft’s data centers and not on a company’s own servers.

The issues surrounding cloud computing have been discussed for a while, and aren’t limited to these sorts of accidents; here’s a post I wrote in 2007 about the rights we ought to have over our cloud data. Partially because of the risks involved, and the risk of leaky data, some kinds of organizations and enterprises simply can’t use cloud computing services. (In the UK, for example, check out the requirements imposed by the Data Protection Act.) At the same time, the Sidekick debacle shows there are clear risks to end-user consumers too.

Despite this, the benefits of cloud computing are obvious, particularly for the organizations that can’t use them: device-independent applications and data we can access and use from anywhere.

Can we have the best of both worlds?

The personal computing model is relatively secure: you install applications on your computer, and they sit on your local hard drive, along with your data. Assuming there hasn’t been a security breach, or you haven’t explicitly provided access to your data over a network or through a direct action like emailing it, it’s safe.

On the other hand, because your applications and data are locked away on your hard drive, you generally have to have direct access to it in order to use them. There are remote desktop solutions like VNC, but these are clunky and fairly useless over a low bandwidth connection.

Web applications that store their data in the cloud overcome this obstacle, but lose the security of sitting on your own computer.

What if there was a halfway house between these two situations?

The personal web server that works

Theoretically, anyone can run their own web server, right now, that allows them to install web applications in a more secure, controlled environment and access them from anywhere. But there are some very good reasons why they don’t:

• You need system administrator skills, usually on top of Linux skills, to do it.
• Web applications – even relatively easy-to-install ones like WordPress or Elgg – are fiddly. There are configuration files, directory permissions and (potentially) source repositories to contend with.
• The web applications you can install on your own server are often not as good as the ones you can get in the cloud.
• When something breaks, it’s your own responsibility to fix it.
• Servers are expensive.

What if we could fix all of these things at once? Enterprises, organizations and individuals could have their own, more secure environment that would allow them to use the cloud applications they needed with fewer security risks, while enjoying the ease-of-use and immediacy that the cloud provides.

One of the reasons everyone’s leaping to copy the iPhone’s app store business model is that it just works. Sure, you’re forced to delegate root control of the phone to iTunes, and the operating system places some seemingly arbitrary restrictions on what applications can and can’t do. But the handset works, and installing software is easier than on any other platform. The truth is, most ordinary users don’t care about those restrictions. Hell, I’m a computer scientist software developer entrepreneur power user, and I’m just happy the thing works. (Context: my previous phone ran Windows Mobile, which doesn’t.)

Imagine if you could get your own server environment that was as easy to use as the iPhone. It would look something like this:

Front end & business model

• You sign up for the service, possibly for a small monthly fee, possibly for free (depending on the service provider). Alternatively, if you’re more technical / an enterprise / an organization, you install it on your own infrastructure. The platform is available for free and could be open source.
• From a secure web-based admin panel, you can add and remove users (although the platform optionally also supports Active Directory and similar standards, as well as OpenID), and install / uninstall applications from a centralized app store with the usual features: ratings, search, similar apps, etc. Installation is one-click, and upgrades are similarly seamless. (That WordPress “what, I have to upgrade again?” problem: solved.)
• Much like the iTunes app store, applications may be free, or may cost a small amount. Applications may impose licensing restrictions based on number of users: for example, the app costs $4.99 for up to 5 users, $19.99 for up to 25, etc.
• As with the iTunes app store, the application store provider takes a cut – and so does the service provider. This creates a strong incentive for multiple vendors to provide hosted services for little cost. It also effectively creates a discount for enterprise, organizational and technical users, who can bypass a service provider. The payment to the web application developer also, for the first time, creates a solid commercial marketplace for high quality web application products, while the free option allows open source vendors to distribute as normal.

Technology

• Behind the scenes, the server runs existing open source technology: Apache, Tomcat, PHP, Perl, Python, Ruby on Rails, MySQL, Postgres, etc. However, there are restrictions on how applications must be structured, behave and share their data. This allows the one-click install and upgrades to function correctly. Importantly, though, users of the system need never worry about the underlying framework.
• The platform has a central data store that all applications may access via an API. This data store is fully exportable, allowing (for example) a datastore stored with a service provider to be moved to an internal setup as an organization expands. As with the iTunes app store, applications are linked to a store account rather than a physical machine, so the application licenses are portable too.

Of course, this wouldn’t replace standard web servers. What it does provide, however, is a simple cloud operating system that simultaneously works in a more secure, dependable way than existing services, would be more acceptable to many organizational users, and provides a genuine business model for web application developers.

The web is now an end user application platform, but still behaves like a lightweight document store. To obtain the level of software customization we all enjoy on our home PCs, a much higher level of technical competence is required. I strongly believe that this situation must change for the web to be a viable commercial application framework.

Related entries

• Social networking: beyond the silo (1)
• Microsoft may rule the open web (3)
• The war for the Web (0)

Twitter went down today at the hand of a denial of service attack (alongside Facebook and Livejournal; the latter has also reported an attack). In the old days, you’d shrug it off and go and look at something else. Today, Twitter is such an integral part of the landscape, and some people’s businesses, that it made BBC News and was commented on all over the Internet.

The headlines are highly strung to say the least:

• Twitter Attack Brings a Day Without Social Media (Gawker)
• A Day (Or Hour) Without Twitter Sucks (Twitterrati)
• Twitter Goes Down, Down, Down (Technologizer, who delightfully illustrated their post with an image of an anvil falling on a bird)

The stress of it all made TechCrunch come over all Mr Humphries:

• Oooh Dramatic! Twitter Gets DDOSed

Meanwhile, away from the hilarity, Dave Winer’s developing rssCloud and people are beginning to talk about Laconi.ca. The only model that makes sense is a distributed one: it’s a fundamentally harder problem to bring down a decentralized network, because there isn’t a single point of failure. So far, for example, DNS has remained pretty robust. As regular readers will know, I strongly believe there are very solid business and development reasons for going decentralized, too.

The web is becoming social, and those conversations are becoming more and more important. A malicious user or group shouldn’t be able to take down our conversation platform – or have the ability to dictate its direction. It’s time to think about a better way to build the social web.

Related entries

• Building the user-centered web (8)
• Social networking: beyond the silo (1)
• Music! (2)

HTML 5 – as-yet unreleased, but shaping up well – contains a specification for finding the current location of the user. The API, if your browser supports it and you grant the web application access, returns your latitude, longitude, elevation, speed and some other details. (If your web-capable device doesn’t have GPS, these details will be estimated using your IP address and other factors.)

A couple of weeks ago, I created a page to test this feature. If your browser is geo capable, this will reveal exactly what data about your location is being sent to web applications that ask for it.

If you’re a developer, here’s how I created the page.

Developing for geolocation in Javascript

The following all occurs in a Javascript code block towards the bottom of the page. You could also include this in the jQuery $() function, if you use that framework – the point is that it needs to execute once the page layout has been written to the browser.

Testing for geolocation capability

To test whether the browser has geolocation capabilities, we need to test for the existence of navigator.geolocation:

if (navigator.geolocation) {
	// Execute geolocation code
} else {
	// Execute code for geolocation not available
}

Getting location data

The actual function to call in order to get the location data is: navigator.geolocation.getCurrentPosition(exportPosition, errorPosition);

Both exportPosition and errorPosition are functions that we must define. The former is called if the location is retrievable; the latter if it isn’t. (You can also omit the second parameter and neglect to have an error handler function.)

The exportPosition function takes one parameter: position. This in turn has a property, coords, which contains all of the major location data:

• position.coords.latitude: the latitude of the user
• position.coords.longitude: the longitude of the user
• position.coords.accuracy: a number representing the accuracy of the latitude and longitude (in metres)
• position.coords.altitude: the altitude of the user in metres above sea level
• position.coords.altitudeAccuracy: the accuracy of the above altitude reading, again in metres
• position.coords.heading: the heading of the user, in degrees clockwise from north
• position.coords.speed: the speed of the user in metres per second

This can then be fed into the functionality of your choice.

For an example of how this can be used, have a look at the source of my test page. Browsers that are known to support geolocation include:

• Firefox 3.5
• Safari in iPhone 3.0

Because geolocation is included in the HTML 5 specification, most browsers will support this soon.

Related entries

• Charging for software in the age of web apps (2)
• Microsoft Web Applications 2010 bring the cloud to the enterprise (1)
• Chrome OS and the web as applications (0)

In advance of the announcement later today, I Started Something have uncovered videos about the new Microsoft Office suite.

Microsoft Office turns to the web

As anticipated, Office 2010 includes web-based versions of applications contained in the suite. These don’t have the complete feature set, but are designed so that company employees can create and make changes to documents (including Word documents, Excel spreadsheets and Powerpoint presentations) on the road.

Web applications: now running in the enterprise

Centralized cloud applications have a difficult time gaining traction in most enterprise environments, and Microsoft have wisely taken note of this: it appears that the web-based versions are installed as part of Sharepoint. By doing this, they’ve allowed organizations to keep tight control of their data, as well as legitimizing web-based applications in the enterprise and revitalizing Sharepoint as an organizational product. In other words this is big news, with sweeping implications across the entire software industry.

Open standards must work for everyone

This is another reason why all open web standards must be browser agnostic. I always argue hard for a transparent browser: one that contains support for web standards, but doesn’t carry any extra baggage for any specific purpose. As web applications move into the enterprise, it’s important that a standard that works on a souped-up Firefox or Chrome browser also works great in Internet Explorer. By integrating web applications into Sharepoint, Microsoft are actually leading the industry, and have made themselves relevant on the web again. In doing so, they’ve opened up an important market, and that can’t be ignored.

Here’s a video introduction (although it keeps going down for me): See What’s New in Microsoft Web Applications 2010.

Related entries

• Microsoft may rule the open web (3)
• The war for the Web (0)
• Charging for software in the age of web apps (2)

It’s not a Google strategy, but they’ve chosen to embrace and extend it: the web is turning into an application platform.

Google announced Chrome OS today – an operating system for netbooks, designed to boot up in seconds directly to a browser. Applications run using HTML 5 standards, which include support for offline applications and advanced interface capabilities.

More than that, it’s an attack – not just on Microsoft, but on the old model for operating systems and home computing. The web allows greater ease of use (no application installs!), lower resource requirements (perfect for those netbook CPUs) and instant connectivity. Social functionality becomes intrinsic to all software on the platform, rather than a product in itself. See Building the User-Centered Web for a detailed analysis of how software will change, and why.

Of course, if this revolution happens through Google Apps (or applications hosted on the Google App Engine), running Google advertising and saving to a central Google Account, well, they’ll just have to live with it. I’ve argued before that Google Wave is a Sharepoint killer, but this move makes that positioning explicit; Google is set to directly take on Microsoft. By making the operating system open source, they’ve invited everyone to join in.

It’ll be an interesting battle: while Windows 7 won’t ship with a browser in Europe, Chrome OS is all browser. More broadly, web applications could help with much-needed cost cutting in places like schools and public institutions, so there’s a lot at stake here.

As regular readers will know, I’m very interested in this change, and I plan on getting my hands dirty helping to build a decentralized user-centered web that, like the web at large, is owned by nobody. There’s still more to be done. Watch this space.

Related entries

• Building the user-centered web (8)
• Charging for software in the age of web apps (2)
• Social networking: beyond the silo (1)

The following post contains my notes for a talk I gave at the Hauser Center for Nonprofit Organizations at Harvard University on June 25, 2009.

What is a social network?

I would like to reclaim some language:

Social is an adjective that means relating to human society and its members.

A network is an interconnected system of things or people.

Therefore, I’d suggest that we can define a social network as just being an interconnected system of people.

The audience of this talk is a social network; so are your friends, colleagues, interest groups and so on. Social networking tools facilitate social networks. The universe of social tools certainly includes web applications with social functionality, but it also includes structured face to face interactions, telephone, post, SMS, email. In other words, the web is just one possible tool for this purpose – albeit a very effective one.

If you build it, they will come

You can’t install a social networking tool and instantly expect usage: Field of Dreams is not a good model for community development. The web is littered with ghost sites created using Ning, Elgg and more that have been established in the hope that a user-base will magically appear; however, if your main selling point is the social network itself, nobody’s going to join until that network of people exists and is actively using it. It’s a chicken-and-egg problem.

Therefore, you either need to have an existing network of people to facilitate interactions between (for example, when Facebook launched at Harvard) or compelling functionality that is useful without a network of existing users (for example, Delicious).

If we’re creating a tool that’s useful for the first user who signs up, without a pre-existing social network, then what we’re really talking is a software application that uses the web as an interface, and happens to have social functionality as one of its features.

The web as applications

When the web was conceived, it consisted of documents and pages linked with hypertext: linked words and phrases that, when clicked, would load another, relevant document. Each page had its own Uniform Resource Locator, which allowed you to return to that specific page at any time. Each page could be a destination in itself, and although the sites (collections of pages) could be linked together through hypertext, each one had no need to know about your activities elsewhere on the web. Why would they? Documents don’t have memory; their role is simply to impart information.

Step forward to today, and the web is not entirely made of pages: applications now represent a large amount of the web. (Princeton WordNet defines an application as “a program that gives a computer instructions that provide the user with tools to accomplish a task”; Google Docs, Remember The Milk, Flickr, Delicious etc are all applications by this definition.)

The benefits are tangible: you can access an application’s functionality from any web-compatible device, anywhere in the world. You’re no longer bound to the software you happen to have installed on a particular machine, and you no longer need to worry about whether you’ve remembered to save a particular file onto a particular drive. Because of historic resource limitations, web applications tend to be easier to use, and entirely bypass the need for IT departments, which have unfortunately earned a reputation for being obstacles to productivity in many organizations.

This change of web usage has been reflected in the ongoing development of HTML, the markup language that all web interfaces are written in. The first four versions were largely orientated towards documents; however, HTML 5, currently in development, is the first version that explicitly contains functionality to support web applications. That includes offline storage and usage, sessions, and more advanced interface features. However, aspects of the document-orientated model remain.

Silos of information

Each application is its own atomic destination with its own URL, and is by default only aware of data created within it. That means we need to register for each application we want to use, fragmenting our accounts over potentially hundreds of products and company data centers, and that the documents, files and data we create within them can’t easily be shared with other applications. On my desktop, I can write a document in Word and open it in OpenOffice, or take a Paint doodle and load it in Photoshop, but there’s no easy, generic way to take my bookmarks from Delicious into another bookmarking tool, or to take my Google Docs and open them in Acrobat.com.

Currently, each web application is like a silo: they exist on their own, and if they interoperate at all, it’s through specific links between applications that have to be individually developed. Certainly, data created in an application stays in that application; sometimes you can check your GMail address book for contacts in order to find existing friends on a service you’ve just signed up to, for example, but it’s rare that you can actually export data fully into another product. As many of these services are free, a significant portion of their business models revolve around being able to control user-contributed data, keep users coming back, and sell user-generated activity data for marketing purposes. (One has to question whether the market for personal details will continue to be profitable, or whether, like the web advertising market before it, it will saturate and crash.)

In a social networking tool, the site model means that your contacts, the information you share and any detailed access permissions all relate solely to the application they were created in. However, collaborative spaces in social web applications are like documents: they’re one of the currencies of the social web. Just as I need to be able to use my wordprocessor of choice to edit a document, I need to be able to use my social tool of choice to collaborate with others.

Turning the model upside down

Right now, we have to register with each application we want to use. What if we required each application we used to register with us, in digital identities under our own control?

What if, using these identities, anyone could connect to anyone else, and anyone could store their data anywhere as long as the storage provider followed the same broad standards?

The web itself would become a social networking tool.

This is far more flexible, and future-proof:

• Your ability to collaborate is not subject to a single company’s success: social functionality and application infrastructure are inherent in the web itself
• The possibilities for collaboration are not subject to technology beyond common open standards, which can evolve
• A wider range of application possibilities is ensured, because web applications gain the ability to interoperate in a general way
• Privacy and user control are established by allowing a person to determine which application has access to which data

By establishing a general standard for social application interactions, the services and technologies used to make connections become less relevant; the Internet is people, one big social network, and users no longer have to worry about how they connect. We can all get on with communicating and collaborating rather than worrying about where we connect.

User-centered identities

Under this model, providing the software that hosts your digital identity becomes big business. This hasn’t gone unnoticed by the main service providers, and they’re already fiercely competing to be your identity on the web:

• Facebook wants your central identity to be a Facebook account (and arguably have made the user-centric model for the web part of their strategy for a very long time)
• Google wants it to be a Google account
• Twitter wants it to be a Twitter account
• Microsoft wants it to be a Live ID
• OpenID want it to be any OpenID-capable URL

Because I use all of these services, the result is a very complicated identity space. These are a subset of my profiles:

• facebook.com/ben.werdmuller
• google.com/profiles/benwerd
• twitter.com/benwerd
• profiles.yahoo.com/u/4IZCH2K6PEV775MICYR3DXMAMQ
• benwerd@gmail.com [my Microsoft ID, ironically]
• benwerd.com [my OpenID]

For identities to be usable as a generic standard, you should be able to use any of these – or all of them. Nobody has just one facet (or persona) comprising their identity; everyone has a collection, representing the different parts of their lives. Ben Werdmuller the web strategist for hire doesn’t need to be connected to Ben Werdmuller the Doctor Who fan, who in turn doesn’t need to be connected to the Oxford resident. They can be connected if I choose to make them, but separating parts of your life is part of a user’s control over their identity.

However, that needs to be context-specific, not application-specific. Currently, for example, my Facebook account tends to be personal, while my Twitter tends to be professional. That doesn’t make sense: in order to write personally on Twitter, I either have to accept the collision of those two parts of my life, or I need to create an entirely separate, fragmented Twitter account. Wouldn’t it be better to be able to control who sees which interactions, and choose tools based on the functionality they add to a conversation? Otherwise you have the situation I present above: one identity per communication context per application. That will quickly become unmanageable, and the web will be littered with dead profiles.

Conversely, I believe the future of the web is in atomic digital identities based on permissive, open standards, linked together as an application framework.

How do we make this work?

Problem to solve: user control

First and foremost, the framework for decentralization must be established – in other words, the actual social mesh standards that will make it possible.

Technical mechanisms need to be established for controlling access to a resource or collaborative space, which should be easy to use without removing any of the flexibility of the platform, and should allow for the maintenance of multiple personas.

Another part of access control is allowing a resource to expire gracefully. It’s important to know when to lose data: sometimes documents, resources, spaces, personas or entire identities may be transient and only required for a certain length of time. There’s no need for everything on the web to exist indefinitely; currently, rigorous indexes like Google ensure that much of it does.

Finally, the tools and standards we create must be permissive of goals, content and structure that we might not have thought of. There certainly doesn’t need to be an overarching structure or taxonomy between individual identity spaces, and constraining the technology to a rigid set of activities and data types would limit the scope of the platform.

Problem to solve: ownership

Existing web applications tend to have a single-ownership model for resources. However, Silona Bonewald rightly pointed out to me that this isn’t always the case, and in a free-flowing social mesh, multiple ownership needs to be represented. For example, all collaborators on a resource should have ownership access, unless they explicitly choose to rescind that right.

In a company environment, a user’s employer may have shared ownership (or full ownership, with author access available to the employee). The same may be true with students in a university environment. On sites like Facebook, the service owner may in reality have some ownership rights over the content.

How can we maintain this granularity, but also retain user control?

Problem to solve: privacy & transparency

There is a very public attitude of "when you put something online, it’s published" in some parts of the software development community, which is a useful concept that gives developers carte blanche to share data freely. In a fully user-controlled environment, this public-or-completely-private binary situation can no longer be the case; a resource may have been published to a few select people. Ignoring this trait disallows the platform’s use in important environments like enterprises or public bodies.

When you sign up to a service, you agree to that service’s terms and conditions and privacy policy. However, your data may be farmed out to a collection of other, secondary services via APIs, without your knowledge or consent.

An important aspect of user control is knowing how your data is used and where it is transmitted by the applications you use, so I propose a simple, human-identifiable and machine-readable mark that:

1. Applies permissions to how my data can be used by applications (like Creative Commons does for shared content)
2. Tells you in a visual way what happens to your data when you visit a site
3. Incorporates multi-ownership

It may be that these issues are addressed within the terms and conditions of a service. However, it’s very unlikely that a user will actually read the full contract. Therefore, a simple graphic icon with a link to a plain-English description, with an underlying microformat for machine-readable use, would be a welcome addition to the user experience. As the web becomes more mesh-like and data moves around more freely, conveying what happens to data owned by less-technical end users will become more and more important.

Problem to solve: platform

Finally, while it’s great having a conversation about this, these ideas aren’t useful to anyone unless someone goes ahead and builds it.

There are some existing projects and thinkers who are on these tracks:

• The Diso Project is turning the WordPress open source blogging tool into a decentralized digital identity through an array of open standards, and the project’s Chris Messina has a lot of wise things to say about its development.
• Laconi.ca is a decentralized microblogging platform, whose Open Microblogging standard may be adaptable into a more widely-scoped technology.
• The Open Stack is a set of developing technologies that address some of the issues.
• Marc Canter’s Open Mesh treatise goes into detail on many of the issues.

All of these are important contributions that strongly address some of the issues; however, we’re still a long way away from the vision of an open, social web.

Conclusion

I believe strongly, for the reasons stated above, that a decentralized, user-centered model for the web is the best way to advance it as an application platform.

Needless to say, I have my own ideas about how to actually build the platform, based on my Making the most of the web principles. However, it has to be a collaborative process: there’s no sense in building an open collaborative standard by yourself. My main concern is that the platform is created and works in an open, lightweight, flexible, easy-to-develop-for way while remaining secure and yielding control to the main user. The result will be an entirely new kind of platform, and presents a unique opportunity for anyone who wants to jump on board.

Images:

• WOW! My 1000 Friends by Cavin was released under a CC Attribution Generic 2.0 License
• Lonely Tree by Jule Berlin was released under a CC Attribution Generic 2.0 License
• Logo 2.0 part II by Stabilo Boss was released under a CC Attribution-Noncommercial-Share Alike 2.0 Generic License
• Upside Down by Johnny Jet was released under a CC Attribution Generic 2.0 License
• Pro Control 24 by Aud1073cH was released under a CC Attribution-Share Alike Generic 2.0 License

Related entries

• Social networking: beyond the silo (1)
• The Internet is People (3)
• User control on the open web (8)

I’ve spent the last week in a series of very interesting mind-sharing meetings. First, the American Association of Colleges & Universities flew me into Washington DC to discuss the future of assessment in higher education with education professionals as well as new techsphere friends like Silona Bonewald and Amber Case. Second, Michael Byrne from Harvard University’s Hauser Center for Nonprofit Organizations invited me to speak there about the open, social web, the notes for which I’ll write up and post here soon. A great big thank you to both organizations for inviting me; I’ve felt privileged to have such insightful and interesting conversations over the last few days.

It seems like the shift in innovation in social tools has gone from developing new and interesting technologies to developing interesting models that happen to use technologies. This is a big step, and in some ways represents the space coming of age. There’s still plenty of technological development and innovation to do, but the platform and concepts are at a point where they can be adapted into all manner of social collaborative spaces, business tools, social experiments, games, art projects, and combinations of those things. It’s becoming a very exciting field to work in.

That said, some adaptation needs to happen, and it’s important to realize that these ideas only work in an effective way when they’re made relevant to the outside world. The social web is extremely political: it imposes an opinion about how the world should be open and social, democratic and centered on individual preferences, but ironically doesn’t allow for differences in that point of view. That makes it very hard for late adopters, enterprises, governments and public organizations to feel the benefit.

Over on Persona Prime, Silona makes this point about technology-inspired government transparency:

Where is the change management?  We are doing some big stuff here and we are poised to make serious mistakes and I see no prelim work being done to prevent this.  Where are the best practices in open govt documents?  All I see are “I want” lists.  I have not seen us doing anything serious to ally [the fears of people who might be wary of transparency].

It would be cool if every Fortune 500 company wanted to be on Twitter, but the reality is that they don’t, and often for very legitimate reasons. If what we’re doing is establishing a new, global, decentralized way to create, share, disseminate and discover information, then we have to take into account the differences in all the decentralized nodes. Embracing different corporate cultures, and different opinions on how communication should be, is part of that. Compromising and addressing the fears of companies and late adopters will build a larger userbase for all our tools, and make the platform much more useful in the long run.

Related entries

• Activity Streams and OAuth: a social web architecture (2)
• How social networks can replace email (11)
• Using game dynamics to drive participation (1)

When Google Wave was previewed last week, most people focused on its Gmail-like user interface and the slick way it handled collaborative spaces in real time. What was more exciting from a web technology point of view, although much less sexy, was the underlying protocol: Google Wave is built on top of an extended version of an open source real-time data standard called XMPP (aka Jabber when it’s used as an instant messaging context). This means that anyone can build servers and end-user applications that are compatible with Wave.

Yesterday, the BBC announced that they were using an open source framework called Hemlock in order to provide real-time web games on their Childrens’ site. Hemlock’s interfaces are built in Flash, but the real-time aspect is powered by XMPP. Developer Ron DeVera explained why over on the Hemlock blog:

XMPP is commonly used to enable chatrooms, like Gtalk or Facebook Chat. However, instead of just sending text, Hemlock lets you send all kinds of data at the speed of instant messaging. Even better, XMPP is designed to scale up and send data to many people at once, so you can push that data to multiple people in real-time.

In other words, it’s perfect for real-time applications on the web.

Traditionally, live web applications have used a development methodology called AJAX, which stands for Asynchronous Javascript And XML. In order to provide the illusion of a real-time application, the web browser needs to continue to ask the server if there’s any new data to ask, via a new AJAX request – which is a waste of bandwidth and resources both on the user and server side. XMPP, on the other hand, pings the client software whenever there’s new data.

Servers already need software to host and process dynamic web pages, as well as a database server to handle most types of user data. To use XMPP, you need to add an XMPP server to the mix – something that in the past was more fiddly than most people were willing to play with. However, times have changed, and ejabberd is free, open source and easy to install on Windows, Mac OS X and Linux servers. (I’m still at the early stages of playing with it; I’ll let you know how I get on.)

The XMPP community is still very small compared to the wider web, but developments like Wave and Hemlock should accelerate growth and interest in it. There are some interesting projects springing up, and lots of libraries available for XMPP development; XMPP programming is a skill developers should get in on now, before it really heats up.

Related entries

• Social networking: beyond the silo (1)
• Google Wave is exciting and transformative (3)
• Chrome OS and the web as applications (0)

1. The rise of social networking
2. Monetization vs. collaboration
3. The open web
4. Fluid collaboration

The rise of social networking

Social forces have been the driving force behind application innovation on the web. Whereas previously we might have looked to advances in computer science for new directions, now some of the most dramatically impactful applications are lightweight, simple, and technologically unimpressive. The best new web applications have centered around collaboration, sharing and discovery with other people.

Correspondingly, enterprises have been relatively quick to pick up on this trend, and software vendors have been quick to grab the market. In an Intranet Journal article earlier this year, Kara Pernice, managing director at the Nielsen Normal Group, had this to say about the rise of social technology on the intranet:

"In the 9 years [the Intranet Design Annual, which highlights the ten best-designed intranets of the year] has been coming out (since 2001), I’ve never seen a change quite as great as this one."

On the Internet at large, social network use is growing at ten times the rate of other activities and now accounts for 10% of all online time, according to Nielsen Online in this March 2009 report (PDF), and is now more popular than email. Jerimiah Owyang has a list of more relevant statistics over on this digest blog post. Executive summary: social networks are big, transformative in terms of how we communicate and share information, and growing at an enormous rate.

Monetization vs. collaboration

Wikipedia defines a “walled garden”, in software terms, as being:

[..] A closed set or exclusive set of information services provided for users (a method of creating a monopoly or securing an information system).

In other words, a walled garden is a system where the data can not easily be imported or exported. These are often also called data silos, after the solid buildings used for secure storage.

Facebook, the #1 social networking site in most western countries, has over 200 million users, including over 30 million who update their profiles at least once a day. The network is free to use, yet their revenue for 2008 has been estimated at around $265 million, despite a decidedly “in progress” revenue strategy.

This has traditionally required a walled garden strategy: the content that users put into Facebook has not been easily removed for export or viewing in other interfaces, in order to preserve revenue from advertising (and – although this is a hunch – revenue from statistical analysis of users’ data). It’s only been in the light of some extremely negative publicity (for example this February 2008 New York Times article) that they have begun to relax this policy and embrace the open direction that much of the rest of the web is heading in.

Speaking personally, I get more enquiries from people wanting to build something “Facebook-like” than anything else, presumably because of its phenomenal popularity. However, this kind of walled garden approach is not conducive to true collaboration; generally people who ask for this are lacking a full understanding of the processes involved in social networking.

According to Nielsen, there are almost 1.6 billion people online. While Facebook’s 200 million sounds like a lot, it’s actually a drop in the digital ocean – so what happens if I want to share a Facebook conversation with someone who hasn’t signed up? The only way is currently to email them a link and force them to register for the service. Facebook would love me to do this, of course, because they get more eyeballs to view their ads and more people to fill in profiles. But what’s the point of even being on the web if you can’t make use of the decentralized communication features that form its backbone?

If I want to collaborate effectively online centering around a resource (which could be a file, a discussion or a pointer to something external), I need to be able to:

• Share that resource with the people who need to see it
• Grant access for them to edit it if required
• Notify them that it’s been shared with them
• Restrict access from everyone else

Furthermore, I need to do this with the lowest possible barrier to entry. My aim is to collaborate, not to get people to use a particular piece of software. By restricting this process, the Facebook model hinders collaboration.

The open web

The web was designed to be an open system, and adheres to principles (notably “every object addressable”, ensuring that every resource on the web has a unique reference address) set out by Doug Engelbart for open hypertext systems generally. Because web pages are interoperable, and all use the same basic standards, any page on the web is allowed to link to any other page on the web, no matter who wrote it or where it is hosted. In many ways that’s the key to why the platform is successful: despite being fragmented across millions of computers throughout the world, it navigates like a cohesive whole and can be viewed using a single piece of browsing software. (The downside to this is that the whole platform lives or dies depending on the capabilities of the browser you use: the sad fact is that Internet Explorer users, who often don’t have a choice because of policy decisions in their working environment, are at a disadvantage.)

While the original web was content-based, the social web is collaborative and centered around live data. However, because web applications are each developed separately using different sets of back-end infrastructure, their data does not adhere to the principle of interoperability – their user interfaces all use the same basic standards and can be viewed in a browser, but the underlying applications and data models tend to not work with each other. When social networks emerged, for example, there was no way to get Livejournal and Friendster, two of the pioneers in the space, to speak the same language; you still can’t add someone as a friend on one social network from another. More recently, this has become apparent in the walled garden approaches of Facebook and others.

Not only does this situation create a bottleneck for application design, and run contrary to the underlying principles that made the web a success, but it’s also a bottleneck to better collaboration. As Tim Berners-Lee, the web’s inventor, put it recently in this essential TED talk, data needs to be linked and interoperable in the same way pages are now. Beyond that, because walled garden services are making money out of the private information we’re loading onto them, there’s a human issue regarding the overall control of that data. Marc Canter, Joseph Smarr and others codified this into a Bill of Rights for users of the social web back in 2007. Though the issue has moved on since then, the underlying principles set out there are essential for open, collaborative, social tools on the web.

While the World Wide Web Consortium works on academically-developed standards for linked data in the form of the semantic web, developers have been getting their game on trying to solve the problems of interoperability between their applications and user control over their data. Application Programming Interfaces (APIs) – published sets of instructions for programmatically querying and extending web applications – have become popular, but in a very walled garden kind of way. Arguably the most successful has been Twitter’s API, which has led to a number of high profile third-party applications like TweetDeck and Tweetie that collectively eclipse Twitter’s own website interface in volume of usage. But these APIs are their own form of walled garden: an application written for Twitter will only work with Twitter, for example. The APIs are not generalized between applications, and as such are not truly open; in many ways they’re a way for services to get more functionality and reach for free.

One of the first attempts to publicize the benefits of truly open data was Marc Canter’s Data Sharing Summit, which I wrote about at the time for ZDNet. Chris Saad’s DataPortability.org attempted (largely successfully) to brand it, and latterly the Open Web Foundation has attracted some of the web’s leading lights in order to create a single organization to handle the creation of a set of open web application standards. Many of these comprise the Open Stack, which I’ve written about before; more generally, Chris Messina has written a very thoughtful overview on the topic.

Fluid collaboration

It used to be that to use the web, you would need to sit down at your computer and log on. Those days are over; the web is becoming more and more ubiquitous, thanks to devices like the iPhone. It’s also being integrated into software that wasn’t previously connected – it’s as easy, for example, to paste the URL of an image into the ‘Insert Image’ dialog box in most word processors as it is to pick an image from your own hard disk. The open, generalized API standards being created by groups like the Open Web Foundation bring us closer to enjoying that level of integration with collaborative social technologies.

The Internet is people, not technology: tools on the web (or anywhere else) facilitate social networks, but are not the network themselves. Currently they consist of destination sites, like Facebook, LinkedIn or Twitter – places that you explicitly have to visit in order to collaborate or share. This is the currently-fashionable model, but it’s a necessarily limited view of how collaboration can take place: all of these sites thrive on the walled garden model and are designed around keeping participation within their walls.

Not everything on the Internet works this way. Email, and increasingly Instant Messaging, are two technologies that generally do not: messages on email, Jabber and to a much lesser extent Skype are peer-to-peer and do not go through a central service:

1. You select the people you wish to collaborate (in this case, email or chat) with. Nobody but the listed recipients will be able to see the content you share with them, and it doesn’t matter if they’re using the same service as you; you don’t have to invite them to join email in the same way you have to invite people to join Facebook.
2. You write your content.
3. You send it.
4. They (hopefully) send content back.
5. The collaborative exchange lasts only as long as it’s useful, and then disappears (but is archived for reference).

Recently, Google announced Wave, a decentralized pairing of protocol and open source web application that took email and IM as its inspirations to redefine how collaborative social technologies could work. Questions have been raised about how a decentralized tool like this can work with corporate data policies present in most large enterprises and public sector organizations, but in some ways they miss the point: Google Wave is best thought of as a proof of concept for how decentralized, transient communities can work in a standard way on the web. In short, websites are a kind of walled garden in themselves: what we will return to is the idea of the web as an open patchwork of people, data and information that links together to form a whole, much stronger than the sum of its parts.

Predicting the future of social networking on the web is hard. However, I believe that as general open social technologies develop and become more commonplace, the “social networking site” will shrink in importance – instead, social network facilitators will become more and more ingrained in all the software you use. This will dramatically increase the types of content and communication that can be used, and present opportunities for much wider, more fluid and – most importantly – more productive collaboration as a whole.

Related entries

• Building the user-centered web (8)
• User control on the open web (8)
• Danger in the cloud: a proposal (8)

The following is an expanded version of the short talk I gave in Oxford yesterday. My original slides follow the transcript.

There’s a new kind of web development afoot, which marries old-school object-orientated programming techniques with the distributed power of the web.

Application Programming Interfaces (APIs) are published sets of instructions for programmatically querying or extending a service. For example, the Google Maps API allows anyone with some development skills to build applications based around Google Maps; the Twitter API allows anyone to build an application that lets you interact with Twitter (like Tweetie or Tweetdeck). The act of allowing anyone to do this is generally thought of as being “open” – as in, open to anybody. While that’s true, in another sense they’re very closed.

The trouble is, if you write a microblogging application using the Twitter API, you’re locked into Twitter. If you want to write an application for another microblogging service, you have to use their API and start from scratch again. The formats produced by each service’s API are proprietary, as are the methods to query them. They’re incompatible with each other by design, because those services don’t really want you moving users’ data around between them. They want users to sign up with them and then stay there – a great proposition for the service, but a lousy one for the users, who have no real way of importing or exporting the content they’ve created.

Furthermore, there are some situations where this one-API-per-service model breaks down completely. Let’s say you’ve joined a social network and you want to check to see which of your friends is already there. The service provider could use a Gmail API to check your address book, but some users will use Hotmail for their email, so they’ll need to use the Hotmail API as well. Repeat for Yahoo, and every single email provider under the sun – there’s no way anyone could possibly write this feature without a generic API that works across all services.

Enter the Open Stack, which is a set of generic APIs designed to provide common tasks:

• eXtensible Resource Descriptor (XRD): allows an application to discover resources and APIs relating to a particular page (for example, an application could check a user’s profile page and discover that they have an OpenID). There’s no point in having open APIs if an application can’t find them, and XRD fills this gap.
• OpenID: allows anyone to log into any service that supports it using the same digital identity (potentially linking that user’s accounts on those services together). A Wordpress.com ID is an OpenID, for example; you can log into any OpenID-compatible site with it.
• OAuth: a way to authenticate access to API-based services without forcing the user to present their username and password for that service. Previously usernames and passwords were passed in order to authenticate APIs, which led to serious security issues. Here a user can easily and securely grant or deny an application’s access to their data. OAuth can also be used to apply granular access controls to a piece of data.
• Activity Streams: a way to broadcast your activity on each service (for example ‘Ben has published a blog post’, a la Facebook’s river) so that it can be aggregated at a central point.
• Portable Contacts: a simple way to transmit collections of contacts, like email address books and lists of friends, between services.
• OpenSocial: provides a framework for hosting remote application widgets on a service.

In programming jargon, a stack is a specific data or platform structure; actually, this is more of a pile of useful technologies that can (in part) be used together.

These generic APIs allow for a more distributed kind of web application. Suddenly, instead of writing support for a particular function from scratch, you can pick up an existing code library and slot it in. You can interact with any service that supports them without writing any further code.

However, there are some unresolved issues that need to be discussed. One major headache is the privacy implications. If I sign up to BensFabApp.com, I agree to the terms of service and privacy policy there. However, behind the scenes it might be accessing the APIs for AppOfEvil.com, which has a very different privacy policy and terms and conditions. How does that second set of terms and conditions apply, and what’s the legality of passing a user’s data across two services with two different sets of terms, when the user only knows about one of them?

A second issue is a user’s control over their data. When BensFabApp.com sends data to AppOfEvil.com, it may get cached and duplicated. What happens if I delete the original data on BensFabApp.com? So far in the open stack there is nothing to handle deletion of content. It’s a fair argument that when something is published publicly you lose control over its distribution; however, if access has been limited using OAuth or another method, there’s no way of reaching out an ensuring its removal from every system it’s been sent to. That would be fine if the data had been sent to another person or application with the user’s express permission, because they’ll be aware of the implications. However, if it’s been sent completely behind the scenes, they have no way of knowing that it was sent to begin with.

These are issues with any API-based service, but generalized APIs are likely to be used more frequently. Solutions to all of these problems will be found, but it’s important to note that they’re not there yet – which serves as a warning to applications developers and an opportunity for anyone who wants to step up and provide them. As this kind of open API becomes commonplace, new kinds of web applications will begin to emerge as developers spend less time reinventing the wheel and more time innovating. It’s just one of the things that makes the web as a platform so exciting to work with.

Related entries

• Activity Streams and OAuth: a social web architecture (2)
• Implementing open standards is too hard (0)
• The mechanics of "open" (0)

It’s a commonly accepted fact that computing is a male-dominated industry, but I was shocked by the scale of the inequality. Okay, this is kind of unscientific, but take a look at these statistics:

• Female population of the world: 49.8%
• Female population of Facebook: 55%
• Female population of social networks as a whole: 54.7%
• Percentage of people awarded undergraduate computer science degrees by PhD-granting institutions in the US and Canada in 2006-7 who were women: 12%

While social media usage is skewed ever so slightly towards women, a whopping 88% of the people who study to learn the skills to build these tools are men. This is at a time when, in science generally, women receiving undergraduate degrees are increasing as a percentage year on year.

Some of the reasons for this have been covered a lot over the past year. This 2007 interview with Aaron Swartz (who worked on Creative Commons and is now behind the awesome government site Watchdog.net) contains some interesting thoughts on discrimination on the basis of both gender and race:

If you talk to any woman in the tech community, it won’t be long before they start telling you stories about disgusting, sexist things guys have said to them. It freaks them out; and rightly so. As a result, the only women you see in tech are those who are willing to put up with all the abuse.

[...] The denial about this in the tech community is so great that sometimes I despair of it ever getting fixed. [...] It’s an institutional problem, not a personal one.

Last year, Chris Messina called out a BusinessWeek article for disproportionately featuring the male participants at Web2Open, a Web 2.0 technology unconference Tara Hunt had predominantly organized. He followed it up this month with another post about the Future of Web Apps as a white boys’ club:

Turns out, white men also don’t have the monopoly on the best speakers – even in the tech industry – yet their ilk continue to make up a highly disproportionate number of the folks who end up on stage. And that means that good content and good ideas and important perspectives aren’t making it into the mix that should be, and as a result, audiences are getting short-changed.

This isn’t just about technology, and it isn’t just about the commercial web. We’re in an era where everything is going online; Barack Obama would arguably not be President of the United States without his engagement with grassroots social media technologies, and he is certainly continuing to embrace them into his Presidency. Yet if those technologies are effectively controlled by a minority of the population, that population’s biases and predispositions seep into how they’re designed, how they’re built, and ultimately how they work in practice.

Although I’ve picked out gender here, the same is doubtless true regarding race and sexuality discrimination in the tech sector, although the numbers haven’t been as widely published. As computing becomes more and more important in society as a whole, it becomes more and more important to ensure the people who help shape it are selected fairly and represent a cross-section of the people it serves.

Update: Lots of really interesting links in the comments, including Katie Piatt’s recommendation of Ada Lovelace Day, which encourages people to blog about women in tech.

Meitar Moscovitz points me to Will the Semantic Web Have a Gender?, a ReadWriteWeb article from last year about the possibility that the semantic web will reflect a predominantly male attitude to the world.

Image by mouton.rebelle and released under a CC-Attribution-Noncommercial license.

Related entries

• The mechanics of "open" (0)

All software has bugs; the trick is to find them and remove them as quickly as possible while balancing against the other development you have to do. This is where bug trackers come in: they allow developers to project manage these activities while allowing users and other members of their organization to submit issues while not interrupting the development workflow.

Even though every software project needs a decent bug tracker, finding one isn’t easy. Everything seems to have a downside:

• Bugzilla, while very powerful, has a near-psychotic user interface – or at least, I nearly went psychotic trying to use it. I’m the technical lead of an open source project with a computer science degree and twenty-five years of programming under my belt; an ordinary user stands no chance of reporting issues in a meaningful way.
• Lighthouse is really simple and integrates well with email and things like Basecamp. Unfortunately, it’s so simple that certain features – like a drop-down menu that allows users or developers to suggest the priority (or complexity) of an issue – have been deliberately left out of the mix with no hope of inclusion. That means there’s no way of saying that a massive privacy flaw is more important than a minor rendering error, except by integrating with another product.

In the end, Trac seems to be the best solution. Integrations with the Eclipse development environment and Subversion source code repository (through plugins) are a definite advantage, and although the web interface leaves a lot to be desired, it’s nothing like Bugzilla.

The perfect bug tracker needs to encompass these two values:

• Project developers must find it a significant help, not a hindrance.
• Users (and project managers) must be able to easily submit useful bug reports, issues and feature requests, and follow progress on them.

Useful bug reports are ones that aren’t duplicated and effectively describe the problem so that the guesswork required on the part of the programmer is minimized. In some ways, both bullet points would be satisfied by a bug submission wizard that falls back to an integrated form once users have become used to the process. At any rate, it requires some empathy on the part of the user interface designer, and more of an emotional approach to the interface design.

In the most perfect world, this would also be fulfilled:

• In an open source project, external developers must find it easy to contribute code through the same interface.

I’d be interested to hear recommendations for any other products – or to hear any bug tracker horror stories from developers and users alike.

Image Lucanus cervus taken by CCCvrcak and distributed under a CC attribution license.

Most Commented Posts

• Twitter DoS and single points of failure (31)
• Gender differences on the new frontier (16)
• Writing in Oxford? (13)