One indication of how powerful it is comes from the following article:

Places allows your friends to tag you when they check in somewhere, and Facebook makes it very easy to say “yes” to allowing your friends to check in for you. But when it comes to opting out of that feature, you  are only given a “not now” option (aka ask me again later). “No” isn’t one of the easy options.

This warning doesn’t come from a computer security forum, the EFF, or a group of interested hackers. It comes from the ACLU, the groundbreaking organization that aims to protect Americans’ civil rights. In other words, Facebook privacy is now being watched by the same group pushing for the closure of Guantanamo Bay. A powerful place to be – but perhaps an indication that Zuckerberg needs to re-assess his take on privacy?

In some ways, it doesn’t matter. Software projects like Status.net and Cliqset (Youtube link) are establishing a resilient, decentralized network where privacy is in users’ hands by default. These new applications are easy to use, accessible and ready for both private individuals and enterprises to pick up – and as such, represent the real future of social data on the web.

Related entries

• Building the user-centered web (8)
• What Facebook users want (0)
• User control on the open web (9)

Building the User-centered Web

By establishing a general standard for social application interactions, the services and technologies used to make connections become less relevant; the Internet is people, one big social network, and users no longer have to worry about how they connect. We can all get on with communicating and collaborating in contextually appropriate ways. In this talk, I’ll discuss how to build a decentralized, user-centered web using existing and emerging technologies. I hope you’ll join me.

If you’d like to see this at the next SXSW, please visit this page to vote.

Paul Adrian also has submitted a talk, this time about the future of journalism, and how technology can help:

Technology Can Create a Press for the People

I believe it is time for a “news” revolution. A new press should produce comprehensive streams of rigorously non-partisan original reporting on the issues that are most important to our lives. Once informed, we the people should have a space where we can discuss the important issues of our times without having to submit to intolerance, deceptive campaigning and fear-mongering. Through the use of technology and new business models, news innovators can provide more credible information and space for civil discussions. The goal is to empower citizens by providing access to superior reporting and the platform for community organization necessary for the People once again to become powerful participants in democracy.

As well as being an award-winning journalist and technology entrepreneur, Paul is an inspiring speaker who is worth listening to. You can vote for his talk over here.

Related entries

• The death of newspapers, and why it matters (2)
• Last.fm isn’t selling your data to the RIAA, apparently (0)
• Facebook location vs the ACLU (0)

Last week, the State of New York passed a bill that bans short-term rentals: specifically, no homeowner or renter may sublet their home for less than a month. The target is sites like AirBNB, an up and coming website that allows travelers to eschew pricey hotels – and their accompanying hotel room occupancy tax – in favor of private homes.

If the governor chooses to pass the legislation (as opposed to veto it), AirBNB will effectively be outlawed, and with it, a grassroots marketplace economy for short-term accommodation. New York State will have cemented hotels and bed & breakfasts as gatekeepers to the city for travelers who can’t stay with friends or relatives.

To me, this is an interesting reaction: it shows, once again, that established gatekeepers are terrified of the Internet. We’re used to that by now in the context of media content – we already know that newspapers, publishers, record companies and movie distributors aren’t as important as they were – but this is a scarcity-driven marketplace. It used to be that finding a safe, clean room in a strange city was a hard problem, so we turned to hotels as a trusted source. Running a hotel is in itself an expensive, tough business, and as a result there were a limited number in any given city, and the price went up according to demand. Although the hotel business is a ruthless game, it’s always been hotels competing with other hotels.

Now, though, we can visit websites like AirBNB and Couchsurfing, where private citizens can offer their homes to travelers, and the site will let us know who we can trust based on other peoples’ experiences. The marketplace has been blown wide open, and it turns out that a lot of us would rather go for a cheaper, friendlier option. I wouldn’t put money on New York blotting out short sublets for long.

Power to the people

We’re going to be seeing a lot more of this, in all kinds of market sectors. We’re already seeing ridesharing sites become popular, for example, blowing up the market previously owned by taxicabs and making it available to anyone who happens to be driving somewhere. Effectively this formalizes hitchhiking, making it both safer and more efficient.

It all comes down to one simple rule: People want to be free.

The Internet is opinionated: as a medium, it inherently works to empower people and eliminate hierarchies in society. It shouldn’t be a surprise that the most popular Internet companies hail from California; their philosophies are direct descendents of the civil rights activism that took place there in the sixties and seventies. In many cases, it’s even the same people. (Or – and here I put up my hand as the son of Berkeley “radicals” – their children.)

Gatekeepers – companies, structures or processes that act as exclusive barriers or filters – are not long for this world. Where gatekeepers exist, they do so because the alternative was inconvenient at the time when the gatekeeper became established – not because they’re inherently better than an empowered population. Those organizations, companies, and even governments, need to look at themselves very carefully and figure out what needs to be changed, before those things are changed for them.

Most Commented Posts

• Gender differences on the new frontier (16)
• Writing in Oxford? (16)
• End of an era, start of a new one (12)

Real-time web applications are networked applications, with web-based user interfaces, that display Internet information as soon as it’s published. Examples include social news aggregators and monitoring tools that continually update themselves with data from an external source. In this tutorial, you will create Pingstream, a small notification tool that uses PHP and JavaScript to communicate over the Extensible Messaging and Presence Protocol (XMPP), a set of XML technologies designed to support presence and real-time-communications functionality.

You can read the whole tutorial here. IBM have made it a featured article, commenting, “bet you have it up and running before lunch.” I hope you find it useful. (And don’t forget to check out my introduction to Activity Streams, also written for IBM.)

Photo: IBM by antonfortunato, released under a Creative Commons license.

Related entries

• An introduction to Activity Streams (0)
• XMPP: powering the real-time, really live web (7)

Enter Activity Streams, an evolving standard that extends Atom for expressing social objects. Although it is a young standard, Activity Streams is fast becoming the de facto method for syndicating activity between web applications. For example, MySpace, Facebook, and TypePad all now produce Activity Streams XML feeds. But this technology isn’t just for the consumer web environment. As corporate intranets and internal software become more social, solid business reasons support implementing Activity Streams as a feature. This article describes Activity Streams in detail, considers its potential uses in enterprise environments, and provides some examples for interpreting Activity Streams feeds using PHP.

The full article is over here.

Related entries

• Write real-time web applications with XMPP, PHP, and JavaScript (0)
• Direct messaging in a social web architecture (3)
• Activity Streams and OAuth: a social web architecture (3)

My apps, then:

• Gmail (email). I used to be a die-hard Mozilla Thunderbird user, but during my Elgg days I switched over. There are probably better email web apps to be using these days; ideally I’d like one that runs on my own infrastructure rather than in Google’s cloud. But with one tweak (I have a separate pane that keeps all starred messages at the top of the screen, so I know what to reply to imminently), the default interface is all I really need.
• Google Calendar (scheduling). I didn’t get into Google Calendar until I figured out how to sync it to my iPhone – and then it became invaluable. I get a reminder of imminent tasks wherever I am. Interoperability with Gmail for event invitations means I have an integrated system for keeping on top of calls and conferencing.
• Producteev (task management). Until I found this, I’d been using Remember the Milk for tasks, which I never really got into, despite buying a pro account. Integration with Google Calendar is perfect, and the iPhone app has its own push notifications. And for my purposes, it’s free, which is even better.
• Freckle (time management) has dramatically simplified the way I bill for my time. The integrated timer means I can effortlessly keep track of how many hours I’m spending on what project, and I get to export unbilled hours to a nicely-formatted automatic invoice. Offline access and the ability to mark invoices as paid would make me even happier.
• Beanstalk (source code management) is by far the best hosted subversion repository provider I’ve found. (Projects can also be hosted using Git.) It integrates with a bunch of different applications, including Basecamp and Zendesk, but so far I’ve only needed to tie it to Lighthouse.
• Lighthouse (issue management) is low on features compared to Trac (which I’ve used for years), and it’s true that I’d prefer an easy-to-use bug tracker that managed to incorporate things like Mylyn integration and bug priority levels. But when it comes to interacting with clients, given the choice between feature-packed and non-developer-friendly, I’ll pick the latter every time. Lighthouse is simple, well-designed and light years less painful to use than a tool like Bugzilla. It’s also proven pretty useful inside teams of developers, although there are usually complaints about missing features.

It should go without saying that I’m not involved with any of these companies, and none of them have paid me for this post. In fact, in the case of Freckle, Beantalk and Lighthouse, I happily pay them. I think subscription or one-off license charges are probably a better way for smaller software houses to fund their web applications, and I’m really glad to see these kinds of premium models become more popular.

I’d love to hear your thoughts on these, and the apps you find useful in your work. Leave a comment or write your own post and tweet it with the hashtag #myprowebapps.

Related entries

• For your consideration at SXSW Interactive (1)
• Devices and desires: why the portable device wars are a red herring (2)
• Some alternative views of the iPad (1)

• Status.net
• DiSo
• 6D
• Kopal
• DSNP
• Noserub
• Appleseed
• OneSocialWeb
• Diaspora (included for completeness, but there isn’t any code yet)

Wow, that’s a lot! And following Diaspora’s flurry of both coverage and cash, you can bet there’ll be plenty more to come. But of all the projects listed above, I’d argue that only Status.net is orientated around consumer need. As a result, it’s the one most likely to survive, become self-sufficient and prosper. Several more – including DiSo and DSNP – are seeking to build out technologies that can support such products, rather than the products themselves. DiSo is certainly working with other vendors and projects, is full of super-smart people, and should do very well.

However, the others are basing their product on ideology and technology rather than a human use case. I worry that a lot of these projects will disappear – which is a shame, because they’re all doing great work.

Here’s a use case distinction I’ve been thinking about:

• A social networking platform allows you to communicate and share with a specific group or community.
• Distributed social networking software allows you to store and organize your own content and – optionally – share it with whoever you like.

Or to put it another way, in social networking platforms, sharing is the feature. In distributed social software, sharing is a feature. The two use cases are genuinely different: rather than being a competition between “monolithic” social communities and distributed social software, they’re used for different things. There is a place for both in the ecosystem – and there’s no real reason why they can’t work together.

As I pointed out in The Internet is People, in order to be successful, any social software you build either has to plug into an existing community, or be useful for the first user who joins. In distributed social software, you only ever have one user: distributed sharing, then, should be a piece of infrastructure that can be plugged into any kind of software.

Related entries

• Building the user-centered web (8)
• Facebook location vs the ACLU (0)
• Direct messaging in a social web architecture (3)

A little pre-history

When I was a kid, I had an Atari 130XE. You’ve probably never heard of it. It was an 8-bit, all-in-one box that booted straight into BASIC; a flexible, well-built, sturdy computer.

There was just one problem: it wasn’t a ZX Spectrum or a Commodore Amiga.

At the time, Britain was undergoing a low-budget computing renaissance. Bedrooms up and down the country were filled with skinny boys (and yes, it was mostly boys) noisily loading games from cassette tapes and dutifully copying down source code listings from specialist magazines. The two engines of this renaissance were the Spectrum and the Amiga, and as such, the games, the tutorials and the social infrastructure were built for these two machines. Perhaps this helped me become more of a creative self-starter: I wrote my own games and stories instead of consuming other peoples’.

Later on, 16 bit computers became popular, and everyone upgraded to the Atari ST: a home machine powerful enough for creatives and musicians, but cool enough for game-playing kids. Except, perhaps inevitably, we had a PC. Running DOS. With a black-and-white Hercules display. Great if you wanted to plug economic figures through a spreadsheet, but lousy if you were a twelve-year-old who was mostly interested in playing The Secret of Monkey Island. Not only was the wholly PC incompatible with the Atari ST, but the PC was actually incompatible with itself: a game that worked on PCs with an EGA or VGA screen wouldn’t work with CGA or Hercules. Back then, the parts inside your computer were at least as important as the operating system you ran or the software you bought.

Plug and Play

Through heavy force and heavy lifting, Microsoft changed all that. Windows 95 was the first widely-accessible operating system that unified hardware platforms. Sure, you had to have an Intel-compatible processor, and it took them a while to get it right (for a while the system was redubbed “plug and pray”), but you didn’t have to mess with configuration files to get your computer working. This was a Big Deal.

Today, we’re used to not having to tinker with our machines. Windows will adapt to just about any hardware you throw it at, and even Linux has become an easy-to-use operating system (relatively speaking).

Better yet, we have data portability: in my house we’re running Windows 7, Mac OS X and Ubuntu, and I can move my documents between them interchangeably. Thanks to the web, and Java before it, we even have applications that don’t care what kind of operating system they run on. For an end user, things just work. That’s exactly how it should be.

Finally, computing is simple, data is interoperable and consumers are in control.

Uh oh: enter the portables

So just as we get a unified computing platform that’s easy to use and relatively simple for consumers to navigate, in comes a new device market that’s as fragmented and consumer-unfriendly as the computing market was in the eighties.

Android. iPhone OS. Windows 7 tablet edition. Windows Embedded Compact. Windows Phone. WebOS. ChromeOS. Kindle OS. Whew! It’s like 1986 all over again.

As a publisher or developer, figuring out which device to build for is a headache. Each one has a different operating system, possibly a different app store (something nobody had to worry about in the eighties), and a different set of underlying technologies. Do you exploit the iPad’s current success and develop for the locked-down Apple platform? Do you take advantage of Amazon’s huge built-in market and write a Kindle app? Do you hold out and wait for HP’s exciting-looking WebOS-powered tablet (which caused a storm recently by publicly moving away from Windows)?

Plug and Play (again)

The truth is, market forces are going to apply the same pressures to the mobile market that the personal computing sector felt in the early nineties. This story has played itself out several times now: one platform will emerge victorious. Judging by the lessons learned by IBM with their Personal Computer architecture, and both Microsoft and Linux for operating systems, it’s likely to be one which is:

• Open: anyone can add it to their system for little cost, allowing hardware manufacturers to maximize profits by concentrating on the device itself rather than the ecosystem around it
• Sustainable: it’s powered by a solid business ecosystem that will ensure the longevity of the platform
• Friendly: it’s a system for everyone, not just hobbyists or developers
• Flexible: it can be used in multiple contexts, from living rooms to science labs

By this measure, Apple is condemned to be a niche player, operating at the premium end of the market. Sure, right now technophiles everywhere are salivating over the iPad, but that will last until someone comes out with something nicer. In any event, Apple’s grasp is limited to the wealthier western nations – there are far more people seeking more affordable devices waiting in the wings in other places. The third world computer revolution is very much underway.

My bet, of course, is on web technologies. But it isn’t necessarily on the Internet: it’s time we separated web technologies from the World Wide Web. Indeed, connectivity isn’t ubiquitous, and isn’t likely to become ubiquitous world-wide for a very long time. Therefore, the ability to download, install and run apps offline, as we always have with software applications, is incredibly important.

With its Chrome Web App Store, Google is leading the way, and showing that it understands what it takes to create a next-generation application platform. It’s also shown leadership over HTML 5, which it is clearly investing in as a genuine method for powering both content and software. The genius is this: anyone can build using web technologies, and web technologies can run on virtually any hardware. Google makes its money through value-added services, like advertising (to allow both device manufacturers and software developers to supplement their incomes), its app store and underlying logic via some powerful APIs. It’s not an operating system, but for most end-users, they’re making the operating system irrelevant: it’s simply the thing that runs the web browser.

My advice: ignore the hardware

Computers as we know them today will always exist, but they won’t be for everybody. If you’re developing for non-technical end users, the plethora of hardware devices available to you is a red herring. You should be thinking of the web as the platform your products will be based on. Make no mistake: you need to become an expert in web technologies now – or, of course, find someone who is.

Images:

• My ‘new’ Atari 130 XE by qnr, released under a Creative Commons license.
• HTC Tablet PC – evolve by nDevilTV, released under a Creative Commons license.
• Amazon Kindle & Sony eBook by jblyberg, released under a Creative Commons license.
• iPad & Friends by Yutaka Tsutano, released under a Creative Commons license
• Web browsers by smemon87, released under a Creative Commons license

Related entries

• So why do we need apps anyway? (3)
• Charging for software in the age of web apps (2)
• Some alternative views of the iPad (1)

NB (May 20, 2010): A lot of my suggestions for web-based apps are part of the Google Chrome Web App Store. In fact, the .crx file used there is a zip file with very similar characteristics to epub. (I assume, as Chromium is open source, that .crx files are also open source – so the web app store is not limited to Google.) This post can be reread as an argument for building for the Web App Store.

At Intersection: Publishing in London the other week, there was a lot of discussion from publishers looking at mobile apps as their mobile publishing solution. Rather than creating ebooks, there seemed to be a general feeling that dedicated applications presented more of an opportunity for richer content, while closing the door to pirates and ensuring that publications remained a paid commodity.

The piracy argument is kind of spurious: although app stores tend to be locked down, this presents a false security blanket for publishers. It only takes one person to crack a store for piracy to be generally possible; technology only ever becomes less secure over time. A cynical person might suggest that the piracy argument is largely spread by the people who own the app stores or provide related services. The people who will suffer are authors and publishers.

Why apps rock

However, there’s definitely an argument for using apps – not just for publishers, but for anyone who wants to create dynamic content. Anyone who’s ever owned an iPhone will tell you that native applications can still provide a smoother, more consistent experience than a web app, without the hassle of remembering website addresses or waiting for pages to load. Tweetie is a million miles better than Twitter’s mobile website – something they themselves acknowledged when they acquired the iPhone application last month.

Above, mobile Twitter is on the left; Tweetie is on the right.

• The app doesn’t need to load its interface from the web; only the underlying data is downloaded, meaning the app can appear instantaneously, loads data faster, and provides a better user experience.
• The mobile web app needs to sit within the browser chrome (URL and search boxes, browser buttons on the bottom, and in my case, a debug toolbar). The app, on the other hand, has a full-screen UI dedicated to Twitter.

Why the web rocks

The mobile landscape right now is a bit like the personal computing landscape circa 1985. There are a bunch of different platforms to code for:

• Apple iPhone and iPad
• Android
• Symbian
• Windows Phone
• Blackberry
• WebOS (now more important in the wake of HP’s acquisition)

Each of these platforms is different under the hood, and must be developed for separately. Most developers and publishers can’t afford to do this – there isn’t a way to write once and cross-compile to many platforms at once. In fact, Apple recently specifically forbade this: if you’re developing an Apple app, you’re doing so natively, or you’re violating that platform’s terms of use.

However, each of these platforms have one thing in common: they support the web.

HTML5 and ePub: a new platform for apps

As you’re probably aware already, the upcoming HTML5 standard revises the web platform to become far more suitable for apps. Improvements include:

• Methods for offline and cached usage (so interfaces can load immediately)
• Built-in databases and storage (so web pages can natively store their own data)
• A paintable canvas element and WebGL 3D graphics functionality (so web pages can display interfaces more like real applications; the 3D shooter Quake II has already been ported to native HTML)
• Native video and audio support (no Flash required)
• Websockets (a more efficient way to connect to Internet data from web pages)
• Built-in support for advanced functionality like geolocation

This is a big deal. Compliant browsers like Firefox, Safari, Chrome and even the upcoming Microsoft Internet Explorer 9 will be able to run applications that look and feel like native software but are powered by web standards. Between those browser engines, that’s most of the mobile platforms covered: those that don’t have an HTML5 browser built in by default should have one available to download. What’s more, both Firefox’s Gecko HTML rendering engine and the WebKit engine that powers both Chrome and Safari are open source, so anyone can pick them up and build software around them.

So sites on the wider web can be more like applications. That’s fantastic news in itself, but what about the app store model? A lot of people depend on that for revenue, and there’s no reason why that should be incompatible with using web standards.

Luckily, it turns out that ePub – the ebook standard – is really just a bunch of XHTML 1.1 pages drawn together in a specialized way and bundled up in a modified zip file. There are already established best practices for buying and selling ebooks.

If the ePub standard was updated to allow HTML5, it would evolve into a format for self-contained, multi-platform apps that could be sold in the same way as ebooks, music, videos, or apps in something like the iTunes App Store. Except app publishers would only need to build once to support many different kinds of mobile platform, thereby reducing the barrier to entry and allowing their budgets to be concentrated on building just one really awesome piece of software instead of spread across multiple devices.

This would be in a lot of peoples’ interests: app publishers, device manufacturers, browser vendors and consumers alike. There’s a lot of money tied up in a venture like this. The only question is, will the International Digital Publishing Forum, which controls the ePub standard, be foresighted enough to see this opportunity?

Update: Steve Jobs weighs in

Apple’s CEO has written a little about why HTML5 is the future of mobile apps (albeit in the context of his platform’s refusal to support Flash):

HTML5, the new web standard that has been adopted by Apple, Google and many others, lets web developers create advanced graphics, typography, animations and transitions without relying on third party browser plug-ins (like Flash). HTML5 is completely open and controlled by a standards committee, of which Apple is a member.

[…] Flash was created during the PC era – for PCs and mice. Flash is a successful business for Adobe, and we can understand why they want to push it beyond PCs. But the mobile era is about low power devices, touch interfaces and open web standards – all areas where Flash falls short.

[…] New open standards created in the mobile era, such as HTML5, will win on mobile devices (and PCs too). Perhaps Adobe should focus more on creating great HTML5 tools for the future, and less on criticizing Apple for leaving the past behind.

Make no mistake: HTML5 is the platform to bet on.

Related entries

• Devices and desires: why the portable device wars are a red herring (2)
• The future of publishing (1)
• Intersection: Publishing is today! (0)

I thought I’d write down some of my takeaways while they’re fresh in my mind:

DRM is misunderstood from both sides.

From some publishers, support was shown for Apple’s locked-down App Store business model, with the assumption that it would prevent piracy. Of course, this isn’t the case. I think Sven Edge put it best to me during the post-debate drinks: “any technological system only becomes less secure over time.” In other words, you cannot assume that any technology is unbreakable; someone will do it. Trusting your business model to DRM is therefore a very bad strategy.

Publisher advocacy of locked-down Digital Rights Management technologies apparently occurs because authors need to be reassured that their work won’t be stolen when it becomes available online. A few authors present disputed this point of view. Regardless of this, more work needs to be done to educate non-technical people around the issues, in a calm way that takes in all points of view and doesn’t attempt to reform the fundamentals of copyright law or rights agreements in the same breath.

The market for electronic publishing is still too fragmented.

Many publishers present were worried about the variety of devices and platforms present on the market, as well as their quality. They simply can’t afford to target all of them, and many are either choosing to wait or work with third-party companies to develop solutions for them. All agreed that a single, open platform that allowed publishers to create content using something approaching their existing skill-sets is desperately required.

There also needs to be an open equivalent for apps, to give publishers a choice, and to allow them to deliver to multiple platforms at once. During the debate, I suggested encapsulating HTML5 (which has all manner of app-friendly capabilities) in the ePub format (which produces stand-alone bundles of content that can be sold and transferred between devices). I intend to write more about this another time.

The publishing industry is following the patterns laid out by the music industry.

Publishers are signing authors rather than books, and are beginning to gather extra revenue through talks and activities surrounding books, just as – for example – musical artists like Madonna are beginning to sign to concert promoters rather than traditional record labels. Together with the DRM arguments above, I think there’s a real danger that the publishing industry could go down exactly the same road. (On the topic of DRM, note that iTunes is now DRM-free – don’t count that any restrictions on iBooks or App Store items will last forever.)

The knowledge gap goes both ways.

The assumptions that geeks take as being gospel are not gospel. The assumptions that publishers take as being gospel are not gospel. Each side needs to listen to the other and contribute to a productive conversation, without demeaning anyone’s expertise or experience. There needs to be both give and take.

To put it another way: the models that govern software do not govern publishing and the models that govern publishing do not govern software. These remain two different businesses, and must be treated as such.

There was some very heated debate yesterday, but also a great deal of very constructive argument. I’m really looking forward to continuing the conversation.

Related entries

• Intersection: Publishing is today! (0)
• Intersection: Publishing (0)
• So why do we need apps anyway? (3)

Quinn Norton has some very smart comments about the blinkered vision of the wealthy middle class people who typically assess the impact of devices like this:

I live a really rich intellectual life and get to do lots of things most poor people don’t, and I appreciate that it’s because almost none of my social group are poor. But sometimes my social group kind of goes crazy and forgets that while they have a lot of power, my class is a whole lot bigger than theirs. And none of them will be buying iPads.

Dave Winer has been testing his for a day, and thinks the revolution is yet to come:

Keep dreaming if you want, but if you give the iPad to your mother expect the light to go on for you. At that exact moment you will realize how poorly prepared it is for that. [...] With the caveat that it’s after one day and I reserve the right to change it at any time: Today’s iPad, the one that I just bought, is just a demo of something that could be very nice and useful at some point in the future. Today it’s something to play with, not something to use. That’s the kind way to say it. The direct way: It’s a toy.

I think Dave’s comment – “a demo of something that could be very nice and useful at some point in the future” – is probably prescient. I am excited about the device, and I do want one, but I’m more interested in where this takes the computer industry as a whole in the future. Apple’s devices are famously locked-down (“The iPad is a LEGO set that can only be assembled into what’s drawn on the box,” as Jarek Piórkowski puts it), but the devices that follow it won’t be, although they will learn from iPad’s design decisions.. Specifically, it will bring about three things:

1. A new kind of smarter, easier, more intuitive portable computer interface
2. The death of Flash and third party plugins for multimedia content on the web (this is a big deal)
3. Tacit approval for the industry to innovate away from the traditional PC model we’ve been working with for decades, and create new information appliances that more easily fit into peoples’ lives and can be used in a more human way

Actually, my last point was kickstarted by the iPhone, but the iPad makes it legit: whereas the former was a “mobile device”, the latter is being marketed and sold as a computer in its own right. Many more will follow.

All these devices with different form factors, designs and operating systems will have two things in common: you can take them with you, and they will run HTML 5+ web applications. The future is going to be very interesting indeed.

Related entries

• Devices and desires: why the portable device wars are a red herring (2)
• iBooks is a killer app for ebooks (1)
• Danger in the cloud: a proposal (8)

So what about direct messaging?

In my previous post, I described content notifications in the social web as being Activity Streams updates in response to requests signed with an OAuth key. Each individual contact would have his or her own OAuth key, and the system would adjust delivered content depending on access permissions I had assigned to them.

A private message in this architecture could just be represented as an item of content restricted to a small set of recipients (in the email use case, this is typically just one), with replies delivered using Salmon. The advantage of this approach is that the message doesn’t have to be text; it can be audio, video, a link to live software, or something else entirely.

However, while this is technically feasible, it may not always be desirable. We know from Google Wave, which also pushes the boundaries of person-to-person messaging, that an open definition of what a message contains can get very messy very quickly. Although I was one of the first people to have one, I no longer check my Wave account regularly. I believe this is mostly a user interface issue: Wave is an awesome collaborative document editor (what I’ve heard described as “a massively multiplayer whiteboard”), but not in any way the evolution of email that its development team claimed.

Therefore, I think it’s useful to think about the difference between a document and a message:

• A message is the body of a communication.
• A document is a bounded representation of some kind of information.

While in many ways they’re the same, I think it makes sense to make a separation on the UI level. As we’re discussing a decentralized architecture here, some kind of semantic marker in our activity stream feed to mark something as a message would be a useful feature.

Messaging “out of the blue”

You know where you are with an email address. Mine is ben@benwerd.com. Anyone who encounters that string of characters, whether on a website like this one, a business card or a scribbled note on a piece of paper, is able to send me a message from anywhere in the world. In the 17 years I’ve had an email address, the list of friendships and business connections I’ve made, and opportunities I’ve received and developed, through this simple mechanism has been uncountable. It’s also likely to continue far into the future.

Compared to this, visiting someone’s social web profile and sending them a message from their web presence is a hassle. Compare these steps:

1. Receive the address of someone’s profile
2. Click the “follow” button either on the profile itself or on the toolbar of your social web compatible browser
3. Wait for the contact to follow you back
4. Send your message

To:

1. Receive someone’s email address
2. Send a message to that address

It’s simple, ubiquitous, decentralized and universally compatible. In fact, it seems hard to improve on, doesn’t it?

However, as this is a thought experiment about how social networking can replace email, let’s see if we can simplify this process somewhat. In my previous post, I discussed how a connection could be established with OpenID and OAuth through a web-based interface on a social web profile. How can we make this as simple as emailing someone, and cut out most of the steps I’ve listed above?

Connecting programmatically

I propose two additions to my previously discussed mechanism. The first is to expand the connection protocol to include a message. If someone connects to me on LinkedIn or Facebook, I receive some explanatory text from them, so it makes sense to include this feature in our decentralized social web architecture. It is likely that this would be an added parameter to the OAuth request token procedure.

The second is to allow connections to be made programmatically through a custom application. Just as we use email clients now, a social web client could automatically send a connection request. In keeping with our principle of using existing technology where possible, this is a simple OAuth connection request from the application, which includes a user message as described above. The application knows our details because we’ve set our preferences, so we’re never visibly redirected to a web browser to complete authentication. (In fact, this could take place using xAuth, a version of the OAuth protocol being developed for just these sorts of browser-free use cases.)

Whether we can send a follow-up message now depends on the receiving party. We have our OAuth token, and while it remains valid, the receiving social web node may choose to ignore any follow-up requests.

Our procedure has become:

1. Obtain address of someone’s social web node (you could even infer it using WebFinger)
2. Send a message to that node, bundled with a connection request

This is significantly better, and is comparable to the simplicity of email.

You may be wondering about the wisdom of adding everyone you contact as a connection. In fact, there’s some precedent for this already in applications like GMail. It’s important to note that not every connection need be a friend: in some ways, you can think of your total list of connections as your contact book. Some are important, some can be safely squirreled away until you need to contact them again. In this context (or any context where people you have a relationship with and people you’ve contacted are merged into one set), an adequate person management interface – or CRM to you and me – becomes important.

Next, and finally: let’s make our distributed social web architecture reliable enough to use in enterprise environments, using message queue protocols like ZeroMQ and AMQP.

Related entries

• Activity Streams and OAuth: a social web architecture (3)
• How social networks can replace email (11)
• Social networking: beyond the silo (1)

Here in the UK, the Digital Economy Bill looks like it’s set to be rushed through Parliament:

There’s plenty to oppose in the Digital Economy Bill, it gives the government the ability to disconnect millions. Schools, libraries and businesses could see their connection cut if their pupils, readers of customers infringe any copyright. But one group likes it, the music industry. In a leaked memo a few days ago they admitted the only way to get the bill through would be to rush it through without a real parliamentary debate. Let’s stop that happening.

According to the Open Rights Group, there have even been questions about the Bill’s compatibility with the Human Rights Act.

The following is a letter I wrote to my MP, Andrew Smith. If you’re a British resident, I recommend you do the same (Boing Boing has a sample letter, but generally it’s a good idea to avoid form letters if you can).

Dear Andrew Smith,

I’m very worried that the Government is planning to rush the Digital Economy Bill into law without a full Parliamentary debate. Despite claims made by the BPI and others, I believe it will have dire consequences for British businesses, and therefore for the economy as a whole.

In short, the Digital Economy Bill provides the mechanism to arbitarily remove anyone’s freedom to communicate – their Internet connection, and potentially access to their website or servers – without due process. This will immediately put us on an uneven footing with countries such as the United States, who are already well ahead of us in terms of digital business. If Britain is to remain competitive in the digital world, this must not go ahead.

This is not to say that piracy should be allowed. It is undeniably a criminal act. However, for an issue that has become so culturally ingrained that it requires precise tactics to undo, these measures are unsubtle and counterproductive.

At the very least, a proper Parliamentary debate must be had.

As a constituent I am writing to you today to ask you to do all you can to ensure the Government doesn’t just rush the bill through and deny us our democratic right to scrutiny and debate. As a digital professional, I would be delighted to help with any questions you might have.

Kind regards,

Ben Werdmuller
ben@benwerd.com

Want to read the Bill for yourself? Here it is.

Related entries

• Dear Member of Parliament, (1)

• Ease of use
• Ubiquity across devices
• Platform, service and infrastructure independence

My argument boiled down to the following statement:

Email has succeeded because it’s open, standard and decentralized; for social networks to replace it, they must also be open, standard and decentralized.

Email is useful because just about everybody has an email address. I can get in touch with my clients in London, my friends here in Oxford or my grandfather in Austin, Texas, with equal ease, even though all of them are using different infrastructure and software provided by different companies. I use Gmail, but there doesn’t need to be any kind of formal agreement between Google and whoever’s providing my grandfather’s email, say. It just works; nobody owns email as a communications method, and anyone can set up an email server. The same is true with websites: anyone can set one up, and nobody owns the web.

For social communications to be as popular and ubiquitous as email, there must be one social web, and it must be owned by nobody. That means that each socially-aware site or application must implement the same social communication standards.

The best standards aren’t dictated: they evolve through common usage. If you look at HTTP (the protocol that the web relies on), SMTP (one of the protocols behind email) and file formats like RSS and HTML, the common thread behind them is that they’re simple. It turns out that through excellent work at companies like Google, Plaxo, SixApart, Twitter, JanRain and – perhaps incredibly – JPMorgan Chase & co, we already have a number of technologies that collectively embody the properties I listed above.

Notes and server architecture for one possible social web

These are my ideas about how these standards might be used. These aren’t intended as replacements for existing social networking platforms or services; rather, they could easily be added as additional features both to those and to many other types of application. The ability to share isn’t a uniquely required feature of social networking software – think about its usefulness in applications like Word or Google Docs, for example.

With email, you use a software client (Outlook, say, or the Gmail web interface) that speaks to an email server which does the hard business of sending and receiving messages to and from the wider Internet. Here, I will be describing a system where everyone has their own node on the social web, which effectively acts as a client and server. Mine might be here at benwerd.com, for example. It’s my website – my profile on the social web – and it’s where I send social communications. That’s the server side. However, it also acts as the client when I’m accessing resources stored on other peoples’ servers.

Establishing connections and granting permissions

Let’s say I want to make a resource available to my clients. With email, I’d send them each a separate copy. This is both insecure and inefficient: I have no control over what happens to that copy, and each time I send it I create a new version. With some back-and-forth, there could easily be ten or twenty individual copies of a document floating around. (I often bounce software specifications – typically Word documents – around with my clients, and this is something that happens to me regularly. Google Docs is probably a better solution, but not everybody has a Google account.)

With the social web, only one version needs to exist, which I own. If my clients have established a connection with me, I can restrict that resource so that only they may see it. The tricky bit is that in order to know if it’s really them, they must be authenticated in some way.

In monolithic systems like Facebook, where everyone uses the same website, that’s easy: my client must be logged in, and we must have established a friend connection. In a decentralized system, that’s a much harder problem, but not insurmountable. Two technologies will help us:

• OpenID: the open, decentralized authentication standard, which currently uses a website address as a kind of universal username
• OAuth: an open protocol that “allows users to share their private resources (e.g. photos, videos, contact lists) stored on one site with another site without having to hand out their username and password.” OAuth provides a secret token to applications that they can use to access authenticated services and resources behind the scenes

Specifically, we’ll need OpenID Connect (or, until that’s up and running, the OpenID / OAuth hybrid protocol), because we’ll be using OpenID to authenticate, OAuth to power our decentralized access permissions, and a number of other protocols and endpoints along the way. It’s much neater if these are all established at once.

Making friends and getting updates

The process would work in the following way. Let’s say I want to make a connection with my friend Marcus Povey.

1. I visit his site, and see that he is displaying a “connect to me” icon, indicating that it is a node on the social web. Later on, perhaps my browser would detect that this was a social web node in the same way that most browsers detect RSS feeds today, and light up an icon. Chris Messina has started a five part series on the browser as a social agent, which is worth a read.
2. Either way, I click on “connect to me”. Marcus’s site prompts me for the address of my profile, which I enter. (Later on, my browser does this bit for me.)
3. My profile address is an OpenID, and through the authentication process my social web node receives an OAuth token from him. No further authentication is required.
4. On his social web node dashboard, Marcus sees that I’ve established a connection with him. He can ignore it, in which case nothing happens, or he can mark me as a friend (or any other arbitrary designation, which could be unique to the software he’s using).
5. My social web node periodically checks for activity updates from Marcus’s, signing each request with that OAuth token so it knows who I am. This may be at my direct request; through repeated polling, RSS-style; or the update may be pushed to me through a PubSubHubbub ping.
6. Depending on the assignation he’s given me, Marcus’s node either responds with just a feed of public activity (if he’s ignored the request), or with additional activity he’s allowed me to see, in Activity Streams format.
7. Marcus can change my assignation or withdraw my OAuth token at any time from his dashboard. (Of course, throughout all this, the OAuth token mechanism is invisible to both users: it’s simply presented as a social connection.)

Embedded content and interacting directly on other social web nodes

Activity Streams is based on Atom, so content for items like blog posts (and resources like photos, using Atom Media) can be embedded directly in the activity feed. (Rob Dolin from Windows Live has some great examples.)

However, not all content is standard enough to be embeddable. In those cases, I can simply click through from Marcus’s activity update to his site, possibly log in again using OpenID, and interact with the content there. Additionally, by allowing users to log directly into his site via OpenID, Marcus can show selected people restricted content even if they don’t have the full range of social web software.

Friends lists and commenting

Further standards help us add extra functionality. If Marcus gives me permission, I might be able to download his contacts via Portable Contacts. Salmon is a protocol for commenting on distributed resources and allowing those comments to find their way upstream to the original, which is compatible with Activity Streams. Using this, I might be able to comment on Marcus’s activity items from within my dashboard and have them show up in his. Through this mechanism, all his friends could have a conversation on his activity stream items.

Reliability

So far, so good: we have a simple technological basis for permissive social communications. But if the social web is really going to replace email, we have to address one of the most important features for enterprise users: reliability. Businesses will not accept their critical communications being subject to fail whales.

In my next posts in the series, then, I’ll discuss person-to-person messaging and the thorny issue of guaranteed delivery.

Related entries

• Direct messaging in a social web architecture (3)
• How social networks can replace email (11)
• The Open Stack and truly open APIs (0)

1. By 2014, social networking services will replace e-mail as the primary vehicle for interpersonal communications for 20 percent of business users.
2. By 2012, over 50 percent of enterprises will use activity streams that include microblogging, but stand-alone enterprise microblogging will have less than 5 percent penetration.
3. Through 2012, over 70 percent of IT-dominated social media initiatives will fail.
4. Within five years, 70 percent of collaboration and communications applications designed on PCs will be modeled after user experience lessons from smartphone collaboration applications.
5. Through 2015, only 25 percent of enterprises will routinely utilize social network analysis to improve performance and productivity.

Social networks replacing email. Really?

I broadly agree with all of these, but that first prediction needs a little more analysis. Let’s think about why email has succeeded:

• Ease of use
• Ubiquity across devices
• Platform, service and infrastructure independence

I access email from my Dell PC, my iPhone, and have in the past used Blackberry phones, Macs, Linux boxes, etc, all the way down to Windows 3.1, using a combination of software that’s included Eudora, Thunderbird, Phoenix, Turnpike, and many more. Right now I use a combination of GMail, Google Apps and self-hosted email addresses; in the past I’ve used Microsoft Exchange in various guises, Yahoo Mail, and so on. No matter which provider or hardware I used, I could email anyone else with an email address, no matter which provider or hardware they used. Email is a completely open, interoperable standard.

Social networking is anything but an open, interoperable standard. If you use Facebook, you can communicate with other people on Facebook, full stop. Even networks based on open source solutions like Elgg are essentially social islands.

What needs to be done?

I strongly believe that social messaging can be significantly more useful to both enterprises and individuals than standard email. Proof-of-concept applications like Google Wave are beginning to show the way: you can make resources available to whoever needs to see them, rather than the current, inherently insecure practice of making copies and sending them out. Whereas email takes inspiration from letters and faxes, the social messaging paradigm is based more closely around conference calls and conversations.

Nonetheless, in a business situation, you need to be reasonably certain your message is going to reach the recipient, and the current platform constraints – only being able to message someone using the same site as you – are untenable. Let’s look again at those email success factors:

• Ease of use
• Ubiquity across devices
• Platform, service and infrastructure independence

Social networks do currently have ease of use. They may approach near-ubiquity across devices only if they create a developer ecosystem around their proprietary APIs, as Twitter has done, but this requires a lot of faith in a single third-party service.

No, I think it comes down to one principle:

Email has succeeded because it’s open, standard and decentralized; for social networks to replace it, they must also be open, standard and decentralized.

Next: real world, technical approaches to this that can be implemented today.

Related entries

• Activity Streams and OAuth: a social web architecture (3)
• Direct messaging in a social web architecture (3)
• Social networking: beyond the silo (1)

• Her Last.fm page contains more polished recordings.
• Facebook has those too, but also contains more rough-and-ready demos, ideas and experiments; it’s also updated the most, probably because she spends more time on it than on any other site.
• The MySpace page seems to exist because you’re expected to have one.

I wonder how common this is? Any other musicians care to comment? Is there one best place to promote yourself as an independent musician?

Related entries

• Facebook location vs the ACLU (0)
• Public IT project hell: let’s make government work for us (1)
• Twitter DoS and single points of failure (9)

A podcast is just an RSS feed with a file enclosure – part of the RSS standard – that points to an MP3 file. Similarly, video podcasts point to video files. An obvious evolution, then, is the pubcast: periodical publications delivered through RSS feeds.

Free publication subscriptions

In the free case, a user would simply subscribe to a public pubcast feed with a compatible reader. The reader software would check regularly for updates, and new publications would be downloaded and fed into the user’s ereader software on release. Easy.

Paid publication subscriptions

In the case of paid publications, there are two options:

An authenticated pubcast feed. When you subscribe to a publication, you get an address to an RSS feed that requires a username and password to download content. (Gmail is an example of an application which already does this.) This authentication ensures that only paid subscribers can access the file, but you could go a step further and watermark the publications themselves.

Activation within the ebook file. The RSS feed itself is public, but each downloaded publication could require an access code to read. This would open the door for public feeds of paid journals, where users could buy each issue individually to read.

Making subscriptions an open standard

Either way, this approach would allow any ereader using any compatible software solution to subscribe to periodicals. It could be used for newspapers, magazines, journals, zines, or new kinds of periodical; they could be hosted anywhere and, in the case of paid content, use any payment provider. I love reading, but dislike monopolies, so this is something I’d like to see.

Related entries

• The future of publishing (1)
• Intersection: Publishing is today! (0)
• Intersection: Publishing (0)

One: this isn’t a device for the tech community. I think Rafe Colburn hits it on the head:

It’s just an iPod Touch with a big screen, but that’s all that many people need from a computer. You can use it to surf the Web, read email, listen to music, watch video, or compose documents. That’s the personal computer use case for many people. And I think a lot of people are going to buy them.

He goes on to discuss the locked-down nature of the device, which I agree is a setback that may have a profound impact on the consumer computing industry. (On the other hand, as Yehuda Katz argues, this is a major win for standards-based web applications.)

Two: for me, the big news wasn’t the iPad at all. It was iBooks: Apple’s new iTunes-like store for ebooks. You may remember that iTunes pretty much revolutionized how we buy music, and this is the same; the books are stored in the open ePub standard, so they’ll play with other ereaders, and the experience is seamless. (You almost certainly won’t need an iPad to buy from iBooks.)

Mashable notes that some big players are on board:

iBooks is backed by big-time launch partners Penguin, Simon and Schuster, HarperCollins, Macmillan and Hachette, all publishing powerhouses in their own rights.

You can think about the iPad as a kind of $499 catwalk model, that other devices will slowly emulate over the next couple of years. But iBooks? That’s a store that anyone will be able to use right away, which just might change the publishing industry forever.

Photo by kennymatic, released under a Creative Commons license.

Related entries

• Devices and desires: why the portable device wars are a red herring (2)
• So why do we need apps anyway? (3)
• The future of publishing (1)

Going out and checking in

I’ve been using Foursquare quite a bit lately (here’s my profile). There’s a lot to be impressed by: not least the level of mobile integration. Foursquare doesn’t make much sense if you’re sitting at your desk, so it’s far easier to see where your friends are and check into a new location from the mobile app.

The way it promotes participation is even more interesting. The designers decided that just seeing where your friends were, and getting personalized travel tips, weren’t enough. You gain points – the same kind you get in Sonic the Hedgehog, say – for checking in at a location, exploring new places and telling the app about venues it’s never seen before. In turn, the points lead to badges, and there’s a weekly leaderboard for the top scorers among your friends. There’s no real tangible value to any of this, but you feel good about joining in. As a result, Foursquare is hugely addictive.

Open source participation

Over on Twitter, I asked:

Could the game dynamics used by apps like Foursquare be harnessed to make a more participative open source community?

Open source projects depend on contributions from their communities. Getting people to participate can be difficult; although many people will join in because it scratches some kind of itch, moral incentives like a place in the credits help. However, adding these kinds of game dynamics over the top could provide an extra push. Currently, the only quantifiable open source contributions are source code patches, and any software project has a lot more going on; this would provide an opportunity to quantify other, equally useful forms of participation.

Game dynamics in the enterprise

Graeme Hunter pointed out to me that this model wouldn’t solely be useful for open source. An internal project communications framework that also incorporated game dynamics could be a very interesting platform for ideas, solutions and internal innovation. He’s right; I think it’s an idea to keep in mind if you’re looking for software to use internally for your project.

There are also implications for online communities, where game dynamics are often already used (to rate individual contributions, for example). What if we used similar ideas for education? Or a community centering around journalism?

Photo by dpstyles, released under a Creative Commons license. It’s of a Target store in Milford, Massachusetts, where they use game dynamics to encourage faster checkout times.

Update: Graeme comments below with an exploration of what a participation framework using game dynamics might involve.

Related entries

• Charging for software in the age of web apps (2)
• Direct messaging in a social web architecture (3)
• Activity Streams and OAuth: a social web architecture (3)

At launch, Data.gov.uk has nearly 3,000 data sets available for developers to build mashups with. The U.S. site, Data.gov, has less than 1,000 data sets today.

[…][Unlike the US equivalent, the site] includes 22 military data sets at launch, including one called Suicide and Open Verdict Deaths in the U.K. Regular Armed Forces.

However, these are raw datasets. As Paul Clarke points out, the site only pays lip service to openness until someone comes along and turns these sets into useful reports and applications:

The only test of real success is: use. Not usefulness. Not theoretical use. Real use. Getting beyond the novelty application, the demonstrator, and the hobby lies at the heart of really untapping the potential of data.gov.uk.

Indeed, the figures that Techcrunch Europe report suggest that turning this data into something useful may be harder than it sounds:

So far over 2,400 developers have registered to test the site and provide feedback, [while] 10 applications have been created.

I left a comment on Paul Clarke’s post pointing out some potential pitfalls that may inhibit innovation, including the government’s insistence on licensing the data under Crown Copyright and their impartiality regarding Twitter. There’s also been some criticism around the lack of a common data format for each feed (although the RDF triple proudly displayed on the front page suggests this is likely to change).

Nonetheless, I believe this represents a huge step forward. Turning raw materials into useful, compelling applications that improve the users’ quality of life requires a huge amount of creativity and talent, and providing the data feeds in the first place is a crucial first step.

You can list all the available datasets here.

Related entries

• The mechanics of "open" (0)
• Public IT project hell: let’s make government work for us (1)
• Direct messaging in a social web architecture (3)