Open data at data.gov.uk

January 21, 2010 | 1 comment

The British equivalent to Obama’s data.gov opened today. Over at ReadWriteWeb, Marshall Kirkpatrick points out the scale of the ambition involved:

At launch, Data.gov.uk has nearly 3,000 data sets available for developers to build mashups with. The U.S. site, Data.gov, has less than 1,000 data sets today.

[…][Unlike the US equivalent, the site] includes 22 military data sets at launch, including one called Suicide and Open Verdict Deaths in the U.K. Regular Armed Forces.

However, these are raw datasets. As Paul Clarke points out, the site only pays lip service to openness until someone comes along and turns these sets into useful reports and applications:

The only test of real success is: use. Not usefulness. Not theoretical use. Real use. Getting beyond the novelty application, the demonstrator, and the hobby lies at the heart of really untapping the potential of data.gov.uk.

Indeed, the figures that Techcrunch Europe report suggest that turning this data into something useful may be harder than it sounds:

So far over 2,400 developers have registered to test the site and provide feedback, [while] 10 applications have been created.

I left a comment on Paul Clarke’s post pointing out some potential pitfalls that may inhibit innovation, including the government’s insistence on licensing the data under Crown Copyright and their impartiality regarding Twitter. There’s also been some criticism around the lack of a common data format for each feed (although the RDF triple proudly displayed on the front page suggests this is likely to change).

Nonetheless, I believe this represents a huge step forward. Turning raw materials into useful, compelling applications that improve the users’ quality of life requires a huge amount of creativity and talent, and providing the data feeds in the first place is a crucial first step.

You can list all the available datasets here.

Public IT project hell: let’s make government work for us

December 3, 2009 | 1 comment

Why does it cost $235 million to integrate a few IT systems?

Johannes Ernst contrasts the Yahoo/Facebook deep integration announcement with the US government’s announcement that they will spend $235 million on integrating incompatible healthcare IT systems, and asks some pertinent questions:

I assume we all agree that an environment in which leading-edge companies innovate on their own to the benefit of their customers is better than one in which the government has to spend large amounts of money to drag along kicking and screaming “participants” — as it is so common in health IT. How do we turn US healthcare IT from the latter to the former?

One might equally substitute education, or local councils, or law enforcement. It’s a widely-accepted truth that public IT endeavors suck, and that enforcing data standards across disparate public bodies is like herding confused, angry cats into a very wet bag. It’s also true that commercial web services have been very good at integrating for the good of their customers, often without any money (let alone $235 million) changing hands.

I do think there’s a false distinction that’s been made here: public bodies and government departments tend to be swamped in a sea of bureaucracy that prevents them from moving or changing as nimbly as many commercial companies. (Of course, as companies begin to become institutionalized through age and size, they also become less nimble: take Microsoft and IBM.) Many of these restrictions are necessary for the simple reason that they’re using our money, and some regulation is required to ensure tax funds are being spent wisely and benefit the wider public good. We don’t want people to just walk off with it.

Our tax dollars at play

It’s also widely-accepted that our tax dollars are not spent wisely, and often don’t benefit the wider public good. Public bodies are full of inefficiencies, in part because of the bureaucracy involved. I’ve certainly worked within university environments where entire departments of people could reasonably be described as incompetent, but had integrated themselves so well into the system that they had become a required port of call in the bureaucratic workflow. I’ve also seen fully private companies formed using university money and resources earmarked for public research, and government grants essentially spent on beer and travel. These are the kinds of inefficiencies and sanctioned fraud that must be stamped out.

Public bodies and private companies are different in one major respect: their stakeholders. It is a legal requirement for shareholders in a company to have access to the company returns, board minutes and so on (although a wider cloak of privacy is often necessary). In a public body, the stakeholders are the public, yet we often don’t have access to details like financial statements, minutes and decision-making rationale. In Britain, an attempt to get government departments to work like commercial companies has resulted in a ridiculous system where departments must pay each other and the British taxpayer often doesn’t have a legal right to the information they produce.

The public is the board

Ultimately, in a democracy, the public should be the board of directors. Genuine public oversight hasn’t been possible before, but transparency and accountability are now possible via the Internet. We don’t need political parties and administrations to be our eyes and ears any more; we need them to be our hands, and act on our behalf. We need to be able to see the inner workings of public bodies: not just the numbers, but the actual internals and decisions. With genuine public oversight in a way that ensures the bodies know they’re being watched, and governments obligated to maintain these bodies for direct public benefit in a way that’s responsive to the public, costs should go down. It’s not perfect – and Switzerland has recently shown us the dangers of having frequent public referendums – but given the spending, inefficiency and fraud inherent in the system, we can no longer trust the government to do this on our behalf.

BarCamp Transparency

April 7, 2009 | 1 comment

One of the outcomes of BarCamp Oxford has been the organisation of a new BarCamp about transparency and ethics – a mix of social media, open government and cyber-activism.

It’s in its early planning stages, but it’ll take place sometime over the summer here in Oxford. If you’re interested, I suggest you take a look at the BarCamp Transparency wiki and throw your name into the ring. I was asked if I’d help organise, and while I can’t provide as much time as I’d like to due to prior commitments, I’ve volunteered to discuss openness in social media, provide web resources and help out with the event itself.

Transparency is hugely important, and becoming more so. As citizens we have more and more demands upon us to surrender our privacy and aspects of our civil liberties, but the government and politicians on all sides have been reluctant to provide more oversight into their activities. Meanwhile, social technologies have the power to enable us to find and share public information, organise ourselves into groups, and have more say in how our country is run.

This is a vital event that already sounds very promising indeed.

The mechanics of "open"

March 9, 2009 | Leave a comment

PanelSince we started Elgg, I’ve always kept a very open philosophy about how the software should work. From the human perspective, we wanted it to be as inclusive as possible, with an easy-to-use interface and innards that allowed you to do very technical things (like, in Elgg 0.x, republishing aggregated RSS) with very little knowhow. From the organizational perspective, we didn’t want there to be a barrier to entry; we released it under the GNU Public License and allowed anyone to download and install it for free. And technically, we allowed anyone to augment, extend and replace its functionality, maintained an open architecture and embraced technologies like FOAF, RSS and so on.

That was five years ago. The world is only now beginning to catch up.

The Silicon Valley Web community is buzzing with “open” ideas: data portability, the open stack, the open mesh, OpenID, OAuth, and so on. There have been two Data Sharing Summits, a bunch of Identity Workshops, and efforts are crystallizing around open activity streams, contacts sharing, and virtually anything else you might want to transfer between web applications. David Recordon, co-creator of OpenID and all-round cheerleader for openness, has predicted that Facebook won’t be a walled garden by 2010.

This is fantastic stuff, which I intend to get even more involved with as the year progresses. Good work is happening all round, and even sleepy behemoths like Microsoft are beginning to take notice.

What worries me slightly is that the work is centered around the Silicon Valley community, and within that is largely built with public-facing commercial websites in mind. Those sites (like Digg, MySpace, the SixApart properties and so on) are awesome without a doubt, but the potential of social technologies falls well beyond the commercial web. People are beginning to use them on intranets, within universities, across governmental departments and so on – places that could use the same approaches, but need to be represented in the discussions.

Their exclusion is not the fault of the people producing the standards and doing this great work; they’re very happily welcoming anyone with a productive contribution to the table. Instead, it falls to those organizations to realize what they’re missing out on and begin to pay more attention to cutting edge technology. The Obama administration is certainly waking up to this, but others – notably the UK government – are extremely reticent to embrace anything open at all.

The technology is falling into place to allow for an open, transparent, knowledge-orientated economy. Now it’s time to look at what else is needed.