The Cabinet Office Organograms project - Moving Linked Data into production and reaping the benefits

Thursday, 1 November 2012, 17:31.

The final edition of the Civil Service Year Book (CSYB) was published in 2009. This bi-annual directory of contacts within government had 674 printed pages, was three centimetres deep and weighed almost two kilogrammes. Creating the publication involved one and a half full time staff collecting information from individual departments, consolidating and proofing in time for publication deadlines.

In 2009 the Cabinet Office re-created the publication digitally. However it still faced the same practical issues needing editorial and other resource with no mechanism for achieving sales to cover these costs.

During 2010/11, with the onset of the transparency agenda, the content of the Civil Service Year Book was well placed to become a key tool within government and a new system of updating this content was envisaged based on Linked Data.

The Vision

To provide a customised organisation chart (organogram) visualisation for human users with the capability to download machine-readable data via a Linked Data API and by SPARQL Query from an RDF store.

To meet new requirements for transparency, financial information such as reporting structures, salary bands and the combined salaries of direct reports would need to be added to the contact details already available in the Civil Service Year Book.

The Linked Data output will gain value over time, enabling users to see the changes to the machinery of government as departments and their responsibilities evolve.

The Solution

The new system was devised by John Sheridan of The National Archives and Jeni Tennison, working as a consultant to TSO with a superb visualisation by Dan Paul Smith. It went live in June 2011.

The system sits upon TSO’s OpenUp® platform services, including:

  • Harvesting – setting up a pipeline to process good quality department data
  • Enriching – extracting RDF
  • Storage – ensuring RDF is stored and available
  • Publishing – making the data available on the web through both an API (to support data re-use) and a visualisation

Around 200 public sector bodies are present in the system and there are plans to extend the initiative to local government in the event of which the total will reach more than 400. New data will be published regularly and re-used by both machines and humans.

The Challenges

It was a complex project, the first occasion on which every government department has published RDF. The long term success of the project depended upon engaging users with low levels of technical knowledge in order to produce the outputs straightforwardly and reliably.

There were also technical challenges to:

  • Generate RDF successfully
  • Align with and where possible, help improve existing organisational processes
  • Cope with an inevitable amount of change as organisations needs evolve
  • To make the process as smooth as possible for users

Technology

In the initial system build, the team hoped to make the process user friendly by using a familiar tool. Microsoft Excel was chosen to provide the template for users (designated officials within each government department) to complete the organisation’s details.

Originally each completed template was converted from CSV to RDF using PHP and XL Wrap. The aim was that users should publish the RDF on their corporate websites from where it would be harvested by using the data.gov.uk registry to identify the location. This proved impractical as several web content management systems were unable to handle RDF.

The Government Secure Intranet technology on which many department’s systems were based would not work with VBA code and the obvious alternative - signing macros - would have required testing and support for each individual organisation. Considering the size of the roll out, this was not practical. As a result VBA script was maintained.

Usability issues with the template and uploads were resolved individually as they become apparent. One irritation to users proved to be the use of two separate validation stages - within the template and on submission to the central website. This led to problems as data passing first stage validation, sometimes later failed validation on online submission without sufficient explanation as to why. This was remedied by locating all validation within the template itself.

Processes

As some data elements such as salary bands and reporting lines are sensitive, officials often need ministerial sign off to proceed. This proved impractical as time-poor ministers were often on the move. The preview process was speeded up and organisers given the ability to upload to a preview server for sign off purposes.

People

The British civil service is well organised in terms of roles and grades. However, with any large organisation there will inevitably (for good operational reasons) be inconsistencies in how reporting is treated. For example, the Ministry of Defence has twin reporting lines for civil servants of the same grade. Some report to the Minister and some elsewhere, so the model had to be tweaked to cope with these types of instances of the real world contradicting the original assumptions in which the organograms were grounded.

Results

These changes to technology, usability and process design have resulted in a low cost, low effort mechanism to enhance transparency for government and publication of a valuable data set.

Although some support desk time is still necessary each month in order to cope with change and special requirements, in general each department is able to complete its own template and upload the result and there are clear directions to support them with any issues.

Next Steps

Finally, although the vision has always been to add value to the dataset by recording its changes over time, the original priority was to publish the first set of organograms to plan.

To record changes over time, each new dataset is published in a separate knowledge base and mapped between the version or iteration, and the department entities. This enables users to see the changes for a department over time using the organogram.

Lessons from the Project

There are three main systems-architecture lessons to be learnt from the organogram project:

  • Always think at the outset about how the system will really be used
  • Create a flexible solution to deal with evolving user needs.
  • The solution must be sustainable, support requirements minimised, and cause little friction for users

To justify a Linked Data project to a business owner it must meet at least one (and preferably more) of the following four criteria:

Funding Criteria

Fulfilment

1. Substantially reduce the cost of publishing

Organograms is lower cost than commensurate alternatives

2. Demonstrably improve value for users

An effective distributed publishing system is better for officials and the online access is much improved for data users

3. Solve a problem which cannot be solved, or cannot be solved as cheaply in another way

For the first time, online, flexible organisation charts are available for the senior civil service

4. Support other agendas (Transparency in Government, Open Access in learned journal publishing)

The project contributes towards the Transparency agenda

 

The Organograms project met the four criteria.

To discuss Linked Data solutions contact: opendata@tso.co.uk

See Richard's presentation slides: