Society of Archivists Conference 2010 – Day 2 (#soa10)

I had a day at the Society of Archivists’ Conference 2010 in Manchester last Thursday; rather a mixed bag. I wasn’t there in time for the first couple of papers, but caught the main strand on digital preservation after the coffee break. It’s really good to see digital preservation issues get such a prominent billing (especially as I understand there few sessions on digital preservation at the much larger Society of American Archivists’ Conference this year), although I was slightly disappointed that the papers were essentially show and tell rehearsals of how various organisations are tackling the digital challenge. I have given exactly this type of presentation at the Society’s Digital Preservation Roadshows and at various other beginners/introductory digital preservation events over the past year.  Sometimes of course this is precisely what is needed to get the nervous to engage with the practical realities of digital preservation, but all the same, it’s a pity that one or more of the papers at the main UK professional conference of the year did not develop the theme a little more and stimulate some discussion on the wider implications of digital archives.  However, it was interesting to see how the speakers assumed familiarity with OAIS and digital preservation concepts such as emulation. I suspect some of the audience were left rather bewildered by this, but the fact that speakers at an archives conference feel they can make such assumptions about audience understanding does at least suggest that some awareness of digital preservation theory and frameworks is at last crawling into the professional mainstream.

I was interested in Meena Gautam’s description of the National Archives of India‘s preparations for receiving digital content, which included a strategy for recruiting staff with relevant expertise. Given India’s riches in terms of qualified IT professionals, I would have expected a large pool of skilled people from which to recruit. But the direction of her talk seemed to suggest that, in actual fact, NAI is finding it difficult to attract the experts they require. [There was one particular comment – that the NAI considers conversion to microfilm to be the current best solution for preserving born-digital content – which seemed particularly extraordinary, although I have since discovered the website of the Indian National Digital Preservation Programme, which does suggest that the Indian Government is thinking beyond this analogue paradigm.]  Anyway, NAI are not alone in encountering difficulties in attracting technically skilled staff to work in the archives sector.  I assume that the reason for this is principally economic, in that people with IT qualifications can earn considerably more working in the private sector.

It was a shame that there was not an opportunity for questions at the end of the session, as I would have liked to ask Dr Gautam how archives could or should try to motivate computer scientists and technicians to work in the area of digital preservation.  Later in the same session, Sharon McMeekin from the Royal Commission on the Ancient and Historical Monuments of Scotland advocated that archives organisations should collaborate to build digital repositories, and I and several others amongst the Conference twitter audience agreed.  But from observation of the real archives world, I would suggest that, although most people agree in principle that collaboration is the way forward, there is very little evidence – as yet at least – of partnership in practice. I wonder just how likely it is that joint repositories will emerge in this era of recession and budget cuts (which might be when we need collaboration most, but when in reality most organisations’ operations become increasingly internally focused).  Since it seems archives are unable to compete in attracting skilled staff in the open market, and – for a variety of reasons – it seems that the establishment of joint digital repositories is hindered by traditional organisational boundaries, I pondered whether a potential solution to both issues might lie in Yochai Benkler‘s third organisational form of commons-based peer-production: as the means both to motivate a community of appropriately skilled experts to contribute their knowledge to the archives sector, and to build sustainable digital archives repositories in common.  There are already of course examples of open source development in the digital archives world (Archivematica is a good example, and many other tools, such as the National Archives of Australia’s Xena and The (UK) National Archives DROID are available under open source licences), since the use of open standards fits well with the preservation objective.  Could the archives profession build on these individual beginnings in order to stimulate or become the wider peer community needed to underpin sustainable digital preservation?

After lunch, we heard from Dr Elizabeth Shepherd and Dr Andrew Flinn on the work of the ICARUS research group at UCL’s Department of Information Studies, of which my user participation research is a small part.  It was good to see the the twitter discussion really pick up during the paper, and a good question and answer session afterwards.  Sarah Wickham has a good summary of this presentation.

Finally, at the end of the day, I helped out with the session to raise awareness of the UK Archives Discovery Network, and to gather input from the profession of how they would like UKAD to develop.  We asked for comments on post-it notes on a series of ‘impertinent questions‘.  I was particularly interested in the outcome of the question based upon UKAD’s Objective 4: In reality, there will always be backlogs of uncatalogued archives.” Are volunteers the answer?  From the responses we gathererd, there does appear to be increasing professional acceptance of the use of volunteers in description activities, although I suspect our use of the word ‘volunteer’ may be holding back appreciation of an important difference between the role of ‘expert’ volunteers in archives and user participation by the crowd.

Read Full Post »

Gloucestershire Archives – Digital Curation from Ingest to Trusted Storage

Last Thursday I was delighted to attend the culminating workshop for the Society of Archivists‘ (SoA) funded digital curation project at Gloucestershire Archives.  As Viv Cothey, the developer employed by Gloucestershire Archives, has noted, “Local authority archivists may well be fully aware of the very many exhortations to do digital curation and to get involved but are frustrated by not knowing where to start”.  Building upon previous work on a prototype desktop ingest packager (GAip), the SoA project set out to create a proof of concept demonstration of a ‘trusted digital store’ suitable for use by a local government record office.  The workshop was an important outreach element of the project, aiming to build up understanding and experience of digital curation principles and workflow amongst archivists in the UK.  I have been involved with the management board for the SoA project, so I was eager to see how the demonstration tools which have been developed would be received by the wider digital preservation and archivist professional communities.

Others are much better qualified than me to evaluate the technical approach that the project has taken, and indeed Susan Thomas has already blogged her impressions over at futureArch.  For me, what was especially pleasing was to see a good crowd of ‘ordinary’ archivists getting stuck in with the demonstration tools – despite the unfamiliarity of the Linux operating system – and teasing out the purpose and process of each of the digital curation tools provided.  I hope that nobody objects to my calling them ‘ordinary’ – I think they will know what I mean, and it is how I would describe myself in this digital preservation context.

Digital preservation research has hitherto clustered around opposite ends of a spectrum.  At one end are the high level conceptual frameworks: OAIS and the like.  At the other end are the practical developments in repository and curation workflow tools in the higher education, national repository, and scientific research communities.  The problem here is the technological jargon which is frankly incomprehensible to your average archivist.  Gloucestershire’s project therefore attempts to fill an important gap in current provision, by providing a set of training tools to promote experimentation and discourse at practitioner level.

I’ll be interested to see the feedback from the workshop, and it’d be good to see some attendee comments here…

Read Full Post »

Public Record Office of Victoria Digital Archive (1)

Developing the Digital Archive at PROV

Australian archival theory emphasises the continuum of recordkeeping, with good records management seen as the core foundation of archival practice. This continuity is well illustrated by the Digital Archive project at the Public Record Office of Victoria (PROV).

Funding for the development of the Digital Archive was won on the back of previous successes in digital recordkeeping, beginning as far back as 1995 in the report ‘Keeping Electronic Records Forever‘ and the resulting specification of a Victorian Electronic Records Strategy (VERS).

With firm backing at a strategic level within the Victorian State government, the project appears to have flourished in spite of its complexity in terms of both the project structure and the technical implementation. In addition to staff contracted specially for the project, external contractors and consultants, PROV also seconded around a fifth of its own staff to the Digital Archive over the life of the implementation project, in order to ensure good staff buy-in and familiarity with the Digital Archive workflows. This contribution was included in annual performance appraisals for the year, and emphasised as important to personal development. Since several of the contract staff were also retained at PROV after the end of the project, knowledge of the system has remained within PROV, and the operation and maintenance of the Digital Archive is now well integrated into the general organisational structure.

Public Record Office of Victoria

Although larger than the typical local Archive Service in the UK, with 67 FTE staff, PROV is not a very large institution and it was felt important to integrate the Digital Archive into existing systems. Technical support for the ongoing operation of the Digital Archive consists of just three staff members. It was noted that OAIS was found useful in establishing a common vocabulary at the very beginning of the project, but was too high level to be of use in the tender and design phases of the project, and also views the Archival Information System as a standalone function rather than fitting into an pre-existing organisation and automated systems (such as the PROV catalogue). This integration – with the PROV archival control database and with existing workflow models for processing new accessions – proved to be one of the most challenging aspects of the whole project. Many of the processes followed in the analogue world can be streamlined in the digital, although a practice of repeated trial runs before completing a transfer has been found to work better than PROV staff attempting to correct errors in the records submitted.

The objectives were to keep digital archives for the long term, defined as up to 100 years – the project found that this concrete period was far easier for the IT contractors to grasp than a vague aspiration of keeping records indefinitely. The solution had to be cost effective over time, and to allow (online) access immediately – unlike the National Archives in Australia, the State of Victoria’s Public Records Act does not include a ’30 Year [closure] Rule’. Building from these considerations and the previous work on VERS, the decision was taken to use a single long-term format, a so-called VEO or Victorian Electronic Object. Essentially this is an XML wrapper containing the original digital object plus a normalised version (for example, PDF/A) and associated metadata to enable management of the digital record over time. This VEO is used as both a Submission Information Package (SIP) and Archival Information Package (AIP) under the OAIS model, and as the basis for the Dissemination Information Package (DIP). The whole package, which may contain anything from a single record to hundreds, each consisting of multiple encodings, is then locked and signed with a digital signature. Future users will be able to verify that each record has remained unaltered since its transfer from the government Agency which deposited it at PROV.

Long-term preservation formats are chosen on the basis of being open specifications and widely used (so that good translation tools are likely to remain available even after the format itself becomes obsolescent in the future). The current specification lists plain text, PDF (with a preference for PDF/A), JPG, JPEG2000, TIFF and MPEG-4. Interestingly, PROV have not yet begun to work with database formats, which appears to be one of the more immediate issues facing West Yorkshire Archive Service if a recent records management audit at Leeds City Council is anything to go by.

Bolstered by strong archival legislation, the PROV Digital Archive workflow pushes as much of the load back onto the depositing government Agencies as possible. Agencies are required to create the VEOs (on transfer) – software suppliers can participate in a compliance programme run by PROV to certify which products meet the VERS specifications and can generate valid VEOs. Currently five products are fully compliant, with several more close to completing the programme.

Unlike the National Archives of Australia, PROV do not maintain a ‘dark archive’. Digital accessions are securely stored on Centera file systems, which cannot be accessed by standard protocols and are continually monitored to detect corruption and repair failures. Off-site back-up copies are automatically updated. Controlling the whole system is a customised version of Documentum’s Enterprise Content Management platform – this proved costly in the short term, and is likely to continue to be a problematic and expensive aspect of the Digital Archive until such time as a public domain engine might become available to run an automated workflow on ingest.

Access to preserved digital records for the public is via the PROV website at http://www.access.prov.vic.gov.au. Different icons indicate the format of individual items – a sheet of paper for analogue archives, a camera for digitised material, a disk for born-digital. Digital archives are delivered to the user in their long-term preservation format, although the full VEO can also be downloaded and the user can manually extract other formats from the VEO. PROV decided not to require users to login to obtain digital archives – this was felt to be an audit control needed in the reading room which was not necessary in the digital world. The Digital Archive currently holds over 300000 VEOs, many of them digitised images from PROV’s own paper-based collections. Use of the PROV website has increased enormously.

Lessons learned from the project and subsequent operation of the Digital Archive included the importance of strong project management and stakeholder communications, and the need to have developers working on site rather than contracting the project overseas. The bulk sometimes encountered in archival accessions has created bottlenecks at certain points along the Digital Archive workflow. Customising off-the-shelf software is expensive. Further training is required for Agency staff who prepare the records for transfer, and simplifications to the procedures may be required.

Read Full Post »

LIFE2 Project Conference, British Library, 26 June 2008

Having read through the original LIFE project documentation, I was looking forward to the project conference for the follow-on research, LIFE2. It is all too common, unfortunately, to hear doom-laden rumours being peddled about the supposed high costs of digital preservation, often in contexts where this truism becomes a convenient excuse to avoid addressing the real challenges of digital curation and preservation. Chris Rusbridge argued in an ARIADNE article that digital preservation being expensive is simply a fallacy. For the local government archives context at least, it seems to me that there is simply insufficient evidence to either support or discredit the assertion. Would LIFE2 offer us an objective tool to assess the likely costs of developing and running a digital preservation service for local government?

LIFE2 promised a revised lifecycle costing model, including mappings to relevant digital preservation standards such as OAIS, clearer element descriptions, and a new set of case studies, including an examination of non-born digital newspaper material. This case study was designed to allow for the comparison of analogue and digital lifecycles and to begin a cost comparison.

Whilst the revised LIFE2 model is more closely aligned in terminology to OAIS, and the elements now appear in a more logical order, I admit I was disappointed that the model does not seem as transferable to the local government archives context as I had hoped. As Neil Beagrie pointed out in his presentation of the costs of curating research data, the decision to exclude infrastructure costs such as the start-up costs of building a digital repository or of maintaining a technology watch service means that the tool could not be used in business cases making the comparison case for or against curating digital material in-house or for outsourcing – the major decision facing me at West Yorkshire Archive Service – although some attempt to address this shortcoming is being made with the development of a Generic Preservation Model (GPM), which will be released at the end of the LIFE2 Project.

The newspaper case study also ran into difficulties around differing patterns of access and the problems of retrospective costing. Although the case study continued using a per entity costing model to assess the relative costs of preserving a digitised newspaper collection with a year’s analogue curation costs of legal deposit newspapers, the results, although interesting, are not truely comparable.

I had hoped I might be able to use the tool to compare the not inconsiderable costs of building and fitting out an archives building for traditional materials conforming to BS5454, with the costs of developing automated tools and digital storage and management capacity for born-digital and hybrid collections. There was discussion at the end of the day about how the LIFE project might progress in a next phase, which – promisingly – included the development of a predictive tool for costing, further case studies and scenario building, and a proposal that comparison studies are made between the costs of a shared preservation service versus an in-house digital repository.

There was also extensive discussion during the panel session about the need to demonstrate the value of digital preservation, particularly to funding bodies. The LIFE tool offers a method for digital repositories to assess costs; different kinds of value assessments are required to convince funders. The point was also made that the more significant properties of digital objects a repository attempts to preserve, the greater the cost – making me ponder on the potential for integrating the lifecycle costing models into preservation planning tools, such as PLATO.

Read Full Post »

Digital Preservation begins… in Italy?

At the DELOS Summer School on Preservation in Digital Libraries, which so far more than lives up to my expectation that it will provide an excellent overview of current and emerging digital preservation research and practices.

A comforting thought for local government Archive Services from the presentations on day 1:

• that digital preservation is as much about organisational and cultural challenges as technical ones. Something we can all start to address – now.

And also a rather more concerning one:

• the emphasis in OAIS on serving a designated community, something which is in any case hard to determine for traditional record offices with geographically defined collection policies. Priscilla Caplan also points out that the OAIS requirement that material preserved in an archival system should remain understandable to that community has not usually been part of the mandate of ordinary libraries or archives.

Read Full Post »