Evaluating Engagement Online

• Digital Impacts: How to Measure and Understand the Usage and Impact of Digital Content, Oxford Internet Institute/JISC, Oxford, 20th May 2011 (#oiiimpacts)
• Beyond Collections: Crowdsourcing for public engagement, RunCoCo Conference, Oxford, 26th May 2011 (#beyond2011)
• Professor Sherry Turkle, Alone Together RSA Lecture, RSA, London, 1st June 2011 (#rsaonline)
  

I’m getting a bit behind with blog postings (again), so here, in the interests of ticking another thing off my to-do list, are a few highlights from various events I’ve attended recently…

It was good to see a couple of fellow archivists at the showcase conference for JISC’s Impact and Embedding of Digitised Resources programme. As searchroom visitor figures continue to fall, it is more important than ever that archivists understand how to measure and demonstrate the usage and impact of their online resources. The number of unique visitor’s to the archive service’s website (currently the only metric available in the CIPFA questionnaire for Archive Services, for instance) is no longer (if it ever was) adequate as a measure of online usage.  As Dr Eric Meyer pointed out in his introduction, one of the central lessons arising from the development of the Toolkit for the Impact of Digitised Scholarly Resources has been that no single metric will ever tell the whole story – a range of qualitative and quantitative methods are necessary to provide a full picture.  The word ‘scholarly’ in the toolkit’s name may be rather off-putting to some archivists working in local government repositories. That would be a shame, because this free online resource is full of very practical and useful advice and guidance. Like the historians caracatured by Sharon Howard of the Old Bailey Online project, archivists are not good at “studying people who can answer back” – the professional archival literature is full of laments about how poor we are at user studies. The synthesis report from the Impact programme, Splashes and Ripples: Synthesizing the Evidence on the Impacts of Digital Resources, is recommended reading; detailed evaluation reports from each of the projects which took part in the programme are also available (at http://www.jisc.ac.uk/whatwedo/programmes/digitisation/impactembedding.aspx).  Many of the recommendations made by the report would be relatively straightforward to implement, yet could potentially transform archive services’ online presence – and the TIDSR toolkit contains the resources to help evaluate the change.  Simple suggestions like picking non-word acronymns to improve project visibility online (like TIDSR – at last I have understood the Internet’s curious aversion to vowels, flickr, lanyrd, tumblr and so on!) and providing simple, automatic citations that are easy to copy or download (although I rather fear that archives are missing the boat on this one). Jane Winters was also excellent on the subject of sustaining digital impact, an important subject for archives whose online resources are perhaps more likely than most to have a long shelf-life. Twitter coverage of the event is available on Summarizr (another one!).

One gap in the existing digital measurement landscape which occurred to me during the Impacts event was the need for metrics which take account, not just of the passive audience of digital resources, but of those who contribute to them and participate in a more active way.  The problem is easily illustrated by the difficulties encountered when using standard quantitative measurement tools with Web2.0 type sites.  Attempting to collate statistics on sites such as Your Archives or Transcribe Bentham through the likes of Google Scholar or Yahoo’s Site Explorer is handicapped by the very flexibility of a wiki site structure, compounded again, I suspect, by the want of a uniquely traceable identity.  Google Scholar, in particular, seems averse to searches on URLs (although curiously, I discovered that although a search for yourarchives.nationalarchives.gov.uk produces 0 hits, yourarchives.nationalarchives.gov.* comes back with 26), whilst sites which invite user contributions are perhaps particularly susceptible to false-positive site inlink hits where they are highlighted as a general resource in blogrolls and the like.

This need to be clearer about what we mean by user engagement and how to measure when we’ve successfully achieved it was also my main take-away from the following week’s RunCoCo Conference – Beyond Collections: Crowdsourcing for Public Engagement.  Like Arfon Smith of the Zooniverse team, I am not very comfortable with the term ‘crowdsourcing’, and indeed many of the projects showcased at the Beyond conference seemed to me to be more technologically-enhanced outreach events or volunteer projects than true attempts to engage the ‘crowd’ (not that there is anything wrong with traditional approaches, but I just don’t think they’re crowdsourcing).  Even where large numbers of people are involved, are they truly ‘engaged’ by receiving a rubber stamp (in the case of the Erster Weltkrieg in Alltagsdokumenten project) to mark their attendance at an open day type event?  Understanding the social dynamics behind even large scale online collaborations is important – the Zooniverse ethical contract bears repeating:

1. Contributors are collaborators, not users
2. Contributors are motivated and engaged by real research
3. Don’t waste people’s time

Podcasts of all the Beyond presentations and a series of short, reflective blog posts on the day’s proceedings are available.

Finally, Professor Sherry Turkle‘s RSA lecture to celebrate the launch of her new book, Alone Together, about the social impact of the Internet, was rather too brief to give more than a glimpse of her current thinking on our technology saturated society, but nevertheless there were some intriguing ideas which have potentially wide-ranging implications for the future of archives. One was the sense that the Internet is not currently serving our human needs.  She also spoke about the tensions between being willing to share and privacy.  Turkle asked what is democracy and what is intimacy without privacy? In response to questions from the audience, Turkle also claimed that people don’t like to say negative things online because it leaves a trace of things that went wrong. If that is true, it might have important implications for what we can expect people to contribute in archival contexts, and the nature of the debate which might take place in contested spaces of memory. Audio of the event is available from the RSA website.

Read Full Post »

Guest Post on UKOLN Cultural Heritage Blog

Today I have a guest post about my research on UKOLN‘s Cultural Heritage Blog.

Read Full Post »

Digital Culture Heresy (musings after #glamwiki)

It’s a brave person who volunteers their opinions to the lions, so if Tom Morgan can argue the case for the National Portrait Gallery before an audience of Wikimedians at the GLAM-WIKI Conference at the British Museum this last weekend [GLAM stands for ‘Galleries, Libraries, Archives, Museums’], then I reckon the least I can do in appreciation is respond in similarly (mildly) provocative manner.

As one of my previous jobs included rights clearance and dealing with publication requests for archives, I have some sympathy with the NPG’s position.  The professional curation (both custody and access) of cultural heritage materials is a complex and expensive business.  Having put in a lot of effort to looking after stuff in a responsible and professional manner, therefore, its extremely irritating (to say the least) when, as a member of staff, you discover that people are making use of these materials without much appreciation, or perhaps even without acknowledgment, of your efforts and their associated costs.  Or, as Tom Morgan put it, “We have a culture of engagement and somebody’s just driven a great big truck through it”.

But I find it harder to empathize with why an organisation would want to invoke an imposed additional layer of intellectual property rights bureaucracy in an attempt to control use of materials where the original author’s copyright has already expired.  Copyright (and here UK archivists face a more complex situation even than gallery curators, since most manuscript materials remain in copyright for far longer terms than artistic works) is frankly a right hassle to administer.  I used to breathe a sigh of relief whenever a document requested for reproduction was clearly and demonstrably out of copyright.  Unfortunately, this didn’t happen very often.  A far more likely scenario involved a lengthy research process in an attempt to trace the current rights holder, often ending with the dreaded ‘orphan work’ issue (known to be in copyright, but the rights owner cannot be identified or contacted).  From the point of view of working for a cultural heritage organisation, I cannot see that if copyright did not already exist that I would have wanted to invent it.  Copyright was hindering, not helping, what I understood as my organisation’s mission to preserve (particularly with regard to born-digital records) and provide access to historical materials.

Attempting to control the use of digitised historical content by enforcing copyright and database rights in the copies and rights management mechanisms therefore seems to me to confuse the real issue.  All the talk of copyright creates confusion by muddling up a need to recognise the organisation’s curatorial efforts with the intellectual property rights of creator(s).  This is why the Europeana Public Domain Charter, which I heard about for the first time at GLAM-WIKI, is so important.  It states that copyright protection is temporary, and that the digitisation of out-of-copyright content should not create new rights in it.  Works that are in the Public Domain in analogue form should continue to be in the Public Domain once they have been digitised.  Yet this does not, as far as I can see, necessarily have to imply that users are granted a free-for-all licence to take what they please from cultural heritage organisations without acknowledgement.  There might even still be a requirement to pay some kind of financial recognition towards the costs involved in digitising fragile materials and in making them available online.  But the emphasis shifts away from paying for a product (copies of the actual cultural heritage content – documents and objects) to acknowledging the costs of the process of providing online access to digital culture.

This shift in emphasis helps to open up an issue – access – in which the organisation and its users have a mutual reinforcing interest, unlike the position over content where the motivations of the organisation and its users tend to be conflicting.  For I would dispute in fact Tom Morgan’s claim that users (Wikipedians in the context in which he was talking) and organisations agree upon the value of cultural heritage materials.  In his Friday keynote, Cory Doctorow exhorted Wikipedians to value use above all else [incidentally, as an archivist, I think that statement is problematic anyway: if current use is primary, how do you ensure that what is preserved in the present for the future is not just what’s currently fashionable?].  Organisations, on the other hand, have a custodial mandate to preserve content, and hence will always seek to control the contexts of its use.

Unlike the disagreements over paying for content, therefore, which tend ultimately to reduce into organisation vs users, re-focusing upon the process of access invites consideration of the points of commonality between audience and organisation.  Both can agree on the importance of access (Tom Morgan said “We’re not shy about access, we really mean it”; one Wikipedian reflected upon the benefits of working with GLAM organisations as “Access to expertise, information and resources that are normally difficult to get to”).   This consensus seems a good starting point for a discussion about how organisation and community can work together to make access more efficient.

In a physical environment, the onus is on the organisation to provide this access infrastructure (gallery space or archive searchrooms, or whatever).  But in the online world, the notion of the passive user or audience can be reconceptualised so that this responsibility for access can be shared.  I use the word responsibility advisedly, for there are responsibilities as well as benefits on both sides of the organisational boundary.  For the user and would-be collaborator, the responsibilities include – as Wikipedian The Land puts it – avoiding “treating it as a hit-and-run raid on image libraries”.  To this end, it was interesting for me to learn at GLAM-WIKI about Wikipedia’s principles, the Five Pillars and that all 6 UK Creative Commons licences require attribution (which is obvious when you look at the licences page, but somehow I’d not cottoned onto it).  For the GLAM, the benefits of opening up access to cultural heritage in a networked environment include wider distribution and re-use of the organisation’s holdings, possibly access to levels of technological expertise which might not otherwise be available to them, and, according to Cory Doctorow at least, the harnessing of public goodwill.  But the responsibilities must include a commitment to greater transparency about the locus of costs in providing digital access to content, and a commitment to keeping these access costs to a minimum.

All of which brings me onto my final heresy, one about digitisation itself.  At the Collections Trust Expert Panel on the Digital Content Supply Chain the week before GLAM-WIKI, Nick Poole gave us a nice little introduction to his supply chain concept; an access process model, in effect.  He explained that digital preservation incorporates both input and output functions – money and resources go in, use and re-use should be the output objectives.  So, as devil’s advocate, I questioned whether the problems that the cultural heritage sector faces in ensuring the sustainability of digitised content might in part be caused by setting the bar too high at the point of entry into the digital supply chain.  In other words, why do sector professionals insist on creating high resolution, large file-size images which we consequently struggle to maintain because of a lack of storage capacity, and which in many cases are all too rarely requested for use?  Here I agree with Tom Morgan at GLAM-WIKI: low resolution copies are adequate for the majority of online usages.  So why do we insist on creating high resolution copies when only a tiny proportion of these will ever be required for publication or other high quality reproduction?  This was rather an off-the-cuff remark, and as my colleagues on the Expert Panel pointed out, there are undoubtedly times when you wouldn’t want run the risk of having to repeat the digitisation process – the rigmarole of conserving and digitising the enormous tithe maps for Tracks in Time might be one example; I guess if you are commissioning specialist photography of a steam locomotive, say, or any large 3D museum piece, this might be another exception.  But on reflection, I do think that the GLAM sector’s lemming-like adherence to gold standard digitisation norms derived from the NOF-Digitise technical standards (now nearly 10 years old) could take some critique.  Not least perhaps because admitting that high resolution digitisation might not be such a holy grail when it comes to encouraging use and re-use of cultural heritage opens the field for community participation in digitised content creation, as part of the shared access paradigm.  Rather than – or perhaps in complement to – investing in expensive professional digitisation, why not encourage users themselves take some of the strain, and endeavour to capitalise upon surely thousands of images already ‘out there’ of our public domain cultural heritage materials?

Other random highlights/thoughts resulting from GLAM-WIKI which I can’t fit into this narrative:

• An inspiring collaboration between the Dutch Nationaal Archief and Wikimedia Nederland
• A remark from Liam Wyatt (@wittylama) about local history not fitting well into the Wikipedia structure which made me ponder whether this is a gap that local archives would be well placed to exploit
• Countering the air of unfettered enthusiasm for all things wiki with a bit of considered scepticism about participation rates in the V&A Museum’s wikipedia collaborations, and wondering whether the return on investment for the organisation (and its partners – there seemed to be a great long list – Collections Trust, MLA, Culture24…) is really worthwhile.
• Neil Wilson’s presentation about the British Library’s data release of bibliographic metadata as CC0 (all copyrights waived), which included the remark that the Library’s metadata strategy tries to move away from using library-specific standards such as MARC towards more cross-domain XML models, for example Dublin Core and RDF.  It struck me that this isn’t a discussion much heard in archival spheres, but maybe I’m wrong?

Presentations from GLAM-WIKI are gradually being added to the slideshare group at http://www.slideshare.net/group/glamwiki.  Enjoy!

Read Full Post »

ECDL 2010 – Day 1 (#ecdl2010)

I am extremely lucky to have been offered a student place helping out at ECDL 2010, the European Conference on Research and Advanced Technology for Digital Libraries. The following are the highlights from day 1 of the conference for this archivist let loose in the virtual stacks:

Susan Dumais‘ keynote presented recent Microsoft research into the temporal dynamics of the web, analysing both changes to content and how people revisit web pages, checking for new content or looking for previously found information. She argued that the current generation of web browsers offer only a static, snapshot view, and went on to illustrate a browser plugin called DiffIE which highlights what has changed on a web page since the user’s last visit. She also presented some initial evaluation of this tool, which indicated that although perceptions of revisitation frequency remained constant, in practice users of the plugin increased their revisitation rate. There are lots of potential applications for this kind of tool for archives – from the presentation of web archives to the user interactions/annotations/ratings examples that Dumais herself gave. She also spoke about the implications of her research to the ranking and presentation of search results, illustrating how the pertinency and hence relevancy of certain terms can decline over time – for example, a user searching for ‘US Open’ this week is more likely to looking for information on the tennis grand slam than the golf tournament. Again, there are some interesting implications here for archival catalogue and document search systems.

Christos Papatheodorou from the Ionian University on Corfu spoke about the mapping of disparate cultural heritage (archives, museums, libraries) XML-based metadata schema to the CIDOC CRM ontology, and went on to describe the transformation of XPath queries submitted to a local (XML) data source into equivalent queries suitable to be submitted to other data sources, via the CIDOC CRM ontology. Having travelled up to Glasgow on the sleeper, arriving at 7 in the morning, I confess I got a bit lost in the technicalities from this point onwards, but the basic idea is to use CIDOC CRM as a mediator between disparate cultural heritage sources marked up in different XML schema. There was an extended worked example using EAD, which was nice to hear. In general, it has been interesting to observe a large number of papers at this conference which report experiments based upon data from cultural heritage rather than scientific domains. All of which tends to reinforce my thoughts after the Society of Archivists’ Conference about attracting technology experts to work in the archives sector: cultural heritage data is complex and thus, it seems, fascinating and intrinsically motivating to work with. We should be more proactive about promoting archival data to this kind of digital research community.

I’d been particularly looking forward to the paper on User-Contributed Metadata for Libraries and Cultural Institutions, although this turned out to be a Drexel University re-working of the Library of Congress flickr Commons experience, albeit concentrating more on user comments and less upon tagging. I was not quite comfortable anyway with the a priori categorization of comments described in the paper (into 1. personal and historical 2. link out (eg to wikipedia) 3. corrections and translations 4. link in (eg adding images to flickr groups) – seems to me that category 1. includes a particularly wide range of possible comment types), plus all the things I wanted to ask about seemed to be listed as ‘future research’. These include a fuller categorization, exploring motivations for adding comments, the presentation of comments in the user interface, and librarians’ (or archivists’) role in moderating user interaction.

I also enjoyed a couple of papers which presented ideas to do with improving information visualisation and user judgement using colours, layout and social navigation, all of which have some potential relevancy to the question of how best to present user-generated content.

Research and Advanced Technology for Digital Libraries, Proceedings of the 14^th European Conference, ECDL 2010, Glasgow, UK, September 2010 is published as Lecture Notes in Computer Science 6273, available via SpringerLink, for those of you who have access.

And I have travelled twice on Glasgow’s baby underground train 🙂

Read Full Post »

Web 2.0 in Local Government

Reading about the Foreign Office and the Treasury’s use of YouTube (see http://www.youtube.com/hmtreasuryuk and http://www.youtube.com/user/ukforeignoffice), government department bloggers, use of RSS and Flickr (for example, http://www.flickr.com/photos/foreignoffice/) in the 30 Year Rule Review got me wondering about the use of Web 2.0 services in West Yorkshire’s local authorities.

So I decided to find out!  The results of my search are, I think, quite interesting. 

All of the five Metropolitan Councils in West Yorkshire make use of RSS on their websites, except, apparently, Bradford.  The Council Press departments are getting into Twitter too, to keep local people up to date with current events in Wakefield, Kirklees and Leeds. 

Blogs were harder to track down – I’m sure there must be plenty of bloggers working in local government, but it seems they don’t want to identify themselves!  http://www.newgarforthlibrary.blogspot.com/ is an example of a blog being used to generate support and give updates on a council building project.  http://www.avhlblog.com/, is written by four members of staff at Aire Valley Homes, one of the arms-length management organisations (ALMOs) in Leeds, managing housing on behalf of the Council.  Blogging doesn’t appear yet to have had the take-up amongst local councillors as it has amongst MPs, although Councillor Clive Hudson’s Cleaner Greener blog is an interesting example, hosted by blogspot under the wakefield.gov.uk domain.  Local councillors’ websites (for instance, the Kirkstall Councillors in Leeds) are usually viewed as political activity, and separated from the ‘official’ council website, so this is an unusual development, which potentially raises all sorts of questions about responsibilties for the comments posted and for the longer-term maintenance of the content.

YouTube doesn’t seem too popular at present, although I did track down a Leeds Initiative channel.  What I did find were plenty of YouTube videos posted by members of the public which were highly critical of the local councils.  Perhaps the councils themselves should consider raising their YouTube presence?

There were a few examples of council Flickr sites, although not as many as I was expecting to find.  One of the most extensive is Kirklees Council’s Economic Development Service’s photostream, although Leeds cultural services departments are also experimenting – but not much content yet – see http://www.flickr.com/groups/leedsmuseumsandgalleries/, http://www.flickr.com/photos/30193899@N04/ and http://www.flickr.com/photos/leedslibraries/.  Aire Valley Homes again showed up their Web 2.0 credentials with http://www.flickr.com/photos/avhl. 

There are also a few dabblings with facebook groups – though you could hardly say that the official facebook groups have taken off in a big way.  Kirklees Council apparently has an overwhelming 24 fans (though, to be fair, I did find also find a posting which intimated it had not been properly advertised as yet), Calderdale Council just 9 fans!  As with YouTube, there were plenty of external groups in evidence with some kind of grudge to bear against the various councils.  There were a few examples of council staff facebook groups – Kirklees Council staff with 91 fans, or Pugneys Country Park in Wakefield, for ‘staff new and old’.  Most of these staff groups seem to be unofficial.  Occasionally the messages they give out leave something to be desired, as with one (closed) council staff group profile which reads “you don’t need to be paranoid and leave the group if you think all facebookers can see it and what we are talking about.  The group can ONLY be viewed by us members.”  Hmm…

Read Full Post »

Muse(um)-ings on Technology

Melbourne

As a gentle introduction to the serious business of my Fellowship, and whilst it still feels like I have left my brain behind somewhere, I’ve spent this afternoon at the Melbourne Museum. The visit was the result of a chance last-minute connection, but fitted into a theme I shall also be looking into at the Powerhouse Museum in Sydney and the Computer History Museum near San Francisco.

One of the challenges for local archives in the UK will be to encourage digital deposits. Few digital preservation projects have addressed this issue (Paradigm is a notable exception), mostly because it isn’t really a problem where there is strong archival legislation or other sticks in place to encourage deposit (such as deposit being a requirement of research funding).

Another problem is that potential depositors don’t seem to see computer-generated records as archives. For them, archives = old stuff, and computers = new stuff. At WYAS, I’ve been doing some research on the first computer purchased by the West Riding County Council in 1957 (see the photo in the blog header) as one possible way of addressing this problem, hoping that people will be interested and surprised to learn that computers have been around (even in local government!) for fifty years.

Could we as archivists learn anything from the way in which the museums profession markets technology for a non-specialist audience? Is there anything we could take from this to hook in potential depositors of digital archives?

Well, it turns out that the Museum Victoria (of which Melbourne Museum is a part) don’t display much of their historical technology collection! However, the CSIRAC, Australia’s first computer and the only complete first-generation computer still intact anywhere in the world, is on display. A (modern) computer program simulates the look and sound of the computer in action, and there is a short video showing reminiscences from those who worked with the computer, some fascinating contemporary footage, and even some music programmed on the CSIRAC.

Melbourne’s Biggest Family Album is an example of what we in the UK would call ‘community archives’ and makes extensive use of Web 2.0 technologies. More relevant perhaps to the digital preservation topic, the Visitor’s Photo Album part of the online exhibition links to the Melbourne Museum group on Flickr and allows visitors to share their photographic experiences of the exhibition. I also had a tour of the museum looking at the way ICT is used in presenting the exhibits, and discussed how Web 2.0 tools can be used to promote usage of the collections, and even bring additional visitors into the museum. Melbourne Museum are experimenting with podcasts by the curators and staff blogs. We also discussed how social tagging can be used in conjunction with traditional collections management databases to encourage visitor engagement – see STEVE – The Museum Social Tagging Project for a wealth of research into the topic. I was also introduced to the Museum’s Archivist and Records Manager, and looked at how the Museum structures and manages its own administrative records, both paper and digital.

Read Full Post »