Chapter 8

The digitisation of New Zealand’s research, heritage and culture

Nigel Stanger

Over the last 25 years, it has become possible to digitise and store an ever-increasing amount and variety of material. New Zealand has been one of the leaders in this area, with early initiatives such as the New Zealand Digital Library in the mid-1990s and the more recent Digital Content Strategy promoting the idea of digitising New Zealand’s research, heritage and culture and making it available online. The government is committed to ensuring ‘New Zealand will be a world leader in using information and technology to realise its economic, environmental, social and cultural goals’.1 They see New Zealanders as world leaders in using information and technology to build globally connected science and technology research communities.

A key benefit of digitising research, cultural and heritage material is improved accessibility. It can be difficult and laborious to find specific items in traditional ‘hard copy’ collections, whereas digital collections can be quickly and easily searched. They can also be made available via the Internet to a much larger audience than was previously possible. Digitisation also removes the access bottleneck arising from there being few physical copies of an item, as many people can access the same digital item simultaneously. Finally, digitisation helps us to preserve fragile historical material by reducing the need for physical access, and hence the likelihood of further physical damage or even loss.

The need to store and manage digital collections of this nature has driven the development of digital libraries and repositories of various kinds, including the aforementioned New Zealand Digital Library. More recently, the launch of the government’s Digital Strategy in 2005 resulted in a nationwide proliferation of digital research repositories at New Zealand tertiary institutions, and ultimately led to the development of the Kiwi Research Information Service (KRIS) by the National Library of New Zealand. These developments have made New Zealand’s research readily available to the wider world.

The same technologies used to build these institutional research repositories are also now being applied in non-academic areas. In 2006, the Cardrona Online Museum was launched, with the aim of storing and making available heritage materials relating to the Cardrona district. The launch attracted strong interest and has led into an ongoing project to develop a similar repository for the Central Otago region. In parallel, the Horowhenua Library Trust and Katipo Communications Ltd, developed the Kete software to facilitate online community collaboration, and recently, the National Library began to harvest and index content from New Zealand websites for its DigitalNZ project.

In this chapter, we will examine these developments, their impact on the dissemination of New Zealand’s research, heritage and culture, and look forward to future developments in this area.

The early days

Project Gutenberg, launched in 1971, was arguably the first digital library in the world, at least in the sense of providing written material in digital form; indexing and discoverability came later as technology advanced in these areas. The project’s aim was to ‘make information, books and other materials available to the general public in forms a vast majority of the computers, programs and people can easily read, use, quote, and search’. They focused on public domain literary and reference works, which were digitised into 7-bit ASCII text to provide maximum portability. They achieved their original goal of 10,000 digitised works in 2001. The library currently contains over 30,000 works, with more being added all the time.

Despite this early start, the limited capacity of available storage devices over the subsequent 20 to 25 years made it difficult to realise a practical digital library. Project Gutenberg itself started with texts of only a few kilobytes. By contrast, the text of a typical 300-page book is about a megabyte (illustrations would increase this considerably), and a modern-day PDF of a technical report with figures typically occupies a few tens to hundreds of kilobytes, depending on the format of the embedded images. It would only take a few thousand of these PDFs — quite a feasible number for a moderately sized tertiary institution — to consume a gigabyte of storage. This may not sound very large by today’s standards, but consider that in 1990 a typical hard drive cost about US$9 per megabyte. A gigabyte of 1990 storage would therefore have cost the better part of US$10,000, which was clearly prohibitive for all but the largest organisations. Fortunately, the capacity of hard drive storage has increased exponentially for many years, doubling roughly every 12 months, with a corresponding exponential drop in cost per unit of storage. The first gigabyte capacity drives appeared in 1995 for about US$800, or about 80c per megabyte. In 2010, a one terabyte drive — 1000 times the capacity! — can be had for just over US$100, or about 0.0001c per megabyte; a drop in cost of almost seven orders of magnitude since 1990.

It is probably no coincidence, then, that the first significant efforts towards building digital libraries coincided with the arrival of relatively cheap, high-volume mass storage devices in the mid-1990s. Several major projects were initiated between 1994 and 1996 at universities across the United States, including Carnegie Mellon, the University of California (Berkeley and Santa Barbara), Stanford, the University of Illinois and the University of Michigan. In New Zealand, the Department of Computer Science at the University of Waikato launched the New Zealand Digital Library (NZDL) project in 1994. The initial aim of the project was modest: to digitally store a collection of New Zealand computer science technical reports, similar to existing US repositories of the time like CS-TR (Computer Science Technical Reports) and arXiV.

The project quickly expanded from this foundation to incorporate other material, and began to develop into a general infrastructure for creating digital libraries of all kinds. Many other collections were added alongside the original computer science collection, including mirrors of Project Gutenberg and the Oxford Text Archive, and the Indigenous Peoples collection. They also expanded from primarily textual documents to incorporate other types of content, such as images, video and music. They also undertook considerable research into techniques for browsing and searching digital library collections, as detailed in the NZDL Publications collection.

The NZDL project culminated in the release of the Greenstone digital library software in 2000. This software provided a construction kit for digital libraries, with an extensible plug-in architecture. Since its release, Greenstone has been used to create many digital libraries around the world.

Today, the original NZDL at Waikato has nearly 60 collections, spanning computing, literature, history, health, education, environment, food and nutrition, social and community development, and cultural heritage. Experiments are being carried out with content in other languages, including Chinese and Arabic. Of particular interest from a New Zealand perspective are the Niupepa collection of historical Māori newspapers published between 1842 and 1932 and the Hamilton Public Library Youth Oral History collection of youth experiences in the 1930s, 1940s and 1950s.

The latter two collections are prime examples of material that would be difficult to find and utilise in non-digital ‘hard copy’ form (e.g. paper, audio tape or film). Traditionally, such collections were stored in hard copy somewhere in the shadowy depths of a library, and access could often only be gained on direct request, effectively requiring potential users of the material to already know that it existed. The library would often have held only one copy, further limiting access to the material. Providing collections in digital form breaks the first constraint by making their contents easily discoverable and indexable by Internet search engines (thus enabling serendipitous discovery without prior knowledge), and by making it accessible from anywhere in the world. It can also break the second constraint by enabling multiple users to access the material simultaneously, or even to make their own digital copies, if permitted.

Putting these two collections into the New Zealand Digital Library has therefore revealed and opened them up to a much broader audience — both domestic and international — than would have otherwise been possible. (Indeed, such collections can be accessed from some surprising places, as we will see when we discuss the Cardrona Online Museum.) These kinds of arguments are also applicable to the outputs from academic research, and form the core of what is known as the open access movement. This movement has driven the construction and adoption of institutional research repositories worldwide, so it is now an opportune moment to look at these developments.

The rise of the institutional repository

The concept of the institutional research repository has its roots in the open access movement. In many ways, open access is to peer-reviewed research outputs what open source is to software; that is, open access literature is ‘digital, online, free of charge, and free of most copyright and licensing restrictions’. In other words, open access removes price barriers such as subscription and licensing fees, and also permission barriers such as copyright and licensing restrictions. This makes research outputs more readily available to those who might not otherwise be able to access them (e.g. due to budgetary constraints); makes publications easier to find (as they are easily searched for and downloaded over the Internet); and broadens the audience beyond the subscription base of a single journal. Of course, open access is not limited to just journal articles; it can be of particular benefit for other types of publication such as conference papers or technical reports, which can sometimes be difficult or even impossible to obtain in printed form.

In order to make research outputs freely available online, we clearly need somewhere to store them: a repository of some sort. Researchers in physics and mathematics realised this very early, and in 1991 founded arXiV, which today contains over half a million articles in physics, mathematics, computer science, quantitative biology, quantitative finance and statistics, all of which are freely available. arXiV is generally considered the first significant open access digital repository, and was a major inspiration for the open access movement that followed it a few years later. During the 1990s several similar repositories were launched, including GenBank in 1992, and Research Papers in Economics (RePec) and PubMed in 1997.

Stevan Harnad effectively laid the groundwork for the modern institutional repository with his ‘subversive proposal’ that researchers should self-archive their research outputs. That is, in addition to publishing them through normal channels, they should also store a freely available digital copy in an online digital repository.1 The intuitive leap Harnad made was to suggest that such repositories could be hosted by the researcher’s institution, hence the term ‘institutional repository’, although at the time, this was hampered by a lack of readily available software for managing such repositories. This lack was ultimately addressed with the release of GNU EPrints by the University of Southampton in 2000 and DSpace by MIT in 2002. These two are the most widely known and used open source packages for managing digital institutional repositories, but there are several other similar open source (e.g. Fedora Commons2) and commercial options (e.g. Digital Commons by The Berkeley Electronic Press).

The arrival of such a plethora of software options, along with practical solutions to issues such as copyright, has, since about 2004, spurred a rapid growth in the number of institutional repository installations. The Directory of Open Access Repositories (OpenDOAR) started keeping statistics on open access repositories in early 2005 and now lists over 1500 repositories worldwide. The actual installed number of repositories is greater than this, however, as OpenDOAR only lists those repositories that have been reported to it by users; that is, it does not automatically discover new repositories.

Just as the international growth in open access repositories began to gather momentum, the New Zealand government launched its previously mentioned Digital Strategy. In parallel with this, the National Library of New Zealand set up an expert working party with representatives from across the research sector to investigate the feasibility of establishing a national repository for New Zealand’s research outputs. The ultimate intent was to improve access to New Zealand’s research outputs, by collaborating with institutions to stimulate the creation of research repositories.

While these efforts were proceeding, two senior University of Otago staff undertook a study tour of ‘digital challenges’ facing universities in the United States. Their report provided the immediate impetus for a digital repository project within the University’s School of Business. Following discussion with network contacts in the National Library of New Zealand and Australia, the School of Business approved funding for a pilot project. Work began on November 7, 2005, with the goal of building a proof of concept open access repository for the School.

The pilot repository was fully implemented and went live within 10 days of assembling the project team (on November 17, 2005). Unbeknownst to the development team, similar projects were already well advanced at other institutions around the country, but the rapid implementation and straightforward goals of the Otago pilot enabled it to leapfrog the rest of the field to become the first publicly available institutional repository of its type in New Zealand (as acknowledged by Professor Arthur Sale of the University of Tasmania, in his keynote speech at the Institutional Repositories Symposium held by the National Library in Wellington on November 23, 2005). This was possible because of a very clear brief to ‘prove the concept’, rather than getting bogged down in details of architecture, policy and procedure. The prototype was also cheap to build, using the free EPrints software, existing hardware and the equivalent of two full-time staff. The Otago implementation showed that creating a successful digital repository need not be a complex or expensive proposition. Six months after the initial launch of the pilot, and with relatively little modification, it was adopted as the official repository for the School of Business, and remains so to this day.

An exciting outcome of the School of Business repository was the ability to make available material that might otherwise be difficult or impossible to access, and thus increase the likelihood of it being cited. About three-quarters of the 735 items in the repository as of March 2010 are documents that might not otherwise be as readily accessible, such as theses, dissertations, seminars, and departmental working or discussion papers. The full text of most of these items is also indexed and searchable through major Internet search engines such as Google, usually within only a couple of days of being made publicly available. The impact of this is evident: there have been over a quarter of a million document downloads since the repository was launched, with nearly 90% of these originating from 205 other countries.

The success of the pilot also attracted attention from other parts of the University. In early 2006, Te Tumu, Otago’s School of Māori, Pacific and Indi­ge­nous Studies, expressed an interest in implementing a repository for their specific needs. They were particularly interested in the use of a digital repository for disseminating their research and other work, as there are relatively few ‘official’ outlets for their discipline. In addition to the usual academic papers, Te Tumu wanted to store multimedia content such as images of traditional crafts and artwork, and video clips of performances. The extensible architecture of EPrints made it simple to cater for these requirements.

Te Tumu repository was implemented in less than a month, and was officially launched on May 3, 2006, becoming the first digital repository for indigenous studies in New Zealand. Interest in the repository is evident with over 54,000 downloads from 134 different countries as of March 2010. The repository currently contains 65 items, including articles, theses, images and videos.

Similar repositories at other tertiary institutions around the country quickly followed those at Otago. Today, all New Zealand universities and many polytechnics have some form of institutional repository, sometimes several. These repositories adopt a broad range of approaches, including:

• Repositories that only accept specific types of research output, e.g. those in the Australasian Digital Theses programme;
• Hosted repositories like coda, which host material from a consortium of tertiary institutions, in order to reduce costs;
• ‘Showcase’ repositories like the Otago University Research Archive, which only include high-quality research outputs, intended to showcase the research output of the institution;
• ‘Working’ repositories like the Otago School of Business repository, which accept a wide range of both quality-assured and non quality-assured research outputs, such as internal discussion papers or technical reports; and
• Repositories relating to a specific subject area, such as Te Tumu repository.

These repositories employ both proprietary (Digital Commons) and open source (DSpace, EPrints) software. Most of the repositories running DSpace are part of a hosted solution developed by the Library Consortium of New Zealand (LCoNZ) and based at the University of Waikato.

2005 and 2006 saw another significant milestone in the development of institutional repositories within New Zealand, when the Tertiary Education Commission announced funding for projects to build institutional repository infrastructure. Three significant projects were funded in this area:

• Coda (mentioned above) produced a hosted repository service for tertiary institutions, based on Digital Commons.
• Institutional Repositories Aotearoa set out to foster the development of a national network of interoperable, open access research repositories. This ultimately led to the successful deployment of institutional repositories at the Universities of Auckland, Canterbury and Waikato.
• Open Access Repositories in New Zealand, shared some goals with Ira, but also aimed to support individual institutional initiatives by building a knowledge base of examples and best practices, thus lowering barriers to entry; and to promote a ‘common road map’ for institutional repositories within New Zealand to encourage institutions to join a national network.

A major spinoff of these projects was the development of the National Library’s Kiwi Research Information Service (KRIS), which was initially launched in January 2008 as the National Research Discovery Service. This is an aggregator that regularly harvests metadata from all New Zealand institutional research repositories, and provides a single search interface. This means that anyone wanting to locate New Zealand research outputs can go to a single location and perform a targeted search across all New Zealand repositories, rather than searching each individually, or relying on a general search engine like Google, which would by default produce a less-focused set of results.

Digital repositories are now well established in New Zealand’s academic sector as a solution for making research outputs publicly accessible. However, this is not the only sector that can benefit from use of digital repositories. Communities have large amounts of historical and heritage material that is often not readily accessible even to those within the community itself, let alone the wider world. A digital repository is an excellent tool for making much of this kind of material available, and there have been several recent and exciting developments in this area in New Zealand.

Community and heritage repositories

For centuries, individuals have collected and preserved all kinds of community and heritage materials, which have provided valuable historical insights into the everyday lives of people around the world. Often these collections and records can have unexpected uses. For example, in a recent study, historical photographs, notebooks and specimens collected and preserved by the public were used to document the environmental effects of increasing temperatures in the Boston area over the previous 100 years.

Many such collections are donated to libraries or museums that have the professional capability to preserve and curate them, but many are also maintained (or neglected) within families or communities. We noted earlier that non-digital collections already suffer from problems of accessibility. Hard copy historical collections housed within libraries and museums are at least accessible to some extent, depending on how fragile or unique they are, but there is very little hope of general accessibility to historical material stored in the community, especially when it is not in digital form.

This has led to the idea of digitising historical material to provide greater access. For example, the UK National Archives digitised the Domesday Book and made it available online. In New Zealand, the National Library launched the Matapihi project in September 2004. This is effectively a cultural analogue of KRIS that provides an aggregated view and search of online collections from several New Zealand cultural organisations, including the Alexander Turnbull Library, Archives New Zealand, the New Zealand Film Archive, Te Papa, the Hocken Library and the Auckland War Memorial Museum. It currently provides online access to over 220,000 items, including images, documents and videos.