After Christmas I tried to reduce my digital pile of recent articles, conference papers, presentations etc. on digital preservation. Interesting initiatives (“a pan European AIP” in the e-Ark project:  wow!) could not prevent that after a few days of reading I ended up slightly in despair: so many small initiatives but should not we march together in a shared direction to get the most out of these initiatives? Where is our vision about this road? David Rosenthals blog post offered a potential medicine for my mood.

He referred to the article of Richard Whitt “Through A Glass, Darkly” Technical, Policy, and Financial Actions to Avert the Coming Digital Dark Ages.” 33 Santa Clara High Tech. L.J. 117 (2017).  http://digitalcommons.law.scu.edu/chtlj/vol33/iss2/1

Mr Whitt works for Google and we have seen on several occasions in the last years that Google finally is interested in digital preservation. Why should Google be interested in digital preservation? Well: if the results of the searches in Google are no longer there – the digital objects -, the fundament of their activity is gone. As Richard Whitt puts it “there is a viable argument that the total value of  the Internet and the World Wide Web in particular, declines significantly in a world without digital preservation”(p. 225).

The challenges

Whitss article is more a booklet about digital preservation than an article  (144 pages!) and gives an up to date overview of the broad range of topics and challenges related to our profession. And he did a good job there!

Read Davids blogpost for some additions and corrections and further thoughts. I would like to add to this that Whitts view is often an American one,  as are many of his sources.  For example the certification tool TRAC (p. 164) is succeeded by the ISO standard 16363 and in Europe this is part of a pyramid  (the European framework) of auditing instruments (DSA, DIN/nestor, ISO), which does not seem to  work in the same manner in the US.  The chapter on copyright describes the US situation and is of less use for Europeans and other parts of the world.

Apart from that he gives a thorough overview of the current state of affairs: an overwhelming and complex set of challenges.

The cure

But the main part of the article is a description of a possible cure to decrease the amount of despair and to help bringing digital preservation forward.  A plea for an organized approach, to create what is called “a deep infrastructure”. To avoid what is often described as the digital dark age.

Whitts ideas are inspired by how the Internet works. He combines the elements in the digital life cycle with an extended OSI model (version Nemeth) in which not only software and hardware layers are present but a Political and Financial Layer were added to it. This results in a model “that helps us uncover the full complexity, so as to better understand and work effectively with it.”

Richard Whitts model describing where preservation challenges need to be solved

In order to build this “deep infrastructure” we need to collaborate and to get organized. The last chapter describes potential areas of collaboration and partners, without summing up concrete organisations.

But could not the existing preservation organisations initiate some activity? Like the Open Preservation Foundation, the Digital Preservation Coalition, nestor, the NCDD,  the IIPC, the NDSA just to name a few (apologies for the European flavour in it). Is not there a challenge for these groups to discuss this framework and start collaborating on different areas to bring digital preservation forward?  A suggestion for iPRES 2017 perhaps?

In my previous blog post I addressed the detection of broken audio files in an automated workflow for ripping audio CDs. For (data) CD-ROMs and DVDs that are imaged to an ISO image, a similar problem exists: how can we be reasonably sure that the created image is complete? In this blog post I will discuss some possible ways of doing this using existing tools, along with their limitations. I then introduce Isolyzer, a new tool that might be a useful addition to the existing methods.

Checksums

A number of techniques exist to verify a newly created ISO image. A seemingly obvious solution would be to do a checksum comparison on both the ISO image and the physical carrier. For instance, the following will work on any Linux system:

md5sum myimage.iso
md5sum /dev/sr0

The first line computes an MD5 checksum from the ISO image; the second line repeats this for the physical carrier. This method is not completely fail-safe. In some tests I did over a year ago, I ran into a a very strange issue where my attempts to image a CD would sometimes result in incomplete reads, and, as a result, truncated ISO images. The problem was most likely caused by faulty hardware (the machine on which I ran those tests more or less died shortly afterwards). Most worryingly, the machine would sometimes return incomplete data, both while creating the ISO image as well as during the subsequent checksum calculation on the physical carrier. The result of this was that the computed checksums were identical in both cases, which meant that the image passed the checksum quality check, even though it was incomplete!

Isovfy

The popular cdrtools library includes a tool called isovfy. Its man page describes it as follows:

isovfy is a utility to verify the integrity of an iso9660 image. Most of the tests in isovfy were added after bugs were discovered in early versions of mkisofs. It isn’t all that clear how useful this is anymore, but it doesn’t hurt to have this around.

I already commented on this tool in an earlier blog post:

The documentation of the tool isn’t very clear about what specific checks it performs. In one of my tests I fed it an ISO image that had its last 50 MB missing (truncated). This did not result in any error or warning message! Most of the reported isovfy errors that I came across in my tests simply reflected the file system on the physical CD not conforming to ISO 9660 (this seems to be pretty common).

You can try this yourself by running isovfy on the following two ISO images:

• This is a small (350 kB) ISO image that is intact
• Here’s the same image with most of its data truncated (the image is only 48 KB)

I ran both images through isovfy (version 3.02a06); both resulted in the following output:

Root at extent 17, 2048 bytes
[0,0]
No errors found

This demonstrates that isovfy is not very useful for detecting truncated ISO files.

Digging into the specs

At this point I decided it was time to start digging into some specs. The ISO 9660 page on the OSDev Wiki gives a good explanation of the internal organisation of an ISO 9660 image. From this I learnt that the Primary Volume Descriptor (which is a data structure that is present on all ISO images) contains two interesting fields:

• Volume Space Size, which is the "number of Logical Blocks in which the volume is recorded";
• Logical Block Size, which is "the size in bytes of a logical block".

In theory, multiplying both figures should give the expected size of the ISO image, and this would provide a useful way to check if data are missing. To test this, I wrote a Python script that parses an ISO’s Primary Volume Descriptor fields, calculates the expected file size and then compares this against the actual file size. Running the script against some 20 ISO images I had lying around showed that for 7 files the expected size was indeed identical to the actual file size. For most images, the actual size turned out to be marginally larger than expected (typically about 300-600 kB). For 3 images, the actual size was about twice the expected size. Digging deeper, I found out that these were hybrid images that contain an Apple partition on top of the ISO 9660 file system. According to this Wikipedia article, these hybrid discs come in two varieties:

1. Hybrid discs that contain an Apple Partition Map (located at 512 bytes into the disc/image).
2. Hybrid discs without a Partition Map. These contain a Master Directory Block (located at 1024 bytes into the disc/image).

In my case all of the 3 hybrid images turned out to be of the first category. Using the information here and here I was able to add detection of such hybrid images to my code, as well as a simple parser for the ‘zero block’ structure that contains two fields that define the partition’s size: Block Size and Block Count. For my hybrid images, multiplying both figures resulted in a value that was close to (but again marginally smaller than) the actual file size.

Finally, I also added detection of the second hybrid disc category (no Partition Map, but Master Directory Block). The Master Directory Block also contains Block Size and Block Count fields that allow one to calculate the size of the file system.

Isolyzer

I wrapped up the results of the above analyses into Isolyzer, which is a dedicated (Python) tool for checking the size of an ISO image. What it does is this:

1. Locate the image’s Primary Volume Descriptor (PVD).
2. From the PVD, read the Volume Space Size (number of sectors/blocks) and Logical Block Size (number of bytes for each block) fields.
3. Calculate the expected file size as ( Volume Space Size x Logical Block Size ).
4. If the image contains an Apple Partition Map, read the Block Size and Block Count fields from the ‘zero block’
5. Calculate the expected file size as ( Block Size x Block Count )
6. If the image contains an Apple Master Directory Block, read its Block Size and Block Count fields
7. Calculate the expected file size as ( Block Size x Block Count )
8. Calculate the final expected file size as the largest value out of any of the above 3 values
9. Compare this against the actual size of the image files.

In addition to this, Isolyzer also extracts and reports technical metadata from the Primary Volume Descriptor and the Zero Block.

Currently the test results are reported in the following format (this may well change in upcoming releases):

<tests>
    <containsISO9660Signature>True</containsISO9660Signature>
    <containsApplePartitionMap>False</containsApplePartitionMap>
    <containsAppleHFSHeader>False</containsAppleHFSHeader>
    <containsAppleMasterDirectoryBlock>False</containsAppleMasterDirectoryBlock>
    <parsedPrimaryVolumeDescriptor>True</parsedPrimaryVolumeDescriptor>
    <sizeExpected>358400</sizeExpected>
    <sizeActual>358400</sizeActual>
    <sizeDifference>0</sizeDifference>
    <sizeAsExpected>True</sizeAsExpected>
    <smallerThanExpected>False</smallerThanExpected>
</tests>

In the above example the sizeExpected field is the size as calculated from the ISO/Apple headers, and sizeActual is the actual size. In this case both are identical. Below some output for a truncated ISO:

<tests>
    <containsISO9660Signature>True</containsISO9660Signature>
    <containsApplePartitionMap>False</containsApplePartitionMap>
    <containsAppleHFSHeader>False</containsAppleHFSHeader>
    <containsAppleMasterDirectoryBlock>False</containsAppleMasterDirectoryBlock>
    <parsedPrimaryVolumeDescriptor>True</parsedPrimaryVolumeDescriptor>
    <sizeExpected>358400</sizeExpected>
    <sizeActual>49157</sizeActual>
    <sizeDifference>-309243</sizeDifference>
    <sizeAsExpected>False</sizeAsExpected>
    <smallerThanExpected>True</smallerThanExpected>
</tests>

So, in this case sizeDifference is negative, and flag smallerThanExpected equals ‘True’ (which indicates a damaged image).

Feedback wanted

At this stage Isolyzer is a bit experimental and pretty rough around the edges, and I wouldn’t recommend it for production use. Nevertheless I’m curious about any feedback on the tool. Do others find this useful? Are things missing (i.e. other hybrid disc types I’m not aware of), or did I get anything completely wrong?

One thing that puzzles me a bit is that for the majority of ISO images I’ve come across, the expected size as calculated by Isolyzer is marginally smaller than the actual size. The difference is typically in the order of about 300-600 kB. I’m not quite sure what’s causing this, although this article mentions that some CD writing software packages add padding bytes when writing a CD. I wasn’t able to verify if this, although this SuperUser answer on validating a burnt DVD suggests it as well. If anyone knows more about this, please let me know!

Download links

Isolyzer can be found here on Github. It can be installed using pip; see the instructions here. For Windows users who cannot/don’t want to install Python I also provided stand-alone Windows binaries, which are available for download here.

To get a better idea of what software tool(s) are best suitable for this task, I created a small dataset of audio files which I deliberately damaged. I subsequently ran each of these files through a set of candidate tools, and then looked which tools were able to detect the faulty files. The first half of this blog post focuses on the WAVE format; the second half covers the FLAC format (at the moment we haven’t decided on which format to use yet).

WAVE dataset

For the WAVE dataset I started out with a small, intact WAVE file. Using a Hex editor I then made the following derivatives of this file:

• frogs-01-last-byte-missing.wav – one byte is missing at the end of the file
• frogs-01-last-2032-bytes-missing.wav – a chunk of 2032 bytes is missing at the end of the file
• frogs-01-byte-missing-at-offset-811537.wav – one byte is missing at offset 811537

Candidate tools, WAVE

The candidate tools I used to analyse the WAVE files are:

• jhove includes a WAVE validation module, which makes it an obvious choice. The tested version is 1.14.6, 2016-05-12.
• shntool is a "multi-purpose WAVE data processing and reporting utility". It was first released in 2000. The tested version is 3.0.7.
• ffmpeg is a popular conversion tool for audio and video formats. The tested version is 3.2.2.
• mediainfo is a widely-used feature extraction tool for audiovisual files. The tested version is v0.7.81.

Note that of the above tools, only Jhove and Shntool are designed to detect problems in WAVE files. Both Ffmpeg and Mediainfo were primarily designed for other purposes (format conversion and technical metadata extraction), and they were not designed to detect defective files! I included these tools here mainly because they are widely used, and I was curious whether they would throw up anything interesting in case of defective files¹. I ran the tools with the following command-line arguments (replacing "foo.wav" with the actual file name):

Jhove

jhove -m WAVE-hul foo.wav

Shntool

shntool info foo.wav

Ffmpeg

ffmpeg -v error -i foo.wav -f null -

Mediainfo

mediainfo foo.wav

I automated this using a simple shell script that runs each tool on all files, and then writes the output to a set of text files.

Results, WAVE

The full output results of each tool can be found here.

Jhove

The ‘Status’ field in Jhove’s output summarises the validation outcome. Here are the results for each file:

So, Jhove was unable to detect any of the damaged files at all!

Shntool

Shntool checks a WAVE on six criteria, which are listed in its output under ‘Possible problems’:

Possible problems:
  File contains ID3v2 tag:    no
  Data chunk block-aligned:   yes
  Inconsistent header:        no
  File probably truncated:    no
  Junk appended to file:      no
  Odd data size has pad byte: n/a

The thing to watch here is the ‘File probably truncated’ item:

So, Shntool was able to detect all damaged files.

Ffmpeg

For our Ffmpeg call we monitor any errors that are sent to the standard error stream. The results:

Interestingly, Ffmpeg reports an error for both files that have 1 byte missing, but it doesn’t for the file that has 2023 bytes missing. This suggests that Ffmpeg is not suitable for detecting broken WAVE files.

Mediainfo

Mediainfo didn’t report errors or warnings for any of these files. This is not surprising, but it does confirm that Mediainfo cannot be used for detecting broken WAVE files.

FLAC dataset

Analogous to the WAVE dataset, I started out with a small, intact FLAC file, which I then butchered into the following derivative files:

• frogs-01-last-byte-missing.flac – one byte is missing at the end of the file
• frogs-01-last-1000-bytes-missing.flac – a chunk of 1000 bytes is missing at the end of the file
• frogs-01-byte-missing-at-offset-651202.flac – one byte is missing at offset 651202

Candidate tools, FLAC

The set of candidate tools is identical to the one used for the WAVE analysis, with two exceptions:

• flac is the reference implementation of the FLAC format. The tested version is 1.3.0.
• Since Jhove does not include a FLAC module, it was not used.

Flac

The Flac tool is able to encode audio to FLAC, and decode and analyze FLAC files. For this tests I ran it with the * -t* (or –test) option:

flac -t foo.flac

This decodes a FLAC without writing the decoded data to a file. Any errors during the decoding process are reported to the standard error stream.

Results, FLAC

The full output results of each tool can be found here.

Shntool

Even though Shntool supports FLAC, it was not able to detect the missing data in any of the files:

So, Shntool does not provide any meaningful information on whether a FLAC is damaged.

Ffmpeg

Here are the results for Ffmpeg:

So, Ffmpeg was able to identify all damaged FLACs.

Mediainfo

Similar to the WAVE results, Mediainfo again didn’t report errors or warnings for any of these files.

Flac

Finally the results for the Flac tool:

So, the Flac tool was able to identify all defective files².

Conclusion

Out of the candidate tools considered here, only Shntool was able to identify all damaged WAVE files in this experiment. As a result, this (ancient!) tool still appears to be the best choice for detecting damaged WAVE files. Surpringly, Jhove was unable to detect any of the damaged files at all, and is probably best avoided for this particular purpose. For FLAC, both the Flac tool (FLAC reference implementation) and Ffmpeg were able to detect all damaged files, and both appear to be suitable tools.

Dataset and scripts

All example files, scripts and raw tool output are available here:

https://github.com/KBNLresearch/detectDamagedAudio

Post scriptum: update on MediaInfo and MediaConch

In response to this post the developers of MediaInfo added support for detecting truncated WAVE files. This should cover all of the damaged WAVE files presented here. Moreover, their Twitter account announced that detection of FLAC flaws is planned for the MediaConch tool, but that they are looking for sponsors for this.

One of the reasons is the following. As reported earlier on iPRES and in several other places, the Ministry of Education, Culture and Science started in 2015  (and financed) a program called Network Cultural Heritage, with a focus on exploring more the Dutch digital cultural heritage, by making our digital collections more visible, more connected with each other and more sustainable. While the NCDD had already started some projects in 2014, this initiative gave a boost to these projects and two of the nominees can be related directly to this Network Cultural Heritage.

A nationwide network

The Award for Research and Innovation went to the project Constructing a network of nationwide facilities together, project-lead Joost van der Nat. Although the Netherlands is a small country, we have a large amount of organisations with a mandate to preserve digital heritage. Not everyone is aware of what others are doing, nor are we always aware how we can benefit from each other. More collaboration in all kinds of areas related to digital preservation can improve efficiency, effectiveness and professionalism for both small and large organisations. Based on common sense, desk research, lots of discussions, interviews and an intense feedback meeting  in 2014 – where for the first time over 80 Dutch preservationists were together, the Analytical Framework for the infrastructure was created. This framework combined requirements from OAIS, legal requirements, policies and quality assurance, R&D, Training and ICT, thus giving an overview of the areas that were important for exercising digital preservation and finding partners to collaborate on this task. But we are all different organisations, as is our digital material and our legal mandate. So it was also important to pay attention in the model to the differences between the various domains.

The second part of the project was a reality check: interviews with the stakeholders in different domains helped identifying the current status of collaboration, whether people were willing to collaborate more by using shared services for digital preservation activities and what activities they thought were so organisation or domain specific that they always will do them on their own or with people in their own domain. The resulting diagram is the starting point for developing a real network of collaboration in digital preservation. Currently an inventory is created of existing services, which can be candidates to be incorporated in this model.

Web archaeology The Digital City

Quite a different project in the Network of Cultural Heritage activities is the Digital City revives of which “web archaeologist” Tjarda de Haan is project lead. In 1994  Amsterdam was the first city in the world with a digital counterpart, when The Digital City (De Digitale Stad) was started, offering its citizens free access to the internet, enabling them to have an email address and to build their own virtual environment. This web presence lasted until 2001 when the website was taken down. Initiated by the Amsterdam Museum, numerous paths have been explored since 2011 to find traces of this Digital City and to make a reconstruction. This project offers an interesting case to raise awareness for digital preservation. It not only shows how “digital archaeology” is able to retrieve a digital environment from the past and what can be learnt of the efforts, tools and technical skills  that need to be in place to enable this, but the results of this archaeology activity also need to be safely preserved for the future. This is a good example of a local initiative that had a need to be served by a “network of nationwide facilities”.  As we want to keep the result accessible too, there will be a close connection with the visibility and usability parts of the Network Cultural Heritage.

The 3^rd Dutch nominee was the submission of Research Data Netherlands (a collaboration of DANS and 4TU). They developed a training course for those responsible for preserving research data: Essentials 4 Data Support.

These initiatives by the partners in the Network Cultural Heritage under project management of the NCDD and by Research Data Netherlands all contributed to more preservation awareness in the Netherlands. Winning the awards are the cherries on the cake!

Uitsnede van: http://nominet-prod.s3.amazonaws.com/wp-content/uploads/2016/03/Map-Of-The-Online-World.jpg

Dat web archivering nuttig is bleek meteen al uit de door Peter de Bode (KB) samengestelde presentatie van web sites die niet meer online te zien zijn maar wel in ons KB web archief aanwezig zijn. Ook de gastheer Tom de Smet van Beeld en Geluid zei dat hun audiovisuele collectie niet meer los te koppelen is van wat er op het web gebeurt. En dat geldt voor veel organisaties in Nederland. Niet alleen worden web collecties aangelegd omdat het gezien wordt als cultureel erfgoed (zoals door de KB) maar er zijn ook veel organisaties die aan web archivering doen om te voldoen aan de Archiefwet. De NDE werkgroep Web Archivering heeft een onderzoek onder 22 Nederlandse instellingen gedaan om te kijken of er al onderling samengewerkt wordt (de N van NDE is immers “netwerk”). Conclusie is dat dit nog aanzienlijk verbeterd kan worden. De overgrote meerderheid doet pas sinds de laatste 3 jaar iets aan web archivering (de KB sinds 2007), waarbij dan een groot gedeelte dit heeft uitbesteed aan commerciële bedrijven (waardoor ze bijvoorbeeld “geen idee” hadden van de tools die gebruikt werden om de websites binnen te halen: een cruciaal en zeer kwetsbaar onderdeel). Naast het verzamelen van web sites, komt ook de (technisch moeilijke) wens om sociale media te gaan bewaren – waar overigens weer allerlei privacy aspecten aan zitten.

Keynote spreker Herbert van de Sompel (onderzoeker bij Los Alamos National Laboratory, tijdelijk werkzaam bij DANS) demonstreerde een aantal technische oplossingen om ook web archieven die alleen ter plekke en dus niet online raadpleegbaar zijn, toch met elkaar te verbinden, zodat gebruikers weten wie wat bewaart en zelfs een overzicht in de tijd kunnen krijgen op basis van verschillende data waarop een snapshot van de website is gemaakt (memento, memgator ). Ook incomplete pagina’s (het is niet altijd mogelijk alles van een website binnen te halen) zou je op deze wijze kunnen “reconstrueren”, waarbij je je natuurlijk wel moet realiseren in hoeverre dit “authentiek” is.

Filip Boudrez van het Stadsarchief Antwerpen doet al sinds de jaren 90 aan web archivering, waarbij bewijsvorming en verantwoording het uitgangspunt vormen. Hij beschreef de overwegingen bij verschillende stappen en de manieren waarop aan kwaliteitscontrole gedaan wordt, waaronder handmatige controle door vrijwilligers. Documentatie over alle besluitvorming wordt nauwkeurig per web site vastgelegd.

Naast web archivering kennen we tegenwoordig ook web archeologie, dankzij het NDE project van Tjarda de Haan, waarbij De Digitale Stad van Amsterdam gereconstrueerd wordt aan de hand van verzamelde informatie op oude dragers in combinatie met nieuwe technieken.

Rafaël Rozendaal, een web kunstenaar gaf zijn visie op hoe zijn kunst bewaard zou moeten blijven en welke maatregelen hij neemt om zijn kunst zo lang mogelijk werkend te houden, onder het motto “als het niet meer werkt, wordt het archief” (bijvoorbeeld omdat de gebruikte tools niet langer in combinatie werken of het beoogde effect bereiken).

Kees Teszelszky (KB afdeling Onderzoek) toonde de eerste Nederlandse website en vertelde over zijn onderzoek naar de context van de KB Web Collectie in relatie tot wat anderen doen en wat er wellicht meer zou kunnen om het Nederlandse web in web collecties vast te leggen. Samenwerking is ook hier weer het toverwoord.

Dat erfgoed instellingen en overheidsinstellingen elkaar daarbij moeten vinden, vertelde Nick Chapel (Erfgoedinspectie) op basis van het binnenkort te verschijnen rapport: hoewel wettelijk verplicht, schiet de Centrale Overheid nog schromelijk tekort wat betreft het archiveren van hun eigen websites en de bijbehorende sociale media.

In een afsluitend panel werden de NCDD Expert groep Webarchivering gepresenteerd, en werden mogelijke initiatieven met het publiek in de zaal bediscussieerd, onder deskundige leiding van de dagvoorzitter Jantje Steenhuis, stadsarchivaris van Rotterdam.  Aan de slag dus maar, de tijd is er rijp voor!

Barbara Sierman

During the preparations for iPRES 2016 the Programme Committee discussed the fact that exactly 20 years ago Preserving Digital Information. Report of the Task Force on Archiving of Digital Information was published. A landmark report by The Commission on Preservation and Access and The Research Libraries Group, published in May 1996. It describes a broad view on digital preservation and is often looked at as one of the first comprehensive reports on this topic.

It was interesting to read it again and I was wondering what the view on preservation was 20 years ago and how this relates to the topics presented at iPRES 2016?

No doubt in 20 years’ time much has changed. For a start, the  (American) view as reflected in the two main principles (p. 22) of the report have little relation with what the iPRES attendants experience. The first principle being

“Information creators/providers/owners have initial responsibility for archiving their digital information objects and thereby ensuring the longterm preservation of those objects”.

I don’t think we have achieved this.  All iPRES attendants were in Bern because this not reality.

The second principle relates to the first:

“Certified digital archives have the right and duty to exercise an aggressive rescue function as a fail-safe mechanism to preserve information objects that become endangered because the creator/provider/owner does not accept responsibility for the preservation function (…).”

This one is  also more complicated than stated in the document, as it now shows that without legal mandate (and even with that legal mandate) it is hard enough to get the digital information. But the amount of digital material has grown immensely since 1996.

The Preserving Digital Information report shows a “vision” on digital preservation. Some of the ingredients of this vision have changed, for example the emphasis that is laid on migration as a preservation strategy is nowadays accompanied by a similar importance for emulation as a strategy. Their concept of  “intellectual integrity” seems to have been incorporated in the OAIS standard.  The notion of “Certified archives” is within reach with WDS-DSA, nestor and ISO 16363 in place. Managing cost is still an issue (not at all discussed at iPRES this year!) other topics still need to be addressed like migration paths “as an integral part of the process or system that generates digital information”( p. 29).

Although there was no presentation on costs at iPRES  2016, there was a wealth of interesting talks about practical solutions related to different materials (emails, web archiving, publications and related materials, PIDs, geodata) , different organisations (expanding from libraries and archives to research data and organizations collecting digital art) and different steps in the preservation process (emulation, quality assurance, risk assessment).

What striked me is the fact that, compared to the 1996 vision, and looking at the individual and valuable contributions at iPRES 2016, we still talk in practical solutions for detailed problems. I don’t deny that a lot of problems need to be solved and that it is interesting to discuss the approaches with your peers at iPRES. But where is the broader view? Do we,  as a community have an updated vision on Digital Preservation in general?

Might it be that the limited funding of the EU for digital preservation programmes resulted into less research on a conceptual level in digital preservation? I’m convinced that the work that was done on emulation in the KEEP (see Internet Archive) project has led to a better acceptance of emulation in the world of digital preservation. The work done in SCAPE on Preservation Planning has improved our thoughts about this OAIS functional entity as well as to the introduction of the concept of Preservation Watch and the nuanced view on migration (“you lose more than you want to”). Of course I’m biased here because I have participated in these projects but I hope that people of for example the E-Ark project will say the same in a few years’ time.

Perhaps we need more occasions for a deliberate reflection on digital preservation. Where we can derive some patterns from the practical solutions, discuss the deviations that organisations made from existing standards and where we include the risks for collection building and preservation caused by the ongoing changes in digital materials.  Could not we use an iPRES conference where so many people with preservation knowledge are together, so that we can develop a vision for the 3^rd decade of digital preservation?

This post was also published at http://digitalpreservation.nl/seeds/20-years-of-digital-preservation/

Vertaling naar de praktijk

OAIS is al ruim 15 jaar de internationale standaard die we gebruiken als we het hebben over digitale duurzaamheid. De gemeenschappelijke taal helpt ons bij het communiceren over complexe problemen. OAIS is de beschrijving van een conceptueel model voor digitale duurzaamheid, geen reeks van voorschriften. Je moet het model dus naar je eigen omgeving vertalen. Hoe weet je of je de standaard goed interpreteert? Als de groep van experts het ergens over eens was, dan was het wel de behoefte aan praktijkvoorbeelden. In het Engels is daar een begin mee gemaakt via een wiki OAIS community. Deze NCDD-bijeenkomst zou wel eens de opmaat kunnen zijn voor een Nederlandse variant [daar wordt aan gewerkt].

OAIS in aluminiumfolie

De experts hadden vooraf hun visie op OAIS gegeven. Wat betekent OAIS voor hen? Dit werd met behulp van aluminiumfolie door iedereen in 2 minuten in een kunstige sculptuur samengevat. Men was het er vrijwel unaniem over eens dat de standaard een goede richtlijn was, goed in elkaar zit en zeker als leidraad bij de inrichting van zowel je organisatie als je systeem gebruikt kan worden. Kritiek komt vaak voort uit onkunde, omdat men stokt bij de beschrijving van het functionele model, of de teksten te letterlijk wil interpreteren. En niet verder leest in de standaard, terwijl het bijbehorende datamodel ook van belang is voor het begrip van de standaard. Er was een gedeeld pleidooi om meer aandacht te geven aan een klein, maar uitermate belangrijk stukje tekst in OAIS: de verantwoordelijkheden van het digitale archief (paragraaf 3.1. voor de kenners).

Voorbeelden gevraagd

Naast hun visie hadden de expert ook hun dilemma’s opgeschreven. Hier kwam vooral naar voren dat men vaak met de praktische vertaling worstelt. Omdat OAIS niet voorschrijft hoe het moet, kan er verschil van interpretatie zijn. Meer voorbeelden van collega’s hoe ze bepaalde richtlijnen naar de (vaak weerbarstige) praktijk vertalen, zou de digitale duurzaamheidscommunity enorm kunnen helpen.

AIP, Pre-Ingest, Designated Community en Access

Is het bijvoorbeeld zo dat een Archival Information Package dat is opgeslagen altijd ongewijzigd moet blijven, of mag men in de loop van de geschiedenis metadata toevoegen en wijzigen? Zou er niet een Pre-Ingest-functie moeten worden toegevoegd aan het OAIS model, omdat alle organisaties worstelen met het verschil tussen wat ze rechtstreeks aangeleverd krijgen en de tussenstappen die nodig zijn voordat het een Submission Information Package is (en dus weer past in het model)? En kunnen we hier dan een onderscheid maken tussen het moment waarop de organisatie de digitale objecten onder zijn hoede genomen heeft en het moment waarop de keuze nog gemaakt moet worden of het materiaal wel geaccepteerd wordt (met andere woorden zit “appraisal” in de pre-ingest of niet)? Het begrip Designated Community – feitelijk de mensen voor wie je het materiaal toegankelijk houdt – is redelijk vaag en ook de Access-functie kent in de praktijk veel meer haken en ogen dan nu in de OAIS standaard verwoord zijn. Via een buitengewoon democratische procedure (je schreef niet alleen op waar je wél maar ook waarover je persé niet wilde discussiëren) werden groepen samengesteld om een selectie van de stellingen nader uit te diepen en om te zetten in een scherp geformuleerde aanbeveling. Waarom dat dan?

Voorbereiding op de review

Wel, er is op dit moment een review proces gestart voor OAIS en dat is bij uitstek de gelegenheid voor de gebruikers van de OAIS-standaard om verbeteringen en wijzigingen voor te stellen. De standaard is oorspronkelijk in de jaren negentig opgesteld door en voor ruimtevaartdeskundigen. De variatie aan gebruikers is echter sindsdien toegenomen: archieven, bibliotheken, data centers, medici, universitaire repositories en ga zo maar door. Wil de standaard ook voor die gebruikers relevant blijven, dan zal die moeten mee veranderen als de wereld van de gebruikers verandert.

Vandaar dat er in internationaal verband een wiki is ingericht om praktische en theoretische kennis over OAIS te verzamelen. Praktijkvoorbeelden dus. Daarnaast kan deze inbreng, net als de resultaten van de NCDD-workshop, leiden tot aanbevelingen voor de nieuwe versie van de standaard. Iedereen kan deelnemen aan de discussie, zodat het proces transparant en open is. Uiteindelijke verzoeken tot aanpassingen dienen via het officiële kanaal ingediend te worden bij review.oais.info.

Maar voordat het tot een nieuwe versie van de standaard komt, duurt nog even omdat eerst alle officiële processen en inspraakrondes doorlopen moeten zijn, zodat de verwachting voor een nieuwe versie van OAIS nu rond 2020 ligt. Genoeg tijd om te discussiëren en te analyseren waar de knelpunten liggen en samen te werken aan een online kennisbron over OAIS in de praktijk – in het Engels of in het Nederlands (zie de nieuwe informatiepagina over OAIS van de NCDD).

Wordt vervolgd

Meer nieuws volgt binnenkort: de resultaten van deze NCDD-workshop brengen we in bij een OAIS-paneldiscussie op de iPRES Conferentie in Bern op 2 oktober. Wordt vervolgd!

Barbara Sierman

Crazy Fixed Layout

As I cannot share the original file for rights reasons, I fired up the Sigil e-book editor and made a handcrafted EPUB that reproduces its behaviour. You can download the file here. If you open it in an e-book viewer, it will probably look perfectly normal at first sight. For example, here’s a screenshot I made using the Calibre viewer:

Next I reduced the width of the viewer window. One would expect the text to re-flow to the new width. Instead this happened:

After increasing the font size, I ended up with this:

I got similar results in Chome’s Readium extension. On my e-Ink reader, a Sony PRS-T2, the book rendered as follows:

However, I wasn’t able to change the font size.

Analysis

The file passes validation in EpubCheck 4.0.1 without errors. However, the output does contain a series of warnings about the use of absolute positions in a stylesheet:

CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (6-2)
CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (24-1)
CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (43-1)
::

To really understand what causes the problem, we need to look inside the file’s HTML and CSS resources. Here’s some of the HTML that underlies the text:

<p id="p01" class="para">This is an <em>EPUB</em> 2 file that uses a fixed layout.</p>
<p id="p02" class="para">This is achieved by placing each line inside a</p>
<p id="p03" class="para"><em>paragraph</em> element. Each <em>paragraph</em> element</p>
<p id="p04" class="para">is placed at a fixed position on the page. Even</p>
<p id="p05" class="para">though this file is valid <em>EPUB</em>, this is a pretty</p>
<p id="p06" class="para"> terrible idea, because in most readers the text</p>
<p id="p07" class="para">will not reflow after resizing the viewer window.</p>

So, every line is wrapped inside a paragraph element, each of which has a unique id selector. These refer to style definitions in the EPUB‘s stylesheet. Here are the definitions for the first two lines:

#p01
{
position:absolute;
left:40px;
top:80px;
letter-spacing:0.42px;
word-spacing:0.1em;
}
#p02
{
position:absolute;
left:40px;
top:120px;
letter-spacing:0.42px;
word-spacing:0.1em;

Each style definition specifies a line’s position on the canvas (left, top); moreover, these co-ordinates are defined as absolute positions. This means that each line is placed at a fixed position, regardless of whether this makes any sense given the actual dimensions of the viewer window (or device), or the user’s preferred font size. It seems that the intention of the producer of the original EPUB (from which I derived my example) was to create some sort of “fixed layout” document. However, this doesn’t make much sense for books with simple, text-only layouts (as in this case). Worse, depending on the viewing device and the user the file may be effectively inaccessible. For example, someone with a visual impairment may only be able to read an EPUB using very large font sizes, which in this case results in garbled text.

Crazy Columns

Things can even get worse. I once came across an EPUB that used similar tricks to achieve a two-column layout. Again I’m not able to share the original file, so I created another EPUB that mimicks its behavour. In the Calibre viewer it looks like this:

As with the first example, the text doesn’t reflow after resizing the viewer window, and increasing the font resulted in this:

This is what I got when I opened the file in my Sony e-Ink reader:

After I increased the font size this happened:

Similarly, when I tried to copy the text in the file to the clipboard, and then pasted it in a text editor, I ended up with this:

This is an EPUB filepage. Even though thisthat uses a two-columnfile is valid EPUB, there’slayout. For each column,no way to establish theevery line is placed atlogical reading order ofa fixed position on thethe text.

Ouch!

Analysis

Again, throwing this file at EpubCheck 4 doesn’t result in any validation errors, although just like the previous file there are some warnings about the use of absolute positions in the stylesheet:

CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (13-1)

A peek inside the HTML reveals the true horrors of this EPUB. This is how the text is encoded:

<div class="pos" style="left: 40px; top: 100px;">This is an <em>EPUB</em> file<div>
<div class="pos" style="left: 260px; top: 100px;">page. Even though this</div>
<div class="pos" style="left: 40px; top: 140px;">that uses a two-column</div>
<div class="pos" style="left: 260px; top: 140px;">file is valid <em>EPUB</em>, there's</div>

So, every line of each column is wrapped in a division element that has a fixed position. The class pos in the stylesheet defines the general layout of each division element. In this case, it specifies that all positions are (again) absolute:

.pos {position:absolute;
} 

Technically this is pretty similar to the first example. Note that the above HTML doesn’t contain any semantic information on the fact that there are two separate columns. Worse, the order of the text in the HTML doesn’t even follow the actual reading order! This also explains the results after copying and pasting. Screen reader applications will not be able to handle this either, which makes books like these inaccessible to many visually impaired users. All of this could have been avoided if the book’s producer had followed the W3C multi-column layout specification.

Conclusion

I don’t know how common (or rare) EPUBs like the above are. They may just be weird edge cases. Nevertheless, their existence indicates that checking for validity alone may not be sufficient to ensure accessibility for all users (in particular those with a visual impairment). In any case, files like these can be identified relatively easily by checking EpubCheck‘s output for the presence of a CSS-017 warning (“CSS selector specifies absolute position”)¹. These examples also underline the importance of guidelines and best practices. Several good resources for making accessible EPUB are available from the EPUB 3 Accessibility Guidelines, including a useful Accessibility QA Checklist. I would also be interested in hearing other people’s experiences with “weird” EPUBs like these.

Postscript

Alberto Pettarin pointed me to his blog post (Current) Fixed Layout eBooks Considered Harmful. Written in 2015, it addresses the problems with current implementations of fixed layouts in EPUB, and if you found this blog post interesting, I would suggest to check out Alberto’s blog as well.

Alberto’s Twitter feed also drew my attention to an interesting EPUB with the program of the recent EPUB Summit in Bordeaux. You can download it here (you need to unzip it first!). The file is interesting because:

1. It does not pass validation by EpubCheck (the mimetype file entry is not the first file resource in the archive)
2. It uses a fixed, multi-column layout that doesn’t scale in either Readium or Calibre‘s viewer (changing the font size has no effect), and I’m wondering if it is usable at all on any handheld devices!

There’s some irony in that this file was published by EDRLab, an organisation that describes itself as “the European headquarter for IDPF and Readium Foundation”, and which mentions “support for people who have print disabilities” as a “key part”of its mission. Oh well …

Link to dataset

The EPUBs used for this blog post are part of the EPUB KB policy testing repository. This is an annotated set of openly licensed EPUB files that were specifically created for testing purposes.

About a month ago the International Digital Publishing Forum, the standards body behind the EPUB format, published an Editor’s Draft of EPUB 3.1. This is meant to be the successor of the current 3.0.1 version. IDPC has set up a community review, which allows interested parties to comment on the draft. The proposed changes relative to EPUB 3.0.1 are summarised in this document. A note at the top states (emphasis added by me):

The EPUB working group has opted for a radical change approach to the addition and deletion of features in the 3.1 revision to move the standard aggressively forward with the overarching goals of alignment with the Open Web Platform and simplification of the core specifications.

As Gary McGath pointed out earlier, this is a pretty bold statement for what is essentially a minor version. The authors of the draft also mention that they expect it “will provoke strong reactions both for and against”, and that changes that raise “strong negative reactions” from the community “will be reviewed for future drafts”.

This blog post is an attempt to identify the main implications of the current draft for libraries and archives: to what degree would the proposed changes affect (long-term) accessibility? Since the current draft is particularly notable for its aggressive removal of various existing EPUB features, I will focus on these. These observations are all based on the 30 January 2016 draft of the changes document.

Removed support for EPUBCFI for linking

The EPUB Canonical Fragment Identifier (EPUBCFI) “defines a standardized method for referencing arbitrary content within an EPUB Publication”. Until EPUB 3.0.1, Reading Systems were required to support EPUBCFI for hyperlinking within and between documents. This requirement is dropped in EPUB 3.1 (although it would still be possible to use EPUBCFI for annotations and bookmarks).

In principle this change could result in problems if an EPUB that uses CFI for hyperlinks is opened in a 3.1 reading system: in that case the hyperlinks would not work. However, according to EPUB editor Matt Garrish, authors simply do not use CFI for hyperlinking. He also mentions a check by Google on their corpus of millions of books, which only turned up a few instances of CFI use. One of these was a link in an EPUB best practices book, while the remaining ones were all part of the EPUB test suite documents. If these results are representative of all EPUBs “in the wild”, the implications of the change would be negligible.

Reduced set of metadata elements in Package Document

EPUB 3.1 imposes restrictions on the metadata elements that can be embedded in the Package Document. Up to version 3.0.1, the full Dublin Core Metadata Element Set was supported, whereas in 3.1 only the dc:identifier, dc:title, dc:language, dc:creator, dc:publisher and dc:type elements are allowed. Additional metadata can be included, but they need to be defined in a separate resource (file), which is referenced from the metadata element using the link element. Below is an example that uses a MARC file:

<link rel="record"
  href="meta/9780000000001.xml" 
  media-type="application/marc"/>

Complicating things further, the EPUB 3.1 Packages draft says:

Linked resources that are not Publication Resources are not subject to Core Media Type requirements [EPUB31] and may be located inside or outside [EPUB31] the EPUB Container. Retrieval of Remote Resources is optional.

So, linked metadata resources can have any possible format, and they may not even be included in the EPUB container. Even though these changes would have no direct consequences for long-term accessibility, they would seriously complicate document processing (e.g. ingest) workflows that rely on the metadata in the Package Document. It would also affect end users who rely on these metadata fields to sort and find their ebooks.

Note: the discussion thread on this topic in the issue tracker is worth checking out, as it contains some excellent additional observations.

Removal of the NCX

EPUB 2 documents contain the NCX file (“Navigation Control file for XML”), which provides a mechanism to navigate a publication. It is essentially a hierarchical table of contents. The NCX was superseded by the Navigation Document in EPUB 3.0.1. However, the NCX was allowed in EPUB 3.01 publications, which was useful for keeping EPUB 3 publications compatible with older (EPUB 2-based) reading systems¹. The 3.1 draft forbids the NCX altogether, which means that such “hybrid” EPUBs are not possible without breaking the specification.

The main consequence of this is that it would make EPUB 3.1 files incompatible with older reading systems. More specifically, basic navigation functionality such as direct access to a chapter from the table of contents would not work.

To get an approximate idea of the impact of this, I had a look at the EPUB 3 support grid, which gives detailed information about the support of specific EPUB 3 features for commonly used devices, apps, and reading systems. This link shows support of the toc nav element, which defines the primary navigational hierarchy in the Navigation Document. Only 55% (34 out of 62) of all tested reading systems fully support the toc nav element, with 37% (23 out of 62) not supporting it at all². This may not be a big deal for users of software-based reading systems (which make up the majority of the support grid), but users of (older) E-ink readers often don’t have the option to upgrade their devices. A good example is this (now discontinued) Sony e-Ink hardware reader. Unfortunately, E-ink devices appear to be underrepresented in the support grid. For example, it contains no information whatsoever on any of the popular Kobo readers.

The proposal to remove the NCX provoked strong reactions in the community review, with one respondent stating it would lead to “dropping support for millions of eInk reading systems”. It would also contradict this statement from the EPUB 3.0.1 specification (emphasis added by me):

The NCX feature defined in [OPF2] is superseded by the EPUB Navigation Document [ContentDocs301]. EPUB 3 Publications may include an NCX (as defined in OPF 2.0.1) for EPUB 2 Reading System forwards compatibility purposes, but EPUB 3 Reading Systems must ignore the NCX.

The explicit reference to EPUB 3 Publications (not EPUB 3.0.1 Publications!!) implies that the statement applies to EPUB 3 in general. Removing the NCX in another EPUB 3 release would be at odds with this.

Removal of the guide Element

The guide element was an optional data structure in EPUB 2 that provided “convenient access” to structural components of a publication. It was deprecated in EPUB 3.0.1. Without any data on the actual usage of this feature, it is difficult to say much about the impact of its complete removal (this was also pointed out by one respondent to the community review).

Removal of the bindings Element

In EPUB 3.0.1 the bindings element could be used to define fallbacks for foreign resources. According to EPUB editor Matt Garrish “this feature is not widely used or supported”, and the impact on accessibility appears to be negligible.

Removal of the switch Element

The switch element in EPUB 3.0.1 allows one to define alternative representations of XML fragments. Here’s an example:

<epub:switch id="cmlSwitch">
   
   <epub:case required-namespace="http://www.xml-cml.org/schema">
      <cml xmlns="http://www.xml-cml.org/schema">
         <molecule id="sulfuric-acid">
            <formula id="f1" concise="H 2 S 1 O 4"/>
         </molecule>
      </cml>
   </epub:case>
   
   <epub:default>
      <p>H<sub>2</sub>SO<sub>4</sub></p>
   </epub:default>
   
</epub:switch>

Here, we have a chemical formula in ChemML format and in standard HTML. ChemML is not natively supported in EPUB, so by default a reader will display the HTML version. However, wrapping both in a switch element would allow a ChemML-capable reader to render that representation instead.

I asked EPUB editor Matt Garrish how an EPUB 3.1-compliant reader would render content that is wrapped in a switch element. He replied that by default all of the switch content would be rendered. So for the example above, a reader would try to render both the HTML and the ChemML versions (with the latter failing on most reading systems). Matt stressed the significance of the switch element, adding that people have been using it, “if not extensively”.

Removal of the trigger Element

The trigger element in EPUB 3.0.1 is used to define simple user interfaces for multimedia content. Since this can be done natively in HTML 5, it is dropped from EPUB 3.1. Here editor Matt Garrish explains that the feature is both “sparsely used” (referring to a survey of publishers) and “poorly supported”.

Miscellaneous changes

Apart from the changes above (which all remove features from the existing specification), the EPUB 3.1 draft also adds a number of new features, and clarifies some existing ones. I won’t go over them in detail, but here’s a brief overview:

• New Core Media Types – adds support for WOFF 2.0 and SFNT fonts.
• Addition of Support for HTML Syntax of HTML5 – adds support for the HTML Syntax of HTML5 (currently only the XHTML syntax is supported³).
• Replacement of EPUB Style Sheets with CSS References – this replaces the current “EPUB Style Sheets profile” by the “official definition” of CSS.
• Addition of WCAG Support – adds the recommendation that all HTML Content Documents conform to the WCAG Guidelines to ensure they are accessible for people with disabilities.

Finally, the draft contains clarifications on Foreign Resource Fallbacks and Scripting Support.

EPUB 3.1 or EPUB 4.0?

By now it should be clear that the aggressive removal of features in EPUB 3.1 would have some far-reaching consequences. This is particularly true for the removal of the NCX, which would make EPUB 3.1 files incompatible with many existing E-ink readers. It would do this by ruling out the option to make backward-compatible “hybrid” files. As Gary McGath pointed out earlier, introducing “radical changes” in what is essentially a minor version is pretty unusual practice for any standard. Nowadays, most software and file formats use some variation of semantic versioning, with version numbers that follow the general form MAJOR.MINOR.PATCH. Here, each component of the version number has a well-defined meaning:

1. MAJOR version is increased in case of incompatible API changes,
2. MINOR version is increased when functionality is added in a backwards-compatible manner, and
3. PATCH version is increased in case of backwards-compatible bug fixes.

Since the current draft includes multiple backward-incompatible changes, this makes me wonder why the editors didn’t name it EPUB 4.0 instead! Kovid Goyal, lead developer of the popular Calibre software, made the following comment on this:

[I]f you want to make backwards incompatible changes, please, dont do it in a point release. From glancing over your changes document, it seems to me that you want to make several breaking changes. That’s great, EPUB 3 could do with some serious breaking. But name it EPUB 4. I really dont want to have tell my users that calibre supports EPUB 3.1 but not EPUB 3.

I agree with Kovid here. Having multiple sub-versions of EPUB 3, with some of them being backward-compatible with EPUB 2, while this backward compatibility is explicitly ruled out in another sub-version, is bound to create a situation that will be incomprehensible for most e-book buyers. Worse, it could even undermine overall confidence in the format. For memory institutions it would also make the management of EPUB 3 publications unnecessarily complicated. Not only would some EPUB 3.1 files not render correctly in an EPUB 3.0.1 reader, the opposite would be true as well.

Flashback

In my 2012 report on EPUB for archival preservation I already mentioned the stability of the EPUB format as a concern:

EPUB 3 shows quite major changes relative to version 2, which raises concerns about the format’s stability over time. These concerns are reinforced by the fact that EPUB 3 is heavily dependent on (X)HTML5 and CSS3, both of which are unfinished “works in progress”, which may undergo various changes before being finalised.

These concerns are once more confirmed by the current EPUB 3.1 draft. However, it remains to be seen how many of these changes will make it to the final version. The community review process is ongoing at this moment, so if you’re getting a little uneasy after reading this blog post, there’s still time to get involved and make your voice heard!

Acknowledgement

Thanks to Matt Garrish for his prompt replies to my questions on Github.

Het wetenschappelijk artikel gaat volgens Mons de huidige centrale plek verliezen ten faveure van de datasets. Niet iedereen was het hiermee eens, maar vanuit een collectieoogpunt zijn deze ontwikkelingen belangrijk. Verzamelen we wel de juiste zaken en sluiten onze activiteiten aan bij wat er in de wereld gebeurt?

Andrew Sallans van het Center for Open Science (COS)  vertelde over de COS   TOP Guidelines om  transparantie en het delen van data te bevorderen. Uitgevers die deze Guidelines volgen, garanderen onder meer dat de datasets van een artikel in een trusted repository worden geplaatst en hergebruikt kunnen worden. Niet zelden hoor je dat er alleen juichverhalen in de wetenschap verteld worden. Om te bevorderen dat ook bijvoorbeeld nul resultaten gepubliceerd kunnen worden, kan een uitgever een “pre-registration” van het onderzoek faciliteren, waarbij het uiteindelijke artikel wordt geplaatst, ongeacht de uitkomst. Door middel van een goed zichtbaar logo in het uiteindelijke artikel is duidelijk dat de uitgever deze open procedure volgde. Inmiddels zijn er 500 journals waarbij de uitgever de TOP Guidelines volgt.

Veel financiers van onderzoek eisen tegenwoordig dat de bijbehorende datasets voor een bepaalde periode, ongeveer 10 jaar, toegankelijk moet blijven voor hergebruik. Het valt in de praktijk nog niet mee om deze datasets binnen universiteiten veilig te stellen in de “Institutional Repositories”. Een paneldiscussie over dit onderwerp maakte duidelijk dat nog veel hobbels genomen moeten worden. Afgezien van een  argument dat “een ander toch niks aan je data heeft”, blijken de meeste onderzoekers niet onwilig datasets te deponeren,  maar weten ze gewoon niet waar te starten. Zij hebben bijvoorbeeld geen verstand van eigendomsrechten rond hun data. Om dit op te lossen zouden ze van begin af aan goed begeleid moeten worden bij het maken van de datasets. Dus ja – door goed opgeleide data stewards. Aanmoedigen van onderzoekers kan ook. Zo standaardiseert 3TU de datasets na ontvangst zodat aanwezige tools gebruikt kunnen worden. Een extra stimulans voor hergebruik van datasets.

Ook Susan Halford (Web Science Institute Southampton) ziet de data als kern van onderzoek, maar voegde daar direct aan toe dat er nogal wat kanttekeningen te plaatsen zijn bij “Big Data”. Zelf een sociologisch onderzoeker, ziet ze met name veranderingen  van uit methodologisch en interdisciplinair oogpunt. Hoe goed zijn de methodieken om Twitter data op waarde te schatten? Deze data is immers door Twitter bewerkt alvorens vrijgegeven. Als onderzoeker heb je geen idee wat je in handen krijgt. Trek het lijntje door en je kunt je als faciliterende instelling afvragen of je wel genoeg informatie geeft aan de onderzoeker als je datasets ter beschikking stelt.

Verschillende lezingen gingen over het juist verwijzen naar datasets om daarmee de “credits” aan de juiste persoon te geven. Nu houden verschillende organisaties als Codata en DataCite zich bezig met het opstellen van regels. Belangrijk zijn ook de onontbeerlijke Persistent Identifiers, zowel voor datasets als voor andere zaken als auteurs (Orcid wordt erg populair) en zelfs voor wetenschappelijke instrumenten.

En uiteindelijk zijn er dan lange termijn archieven die deze zaken bewaren. De datasets die niet alleen meer ter verificatie bewaard worden maar vooral ook voor hergebruik. Soms is voor hergebruik specifieke software voor nodig die de onderzoeker – soms jaren geleden – heeft gemaakt (gekscherend “professorware” genoemd). Niet zelden staat deze informatie nog op verouderde dragers als Cd’s en floppy disks. De British Library initieerde (net als de KB overigens) een project  Flashback, waarbij de inhoud op deze verouderde dragers wordt veilig gesteld en deelde hun ervaringen. Alle lezingen zijn online beschikbaar.

Of onderzoeksdata nu belangrijker wordt dan de publicatie of niet, er zal altijd een relatie blijven met een verhaal over het bijbehorende onderzoek – in boek, in artikel, op een website. Zonder twijfel  verandert de traditionele manier van publiceren over wetenschappelijk onderzoek en dat zal ook voor ons gevolgen hebben.

Barbara Sierman