KB Research

Research at the National Library of the Netherlands

Page 11 of 13

iPRES 2013

Author: Barbara Sierman

The iPRES2013 conference took place in beautiful Lisbon, together with the Dublin Core 2013 conference. In total there were around 400 people, from 38 countries.  Each conference had its own program. But the three (shared) key note speakers draw the attention from both the bibliographic people and the digital preservation in the room and sketched their views on important challenges we need to work on collaboratively. Gildas Illien (BnF) strongly advocated that  bibliographic people and digital preservation people would be more cooperative, as they both are trying to make the collections accessible but from a different angle. The user expectations should be leading in both fields and, if so, will require more collaboration in the organizations. Management need to be convinced of this. Paul Bertone from the European Bioinformatics Institute explained the recent breakthrough in storage: storage in DNA, which might be a solution for massive storage of data. And finally Carlos Morais Pires, from the European Commission, talked about Horizon 2020 and data infrastructures (and here – as libraries we need to point this out again and again: data is not only restricted to scientific data generated by instruments, but also the big data collections in libraries and data centres for social sciences ! Carlos Morais Pires immediately agreed on this and changed his slide.)

IMG_3486

Barbara Sierman speaking at iPRES

All presentations can be found on http://purl.pt/24107 ,  covering a wide range of aspects. There are simply so many aspects related to digital preservation ( webarchiving, preservation policies, open source preservation systems, trust, storage, and so on…). I can only advise you to have a look at the above mentioned URL.

Is there a trend to be discovered in all these presentations? To me, they demonstrate there is a lot of national and international collaboration nowadays. The European projects like Blog4Ever, SCAPE, APARSEN, ENSURE and Timbus, national initiatives like Goportis and international collaboration in the 4C project,  they all bring together people from various disciplines . No longer is it only about libraries, archives and data centers, but institutional repositories, health care and business are now also tackling the problem and are presenting their views.  The presentations reflect a greater self-confidence of the digital preservation community; we don’t have the answers to all challenges but we are developing a methodological way to deal with them: the development of standards, life cycle models, cost models, monitoring of the environment, lending from other communities to create tools etc. And most important of all, we know how to find each other.

IMG_2866

Organiser José Borbinha with all varietes of the conference badge on his shirt

But there was also another topic, mainly raised in discussions and during breaks. Our own organisations. The elephant in the room is the fact that our own organisations will need to deal with both analogue and digital material, while the expertise in dealing with analogue material is far more developed in the organisation then the competence of dealing with digital material. Someone said to me “these are different people”.  May be that is the case. Look at the sometimes heated debates  around reading e-books or preferring the paper ones. I like both and don’t think the paper book will disappear. So as a reader I will integrate both worlds and sometimes prefer a paper book above an e-book. This is the world we need to deal with, and organizations need to integrate both worlds. It will require training to have employees that are both  familiar with digital as well as print collections. This is a management challenge, but as digital preservation people we cannot close our eyes for it. We need to convince our management and as the keynote speaker Gildas Illien said (paraphrased by me): “ We need to show our added value. Use the rest of the world to convince your management.” This is how we as digital preservation people can exploit  our  existing collaboration structures!

A deadly sin

Author: Barbara Sierman
Originally posted on: http://digitalpreservation.nl/seeds/a-deadly-sin/

At last week’s iPRES2013 conference in Lisbon, a talk was given about an experiment on the migration of WARC files, done by Tessella, called Studies on the scalability of web preservation. One remark in the talk caused some rumour, namely the fact that the presenter suggested to adapt the WARC file and deviate from the standard. Why did they? Because – as we were told –  the current version of the Wayback Machine software, that enables you to render the WARC file format, is not optimal for rendering WARC files with conversion records. But tweaking the format of the Archival Information Package and store this for the long term is not the way we should go. We preserve information for long term. Our future custodians will not understand this (unless they are told so via metadata and even then) and will assume if they see a WARC format, all the rules in the standard are taken into account. Deviating from this is wrong, in fact it is almost a deadly sin.

After reading the corresponding publication (the conference papers are published by the Portugese National Library as a free e-book), I saw that things were less straight forward. The approach Tessella chose was to create two WARC files: a correct WARC according to the standards and an adapted  WARC for access. From the article:

 This required the development of two different workflows for creating migrated WARC files: one, which is formally correct according to the WARC standard, and maintains the integrity of the WARC schema, and a second which is more pragmatic,  and produces a file that can be displayed correctly by current WARC viewers. This pragmatic workflow can also be used for the migration of container formats  which do not support conversion records, such as ARC files.

So what should one do in the case an ISO  standard does not meet ones requirements? In this case the WARC standard is maintained by the BnF , which can easily be seen if one looks for the standard itself. This is especially mentioned on the internet so that people can get in touch. Another approach is to look for interested parties in the Wayback Machine software, which every one who is involved in web archiving knows, is the Internet Archive. And there is the IIPC, the International Internet Preservation Coalition that is currently initiating a developers working group to improve the Wayback Machine software. So if you have some problems with the standards, think about the millions of precious digital objects  that need to be preserved in that format and get in touch with the community. But don’t tweak the format itself!

KB at DH2013

So, how do you summarise a 4-day conference with 159 papers, 52 posters, 13 workshops and 9 panels in one blogpost? You don’t… But I am going to try anyway!

I had the pleasure to attend and present a poster at the DH2013 conference this year, which took place two weeks ago in Lincoln, Nebraska. It was my first time attending the event and I was not disappointed! After a 14 hour trip (and a good night’s sleep) I started off my DH2013 experience with a wonderful workshop about Voyant, a web-based reading and analysis environment for digital text. All the material from the workshop is available online: http://hermeneuti.ca/workshop/dh13.

DH2013 logo

DH2013 logo

After some introductions of the people there, but also the tool, we formed groups to discuss how Voyant would be of use in our work. I was happy to see that we had quite a big group of librarians there, so of course we discussed how we could either show our own data to the users, but also how we can introduce the tool to students or professors for the university libraries. I’d love to see what our data looks like, so that’s a nice task for the coming months!

The main part of the conference started on my second day and I mainly spent my hours in the various short paper sessions that were held in the conference hotel. There were five papers in a sessions, each with a similar topic. And, being a text junkie, I listened to a lot of text analysis and stylometry, but I also found the time to visit papers on how to best serve researchers with tools, environments and other helpful equipment.

Some of my highlights included the paper of Anna Jobin and Fredric Kaplan on Google’s adwords lexicon, which not only consists of expensive and cheap words, but also of misspelled and non-existents words. The team at the DHLab of EPFL is undertaking a case study on the linguistic effects of autocompletion alghoritms and keyword bidding.

During the poster presentation

During the poster presentation

Another paper that stuck with me was that of our colleague from the British Library, Nora McGregor, who introduced their Digital Scholarship Training Programme. The BL has set up a training programme, consisting of 15 courses, all about anything digital. Their (not so digital?) curators can take a class on, for example, HTML, metadata formats, the BL’s digital collections or linked data. These low level courses can be taken by everyone in the library with an interest in the digital world.

Most of the people took about three courses, but there was also someone who completed the entire programme. Food for thought here at the KB! We have, what we call, kennis sessies (knowledge exchange sessions) and also offer some excellent courses in-house on copyright and digital preservation, but we have never looked at these as an entire course load. Perhaps we should!

KB and BL Poster

KB and BL Poster

And then the absolute top highlight of my DH2013 experience, our poster! The organisation arranged for a very nice and spacious room where we had our poster up on our own board, giving us the opportunity to have a big crowd of people around us. Luckily, this was also the case for us! I spoke to many people about our data (there actually ís an interest in Dutch data in the US!) and about what they would expect from a national library like ours or the BL. Want to leave your feedback as well? Please fill out our survey and help us improve!

So, with this blogpost I have NOT done justice to the conference at all, but have given you a very short overview of some of my wonderful experiences while in Nebraska. Please do read other posts to learn about the rest of the talks, such as the wonderful keynotes by David Ferriero, Willard McCarthy, and my favourite, Isabel Galina.

I met many very interesting people while there and was happy to find out I was accompanied by a lot of librarians. However, most of them were from university libraries. So, national librarians, we need you! Want to share experiences on how we work with digital humanists? Please do get in touch! And researchers? I just want to mention our survey once more.

Identification of PDF preservation risks: the sequel

Author: Johan van der Knijff
Originally posted on: http://www.openplanetsfoundation.org/blogs/2013-07-25-identification-pdf-preservation-risks-sequel

Last winter I started a first attempt at identifying preservation risks in PDF files using the Apache Preflight PDF/A validator. This work was later followed up by others in two SPRUCE hackathons in Leeds (see this blog post by Peter Cliff) and London (described here). Much of this later work tacitly assumes that Apache Preflight is able to successfully identify features in PDF that are a potential risk for long-term access. This Wiki page on uses and abuses of Preflight (created as part of the final SPRUCE hackathon) even goes as far as stating that “Preflight is thorough and unforgiving (as it should be)“. But what evidence do we have to support such claims? The only evidence that I’m aware of, are the results obtained from a small test corpus of custom-created PDFs. Each PDF in this corpus was created in such a way that it includes only one specific feature that is a potential preservation risk (e.g. encryption, non-embedded fonts, and so on). However, PDFs that exist ‘in the wild’ are usually more complex. Also, the PDF specification often allows you to implement similar features in subtly different ways. For these reasons, it is essential to obtain additional evidence of Preflight‘s ability to detect ‘risky’ features before relying on this tool in any operational setting.

Adobe Acrobat Engineering test files

Shortly after I completed my initial tests, Adobe released the Acrobat Engineering website, which contains a large volume of test documents that are used by Adobe for testing their products. Although the test documents are not fully annotated, they are subdivided into categories such as Multimedia & 3D Tests and Font tests. This makes these files particularly useful for additional tests on Preflight.

Methodology

The general methodology I used to analyse these files is identical to what I did in my 2012 report: first, each PDF was validated using Apache Preflight. As a control I also validated the PDFs with the Preflight component of Adobe Acrobat, using the PDF/A-1b profile. The table below lists the software versions used:

Software Version
Apache Preflight 2.0.0
Adobe Acrobat 10.14
Acrobat Preflight 10.1.3 (090)

Re-analysis of PDF Cabinet of Horrors corpus

Because the current analysis is based on a more recent version of Apache Preflight than the one used in the 2012 report (which was 1.8.0), I first re-ran the analysis of the PDFs in the PDF Cabinet of Horrors corpus. The main results are reproduced here. The main differences with respect to that earlier version are:

  1. Apache Preflight now has an option to produce output in XML format (as suggested by William Palmer following the Leeds SPRUCE hackathon)
  2. Better reporting of non-embedded fonts (see also this issue)
  3. Unlike the earlier version, Preflight 2.0.0 does not give any meaningful output in case of encrypted and password-protected PDFs! This is probably a bug, for which I submitted a report here.

Analysis Acrobat Engineering PDFs

Since the Acrobat Engineering site hosts a lot of PDFs, I only focused on a limited subset for the current analysis:

  1. all files in the General section of the Font Testing category;
  2. all files in the Classic Multimedia section of the Multimedia & 3D Tests category.

The results are summarized in two tables (see next sections). For each analysed PDF, the table lists:

  • the error(s) reported by Adobe Acrobat Preflight;
  • the error code(s) reported by Apache Preflight (see Preflight’s source code for a listing of all possible error codes);
  • the error description(s) reported by Apache Preflight in the details output element.

For the sake of readability, the tables only list those error messages/codes that are directly related to font problems, multimedia, encryption and JavaScript. The full output for all tested files can be found here.

Fonts

The table below summarizes the results of the PDFs in the Font Testing category:

Test file Acrobat Preflight error(s) Apache Preflight Error Code(s) Apache Preflight Details
EmbeddedCmap.pdf Font not embedded (and text rendering mode not 3) ; Glyphs missing in embedded font 3.1.3 Invalid Font definition, FontFile entry is missing from FontDescriptor for HeiseiKakuGo-W5
TEXT.pdf Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font ; TrueType font has differences to standard encodings but is not a symbolic font; Wrong encoding for non-symbolic TrueType font 3.1.5; 3.1.1; 3.1.2; 3.1.3; 3.2.4 Invalid Font definition, The Encoding is invalid for the NonSymbolic TTF; Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor; Invalid Font definition, FontFile entry is missing from FontDescriptor for Arial,Italic(repeated for other fonts); Font damaged, The CharProcs references an element which can’t be read
Type3_WWW-HTML.PDF 3.1.6 Invalid Font definition, The character with CID”58″ should have a width equals to 15.56599 (repeated for other fonts)
embedded_fonts.pdf Font not embedded (and text rendering mode not 3); Type 2 CID font: CIDToGIDMap invalid or missing 3.1.9; 3.1.11 Invalid Font definition; Invalid Font definition, The CIDSet entry is missing for the Composite Subset
embedded_pm65.pdf 3.1.6 Invalid Font definition, Width of the character “110” in the font program “HKPLIB+AdobeCorpID-MyriadRg”is inconsistent with the width in the PDF dictionary (repeated for other font)
notembedded_pm65.pdf Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font 3.1.3 Invalid Font definition, FontFile entry is missing from FontDescriptor for TimesNewRoman (repeated for other fonts)
printtestfont_nonopt.pdf* ICC profile is not valid; ICC profile is version 4.0 or newer; ICC profile uses invalid color space;ICC profile uses invalid type Preflight throws exception (exceptionThrown), exits with message ‘Invalid ICC Profile Data’
printtestfont_opt.pdf* ICC profile is not valid; ICC profile is version 4.0 or newer; ICC profile uses invalid color space; ICC profile uses invalid type Preflight throws exception (exceptionThrown), exits with message ‘Invalid ICC Profile Data’
substitution_fonts.pdf Font not embedded (and text rendering mode not 3) 3.1.1; 3.1.2; 3.1.3 Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor; Invalid Font definition, FontFile entry is missing from FontDescriptor for Souvenir-Light(repeated for other fonts)
text_images_pdf1.2.pdf Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font; Width information for rendered glyphs is inconsistent 3.1.1; 3.1.2 Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor

As this document doesn’t appear to have any font-related issues it’s unclear why it is in the Font Testing category. Errors related to ICC profiles reproduced here because of relevance to Apache Preflight exception.

General observations

An intercomparison between the results of Acrobat Preflight and Apache Preflight shows that Apache Preflight’s output may vary in case of non-embedded fonts. In most cases it produces error code 3.1.3 (as was the case with the PDF Cabinet of Horrors dataset), but other errors in the 3.1.x range may occur as well. The 3.1.6 “character width” error is something that was also encountered during the London SPRUCE Hackathon, and according to the information here this is most likely the result of the PDF/A specification not being particularly clear. So, this looks like a non-serious error that can be safely ignored in most cases.

Multimedia

The next table shows the results for Multimedia & 3D Tests category:

Test file Acrobat Preflight error(s) Apache Preflight Error Code(s) Apache Preflight Details
20020402_CALOS.pdf 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
Disney-Flash.pdf Contains action of type JavaScript; Document contains JavaScripts; Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Form field does not have appearance dict; Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry 1.0; 1.2.1 No multimedia-related errors; Preflight did report syntax and body syntax error
Jpeg_linked.pdf Document is encrypted; Encrypt key present in file trailer; Named action with a value other than standard page navigation used; Incorrect annotation type used (not allowed in PDF/A); Font not embedded (and text rendering mode not 3) 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
MultiMedia_Acro6.pdf Document is encrypted; EmbeddedFiles entry in Names dictionary; Encrypt key present in file trailer; PDF contains EF (embedded file) entry; Incorrect annotation type used (not allowed in PDF/A) 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
MusicalScore.pdf CIDset in subset font is incomplete; CIDset in subset font missing; Contains action of type JavaScript; Document contains JavaScripts; Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry; Type 2 CID font: CIDToGIDMap invalid or missing 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
SVG-AnnotAnim.pdf Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry 5.2.1; 1.2.9 Forbidden field in an annotation definition, The subtype isn’t authorized : SVG; Body Syntax error, EmbeddedFile entry is present in a FileSpecification dictionary
SVG.pdf Contains action of type JavaScript; Document contains JavaScripts; Font not embedded (and text rendering mode not 3); Form field has actions; PDF contains EF (embedded file) entry 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
ScriptEvents.pdf Contains action of type JavaScript; Document contains JavaScripts; Font not embedded (and text rendering mode not 3); Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
Service Form_media.pdf Contains action of type JavaScript; Contains action of type ResetForm; Document contains JavaScripts; Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font; Incorrect annotation type used (not allowed in PDF/A); Named action with a value other than standard page navigation used; PDF contains EF (embedded file) entry 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
Trophy.pdf Contains action of type JavaScript; Document contains JavaScripts; Font not embedded (and text rendering mode not 3); Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
VolvoS40V50-Full.pdf Preflight exits with: “An error occurred while parsing a contents stream. Unable to analyze the PDF file” 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
gXsummer2004-stream.pdf File cannot be loaded in Acrobat (damaged file) 1.0; 1.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
phlmapbeta7.pdf Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
us_population.pdf Preflight exits with: “An error occurred while parsing a contents stream. Unable to analyze the PDF file” 1.0; 1.2.1 No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
movie.pdf Incorrect annotation type used (not allowed in PDF/A) 5.2.1 Forbidden field in an annotation definition, The subtype isn’t authorized : Movie
movie_down1.pdf Incorrect annotation type used (not allowed in PDF/A) 5.2.1 Forbidden field in an annotation definition, The subtype isn’t authorized : Movie
remotemovieurl.pdf Font not embedded (and text rendering mode not 3); Incorrect annotation type used (not allowed in PDF/A) 5.2.1; 3.1.1; 3.1.2; 3.1.3 Forbidden field in an annotation definition, The subtype isn’t authorized : Movie; Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor; Invalid Font definition, FontFile entry is missing from FontDescriptor for Arial

General observations

The results from the Multimedia PDFs are interesting for several reasons. First of all, these files include a wide variety of ‘risky’ features, such as multimedia content, embedded files, JavaScript, non-embedded fonts and encryption. These were successfully identified by Acrobat Preflight in most cases. Apache Preflight, on the other hand, only reported non-specific and fairly uninformative errors (1.0 + 1.2.1) for 12 out of 17 files. Even thoughPreflight was correct in establishing that these files were not valid PDF/A-1b, it wasn’t able to drill down to the level of specific features for the majority of these files.

Summary and conclusions

The re-analysis of the PDF Cabinet of Horrors corpus, and the subsequent analysis of a sub-set of the Adobe Acrobat Engineering PDFs shows a number of things. First, Apache Preflight 2.0.0 does not properly identify encryption and password-protection. This looks like a bug that is probably easily fixed. Second, the analysis of theFont Testing PDFs from the Acrobat Engineering site revealed that non-embedded fonts may result in a variety of error codes in Apache Preflight (assuming here that the Acrobat Preflight results are accurate). So, when usingApache Preflight to check font embedding, it’s probably a good idea to treat all font-related errors (perhaps with the exception of character width errors) as a potential risk. The more complex PDFs in the Multimedia category proved to be quite challenging to Apache Preflight: for most files here, it was not able to identify specific features such as multimedia content, embedded files, JavaScript and non-embedded fonts. This is not necessarily a problem if Apache Preflight is used for its intended purpose: verify if a PDF conforms to PDF/A-1. However, it does rather limit its use as a tool for profiling heterogeneous PDF collections for specific preservation risks at this stage. This may well change with future versions; in fact the specificity of Preflight‘s validation output already improved considerably since version 1.8.0. In the meantime it’s important to keep the expectations about the tool’s capabilities realistic, in order to avoid some potential unintended misuses.

Links

Europeana Newspapers Refinement & Aggregation Workshop

The KB participates in the Europeana Newspapers project that has started in February 2012. The project will enrich 18 million pages of digitised newspapers with Optical Character Recognition (OCR), Optical Layout Recognition (OLR) and Named Entity Recognition (NER) from all over Europe and deliver them to Europeana. The project consortium consists of 18 partners from all over Europe: some will provide (technical) support, while other will provide their digitised newspapers. The KB has two roles: we will not only deliver 2 million of our newspaper pages to Europeana, but we will also enrich ours and the newspapers of other partners with NER.

Untitled

Europeana Newspapers Workshop in Belgrade

In the last months, the project has welcomed 11 new associated partners and to make sure they can benefit as much as possible from the experiences of the project partners the University Library of Belgrade and LIBER jointly organised a workshop on refinement and aggregation on 13 and 14 June. Here, the KB (Clemens Neudecker and I) presented the work that is currently being done to make sure that we will have Named Entities for several partners. To make sure that the work that is being done in the project also benefits our direct colleagues, we were joined by someone from our Digitisation department.

The workshop started with a warm welcome in Belgrade by the director of the library, Prof. Aleksandar Jerkov. After a short introduction into the project by the project leader Hans-Jörg Lieder from the State Library Berlin, Clemens Neudecker from the KB presented the refinement process of the project. All presentations will be shared on the project’s Slideshare account. The refinement of the newspapers has already started and is being done by the University of Innsbruck and the company CCS in Hamburg. However, it was still a big surprise when Hans-Jörg Lieder announced a present for the director of the University Library Belgrade; the first batch of their processed newspapers!

Giving a gift of 200,000 digitised and refined newspapers to our Belgrade hosts

Giving a gift of 200,000 digitised and refined newspapers to our Belgrade hosts

The day continued with an introduction into the importance of evaluation of OCR and OLR and a demonstration of the tools used for this by Stefan Pletschacher and Cristian Clausner from the University of Salford. This sparked some interesting discussions in the break-out sessions on methods of evaluation in the libraries digitising their collections. For example, do you tell your service provider what you will be checking when you receive a batch? You could argue that the service provider would then only fix what you check. On the other hand if that is what you need to reach your goal it would save a lot of time and rejected batches.

After a short getting-to-know-each-other session the whole workshop party moved to the Nikola Tesla Museum nearby where we were introduced to their newspaper clippings project. All newspaper clippings collected by Nikola Tesla are now being digitised for publication on the museum’s website. A nice tour through the museum followed with several demonstrations (don’t worry, no one was electrocuted) and the day was concluded with a dinner in the bohemian quarter.

Breakout groups at the Belgrade Workshop

The second day of the workshop was dedicated solely to refinement. I kicked off the day with the question ‘What is a named entity?’. This sounds easy, but can provide you with some dilemmas as well. For example, a dog’s name is a name, but do you want it to be tagged as a NE? And what do you do with a title such as Romeo and Juliet? Consistency is key in this and as long as you keep your goal in mind while training your software you should end up with the results you are looking for.

Claus Gravenhorst followed me with his presentation on OLR at CCS, by using docWorks, with which they will process 2 million pages. It was then again our turn with a hands-on session about the tools we’re using, which are also available on Github. The last session of the workshop was a collaboration between Claus Gravenhorst from CCS and Günter Mühlberger from the University of Innsbruck who gave us a nice insight into their tools and the considerations made when working with digitised newspapers. For example, how many categories would you need to tag every article?

Group photo from the Europeana Newspapers workshop in Belgrade

All in all, it was a very successful workshop and I hope that all participants enjoyed it as much as I have. I at least am happy to have spoken to so many interesting people with new experiences from other digitisation projects. There is still much to learn from each other and projects like Europeana Newspapers contribute towards a good exchange of knowledge between libraries to ensure our users get the best experience when browsing through the rich digital collections.

Europeana – the case for funding: tweet #AllezCulture

A drastic cut was made in the budget for the Connecting Europe Facility (CEF) from 9 billion to 1 billion euros. This will hit Europeana, the infrastructure supporting Europe’s free digital library, museum and archive, very hard. Europeana is now being asked to put the case for funding under the revised guidelines for CEF, which were issued 28 May 2013. Europeana will face severe competition for the available funding from other digital service infrastructure such as e-Justice, e-Health and Safer Internet. All good causes in their own right, but the wonderful digital culture infrastructure that has been built in the last decade will soon get squashed if we do not speak out now!  So here goes: 

europeana

Here is a summary of the three arguments for funding:

1 Europeana supports economic growth.
Some Impact Indicators:

  • To date, 770 businesses, entrepreneurs, educational and cultural organisations are exploring ways of including Europeana information in their offerings (websites, apps, games etc.) through our API. See examples such as inventingeurope.eu and www.zenlan.com/collage/europeana.
  • Digital heritage creates jobs – in Hungary, for example, over 1,000 graduates are now involved in digitising heritage that will feed in to Europeana. Historypin in the UK predicts it will double in size with the availability of more open digital cultural heritage.

2. Europeana connects Europe. 

People often speak about closing the digital divide and opening up culture to new audiences but very few can claim such a big contribution to those efforts as Europeana’s shift to cultural commons.’ Neelie Kroes, Vice President of the Commission

3. Europeana makes Europe’s culture available for everyone.

In 2012, all 20m Europeana records were released under a Creative Commons Zero public domain dedication making them available for re-use both commercially and non-commercially. Europeana’s CC0 release is a ‘coup d’état’ that ‘will help to establish a precedent for other galleries, libraries, archives and museums to follow – which will in turn help to bring us that bit closer to a joined up digital commons of cultural content that everyone is free to use and enjoy.Jonathan Gray, Open Knowledge Foundation.

For those unaware of Europeana – here is what they do: 

Europeana has been transformative in opening up data and access to cultural heritage and now leads the world in accessible digital culture that will fuel
Europe’s digital economy. Through Europeana today, anyone can explore
27 million digitised objects including books, paintings, films and audio.

Europeana is a catalyst for change for cultural heritage

– Because they make cultural heritage accessible online.

– Because they have standardised the data of over 2,200 organisations, covering all European countries and 29 European languages.

– Because they provide creative industries and business start-ups with rich, interoperable material, complete with copyright information.

– And because they ensure that every citizen, whether young or old, privileged or deprived, can be a digital citizen.

So please support Europeana by tweeting, blogging, facebooking and whatever other media you like, using the hashtag #AllezCulture!

EPUB for archival preservation: an update

Author: Johan van der Knijff
Originally posted on: http://www.openplanetsfoundation.org/blogs/2013-05-23-epub-archival-preservation-update

Last year (2012) the KB released a report on the suitability of the EPUB format for archival preservation. A substantial number of EPUB-related developments have happened since then, and as a result some of the report’s findings and conclusions have become outdated. This applies in particular to the observations onEPUB 3, and the support of EPUB by characterisation tools. This blog post provides an update to those findings. It addresses the following topics in particular:

  • Use of EPUB in scholarly publishing
  • Adoption and use of EPUB 3
  • EPUB 3 reader support
  • Support of EPUB by characterisation tools

In the following sections I will briefly summarise the main developments in each of these areas, after which I will wrap up things in a concluding section.

Use of EPUB in scholarly publishing

Although scholarly publishing is still dominated by PDF, the use of EPUB in this sector is on the rise. This blog post by Todd Carpenter gives the following examples:

At the time of writing, the above publishers are all using EPUB 2.

Adoption and use of EPUB 3

Over the last year a number of organisations that are representing the publishing industry have expressed their support of EPUB 3. The Book Industry Study Group (BISG) is a trade association for companies in the publishing industry. Last year (August 2012) BISG released a policy statement in which it endorsed “EPUB 3 as the accepted and preferred standard for representing, packaging, and encoding structured and semantically enhanced Web content — including XHTML, CSS, SVG, images, and other resources — for distribution in a single-file format“. Early this year (March 2013) the International Publishers Association (IPA) issued a press releasethat also endorsed EPUB 3 as a “preferred standard format for representing HTML and other web content for distribution as single-file publications“. IPA represents over 60 national publishing organisations from more than 50 countries. Finally, the European Booksellers Federation recently released a report on the interoperability of eBook Formats. Its authors did a comparison of the features and functionality provided by EPUB 3, Amazon’s KF8 (Kindle) and Apple’s e-book formats. They concluded that EPUB 3 “clearly covers the superset of the expressive abilities of all the formats“, and that there is “no technical or functional reason not to use and establish EPUB 3 as an/the interoperable (open) ebook format standard“. This all suggests that EPUB 3 is widely supported by the publishing industry.

Having said that, the actual use of EPUB 3 is still limited at this stage, even though some publishers have already started using the format. Earlier this year technical publisher O’Reilly started releasing all their new eBook bundles in EPUB 3 format. The announcement mentions that their backlist will be updated as well. Interestingly, they decided to create “hybrid” EPUBs that are backward-compatible with EPUB 2. In November 2012 publisher Hachette also announced the launch of their EPUB 3 program.

EPUB 3 reader support

At this time reader support for EPUB 3 is still limited, but there have been a number of significant developments since the second half of 2012:

Support of EPUB by characterisation tools

The 2012 report concluded that EPUB was not optimally supported by characterisation tools. This situation has improved quite a lot since that time.

Identification

EPUB is now included in PRONOM, and has a corresponding DROID signature. This means that Fido should now be able to identify the format as well. On a side note, PRONOM doesn’t differentiate between EPUB 2 and 3, and it appears that the current record (which is only an outline record anyway) either combines both versions, or only refers to EPUB 2. PRONOM should probably be more specific on this.

Validation and feature extraction

The 2012 report included tests of 2 EPUB validator tools: epubcheck and flightcrew. While testing epubcheck in 2012, I was’t entirely happy with the rather unstructured output that the tool produced. Also, I couldn’t find any tool that was capable of extracting technical meta-information about an EPUB, like the presence of encryption or other digital rights management technology (feature extraction). Happily, starting with version 3.0 epubcheck is capable of extracting this kind of information. Moreover, it added an option to report its output in structured XML format that follows the JHOVE schema. I haven’t done any elaborate testing, but a quick run on some ofthese EPUB 3 samples showed that epubcheck was able to identify font obfuscation, in which case a property hasEncryption (valuetrue) is reported. I wasn’t able to find any EPUB files with DRM, so I cannot confirm if epubcheck detects this as well.

Flightcrew

As for flightcrew, no new versions of that tool have been released since August 2011, and it looks like it is not under any active development.

Discussion and conclusions

Since the release of the KB report on the suitability of EPUB for archival preservation the EPUB landscape has changed rather a lot. First, a number of academic publishers have started to offer scholarly content in this format. Although EPUB 3 is still in its early stages, various organisations representing the publishing industry have explicitly expressed their support of EPUB 3. A number of software applications now exist that are able to read the format, and work on a high-performance open source EPUB 3 Software Development Kit is backed by major players in the digital publishing industry (including e-reader manufacturers such as Kobo and Sony). EPUB support by characterisation tools has improved as well, mostly thanks to a number of recent enhancements ofepubcheck. So, overall, EPUB‘s credentials as a preservation format appear to have improved quite a bit over the last year. In the case of EPUB 3 it’s still too early to say anything about actual adoption, but the conditions for adoption to happen look pretty favourable. This is something I will get back to in my next update, perhaps in another year from now.

Useful links

Re:publica 2013: In/side/out

From 6-8 May the Re:publica conference was held in Berlin – one of the largest international conferences in the field of blogging, social media and our society in the digital age. In numbers alone the event is already quite impressive: with over 5.000 attendees, 300 speakers, 3 Tb of video footage or 95 hours of sessions, talks, and panel discussions and around 27.600 social media activities, it was hard to keep up: already before the conference started, the Twitter stream of #rp13 exploded with tweets!

republica_1          republica_2Photos by Gregor Fischer (https://www.flickr.com/photos/re-publica/)

But it was not only the size of the conference that was impressive: there was also a large number of interesting speakers and talks. Unfortunately my German was not sufficient to follow every detail, so I’ve mainly attended the sessions in English. This did make for an easier choice, as there were often around 10 parallel sessions and workshops. For most of the  presentations a video is online – a good overview of all sessions is available here: http://michaelkreil.github.io/republicavideos/.

Several talks focused on the differences in internet access and internet censorship worldwide. During the session ‘ 403 Forbidden: A Hands on Experience of the Iranian Internet’ you could gain more insight in how the Iranian internet is censored by participating in a quiz. Several websites and newpaper images were shown, and you had to guess which sites are blocked and which photos are manipulated.  The results were often surprising (for example, the sites of the KKK and Absolut Vodka are blocked, but the sites of Smirnoff Vodka and the American Nazi party are not), showing how internet filtering in Iran is both arbitrary and targeted, designed to make you feel insecure online.

michella obama

#403forbidden: Iranian photo manipulation of Michelle Obama’s dress

Another interesting talk in this area was ‘Internet Geographies: Data Shadows and Digital Divisions of Labour’ (unfortunately the video of this talk is not online yet). Mark Graham of the Oxford Internet Institute showed how, despite the initial hope that internet would offer everyone around the world the opportunity to share their knowledge, there are significant concentrations of knowledge. For example, Europe has just over 10% of the world’s population, but produces nearly 60% of all Wikipedia articles. There are even more Wikipedia articles written about Antarctica than about any country in South America or Africa. He illustrated these numbers with great visualisations, most of which you can find through this blog: http://www.zerogeography.net/2011/09/geographies-of-worlds-knowledge.html

In ‘Investigation 2.0’ Stephanie Hankey and Marek Tuszynksi of Tactical Tech discussed ways in which individual hackers and people working with data visualisation are nowadays working together with journalists to make certain underexposed data visible and present it in a way that gives us more insight into complex social and political issues. A good example of this is the visualisation that was made of drone strikes in Pakistan since 2004: http://drones.pitchinteractive.com.

Data visualisation can also be done through food, which the three women behind the cool website http://bindersfullofburgers.tumblr.com call ‘Data cuisine’ (or How to get juicy data from spreadsheets). They are working on visualising data that is usually thought of as boring and dry, such as election results, in creative and appealing ways. In Binders full of Burgers they displayed the US election results of 2012 with burgers and fries, while they used waffles, candy and fruit to show the local Berlin elections (http://wahlwaffeln.tumblr.com/). They also made their own talk more appealing by handing out cookies for audience participation :)

Then there was a good overview presentation by Joris Pekel of the Open Knowledge Foundation on Open Data & Culture – Creating the Cultural Commons. After presenting some impressive numbers of available online records, media files and metadata from Europeana, DPLA and Wikimedia Commons, he focused on the possibilities of enabling the public to connect and contextualise open data in the future through initiatives such as OpenGLAM. His talk is also available from Slideshare at http://t.co/bpqBEB28sb.

A very direct way of promoting open data, transparency and hacker culture was presented by Daniela B. Silva in Hacker culture on the road. In 2011, a Brazilian hacker community came up with the plan of starting a Hacker bus: they crowdfunded the budget for buying an old school bus and started travelling around Brazil, organising local events focused on promoting government transparency and hacker activism. Volunteer hackers, digital activists, lawyers, artists and journalists joined the bus, listened to the needs of the people they met and helped them to develop answers with the help of available technology, such as local street mapping applications and apps but also new local legislations.

Finally, one of the most fascinating talks was that of Neil Harbisson and Moon Ribas on Life with extra senses – How to become a cyborg. They showed ways of extending and creating new senses and perceptions by applying technology to the human body, as well as the artistic projects they created based on these sensory extensions. For example, due to a medical condition Neil Harbisson only sees in black and white, so he developed an electronic eye that transforms the colours around him into sound, allowing him to experience colour in a different way.  Together they have established the Cyborg Foundation to further promote their activities and help more people to become a cyborg, because they believe all humans should have the right to extend their senses and perceptions. Interestingly they see this as a way for people to get closer to nature instead of alienating themselves from it, since many animals have very different sensory perceptions than we do.

935450_502820113100275_1127485294_n

 

 

 

 

 

 

 

 

Neil Harbisson and Moon Ribas: when Neil scans Moon’s dress with his Eyeborg, he will hear the song ‘Moon River’ (Photo by Tony Sojka, https://www.flickr.com/photos/re-publica/)

All this is just a fraction of everything that was presented at Re:Publica: at http://michaelkreil.github.io/republicavideos/ you can find videos of nearly every talk – and if you speak German, you’re even more lucky!

Digital Humanities at the National library

About two months ago the Journal for Library Associations published an issue completely about Digital Humanities in libraries. Enthusiastically I printed all the open access articles (I know, not very nature conscious of me…) and put them on my desk. As it often goes with papers on desks, they’ve been lying there ever since. This changed this morning as I took out the stack and started reading them. And I loved it! Article after article I took out my highlighter and marked sentences and paragraphs that sounded too familiar to me, working in a research library with an interest in Digital Humanities.

The KB has started to look at Digital Humanities (DH) as a topic not so long ago, but has been involved with DH related projects for quite some time, although there were not called DH at the time. Examples are the CATCH projects that started in 2004, but since the beginning of our digitising-days our material is used by a variety of people and institutions. However, the KB is special in a way when it comes to doing DH research. We are the national library of the Netherlands and are thus not connected to a specific university or research institution. This means that we do not employ our own researchers. We do have a Research department, but most of the people here do not dive into our content, but do research to ensure the public can do this.

catchplus

The continuation of CATCH, CATCHPlus, where prototypes are converted to reliable tools

Although the JLA I read only discusses university libraries, and their associated researchers, this does not mean that the articles from for example Miriam Posner or Bethany Nowviskie are not relevant for us. The, often mentioned, lack of flexibility that is apparently inherent to a library also exist here and the desire to only publish something once it is perfect is something I too can relate to. Working with digitised material is never perfect. The software is not perfect, so how can the outcomes be? Nonetheless, the KB chose to show these imperfections in our OCR by opening up the texts to the public, including all mistakes and an estimate of accuracy.

kbnewspaper

A KB newspaper article. The OCR quality of this article is estimated at 84,9% character accuracy.

Being a research institute, with a large digital corpus that we are more than happy to share, without our own researchers (apart from the occasional research fellow), the KB not only faces the challenges of the university libraries as mentioned by Miriam Posner in her article (i.e. inflexibility, lack of time, authority, and incentive, overcautionesness, etc.), but I believe another crucial element can be added to this list: No affiliated researchers. Until not so long ago, when a researcher wanted to use (a section) of our digitised sets he/she would find someone from within the library who could help them get it. There was no official route to obtain the data or one contact person for a specific set, so it could be possible that people left the KB with hard disks full of images or that they tracked one of our employees down at a conference and badgered them until they got an e-mail with instructions on how to harvest a collection. Luckily, this has changed with the creation of the Data Services team.

The Data Services team are the go-to guys when it comes to our digital sets. They have taken up the responsibilities of advertising our datasets on our (unfortunately only in Dutch) website, at events and conferences and on the Dutch Open Data community, such as Open Data Nederland. We hope these efforts will lead to interesting use of our data and perhaps even some enrichments that we might implement in the future (OCR correction anyone?). But how can we be sure that our data does indeed gets used and that we reach the people who might be interested? And how do we know if our methods is in fact what they are looking for?

photo-1

The KB at the CLIN2013 conference.

This issue is one that I would imagine is easier to solve when you can simply walk to the other side of the building, knock on some doors and talk to researchers of whom you know their interests, because they teach Data Mining at your university. Unfortunately, we are not in that position, apart from the people who have asked for our data and those that will come to our (currently a work in progress) KB Lab. Now that we have the  instructions to harvest our sets available on the website, less and less people will probably be doing this, leaving us more in the dark about what interesting things are happening with the digitised Early Dutch Books Online or the ANP radio bulletins.

So, how do we get and stay in touch with interested parties that might contribute to the enrichment of our collections? How can we be sure that what we are doing is in fact what researchers need? How much can and do we want to adapt our methods to fit the need of researchers? For example, do we want to offer all possible data formats if there is a demand for it or is that something that the scholars might be able to tackle themselves? (Solutions and ultimate answers of course always welcome in the comment section below!)

We are undertaking several activities to try to find answers and also our place in the wonderful world of Digital Humanities. The establishment of our own KB Lab, where we will to work with scholars who wish to do something with our data, is one such activity. Another is the poster session that we will present together with the BL Labs project at the DH2013 conference this summer. Our aim there is to talk to different researchers about our collections and their ways of working. What types of collections they would like, what data format they would love to see, but also what they would like to do with our data. So, if you’re around in Nebraska, please come and find me at the posters and let’s talk this through!

Performance measurement – A state of the art review

Author: Henk Voorbij

The Handboek Informatiewetenschap (Handbook Information Science) contains about 150 contributions from experts on various subjects, such as budgeting, collection management, open access, website archiving, digitization of heritage collections, information retrieval software and gaming in the library. The Handbook is published as a loose-leaf paper version and as an online version (http://www.iwabase.nl/iwa-base/). Unfortunately, it is not well known.

c_iw

Articles are being updated every ten years. Recently, I was requested to update the article on Performance Indicators, originally published in 2000 and authored by Peter te Boekhorst. My contribution starts with definitions of core concepts (for example: what is the difference between statistics and performance indicators). It continues with general guidelines that may be helpful to libraries which aim to develop their own instrument for performance measurement. Among these are models such as the classical system approach (input – throughput – output – outcomes) and the Balanced Scorecard, and international standards, such as provided by ISO and IFLA. In the Netherlands, there are well developed benchmark instruments for university libraries and libraries from universities of applied sciences. I am involved with the development of these systems and analysis of the data since many years and describe my experiences in depth, in order to provide an example of the caveats and benefits of performance measurement. The last three chapters address potential additions to traditional performance indicators: user surveys, web statistics and outcomes.

Updating an earlier version offers an excellent opportunity to depict the progress in the field. Two things struck me most. One is the fast rise of the concept ‘Key Performance Indicators’. There’s no agreement in the literature of what this concepts actually means. Some use it loosely and do not make a genuine distinction between performance indicators and key performance indicators. Others have very pronounced ideas of its meaning: there should be no more than ten KPI’s, they should be measured daily or weekly, they should be coupled to critical success factors rather than strategic goals, and they should be understood by a fourteen year old child. The other thing is the growing interest in outcomes, the popularity of LibQual+ as an instrument to measure user satisfaction and the upsurge of new technologies such as web analytics. I can’t wait to see the 2023 version of the paper.

« Older posts Newer posts »

© 2018 KB Research

Theme by Anders NorenUp ↑