In my previous blog post I addressed the detection of broken audio files in an automated workflow for ripping audio CDs. For (data) CD-ROMs and DVDs that are imaged to an ISO image, a similar problem exists: how can we be reasonably sure that the created image is complete? In this blog post I will discuss some possible ways of doing this using existing tools, along with their limitations. I then introduce Isolyzer, a new tool that might be a useful addition to the existing methods.

Checksums

A number of techniques exist to verify a newly created ISO image. A seemingly obvious solution would be to do a checksum comparison on both the ISO image and the physical carrier. For instance, the following will work on any Linux system:

md5sum myimage.iso
md5sum /dev/sr0

The first line computes an MD5 checksum from the ISO image; the second line repeats this for the physical carrier. This method is not completely fail-safe. In some tests I did over a year ago, I ran into a a very strange issue where my attempts to image a CD would sometimes result in incomplete reads, and, as a result, truncated ISO images. The problem was most likely caused by faulty hardware (the machine on which I ran those tests more or less died shortly afterwards). Most worryingly, the machine would sometimes return incomplete data, both while creating the ISO image as well as during the subsequent checksum calculation on the physical carrier. The result of this was that the computed checksums were identical in both cases, which meant that the image passed the checksum quality check, even though it was incomplete!

Isovfy

The popular cdrtools library includes a tool called isovfy. Its man page describes it as follows:

isovfy is a utility to verify the integrity of an iso9660 image. Most of the tests in isovfy were added after bugs were discovered in early versions of mkisofs. It isn’t all that clear how useful this is anymore, but it doesn’t hurt to have this around.

I already commented on this tool in an earlier blog post:

The documentation of the tool isn’t very clear about what specific checks it performs. In one of my tests I fed it an ISO image that had its last 50 MB missing (truncated). This did not result in any error or warning message! Most of the reported isovfy errors that I came across in my tests simply reflected the file system on the physical CD not conforming to ISO 9660 (this seems to be pretty common).

You can try this yourself by running isovfy on the following two ISO images:

• This is a small (350 kB) ISO image that is intact
• Here’s the same image with most of its data truncated (the image is only 48 KB)

I ran both images through isovfy (version 3.02a06); both resulted in the following output:

Root at extent 17, 2048 bytes
[0,0]
No errors found

This demonstrates that isovfy is not very useful for detecting truncated ISO files.

Digging into the specs

At this point I decided it was time to start digging into some specs. The ISO 9660 page on the OSDev Wiki gives a good explanation of the internal organisation of an ISO 9660 image. From this I learnt that the Primary Volume Descriptor (which is a data structure that is present on all ISO images) contains two interesting fields:

• Volume Space Size, which is the "number of Logical Blocks in which the volume is recorded";
• Logical Block Size, which is "the size in bytes of a logical block".

In theory, multiplying both figures should give the expected size of the ISO image, and this would provide a useful way to check if data are missing. To test this, I wrote a Python script that parses an ISO’s Primary Volume Descriptor fields, calculates the expected file size and then compares this against the actual file size. Running the script against some 20 ISO images I had lying around showed that for 7 files the expected size was indeed identical to the actual file size. For most images, the actual size turned out to be marginally larger than expected (typically about 300-600 kB). For 3 images, the actual size was about twice the expected size. Digging deeper, I found out that these were hybrid images that contain an Apple partition on top of the ISO 9660 file system. According to this Wikipedia article, these hybrid discs come in two varieties:

1. Hybrid discs that contain an Apple Partition Map (located at 512 bytes into the disc/image).
2. Hybrid discs without a Partition Map. These contain a Master Directory Block (located at 1024 bytes into the disc/image).

In my case all of the 3 hybrid images turned out to be of the first category. Using the information here and here I was able to add detection of such hybrid images to my code, as well as a simple parser for the ‘zero block’ structure that contains two fields that define the partition’s size: Block Size and Block Count. For my hybrid images, multiplying both figures resulted in a value that was close to (but again marginally smaller than) the actual file size.

Finally, I also added detection of the second hybrid disc category (no Partition Map, but Master Directory Block). The Master Directory Block also contains Block Size and Block Count fields that allow one to calculate the size of the file system.

Isolyzer

I wrapped up the results of the above analyses into Isolyzer, which is a dedicated (Python) tool for checking the size of an ISO image. What it does is this:

1. Locate the image’s Primary Volume Descriptor (PVD).
2. From the PVD, read the Volume Space Size (number of sectors/blocks) and Logical Block Size (number of bytes for each block) fields.
3. Calculate the expected file size as ( Volume Space Size x Logical Block Size ).
4. If the image contains an Apple Partition Map, read the Block Size and Block Count fields from the ‘zero block’
5. Calculate the expected file size as ( Block Size x Block Count )
6. If the image contains an Apple Master Directory Block, read its Block Size and Block Count fields
7. Calculate the expected file size as ( Block Size x Block Count )
8. Calculate the final expected file size as the largest value out of any of the above 3 values
9. Compare this against the actual size of the image files.

In addition to this, Isolyzer also extracts and reports technical metadata from the Primary Volume Descriptor and the Zero Block.

Currently the test results are reported in the following format (this may well change in upcoming releases):

<tests>
    <containsISO9660Signature>True</containsISO9660Signature>
    <containsApplePartitionMap>False</containsApplePartitionMap>
    <containsAppleHFSHeader>False</containsAppleHFSHeader>
    <containsAppleMasterDirectoryBlock>False</containsAppleMasterDirectoryBlock>
    <parsedPrimaryVolumeDescriptor>True</parsedPrimaryVolumeDescriptor>
    <sizeExpected>358400</sizeExpected>
    <sizeActual>358400</sizeActual>
    <sizeDifference>0</sizeDifference>
    <sizeAsExpected>True</sizeAsExpected>
    <smallerThanExpected>False</smallerThanExpected>
</tests>

In the above example the sizeExpected field is the size as calculated from the ISO/Apple headers, and sizeActual is the actual size. In this case both are identical. Below some output for a truncated ISO:

<tests>
    <containsISO9660Signature>True</containsISO9660Signature>
    <containsApplePartitionMap>False</containsApplePartitionMap>
    <containsAppleHFSHeader>False</containsAppleHFSHeader>
    <containsAppleMasterDirectoryBlock>False</containsAppleMasterDirectoryBlock>
    <parsedPrimaryVolumeDescriptor>True</parsedPrimaryVolumeDescriptor>
    <sizeExpected>358400</sizeExpected>
    <sizeActual>49157</sizeActual>
    <sizeDifference>-309243</sizeDifference>
    <sizeAsExpected>False</sizeAsExpected>
    <smallerThanExpected>True</smallerThanExpected>
</tests>

So, in this case sizeDifference is negative, and flag smallerThanExpected equals ‘True’ (which indicates a damaged image).

Feedback wanted

At this stage Isolyzer is a bit experimental and pretty rough around the edges, and I wouldn’t recommend it for production use. Nevertheless I’m curious about any feedback on the tool. Do others find this useful? Are things missing (i.e. other hybrid disc types I’m not aware of), or did I get anything completely wrong?

One thing that puzzles me a bit is that for the majority of ISO images I’ve come across, the expected size as calculated by Isolyzer is marginally smaller than the actual size. The difference is typically in the order of about 300-600 kB. I’m not quite sure what’s causing this, although this article mentions that some CD writing software packages add padding bytes when writing a CD. I wasn’t able to verify if this, although this SuperUser answer on validating a burnt DVD suggests it as well. If anyone knows more about this, please let me know!

Download links

Isolyzer can be found here on Github. It can be installed using pip; see the instructions here. For Windows users who cannot/don’t want to install Python I also provided stand-alone Windows binaries, which are available for download here.

To get a better idea of what software tool(s) are best suitable for this task, I created a small dataset of audio files which I deliberately damaged. I subsequently ran each of these files through a set of candidate tools, and then looked which tools were able to detect the faulty files. The first half of this blog post focuses on the WAVE format; the second half covers the FLAC format (at the moment we haven’t decided on which format to use yet).

WAVE dataset

For the WAVE dataset I started out with a small, intact WAVE file. Using a Hex editor I then made the following derivatives of this file:

• frogs-01-last-byte-missing.wav – one byte is missing at the end of the file
• frogs-01-last-2032-bytes-missing.wav – a chunk of 2032 bytes is missing at the end of the file
• frogs-01-byte-missing-at-offset-811537.wav – one byte is missing at offset 811537

Candidate tools, WAVE

The candidate tools I used to analyse the WAVE files are:

• jhove includes a WAVE validation module, which makes it an obvious choice. The tested version is 1.14.6, 2016-05-12.
• shntool is a "multi-purpose WAVE data processing and reporting utility". It was first released in 2000. The tested version is 3.0.7.
• ffmpeg is a popular conversion tool for audio and video formats. The tested version is 3.2.2.
• mediainfo is a widely-used feature extraction tool for audiovisual files. The tested version is v0.7.81.

Note that of the above tools, only Jhove and Shntool are designed to detect problems in WAVE files. Both Ffmpeg and Mediainfo were primarily designed for other purposes (format conversion and technical metadata extraction), and they were not designed to detect defective files! I included these tools here mainly because they are widely used, and I was curious whether they would throw up anything interesting in case of defective files¹. I ran the tools with the following command-line arguments (replacing "foo.wav" with the actual file name):

Jhove

jhove -m WAVE-hul foo.wav

Shntool

shntool info foo.wav

Ffmpeg

ffmpeg -v error -i foo.wav -f null -

Mediainfo

mediainfo foo.wav

I automated this using a simple shell script that runs each tool on all files, and then writes the output to a set of text files.

Results, WAVE

The full output results of each tool can be found here.

Jhove

The ‘Status’ field in Jhove’s output summarises the validation outcome. Here are the results for each file:

So, Jhove was unable to detect any of the damaged files at all!

Shntool

Shntool checks a WAVE on six criteria, which are listed in its output under ‘Possible problems’:

Possible problems:
  File contains ID3v2 tag:    no
  Data chunk block-aligned:   yes
  Inconsistent header:        no
  File probably truncated:    no
  Junk appended to file:      no
  Odd data size has pad byte: n/a

The thing to watch here is the ‘File probably truncated’ item:

So, Shntool was able to detect all damaged files.

Ffmpeg

For our Ffmpeg call we monitor any errors that are sent to the standard error stream. The results:

Interestingly, Ffmpeg reports an error for both files that have 1 byte missing, but it doesn’t for the file that has 2023 bytes missing. This suggests that Ffmpeg is not suitable for detecting broken WAVE files.

Mediainfo

Mediainfo didn’t report errors or warnings for any of these files. This is not surprising, but it does confirm that Mediainfo cannot be used for detecting broken WAVE files.

FLAC dataset

Analogous to the WAVE dataset, I started out with a small, intact FLAC file, which I then butchered into the following derivative files:

• frogs-01-last-byte-missing.flac – one byte is missing at the end of the file
• frogs-01-last-1000-bytes-missing.flac – a chunk of 1000 bytes is missing at the end of the file
• frogs-01-byte-missing-at-offset-651202.flac – one byte is missing at offset 651202

Candidate tools, FLAC

The set of candidate tools is identical to the one used for the WAVE analysis, with two exceptions:

• flac is the reference implementation of the FLAC format. The tested version is 1.3.0.
• Since Jhove does not include a FLAC module, it was not used.

Flac

The Flac tool is able to encode audio to FLAC, and decode and analyze FLAC files. For this tests I ran it with the * -t* (or –test) option:

flac -t foo.flac

This decodes a FLAC without writing the decoded data to a file. Any errors during the decoding process are reported to the standard error stream.

Results, FLAC

The full output results of each tool can be found here.

Shntool

Even though Shntool supports FLAC, it was not able to detect the missing data in any of the files:

So, Shntool does not provide any meaningful information on whether a FLAC is damaged.

Ffmpeg

Here are the results for Ffmpeg:

So, Ffmpeg was able to identify all damaged FLACs.

Mediainfo

Similar to the WAVE results, Mediainfo again didn’t report errors or warnings for any of these files.

Flac

Finally the results for the Flac tool:

So, the Flac tool was able to identify all defective files².

Conclusion

Out of the candidate tools considered here, only Shntool was able to identify all damaged WAVE files in this experiment. As a result, this (ancient!) tool still appears to be the best choice for detecting damaged WAVE files. Surpringly, Jhove was unable to detect any of the damaged files at all, and is probably best avoided for this particular purpose. For FLAC, both the Flac tool (FLAC reference implementation) and Ffmpeg were able to detect all damaged files, and both appear to be suitable tools.

Dataset and scripts

All example files, scripts and raw tool output are available here:

https://github.com/KBNLresearch/detectDamagedAudio

Post scriptum: update on MediaInfo and MediaConch

In response to this post the developers of MediaInfo added support for detecting truncated WAVE files. This should cover all of the damaged WAVE files presented here. Moreover, their Twitter account announced that detection of FLAC flaws is planned for the MediaConch tool, but that they are looking for sponsors for this.

Crazy Fixed Layout

As I cannot share the original file for rights reasons, I fired up the Sigil e-book editor and made a handcrafted EPUB that reproduces its behaviour. You can download the file here. If you open it in an e-book viewer, it will probably look perfectly normal at first sight. For example, here’s a screenshot I made using the Calibre viewer:

Next I reduced the width of the viewer window. One would expect the text to re-flow to the new width. Instead this happened:

After increasing the font size, I ended up with this:

I got similar results in Chome’s Readium extension. On my e-Ink reader, a Sony PRS-T2, the book rendered as follows:

However, I wasn’t able to change the font size.

Analysis

The file passes validation in EpubCheck 4.0.1 without errors. However, the output does contain a series of warnings about the use of absolute positions in a stylesheet:

CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (6-2)
CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (24-1)
CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (43-1)
::

To really understand what causes the problem, we need to look inside the file’s HTML and CSS resources. Here’s some of the HTML that underlies the text:

<p id="p01" class="para">This is an <em>EPUB</em> 2 file that uses a fixed layout.</p>
<p id="p02" class="para">This is achieved by placing each line inside a</p>
<p id="p03" class="para"><em>paragraph</em> element. Each <em>paragraph</em> element</p>
<p id="p04" class="para">is placed at a fixed position on the page. Even</p>
<p id="p05" class="para">though this file is valid <em>EPUB</em>, this is a pretty</p>
<p id="p06" class="para"> terrible idea, because in most readers the text</p>
<p id="p07" class="para">will not reflow after resizing the viewer window.</p>

So, every line is wrapped inside a paragraph element, each of which has a unique id selector. These refer to style definitions in the EPUB‘s stylesheet. Here are the definitions for the first two lines:

#p01
{
position:absolute;
left:40px;
top:80px;
letter-spacing:0.42px;
word-spacing:0.1em;
}
#p02
{
position:absolute;
left:40px;
top:120px;
letter-spacing:0.42px;
word-spacing:0.1em;

Each style definition specifies a line’s position on the canvas (left, top); moreover, these co-ordinates are defined as absolute positions. This means that each line is placed at a fixed position, regardless of whether this makes any sense given the actual dimensions of the viewer window (or device), or the user’s preferred font size. It seems that the intention of the producer of the original EPUB (from which I derived my example) was to create some sort of “fixed layout” document. However, this doesn’t make much sense for books with simple, text-only layouts (as in this case). Worse, depending on the viewing device and the user the file may be effectively inaccessible. For example, someone with a visual impairment may only be able to read an EPUB using very large font sizes, which in this case results in garbled text.

Crazy Columns

Things can even get worse. I once came across an EPUB that used similar tricks to achieve a two-column layout. Again I’m not able to share the original file, so I created another EPUB that mimicks its behavour. In the Calibre viewer it looks like this:

As with the first example, the text doesn’t reflow after resizing the viewer window, and increasing the font resulted in this:

This is what I got when I opened the file in my Sony e-Ink reader:

After I increased the font size this happened:

Similarly, when I tried to copy the text in the file to the clipboard, and then pasted it in a text editor, I ended up with this:

This is an EPUB filepage. Even though thisthat uses a two-columnfile is valid EPUB, there’slayout. For each column,no way to establish theevery line is placed atlogical reading order ofa fixed position on thethe text.

Ouch!

Analysis

Again, throwing this file at EpubCheck 4 doesn’t result in any validation errors, although just like the previous file there are some warnings about the use of absolute positions in the stylesheet:

CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (13-1)

A peek inside the HTML reveals the true horrors of this EPUB. This is how the text is encoded:

<div class="pos" style="left: 40px; top: 100px;">This is an <em>EPUB</em> file<div>
<div class="pos" style="left: 260px; top: 100px;">page. Even though this</div>
<div class="pos" style="left: 40px; top: 140px;">that uses a two-column</div>
<div class="pos" style="left: 260px; top: 140px;">file is valid <em>EPUB</em>, there's</div>

So, every line of each column is wrapped in a division element that has a fixed position. The class pos in the stylesheet defines the general layout of each division element. In this case, it specifies that all positions are (again) absolute:

.pos {position:absolute;
} 

Technically this is pretty similar to the first example. Note that the above HTML doesn’t contain any semantic information on the fact that there are two separate columns. Worse, the order of the text in the HTML doesn’t even follow the actual reading order! This also explains the results after copying and pasting. Screen reader applications will not be able to handle this either, which makes books like these inaccessible to many visually impaired users. All of this could have been avoided if the book’s producer had followed the W3C multi-column layout specification.

Conclusion

I don’t know how common (or rare) EPUBs like the above are. They may just be weird edge cases. Nevertheless, their existence indicates that checking for validity alone may not be sufficient to ensure accessibility for all users (in particular those with a visual impairment). In any case, files like these can be identified relatively easily by checking EpubCheck‘s output for the presence of a CSS-017 warning (“CSS selector specifies absolute position”)¹. These examples also underline the importance of guidelines and best practices. Several good resources for making accessible EPUB are available from the EPUB 3 Accessibility Guidelines, including a useful Accessibility QA Checklist. I would also be interested in hearing other people’s experiences with “weird” EPUBs like these.

Postscript

Alberto Pettarin pointed me to his blog post (Current) Fixed Layout eBooks Considered Harmful. Written in 2015, it addresses the problems with current implementations of fixed layouts in EPUB, and if you found this blog post interesting, I would suggest to check out Alberto’s blog as well.

Alberto’s Twitter feed also drew my attention to an interesting EPUB with the program of the recent EPUB Summit in Bordeaux. You can download it here (you need to unzip it first!). The file is interesting because:

1. It does not pass validation by EpubCheck (the mimetype file entry is not the first file resource in the archive)
2. It uses a fixed, multi-column layout that doesn’t scale in either Readium or Calibre‘s viewer (changing the font size has no effect), and I’m wondering if it is usable at all on any handheld devices!

There’s some irony in that this file was published by EDRLab, an organisation that describes itself as “the European headquarter for IDPF and Readium Foundation”, and which mentions “support for people who have print disabilities” as a “key part”of its mission. Oh well …

Link to dataset

The EPUBs used for this blog post are part of the EPUB KB policy testing repository. This is an annotated set of openly licensed EPUB files that were specifically created for testing purposes.

About a month ago the International Digital Publishing Forum, the standards body behind the EPUB format, published an Editor’s Draft of EPUB 3.1. This is meant to be the successor of the current 3.0.1 version. IDPC has set up a community review, which allows interested parties to comment on the draft. The proposed changes relative to EPUB 3.0.1 are summarised in this document. A note at the top states (emphasis added by me):

The EPUB working group has opted for a radical change approach to the addition and deletion of features in the 3.1 revision to move the standard aggressively forward with the overarching goals of alignment with the Open Web Platform and simplification of the core specifications.

As Gary McGath pointed out earlier, this is a pretty bold statement for what is essentially a minor version. The authors of the draft also mention that they expect it “will provoke strong reactions both for and against”, and that changes that raise “strong negative reactions” from the community “will be reviewed for future drafts”.

This blog post is an attempt to identify the main implications of the current draft for libraries and archives: to what degree would the proposed changes affect (long-term) accessibility? Since the current draft is particularly notable for its aggressive removal of various existing EPUB features, I will focus on these. These observations are all based on the 30 January 2016 draft of the changes document.

Removed support for EPUBCFI for linking

The EPUB Canonical Fragment Identifier (EPUBCFI) “defines a standardized method for referencing arbitrary content within an EPUB Publication”. Until EPUB 3.0.1, Reading Systems were required to support EPUBCFI for hyperlinking within and between documents. This requirement is dropped in EPUB 3.1 (although it would still be possible to use EPUBCFI for annotations and bookmarks).

In principle this change could result in problems if an EPUB that uses CFI for hyperlinks is opened in a 3.1 reading system: in that case the hyperlinks would not work. However, according to EPUB editor Matt Garrish, authors simply do not use CFI for hyperlinking. He also mentions a check by Google on their corpus of millions of books, which only turned up a few instances of CFI use. One of these was a link in an EPUB best practices book, while the remaining ones were all part of the EPUB test suite documents. If these results are representative of all EPUBs “in the wild”, the implications of the change would be negligible.

Reduced set of metadata elements in Package Document

EPUB 3.1 imposes restrictions on the metadata elements that can be embedded in the Package Document. Up to version 3.0.1, the full Dublin Core Metadata Element Set was supported, whereas in 3.1 only the dc:identifier, dc:title, dc:language, dc:creator, dc:publisher and dc:type elements are allowed. Additional metadata can be included, but they need to be defined in a separate resource (file), which is referenced from the metadata element using the link element. Below is an example that uses a MARC file:

<link rel="record"
  href="meta/9780000000001.xml" 
  media-type="application/marc"/>

Complicating things further, the EPUB 3.1 Packages draft says:

Linked resources that are not Publication Resources are not subject to Core Media Type requirements [EPUB31] and may be located inside or outside [EPUB31] the EPUB Container. Retrieval of Remote Resources is optional.

So, linked metadata resources can have any possible format, and they may not even be included in the EPUB container. Even though these changes would have no direct consequences for long-term accessibility, they would seriously complicate document processing (e.g. ingest) workflows that rely on the metadata in the Package Document. It would also affect end users who rely on these metadata fields to sort and find their ebooks.

Note: the discussion thread on this topic in the issue tracker is worth checking out, as it contains some excellent additional observations.

Removal of the NCX

EPUB 2 documents contain the NCX file (“Navigation Control file for XML”), which provides a mechanism to navigate a publication. It is essentially a hierarchical table of contents. The NCX was superseded by the Navigation Document in EPUB 3.0.1. However, the NCX was allowed in EPUB 3.01 publications, which was useful for keeping EPUB 3 publications compatible with older (EPUB 2-based) reading systems¹. The 3.1 draft forbids the NCX altogether, which means that such “hybrid” EPUBs are not possible without breaking the specification.

The main consequence of this is that it would make EPUB 3.1 files incompatible with older reading systems. More specifically, basic navigation functionality such as direct access to a chapter from the table of contents would not work.

To get an approximate idea of the impact of this, I had a look at the EPUB 3 support grid, which gives detailed information about the support of specific EPUB 3 features for commonly used devices, apps, and reading systems. This link shows support of the toc nav element, which defines the primary navigational hierarchy in the Navigation Document. Only 55% (34 out of 62) of all tested reading systems fully support the toc nav element, with 37% (23 out of 62) not supporting it at all². This may not be a big deal for users of software-based reading systems (which make up the majority of the support grid), but users of (older) E-ink readers often don’t have the option to upgrade their devices. A good example is this (now discontinued) Sony e-Ink hardware reader. Unfortunately, E-ink devices appear to be underrepresented in the support grid. For example, it contains no information whatsoever on any of the popular Kobo readers.

The proposal to remove the NCX provoked strong reactions in the community review, with one respondent stating it would lead to “dropping support for millions of eInk reading systems”. It would also contradict this statement from the EPUB 3.0.1 specification (emphasis added by me):

The NCX feature defined in [OPF2] is superseded by the EPUB Navigation Document [ContentDocs301]. EPUB 3 Publications may include an NCX (as defined in OPF 2.0.1) for EPUB 2 Reading System forwards compatibility purposes, but EPUB 3 Reading Systems must ignore the NCX.

The explicit reference to EPUB 3 Publications (not EPUB 3.0.1 Publications!!) implies that the statement applies to EPUB 3 in general. Removing the NCX in another EPUB 3 release would be at odds with this.

Removal of the guide Element

The guide element was an optional data structure in EPUB 2 that provided “convenient access” to structural components of a publication. It was deprecated in EPUB 3.0.1. Without any data on the actual usage of this feature, it is difficult to say much about the impact of its complete removal (this was also pointed out by one respondent to the community review).

Removal of the bindings Element

In EPUB 3.0.1 the bindings element could be used to define fallbacks for foreign resources. According to EPUB editor Matt Garrish “this feature is not widely used or supported”, and the impact on accessibility appears to be negligible.

Removal of the switch Element

The switch element in EPUB 3.0.1 allows one to define alternative representations of XML fragments. Here’s an example:

<epub:switch id="cmlSwitch">
   
   <epub:case required-namespace="http://www.xml-cml.org/schema">
      <cml xmlns="http://www.xml-cml.org/schema">
         <molecule id="sulfuric-acid">
            <formula id="f1" concise="H 2 S 1 O 4"/>
         </molecule>
      </cml>
   </epub:case>
   
   <epub:default>
      <p>H<sub>2</sub>SO<sub>4</sub></p>
   </epub:default>
   
</epub:switch>

Here, we have a chemical formula in ChemML format and in standard HTML. ChemML is not natively supported in EPUB, so by default a reader will display the HTML version. However, wrapping both in a switch element would allow a ChemML-capable reader to render that representation instead.

I asked EPUB editor Matt Garrish how an EPUB 3.1-compliant reader would render content that is wrapped in a switch element. He replied that by default all of the switch content would be rendered. So for the example above, a reader would try to render both the HTML and the ChemML versions (with the latter failing on most reading systems). Matt stressed the significance of the switch element, adding that people have been using it, “if not extensively”.

Removal of the trigger Element

The trigger element in EPUB 3.0.1 is used to define simple user interfaces for multimedia content. Since this can be done natively in HTML 5, it is dropped from EPUB 3.1. Here editor Matt Garrish explains that the feature is both “sparsely used” (referring to a survey of publishers) and “poorly supported”.

Miscellaneous changes

Apart from the changes above (which all remove features from the existing specification), the EPUB 3.1 draft also adds a number of new features, and clarifies some existing ones. I won’t go over them in detail, but here’s a brief overview:

• New Core Media Types – adds support for WOFF 2.0 and SFNT fonts.
• Addition of Support for HTML Syntax of HTML5 – adds support for the HTML Syntax of HTML5 (currently only the XHTML syntax is supported³).
• Replacement of EPUB Style Sheets with CSS References – this replaces the current “EPUB Style Sheets profile” by the “official definition” of CSS.
• Addition of WCAG Support – adds the recommendation that all HTML Content Documents conform to the WCAG Guidelines to ensure they are accessible for people with disabilities.

Finally, the draft contains clarifications on Foreign Resource Fallbacks and Scripting Support.

EPUB 3.1 or EPUB 4.0?

By now it should be clear that the aggressive removal of features in EPUB 3.1 would have some far-reaching consequences. This is particularly true for the removal of the NCX, which would make EPUB 3.1 files incompatible with many existing E-ink readers. It would do this by ruling out the option to make backward-compatible “hybrid” files. As Gary McGath pointed out earlier, introducing “radical changes” in what is essentially a minor version is pretty unusual practice for any standard. Nowadays, most software and file formats use some variation of semantic versioning, with version numbers that follow the general form MAJOR.MINOR.PATCH. Here, each component of the version number has a well-defined meaning:

1. MAJOR version is increased in case of incompatible API changes,
2. MINOR version is increased when functionality is added in a backwards-compatible manner, and
3. PATCH version is increased in case of backwards-compatible bug fixes.

Since the current draft includes multiple backward-incompatible changes, this makes me wonder why the editors didn’t name it EPUB 4.0 instead! Kovid Goyal, lead developer of the popular Calibre software, made the following comment on this:

[I]f you want to make backwards incompatible changes, please, dont do it in a point release. From glancing over your changes document, it seems to me that you want to make several breaking changes. That’s great, EPUB 3 could do with some serious breaking. But name it EPUB 4. I really dont want to have tell my users that calibre supports EPUB 3.1 but not EPUB 3.

I agree with Kovid here. Having multiple sub-versions of EPUB 3, with some of them being backward-compatible with EPUB 2, while this backward compatibility is explicitly ruled out in another sub-version, is bound to create a situation that will be incomprehensible for most e-book buyers. Worse, it could even undermine overall confidence in the format. For memory institutions it would also make the management of EPUB 3 publications unnecessarily complicated. Not only would some EPUB 3.1 files not render correctly in an EPUB 3.0.1 reader, the opposite would be true as well.

Flashback

In my 2012 report on EPUB for archival preservation I already mentioned the stability of the EPUB format as a concern:

EPUB 3 shows quite major changes relative to version 2, which raises concerns about the format’s stability over time. These concerns are reinforced by the fact that EPUB 3 is heavily dependent on (X)HTML5 and CSS3, both of which are unfinished “works in progress”, which may undergo various changes before being finalised.

These concerns are once more confirmed by the current EPUB 3.1 draft. However, it remains to be seen how many of these changes will make it to the final version. The community review process is ongoing at this moment, so if you’re getting a little uneasy after reading this blog post, there’s still time to get involved and make your voice heard!

Acknowledgement

Thanks to Matt Garrish for his prompt replies to my questions on Github.

Yesterday (7 December) we released version 1.16.0 of the jpylyzer tool, which is this year’s third release of the software (excluding bugfix releases). This blog post gives a brief overview of the main jpylyzer improvements that have been implemented over this year.

Changes in XML output

The 1.14 release introduced two output improvements. Most importantly, an XML Schema Definition (XSD) was created. The schema formally defines the output format, and it also makes it possible to validate output files. In addition, a namespace declaration was added. These changes make the post-processing of jpylyzer‘s output more straightforward.

The 1.16 release added the statusInfo element, which tells you whether the validation completed without any internal errors. It contains the following sub-elements:

• success: a Boolean flag that indicates whether the validation attempt
  completed normally (“True”) or not (“False”). A value of “False” indicates
  an internal error that prevented jpylyzer from validating the file.
• failureMessage: if the validation attempt failed (value of success
  equals “False”), this field gives further details about the reason of the failure.

This means that the general structure of the output now looks like this:

Recursive traversal of directory trees

Another feature that was introduced with the 1.14 release is the --recurse option. This allows one to recursively traverse a directory tree. The code for this feature was created by Adam Retter, Jaishree Davey and Laura Damian of The National Archives (UK).

Memory mapping

The 1.15 release introduced the use of memory mapping for reading input images. This results in better performance when processing (very) large files. Images that would cause a memory error in previous versions are now handled without any problem. Also, the processing of very large files can be significantly faster than in earlier releases, and is less prone to freezing other processes that are simultaneously running on the machine. This improvement was suggested by Stefan Weil of Mannheim University Library, and the changes are based on a patch he submitted.

Two examples illustrate the benefits of this change:

• This 2 GB image
  resulted in a memory error with jpylyzer 1.14.2 on a Windows machine with 4 GB RAM. The latest versions process the file without problems.
• On a Linux Mint machine with 8 GB RAM, this 6.7 GB image
  also resulted in a memory error. Again, the current version handles the file without any problem.

This doesn’t mean that memory errors are now a thing of the past entirely; they may still occur under some circumstances. For instance, a test with the 6.7 GB image failed on a Linux Mint machine with 4 GB RAM. So it seems prudent to make sure that the amount of available RAM always exceeds the maximum image size by a fairly wide safety margin. Also, chip architecture and operating system may put further constraints on the amount of memory than can be mapped at a time.

Improved exception handling

Prior to release 1.16.0, an exception during the processing of an image could cause jpylyzer to crash. For example, an extremely large image can result in an internal memory error, and this would grind jpylyzer to a halt. This is particularly problematic when using the new --recurse option: in this case a single jpylyzer invocation may involve the processing of thousands of images at a time. One single (e.g. extremely large) image could then result in unusable output; moreover, it would be difficult to identify which image caused the crash in the first place! Release 1.16.0 introduces improved exception handling that allows jpylyzer to handle such situations more gracefully.

Robustness

The combined effect of the exception handling, memory mapping and status output should make jpylyzer releases from 1.16.0 onwards significantly more robust than previous versions. As an example, here’s some (simplified) output for a 6.5 GB JP2 that caused a memory error:

<?xml version='1.0' encoding='UTF-8'?>
<jpylyzer>
    <toolInfo>
        <toolName>jpylyzer.py</toolName>
        <toolVersion>1.16.0</toolVersion>
    </toolInfo>
    <fileInfo>
        <fileName>AS16-P-4102.jp2</fileName>
        <filePath>/home/johan/testJpylyzer/AS16-P-4102.jp2</filePath>
        <fileSizeInBytes>6745365021</fileSizeInBytes>
        <fileLastModified>Wed Dec  2 20:05:29 2015</fileLastModified>
    </fileInfo>
    <statusInfo>
        <success>False</success>
        <failureMessage>memory error (file size too large)</failureMessage>
    </statusInfo>
    <isValidJP2>False</isValidJP2>
    <tests/>
    <properties/>
</jpylyzer>

Previous versions would simply crash in this situation. Now, automated workflows can simply check for the value of the success field to verify the status of the validation. More importantly, if the jpylyzer invocation involved multiple input files (e.g. through the --recurse option), errors like these will not stop the processing of the remaining files.

64-bit Windows binaries

Finally, from version 1.15.1 onwards we are now providing 64 bit Windows binaries of jpylyzer (previously only 32-bit binaries were available).

Links

Jpylyzer website

The KB has quite a large collection of offline optical media, such as CD-ROMs, DVDs and audio CDs. We’re currently investigating how to stabilise the contents of these materials using disk imaging. During the initial phase of this work I did a number of tests with various open-source tools. It’s doubtful whether we’ll end up using these same tools in our actual workflows. The main reason for this is the sheer size of the collection, which we estimated at some 15,000 physical carriers; possibly even more. At those volumes we will need a solution that involves the use of a disk robot, and these often require dedicated software (we still need to investigate this more in-depth).

Nevertheless, throughout the initial testing phase I was surprised at the number of useful tools that are available in the open source domain. Since this will probably be of interest to others as well, I decided to polish a selection from my rough working notes into a somewhat more digestible form (or so I hope!). I edited my original notes down to the following topics:

• How to figure out the device path of the CD drive
• How to create an ISO image from a CD-ROM or DVD
• How to check the integrity of the created ISO image
• How to extract audio from an audio CD

In addition there’s a final section that covers my attempts at imaging a multisession / mixed mode CD. The result of this particular exercise wasn’t all that successful, but I included it anyway, as some may find it useful. All software mentioned here are open-source tools that are available for any modern Linux distribution (I’m using Linux Mint myself). Some can be used under Windows as well using Cygwin.

Find the device path of the CD drive (Linux)

The majority of the tools covered by this blog post need the device path of the CD drive as a command-line argument. Under Linux you can usually find this by inspecting the output of the following command (run this while a CD or DVD is inserted in your drive):

mount|grep ^'/dev'

If all goes well, the result will look similar to this:

/dev/sda1 on / type ext4 (rw,errors=remount-ro)
/dev/sr0 on /media/johan/REBELS_0 type iso9660
(ro,nosuid,nodev,uid=1000,gid=1000,iocharset=utf8,mode=0400,dmode=0500,uhelper=udisks2)

So, in this case the path to the CD drive is /dev/sr0 (if you have multiple optical drives you may also see /dev/sr1, and so on).

Finding the device path on Windows (Cygwin)

For some reason the mount command doesn’t result in the printing of any device paths in CygWin. Instead, try this:

ls /dev/

Which produces a list of all devices:

clipboard  dsp   mqueue  random  sda2  sdc1    stdin   ttyS2
conin      fd    null    scd0    sdb   shm     stdout  urandom
conout     full  ptmx    sda     sdb1  sr0     tty     windows
console    kmsg  pty0    sda1    sdc   stderr  ttyS0   zero

In the above output both sr0 and scd0 point to the CD drive, and either the full paths /dev/sr0 or /dev/scd0 will work (again in case of multiple drives you may be looking for /dev/sr1 or /dev/scd1).

In all examples below I assumed that the device path is /dev/sr0; substitute your own path if necessary.

Create ISO image of a CD-ROM or DVD

A number of tools allow you to create an (ISO¹) image from a CD-ROM or DVD. Although generic Unix data copying and recovery tools like dd and ddrescue are often used for this, various people have pointed out that the result may be unreliable because they only perform limited error checking. See for example the comments here and here; both recommend to use the readom tool, which is part of the cdrkit library. My own experience with readom is that while it works great in most cases, it is less suitable for CD-ROMs that are damaged or otherwise degraded. In those cases ddrescue is often a better choice. So below I’ll first show how to use readom, followed by a ddrescue example that specifically addresses the recovery of a CD-ROM gives read errors in readom.

Running readom

The documentation recommends to always run readom as root. Also, before running readom, the CD or DVD must be unmounted². So, after inserting the CD or DVD, first enter this:

umount /dev/sr0

Then run readom as root:

sudo readom retries=4 dev=/dev/sr0 f=mydisk.iso

Here the value of the retries parameter defines the number of attempts that readom will make at trying to recover unreadable sectors. The default value is 128, which can result in huge processing times for CDs that are seriously damaged. The f parameter sets the name of the image file that is created. If all goes well the following output is printed to the screen at the end of the imaging process:

Read  speed:  4234 kB/s (CD  24x, DVD  3x).
Write speed:     0 kB/s (CD   0x, DVD  0x).
Capacity: 309104 Blocks = 618208 kBytes = 603 MBytes = 633 prMB
Sectorsize: 2048 Bytes
Copy from SCSI (10,0,0) disk to file 'mydisk.iso'
end:    309104
addr:   309104 cnt: 44
Time total: 259.287sec
Read 618208.00 kB at 2384.3 kB/sec.

When read errors happen: try ddrescue

If the source medium is in a bad condition or otherwise damaged, readom will most likely terminate prematurely with read errors. If this happens, you may get better results with ddrescue. There are two reasons for this:

1. Unlike readom, which usually gives up pretty soon after the first read error occurs, ddrescue was specifically designed to deal with source media that contain errors. Consequently, it is much more persistive in such cases.
2. If you try to read a defective source medium using two different CD drives (let’s call them A and B), it is not uncommon to find that some sectors that result in read errors on drive A are read correctly by drive B (and vice versa). With ddrescue it is possible to take advantage of this.

The ddrescue Manual manual gives a (very concise) example of how this works. Based on this I created the following, more detailed example.

First we run ddrescue with the following command line:

ddrescue -b 2048 -r4 -v /dev/sr0 mydisk.iso mydisk.log

Here -b sets the block size (which is typically 2048 bytes for a CD-ROM); -r4 sets the maximum number of retries in case of bad sectors to 4³, and -v activates verbose output mode. File mydisk.log is a so-called mapfile (known as logfile in ddrescue versions prior to 1.20). The mapfile contains (among a few other things) information on the recovery status of blocks of data. After running the above command on a faulty CD-ROM, we end up with output that looks like this:

GNU ddrescue 1.17
About to copy 624918 kBytes from /dev/sr0 to mydisk.iso
    Starting positions: infile = 0 B,  outfile = 0 B
    Copy block size:  32 sectors       Initial skip size: 32 sectors
Sector size: 2048 Bytes

Press Ctrl-C to interrupt
rescued:   624871 kB,  errsize:   47104 B,  current rate:        0 B/s
   ipos:   508162 kB,   errors:       3,    average rate:     592 kB/s
   opos:   508162 kB,    time since last successful read:    12.3 m
Finished

From this we can see the following:

• The CD-ROM contains 624918 kBytes of data (2nd line from top).
• Only 624871 kBytes were extracted (‘rescued’ field)
• A total of 47104 bytes were not rescued (‘errorsize’ field)
• 3 errors occurred while reading the CD (‘errors’ field)

However, it is often possible to improve the result by additional runs of ddrescue using either different options, or other hardware. First we’ll see if we can improve things by re-running in direct disc access mode (this does not work on some systems, in which case ddrescue will report a warning). So we use the following command⁴:

ddrescue -d -b 2048 -r1 -v /dev/sr0 mydisk.iso mydisk.log

Here the -d switch activates direct disc access, which bypasses the kernel cache (note that the number of retries is set to 1 in the above example). Running the command causes ddrescue to update both the ISO and the mapfile. The screen output now looks like this:

GNU ddrescue 1.17
About to copy 624918 kBytes from /dev/sr0 to mydisk.iso
    Starting positions: infile = 0 B,  outfile = 0 B
    Copy block size:  32 sectors       Initial skip size: 32 sectors
Sector size: 2048 Bytes

Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:   624871 kB,  errsize:   47104 B,  errors:       3
Current status
rescued:   624912 kB,  errsize:    6144 B,  current rate:        0 B/s
   ipos:   508162 kB,   errors:       3,    average rate:     1706 B/s
   opos:   508162 kB,    time since last successful read:       7 s
Finished

What we see here:

• 624871 kBytes were extracted (previously this was 624871)
• Consequently ‘errsize’ has gone down from to 47104 to 6144 bytes

So this is better, but still not perfect. So let’s try if we can improve the results by using a different CD-reader. At this point I hooked up an external USB CD-drive, and moved my faulty CD-ROM from the internal reader to the external one. In this case my external drive is mapped under device path /dev/sr2 (re-run the aforementioned steps to find the device path if necessary). This gives the following command-line:

ddrescue -d -b 2048 -r4 -v /dev/sr2 mydisk.iso mydisk.log

Now the output looks like this:

GNU ddrescue 1.17
About to copy 624918 kBytes from /dev/sr2 to mydisk.iso
    Starting positions: infile = 0 B,  outfile = 0 B
    Copy block size:  32 sectors       Initial skip size: 32 sectors
Sector size: 2048 Bytes

Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:   624916 kB,  errsize:    2048 B,  errors:       1
Current status
rescued:   624918 kB,  errsize:       0 B,  current rate:      682 B/s
   ipos:   106450 kB,   errors:       0,    average rate:      682 B/s
   opos:   106450 kB,    time since last successful read:       0 s
Finished

From the output we can see that after re-running ddrescue with the external drive, both ‘errsize’ and the number of errors have gone down to 0. In other words: all of the contents of the CD have been rescued without any errors. Yay!

In the above example I used two different CD readers that were connected to the same machine, but you could use as many readers as you like. It also possible to do the first run on one machine, transfer the ISO image and the mapfile to another machine, and then re-run ddrescue there (this even works across OS platforms).

Check integrity of ISO image against physical CD-ROM or DVD

You can use check the integrity of the created ISO image by computing a checksum on both the ISO file and the physical carrier, and then comparing both:

md5sum mydisk.iso
md5sum /dev/sr0

Note that the aforementioned Aaron Toponce article claims that readom already does a checksum check. If true, the additional check would be overkill (especially given that computing a checksum on a physical CD or DVD is time consuming). However, I couldn’t find any confirmation of this in either readom‘s documentation nor its source code (although I found the source hard to read, so I may have simply overlooked it).

Verify ISO image

In theory, there shouldn’t be any need for additional quality checks on an ISO image once its integrity against the physical carrier is confirmed by the checksum. However, since cdrkit includes an isovfy tool that claims to " verify the integrity of an iso9660 image", I decided I might as well give it a try. It works by entering:

isovfy mydisk.iso

Here’s some example output:

Root at extent 13, 2048 bytes
[0 0]
No errors found

The documentation of the tool isn’t very clear about what specific checks it performs. In one of my tests I fed it an ISO image that had its last 50 MB missing (truncated). This did not result in any error or warning message! Most of the reported isovfy errors that I came across in my tests simply reflected the file system on the physical CD not conforming to ISO 9660 (this seems to be pretty common). Based on this it looks like isovfy isn’t very useful after all.

Get information about an ISO image

Isoinfo

The Primary Volume Descriptor (PVD) of an ISO 9660 file system contains general information about the CD or DVD. The isoinfo tool (which is also part of cdrkit) is able to print the most important PVD fields to the screen:

isoinfo -d -i mydisk.iso

Result:

CD-ROM is in ISO 9660 format
System id: 
Volume id: REBELS_0
Volume set id: 
Publisher id: 
Data preparer id: 
Application id: NERO - BURNING ROM
Copyright File id: 
Abstract File id: 
Bibliographic File id: 
Volume set size is: 1
Volume set sequence number is: 1
Logical block size is: 2048
Volume size is: 333151
Joliet with UCS level 3 found
NO Rock Ridge present

You can also run isoinfo directly on the physical carrier:

isoinfo -d -i /dev/sr0

To get a listing of all files and directories that are part of the filesystem, use this:

isoinfo -f -i mydisk.iso

Result:

/AUTORUN.EXE;1
/AUTORUN.INF;1
/DISK0
/LICENSE2.TXT;1
/LICENSEF.TXT;1
/LICENSEU.TXT;1
/SETUP.EXE;1
/DISK0/CONTROLS.CFG;1
/DISK0/DISK0;1
::
::
etc

It looks like all items that are followed by ;1 are files, and those that aren’t are directories. Also, the -l option can be used for a detailed list that includes additional file attributes (size, date, etc.).

Disktype

The disktype tool is particularly useful for identifying hybrid disc images that combine multiple file systems. For example:

disktype bewaarmachine.iso

This results in:

--- bewaarmachine.iso
Regular file, size 342.1 MiB (358727680 bytes)
Apple partition map, 2 entries
Partition 1: 1 KiB (1024 bytes, 2 sectors from 1)
  Type "Apple_partition_map"
Partition 2: 172.4 MiB (180773376 bytes, 353073 sectors from 346957)
  Type "Apple_HFS"
  HFS file system
    Volume name "de bewaarmachine"
    Volume size 172.4 MiB (180764672 bytes, 44132 blocks of 4 KiB)
ISO9660 file system
  Volume name "BEWAARMACHINE_PC"
  Application "TOAST ISO 9660 BUILDER COPYRIGHT (C) 1993-1996 MILES SOFTWARE GMBH - HAVE A NICE DAY"
  Data size 169.4 MiB (177641472 bytes, 86739 blocks of 2 KiB)

In this example we have an image that contains both an ISO 9660 and an Apple HFS filesystem. Disktype can also be run directly on the physical carrier, using:

disktype /dev/sr0

Rip audio CD with cdparanoia

The data structure of an audio CD is fundamentally different from a CD-ROM or DVD, and because of this its content cannot be stored as an ISO image. The most widely-used approach is to extract (or "rip") the audio tracks on a CD to separate WAVE files. A complicating factor here is that the way audio is encoded on a CD tends to obscure (small) read errors during playback. As a result, a single linear read will not result in a reliable transfer of the audio data. More details can be found in this excellent article by Alexander Duryee. Duryee recommends a number of extraction tools that overcome this problem using sophisticated verification and correction functionality. One of these tools is the cdparanoia ripper. As an example, the following command can be used to rip a CD in batch mode, where each track is stored as a separate WAVE file:

cdparanoia -B -L

or:

cdparanoia -B -l

The -L switch results in the generation of a detailed log file; -l produces a summary log (name: cdparanoia.log)⁵. File names are generated automatically like this:

track01.cdda.wav
track02.cdda.wav
track03.cdda.wav

Here is a link to an example log file. The output may look a little weird at first sight, which is because cdparanoia reports all status and progress information as symbols and smilies, respectively. Their meaning is explained in the documentation.

Extract data from multi-session / mixed mode CDs

Some CDs combine data and audio tracks. Examples are "enhanced" audio CDs that include software or movies as bonus material, as well as many ’90s video games. Even though the data part of such CDs is typically compatible with an ISO 9660 file system, the audio tracks are not. Since there is no good, open and mature file format to describe the contents of a CD precisely, such CDs pose a particular challenge. In addition, tools such as readom and ddrescue typically only recognise the first session on a multisession CD, which means that are not suitable.

I’ve done some (pretty limited) testing on mixed-mode CDs using the cdrdao tool. This didn’t result in a satisfactory way to process these carriers. However, since many people appear to be struggling with this, I’ll briefly report my results so far.

Identifying multisession CDs

We can use cdrdao to figure out the number of sessions on a CD. First unmount the disc (this is important!):

umount /dev/sr0

Then run:

cdrdao disk-info --device /dev/sr0

Here’s the result I got for a mixed-mode audio/data CD:

CD-RW                : no
Total Capacity       : n/a
CD-R medium          : n/a
Recording Speed      : n/a
CD-R empty           : no
Toc Type             : CD-DA or CD-ROM
Sessions             : 2
Last Track           : 18
Appendable           : no

The value of Sessions is 2, which indicates this is a multi-session CD.

Imaging

This article on the Linux Reviews site contains instructions on how to rip a mixed-mode CD using cdrdao. I followed these instructions in an attempt to make a copy of They Might Be Giants’ "No" album (which contains some video content). First I unmounted the disk:

umount /dev/sr0

Then I ran ran cdrdao with the following arguments:

cdrdao read-cd --read-raw --datafile no.bin --device /dev/sr0 --driver generic-mmc-raw no.toc

The result of this is a disc image in BIN/TOC format. The .toc file looks like this:

CD_DA


// Track 1
TRACK AUDIO
NO COPY
NO PRE_EMPHASIS
TWO_CHANNEL_AUDIO
ISRC "USIR70200001"
FILE "no.bin" 0 02:10:53


// Track 2
TRACK AUDIO
NO COPY
NO PRE_EMPHASIS
TWO_CHANNEL_AUDIO
ISRC "USIR70200002"
FILE "no.bin" 02:10:53 02:17:34

::
etc

Closer inspection showed that only the audio tracks were copied, not the data track! As a comparison, below example from the cdrdao documentation shows the expected output:

CD_ROM
 TRACK MODE1
 DATAFILE "data_1"
 ZERO 00:02:00 // post-gap

TRACK AUDIO
 SILENCE 00:02:00 // pre-gap
 START
 FILE "data_2.wav" 0

TRACK AUDIO
 FILE "data_3.wav" 0

In particular I would expect the .toc file to start with CD_ROM, and I would also expect one TRACK MODE1 item for the data part of the disk. It’s not clear to me why my test produced a different result. Interestingly, I was able to make 2 separate images of the audio and data components of the disc by adding the –session option:

    cdrdao read-cd --read-raw --session 1 --datafile no1.bin --device /dev/sr0 --driver generic-mmc-raw no1.toc

And then:

    cdrdao read-cd --read-raw --session 2 --datafile no2.bin --device /dev/sr0 --driver generic-mmc-raw no2.toc

Running cdrdao twice like this, I was able to create two separate images with the audio and file system data, respectively.

Post-processing of BIN/TOC files

In the above example, both images (including the data track) have the BIN/TOC format, which is not easily accessible. It is possible to convert these files to something more useful with the bchunk tool.

First convert the .toc files to .cue format. For this we use the toc2cue tool (which is part of cdrdao):

toc2cue no2.toc no2.cue

Next use bchunk to convert the BIN/TOC to an ISO file (the last argument is the basename for any output files created by bchunk):

bchunk no2.bin no2.cue no2

In this case this resulted in file no201.iso, which is a mountable ISO image.

For audio images bchunk has a -w option, which creates output in WAVE format. Just use this:

bchunk -s -w no1.bin no1.cue no1

Note the use of the -s switch, which does a byte swap on the audio track samples. I initially omitted this, and ended up with WAVE files that all played as static noise! This strikes me as odd, since according to its specification the WAVE format is little-Endian by definition.

Additional material

• The rough, unedited notes on which this blog post is based can be found here (they contain some additional material that I left out here for readability).

• Here’s an experimental Python script that verifies if the file size of a CD / DVD ISO 9660 image is consistent with the information in its Primary Volume Descriptor. This can be useful for detecting incomplete (e.g. truncated) ISO images.

• The User Manual of ddrescue gives some useful additional examples of how this tool can be used to recover data from a faulty CD-ROM.

Apache Preflight

During the SCAPE project we did a number of experiments with the PDF/A validator that is part of the open-source Apache PDFBox library (incidentally it is also called Preflight). Throwing the PDF of our last example at Apache Preflight results in the following output¹:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<preflight name="Jpeg_linked.pdf">
  <executionTimeMS>9792</executionTimeMS>
  <isValid type="PDF/A1-b">false</isValid>
  <errors count="96">
    <error count="1">
      <code>3.1.3</code>
      <details>Invalid Font definition, CourierNewPSMT: FontFile entry is missing from FontDescriptor</details>
    </error>
    <error count="1">
      <code>7.11</code>
      <details>Error on MetaData, PDF/A identification schema http://www.aiim.org/pdfa/ns/id/ is missing</details>
    </error>
    <error count="3">
      <code>6.2.1</code>
      <details>Action is forbidden, GoToPage isn't authorized as named action</details>
      <page>0</page>
    </error>
     
    ::
    ::

    <error count="1">
      <code>1.4.2</code>
      <details>Trailer Syntax error, The trailer dictionary contains Encrypt</details>
    </error>
  </errors>
</preflight>

Assessment against a technical profile / policy

By post-processing Preflight’s XML output further, it is possible to automatically evaluate PDFs against a user-defined set of features (i.e. a technical profile, equivalent to what was known as a control policy in the SCAPE project). This is pretty straightforward if you express all features (or policy elements) as Schematron rules. Here’s an example of a Schematron rule that checks for encryption:

<?xml version="1.0"?>
<!--
Schematron rules for policy-based  validation of PDF, based on output of Apache Preflight.
-->
<s:schema xmlns:s="http://purl.oclc.org/dsdl/schematron">
  
  <s:pattern name="Checks for encryption">        
    <s:rule context="/preflight/errors/error">
      <s:assert test="not(code = '1.0' and contains(details,'password'))">Open password</s:assert>
      <s:assert test="not(code = '1.4.2')">Encryption</s:assert>
    </s:rule>
  </s:pattern>

</s:schema>

Rules can be defined for other features as well (e.g. multimedia, fonts), which makes it possible to test against custom policies. The figure below illustrates the general procedure:

A simple demo (based on Shellscript) that implements the above workflow can be found here.

Test with Govdocs1 corpus

As part of the SCAPE work, we tested whether we could use Preflight in this way to assess a large set of PDFs. For this we used about 15,000 PDFs from the Govdocs1 corpus. We tried to assess these PDFs against a user-defined policy, which was made up of the following elements:

1. No encryption or password protection
2. Fonts must be embedded and complete
3. No JavaScript
4. No embedded files (i.e. file attachments)
5. No multimedia content (audio, video, 3-D objects)
6. File should be valid PDF².

The somewhat disappointing result of this excercise was that only 26% of all PDFs in the dataset satisfied all criteria in our test policy! Closer inspection of the Preflight output showed the majority of validation errors that caused this to be related to fonts. Preflight is able to report on many different font-related errors, but their exact meaning is not always clear, and neither is their impact on the rendering process. This made it difficult to establish whether the results reflected the quality of the PDFs, or perhaps our assessment was too strict on font errors.

Way forward: VeraPDF

The original report on the Govdocs1 analysis ended with the following conclusions:

These preliminary results show that policy-based assessment of PDF is possible using a combination of Apache Preflight and Schematron. However, dealing with font issues appears to be a particular challenge. Also, the lack of reliable tools to test for overall conformity to PDF (e.g. ISO 32000) is still a major limitation. Another limitation of this analysis is the lack of ground truth, which makes it difficult to assess the accuracy of the results.

Earlier this year work started on VeraPDF, an open-source PDF/A validator that -like Preflight- will be part of the PDFBox library. Its development is funded by the EU PREFORMA project. The consortium that is behind the software includes the PDF Association, whose member base covers a wide spectrum of vendors that already implement PDF technology. Although still in its early stages, it’s interesting to see how the VeraPDF work could help in solving the issues that we identified as part of the SCAPE work.

Font issues

As I’m writing this, font checks haven’t been implemented yet in the VeraPDF code. Nevertheless, validation profiles already exist for a number of aspects of PDF/A. These profiles contain one or more validation rules, and each rule explicitly references its corresponding clause in the PDF/A standard. For example, have a look at this rule on images:

<?xml version="1.0" encoding="UTF-8"?>
<profile xmlns="http://www.verapdf.org/ValidationProfile" model="org.verapdf.model.PDFA1a">
    <name>ISO 19005-1:2005 - 6.2.4 Images - Alternates</name>
    <description></description>
    <creator>veraPDF Consortium</creator>
    <created>2015-06-16T22:22:45Z</created>
    <hash>sha-1 hash code</hash>
    <rules>
        <rule id="6-2-4-t01" object="PDXImage">
            <description>An Image dictionary shall not contain the Alternates key</description>
            <test>Alternates_size == 0</test>
            <error>
                <message>Alternates key is present in the Image dictionary(</message>
            </error>
            <reference>
                <specification>ISO19005-1</specification>
                <clause>6.2.4</clause>
            </reference>
        </rule>
    </rules>
</profile>

Here, the fields in the clause field in the reference element refers to a specific clause in the PDF/A-1 (ISO 19005-1) specification. This makes the errors much easier to interpret, since they are directly linked to the standard. I expect that this will make the interpretation of font-related errors much clearer as well.

Conformance to canonical PDF

A PDF may satisfy all requirements of PDF/A, and still be broken. An example is file veraPDFHiResWrongObjectID.pdf. If you open it in Acrobat you will see this:

Neverheless, Apache Preflight considers this to be “valid” PDF/A:

The reason for this is that the structure of this file is broken at a deeper level than the (relatively high-level) PDF/A profiles³. Although the current funding for VeraPDF only addresses PDF/A, the veraPDF Technical and Functional Specification stresses that its validation model is extensible, and this would ultimately allow more elaborate validation. From p. 16 of the document:

The veraPDF model encourages plug-ins for parsing not only PDF/A-related third-party data structures (…), but also for other features in ISO 32000, other ISO standards for PDF such as PDF/E or PRC, images, and for embedded content such as rich media or attachments (…)

This suggests that ultimately, VeraPDF has the potential to develop into a full-fledged canonical (ISO 32000) PDF validator. Obviously this would be a huge task that would require substantial additional effort and funding, but it’s encouraging to see that the overall design already allows for such a move.

Ground truth

During the SCAPE project we often struggled to find suitable openly licensed test files. In fact, for much of the policy-based assessment work we relied on files on the Adobe Acrobat Engineering website, which is a true treasure trove of PDFs with exotic features. Or rather was, as the site’s been offline for at least several weeks now, and it’s unclear when (if?) it will be back⁴. Back in 2013, the BL’s Andy Jackson already inquired about the license terms of those files, and Adobe’s response was that although the files were free to use, redistribution was not allowed. Fast-forward two years, and the files are gone! Internet Archive has several snapshots of the site, but they are incomplete and do not include all sample files.

This poignantly illustrates the importance of test data that are available under a sufficiently open license that allows redistribution. I’m happy to see that the VeraPDF initiative includes work on the production a number of openly-licensed test corpora (see also sections CE 3.2 and TS 6.2 of the Technical and Functional Specification).

Conclusion

In this blog series I’ve given a brief overview of some preservation risks of the PDF format, and I showed how PDF/A validators can be used to identify such risks, even in files that are not really PDF/A. I also explained the main problems we encountered while trying to use the open-source Apache Preflight PDF/A validator to identify preservation risks in a large collection of PDFs. The new VeraPDF initiative is still in its early stages, but it appears to be addressing most of these issues. Therefore it would be interesting to apply it to some of the datasets that we used for SCAPE, once the software is more fully developed.

Update on Adobe Acrobat Engineering website

Adobe’s Leonard Rosenthol has commented on the status of the Acrobat Engineering website. Here he explains that Adobe “are working to address the licensing and distribution of those files, which is one reason that site has gone offline”. He also adds that they “hope to have it back as soon as possible”. So this looks like good news after all!

Further resources

• Why PDF/A validation matters, even if you don’t have PDF/A (Part 1)

• Portable Document Format in the OPF File Format Risk Registry

• The Archivist’s PDF Cabinet of Horrors (test corpus)

The purpose of this 2-part blog is twofold: first it is an attempt to give an accessible overview of the earlier work on PDF preservation risks, stressing the importance of PDF/A validator tools in detecting these risks. Second, it provides some tentative suggestions of how the ongoing work on the new VeraPDF PDF/A validator could close some of the gaps and limitations of the SCAPE work.

Preservation risks of PDF

The PDF format has a number of features that don’t sit well with the aims of long-term preservation and accessibility. This includes encryption and password protection, external dependencies (e.g. fonts that are not embedded in a document), and reliance on external software. More details can be found in the PDF entry of the OPF File Format Risk Registry. Below are some examples; I included download links, so you can try them out for yourself.

Document Open password

If you try to open file encryption_openpassword.pdf in Adobe Acrobat, you end up with this dialog:

Without the password, the file cannot be opened at all.

Print password

File encryption_noprinting.pdf can be opened normally, but you cannot print it:

Embedded Quicktime movie

File embedded_video_quicktime.pdf contains multimedia content in Quicktime format. Acrobat cannot render this format natively, and relies on an external player. This is what happened when I opened the file on my PC:

After I clicked on Get Media Player, I was taken here:

I wasn’t able to configure Acrobat to use a media player that supports Quicktime ¹.

External reference to multimedia file

The file movie.pdf contains references to external multimedia files. If you click on any of them you get an error like this one:

Font not embedded

File calistoMTNoFontsEmbedded.pdf uses Calisto MT, but the font is not embedded. Since Calisto MT is a Windows system font, the file looks fine on my Windows PC:

The font does not come pre-installed with common Linux distros, and as a result the file looks quite a bit different on my Linux machine:

3D content

The file digitally_signed_3D_Portfolio.pdf contains 3D artwork. Acrobat correctly renders the 3D content, which can be manipulated interactively by the user:

However, Acrobat aside, the majority of PDF readers don’t support 3D content, with the result that in other readers you may end up with something like this:

Detecting risky features

Archives or libraries may want to check their PDFs for one or more features like those shown above. Reasons for doing so include:

• Pre-ingest checks against an institutional policy (e.g. an archive may not accept PDFs that are password protected)

• Profiling of existing collections for preservation risks (e.g. embedded multimedia content in hard-to-render formats)

For this quite a few useful software tools are already available. For example, qpdf gives detailed information about encryption and password protection:

Similarly, the pdffonts tool that is part of xpdf is useful for checking whether fonts in a PDF are embedded:

As the number of features you want to check for increases, this approach becomes increasingly cumbersome: most of tools only cover some features, so you rapidly end up having to deal with a multitude of software tools and output formats. So you may ask yourself if there’s a way to do this more efficiently.

PDF/A validation

This is where PDF/A enters the picture. The PDF/A standards are nothing more than a set of profiles that impose some restrictions on a PDF, ruling out features that are not well-suited to long-term accessibility. Unsurprisingly, these include the very same features that we are interested in here, such as encryption, non-embedded fonts, multimedia content, and so on. Several tools exist that compare a PDF against PDF/A and report any deviations. These PDF/A validators are typically used to verify “true” PDF/A files; however, they can also be used to detect user-specified risky features in regular PDFs.

The professional version of Adobe Acrobat has a PDF/A validator built into its Preflight tool. After opening a PDF in Acrobat, it allows you to verify its compliance with a number of profiles, including PDF/A (currently A-1, 2 and 3):

This results in output as shown here:

This PDF² (which isn’t a PDF/A) violates the PDF/A-1a profile in several ways, but supposing we’re only interested in encryption and non-embedded fonts, the relevant information can be extracted from Preflight’s output quite easily. This example demonstrates the overall feasibility of identifying preservation risks with a PDF/A validator, but it is not scalabe to situations where you need to verify large volumes of PDFs. This will be the main focus of the second part of this blog series.

To get a better view of the formats in our collection, we did an analysis of the “top 50” most prevalent file extensions in our e-Depot: what are the corresponding formats, can these formats be automatically identified, and can we render them in our reading rooms? This blog post summarises the main findings of this work.

Extension counts

As a first step, we compiled a list with file counts for every unique file extension in our e-Depot. Importantly, we did this for all files on the file system, including main files, supplemental content and (original) metadata files. The following chart shows the number of files for every extension, sorted in descending order (note that the vertical axis has a logarithmic scale):

The total number of of unique extensions is no less than 1163. Somewhat surprisingly, .gif turned out to be the most prevalent extension at 34 million files¹. Altogether, the 10 most common extensions make up 99% of al files in the e-Depot. There is a long tail of extensions of which less than 10 file objects exist, and these make up for over half of all unique extensions. In the remainder of this blog we will take a closer look at the “top 50” of all file extensions. The full list is too large to include in this blog post, but you can view it as as a table at the following link:

50 most prevalent formats in KB e-Depot by file extension

Analysis of sample dataset

For each extension we extracted about 20 sample files². We then tried to identify each file with Apache Tika (version 1.4) in detector mode. The third column of our table shows the results for each extension. A manual inspection of selected samples revealed some further details, which are listed in the fourth column of the table (you may need to use the scrollbar at the bottom to view it). One interesting finding was that Matlab Figure files were misidentified by Apache Tika as either application/x-xfig or image/jpeg.

Further analysis of the contents of the 22 ZIP files in our test dataset yielded some additional formats:

Because of the small sample size (and also the fact that the ZIP files were taken from similar batches), these results cannot be taken as representative. Nevertheless, it does show that the identification of scientific text-based data formats such as mol or cif often isn’t very informative. Automatic identification of such formats is difficult anyway, because they typically don’t have unique patterns or header fields.

Accessibility in reading rooms

Finally we wanted to know to what extent the PCs in our reading rooms support our most common formats. To find out, we simply plugged a thumb drive with our test dataset into one of these PCs, and tried to open sample files for each extension in our “top 50” (and those found in the ZIP files as well). To make the results of this exercise easier to digest, we grouped all extensions into 12 format categories. The table below shows the main results for each category:

The main conclusion is that most formats in our “Top 50” are sufficiently accessible. Nevertheless, there is some room for improvement:

• The currently installed version of Microsoft Office (2010) does not support all previous versions of some of the Office formats. According to Microsoft’s documentation there’s no support for Powerpoint 95 presentations, and the documentation is not clear on Word 95 and earlier either. From the current analysis we cannot establish whether we have these old formats in our collection, so this may need further work in the future.
• Comma-delimited text files are not read correctly by Excel. This is caused by region-specific settings of the PCs in the reading rooms, which cause Excel to expect a semicolon as a separator instead of a comma (the comma is used as a decimal separator in Dutch!). This could be improved by changing the configuration of the PCs (but a side-effect would be that semicolon-separated files would then go wrong instead!).
• The applications that are currently available for the “plain” text formats are not that great for scripts, large data files and files that have non-Windows line endings. This could be easily solved by installing a more sophisticated text editor such as Notepad++.
• As part of the scientific binary data category, we came across some 1800 Matlab Figure files. This is a proprietary format that requires the Matlab software, which is not available in our reading rooms. So, essentially these files are not accessible to our users. Whether we will take any action on this is a different matter, since Matlab licences are expensive and the number of files is relatively small anyway.

Links

• 50 most prevalent formats in KB e-Depot by file extension

• All file extensions in e-Depot and corresponding file counts

Acknowledgements

Victor van der Wolf prepared the file extension counts; Danny Stephan prepared the database queries for the sample dataset. Barbara Sierman came up with the initial idea of a “file formats top 50”.

Back in 2012 the KB conducted a first investigation of the suitability of the EPUB format for long-term preservation. The KB will soon start receiving publications in this format, and in anticipation of this, our Collection Care department has formulated a policy on the minimum requirements an EPUB must meet to ensure long-term accessibility. The policy largely follows the recommendations from the 2012 report. This blog explores to what extent it is possible to automatically assess the EPUBs that we receive against our policy using a combination of the Epubcheck tool and Schematron rules.

KB EPUB policy

The KB’s policy on EPUB is made up of the following objectives:

1. File must be valid EPUB (either version 2 or 3)

   Rationale: this minimises the risk of interoperability problems.

2. File may not contain DRM or encryption

   Rationale: this minimises the risk that files become inaccessible. An edge case here is font obfuscation, which mangles some leading bytes in embedded fonts. This technology is merely meant as a stumbling block to discourage third parties from re-using embedded fonts, and it doesn pose a serious threat to long-term accessibility.

3. File may not contain foreign resources

   Rationale: the Core Media Types define a set of file formats that must be supported by all conforming EPUB readers. Foreign resources are resources that are not part of this set, and the KB’s policy is to not accept them. This requirement minimises the risk of accepting files that contain content that may not be rendered correctly by some readers.

4. File may not contain DTBook content

   Rationale: EPUB 2 offered the option to use the DTBook (DAISY Digital Talking Book) format as an alternative to XHTML 1.1. Support for DTBook was dropped in EPUB 3. Support is already limited with current EPUB reading software: both the popular Calibre and Readium viewers are unable to process EPUBS with DTBook content (although my Sony Reader device handles them without problems). This does not bode well for the future.

Automated conformance checking

To check if an EPUB conforms to the above policy, we need to:

1. test for validity against the format’s standard;
2. extract technical information that tells us something about DRM and file resources inside the EPUB;
3. assess the results of steps 1 and 2 against our policy.

The Epubcheck validator is the obvious candidate for steps 1 and 2. Since Epubcheck is capable of reporting its results in XML format, we can use Schematron rules for the final assessment step. The general approach is similar to earlier work on the JP2 and PDF formats, as well as the British Library’s Flint tool.

Test data

For testing, we first need a corpus of files that are known violate one or more objectives of our policy. As this turned out to be more difficult than expected, I created a small set of test files. Some of the files in this dataset were created from scratch; others were taken directly or adapted from existing openly licensed datasets. The following table lists the main characteristics of the files in the dataset¹:

Apart from the above files, the dataset also includes:

• the full source of each test file (as a directory structure);
• a bash script that automatically builds (zips) all directories to EPUBs;
• another bash script that analyses all EPUBs with version 3 and 4 of Epubcheck;
• the full Epubcheck output of each file.

All files are openly licensed, and by adapting the existing tests it is pretty straightforward to add new ones.

Analysis with Epubcheck

The first question that we need to answer here is whether Epubcheck’s output is sufficiently detailed for our needs. So, as a first step I analysed all files in the dataset with Epubcheck. I did this using both Epubcheck 3.0.1 (the current stable version) and the alpha 11 release of Epubcheck 4.0.0. The full output can be found here. In the following sections I will address each of the objectives of the KB policy.

Encryption objective

For the ‘fake’ encrypted file Epubcheck’s output contains a hasEncryption property. Moreover, the messages element in the output contains an error message that refers to the encrypted resource. In Epubcheck 3 this is:

ERROR: : OPS/XHTML file OEBPS/Text/pdfMigration.html cannot be decrypted

A double-check with a ‘real’ encrypted EPUB (which is proprietary and could not be included in the dataset) confirmed that each encrypted resource produces an error message of the general form:

ERROR: : $fileType file $fileName cannot be decrypted

Here, $fileType and $fileName refer to the file type and name of the affected resource. The ‘fake’ encrypted file also resulted in some additional error messages about undefined fragment identifiers, but these look like secondary errors that result from Epubcheck ’s inability to decrypt the encrypted resource.

The behaviour of Epubcheck 4 is similar, although the error message is slightly different:

RSC-004, ERROR, [File 'OEBPS/Text/pdfMigration.html' could not be decrypted.],epub20_minimal_encryption.epub

The file with the obfuscated fonts also results in a hasEncryption entry in Epubcheck’s output. Epubcheck (both versions 3 and 4) doesn’t provide any direct clue that the encryption in this file is limited to some obfuscated fonts. For our policy-based assessment we can therefore ignore the hasEncryption entry, and simply check for the presence of “cannot be decrypted” error messages (see above).

DTBook objective

Epubcheck’s output does not give any explicit clue to the presence of DTBook content. However, Epubcheck 3 does report a read error on the corresponding file resource:

ERROR: : I/O error reading OEBPS/hauy-2005-1.xml: Stream closed

Epubcheck 4 does not report this error. A check of the DTBook resource confirmed that it is valid against version 2 of the DTBook Document Type Definition (I checked this using both JHOVE and an online XML validator). This suggests that Epubcheck 3 doesn’t properly recognise (cannot parse?) DTBook content, and incorrectly flags EPUBs that hold this as “Not well-formed”. The behaviour of Epubcheck 4 is correct (see also this issue report).

Foreign resources objective

The test dataset contains 3 files with foreign resources (resources that are not on the list of Core Media Types). In the first one I simply replaced a PNG image by a JP2 (and updated the manifest and the reference in the text accordingly). This results in the following validation error (Epubcheck 3):

ERROR: /OEBPS/Text/pdfMigration.html(20): non-standard image resource 'OEBPS/Images/pdfVenn.jp2' of type 'image/jp2'

And in Epubcheck 4;

MED-003, ERROR, [Non-standard image resource of type image/jp2 found.], OEBPS/Text/pdfMigration.html (20-63) 

This error also causes the validation to fail. The EPUB specification allows the use of foreign resources, but only if they have a Core Media fallback. I created two additional test files that use the original PNG image as a fallback³; I then updated the manifest of these files accordingly. Epubcheck validates both files as “Well-formed”, but gives no information whatsoever on the presence of foreign resources. Somewhat alarmingly, both Calibre and Readium failed to read either of these files correctly: the (fall back) image was not shown in both cases. As it turns out, very few EPUB readers support manifest fallbacks, even though this feature has been part of the EPUB specification for a long time (at least since EPUB 2).

Translating the policy to Schematron rules

If Epubcheck were able to address all apects of the KB’s policy, it would be possible to translate each of its objectives into a Schematron rule. As Epubcheck doesn’t yet provide the required information on foreign resources and DTBook content, for now we can only do this for the validity and encryption objectives. The Schematron rule for validity is:

<s:pattern name="wellFormed">
  <s:rule context="/jh:jhove/jh:repInfo">
    <s:assert test="(jh:status = 'Well-formed')">Not well-formed epub</s:assert>
  </s:rule>
</s:pattern>

For the encryption objective we have this:

<!-- This rule rules out encrypted content, but permits font obfuscation-->  
<s:pattern name="encryptedResources">
  <s:rule context="/jh:jhove/jh:repInfo/jh:messages">
    <s:assert test="count(jh:message[contains(.,'cannot be decrypted')]) = 0">Contains encrypted resources</s:assert>
  </s:rule>
</s:pattern>

Alternatively, we could have created a rule that uses the hasEncryption property here, but that would cause any files with obfuscated fonts to fail the assessment. The corresponding schema (adapted from the BL’s Flint tool) can be found here. It is designed to work with Epubcheck 3 only ⁴.

Demo

I created a simple EPUB policy-based validation demo. It is a shell script that validates all EPUB files in a user-defined directory with Epubcheck, and subsequently assesses Epubcheck’s output against a user-defined schema. Note that the purpose of the script is just to demonstrate the general procedure; it is not recommended for operational use.

Possible Epubcheck enhancements

The above tests demonstrate that currently Epubcheck is able to cover two aspects of the KB’s policy on EPUB: validity and encryption. However, its output doesn’t provide the information we need on the presence of foreign resources and DTBook content. It would be useful if Epubcheck could be extended with an option that reports all resources in an EPUB with their corresponding media types. This information can be extracted from the manifest element of an EPUB’s Package Document. For a file with both DTBook content and a foreign resource (a JP2 file) this looks something like this:

<manifest>
    <item href="toc.ncx" id="ncx" media-type="application/x-dtbncx+xml" />
    <item href="Images/pdfVenn.jp2" id="pdfVennJP2" media-type="image/jp2" fallback="pdfVennPNG" />
    <item href="Images/pdfVenn.png" id="pdfVennPNG" media-type="image/png" />
    <item href="hauy-2005-1.xml" id="opf3" media-type="application/x-dtbook+xml" />
</manifest>

Some simple Schematron rules on the media-type attribute would make it possible to filter this for the presence of DTBook content (where media-type is application/x-dtbook+xml) or foreign resources (which have a media-type value that is not on the Core Media Types list). Another solution would be to add properties like hasDTBook and hasForeignResources to Epubcheck’s output. This solution is less generic, but possibly more user-friendly.

Finally, the presence of DTBook resources ⁵ incorrectly causes the validation to fail in Epubcheck 3; this has been fixed in Epubcheck 4.

Links

• Epubcheck
• EPUB KB policy testing dataset, includes full source, build and analysis scripts.
• Policy-based validation demo based on Epubcheck
• EPUB 3 Support Grid – comparison of support of EPUB 3.0 features by different reading systems