Metadata Storage for DjVu files
A White Paper
By James Rile, James Rile Associates, December 4, 2001
We have created a method and a standard architecture for embedding metadata in DjVu files, which is supported by the DjVu file parser of the DjVuSearch product and by the Metadata Import Utility that we have developed.
Metadata is information about documents, stored in individual data fields. These fields define, classify and categorize the documents. Historically these data fields have resided in databases, apart from the documents themselves. DjVuSearch embeds the data fields directly into the document, making them portable. In other words, they take their metadata with them. As another alternative to the database-storage of metadata, DjVuSearch permits you to store the metadata in companion text files.
Each document format supported by DjVuSearch has its own structure for storing embedded metadata. For example, HTML has metatags and PDF has the DocInfo field dictionary. The DjVu file format lacks a formal metadata structure, but the open architecture of the DjVu format permits us to create our own standard for metadata storage inside of DjVu files.
The purpose and benefit of storing metadata in the DjVu file structure is that it will not be lost (left behind) during INDIRECT<->BUNDLED conversions, as well as when downloading a DjVu file with the Viewer plug-in. Because metadata is an integrated part of the DjVu file, the DjVu file is portable, as it carries its metadata with it at all times. As a result, for example, a DjVu file can be downloaded from the web and re-indexed on the desktop or on CD, and the metadata will still be present. We call this form of metadata storage Embedded Metadata.
In our standard architecture, metadata fields are stored with tag delimiters. For example, this document in the DjVu format has title, author and date fields that are expressed and stored as:
<title>Metadata Storage for DjVu files</title>
Metadata is stored in our standard architecture at three levels: document-level, shared-page-level and page-level.
Metadata Storage Architecture:
There are two global .iff files per document. The names of these files are docmeta.iff and pagemeta.iff.
The docmeta.iff file contains document-wide metadata and will be used when indexing the document as a whole. This file is mentioned in the DIRM chunk of the DjVu file as an included object.
The pagemeta.iff file contains shared page-level metadata. It is referenced by the metachunk of the DjVu page.
The metachunk of each DjVu page in a new section called JRMD contains unique page-level metadata, along with the reference to pagemeta.iff, if present.
External Metadata Storage Option:
We call this form of metadata storage External Metadata because the metadata is stored in external .txt files. These files will be left behind if the DjVu files are transformed, since they are not embedded in the DjVu file.
Instead of embedding document-wide metadata in docmeta.iff, it is possible to store this metadata externally in a docmeta.txt file.
Instead of embedding shared-page-level metadata in pagemeta.iff, it is possible to store this metadata externally in a pagemeta.txt file.
Instead of embedding unique page-level metadata in the metachunk of a DjVu page, it is possible to store this metadata externally in a page_xxx.djvu.txt file
Combination Embedded and External Metadata Storage is Possible
It is possible for some metadata to be Embedded and some metadata to be External. For example, document-level metadata can be stored internally in a docmeta.iff file, while page-level data is stored externally in page_xxx.djvu.txt files.
We recommend that unless you have specific reasons for doing this, you go ahead and embed the metadata in the DjVu files.
External Metadata can be converted to Embedded Metadata, as detailed below.
Metadata Embedding Utility
The utility will take metadata that is stored in external text files, and store them internally in the DjVu file.
The program will expect the following as the input file structure for Embedding:
BUNDLED and SINGLE:
* there are no restrictions on what <doc_name> should be.
*page_xxx.djvu (1 per page)
* page can be any name.
** when a page is indexed at the page-level, then the name should end with an
underscore followed by a number that is equal to the DjVu page number, and which is
part of the DjVu page file name.
Example layouts for the Metadata Import Utility
Indirect - Document Level:
Indirect - Page Level:
Sample metadata fields in the DjVuSearch Model Application:
Bundled and Indirect - Document Level:
Indirect - Page Level:
Docmeta.iff (permits optional document-level search-indexing instead when present)
Pagemeta.iff (shared page-level metadata)
Page_xxx.djvu (JRMD chunk) (unique page-level metadata)
Methods of Search-Indexing DjVu Files with Embedded Metadata
This is what happens at the search-indexing stage:
The bundled DjVu file is indexed directly. The parser looks for a file with name docmeta.iff in the bundle. The contents of this file (if any) will be used as document-level metadata. Metachunks in the pages will be ignored along with the shared-page-level metafile.
INDIRECT documents indexed as a document:
The index.djvu file is indexed. The parser will recognize that index.djvu is an index for the INDIRECT document, will look for a docmeta.iff file, and will use it as document-level metadata. Metachunks in the pages will be ignored along with the pagemeta.iff file, if any.
INDIRECT documents indexed page-by-page:
The index.djvu file is not indexed. The individual DjVu pages are indexed instead. The parser will find a metachunk in the page file, will decode it, retrieve the metadata, and notice the name pagemeta.iff, which it will load next. docmeta.iff will be completely ignored in this scenario. If the name pagemeta.iff has not been found (referenced) in the metachunk, it won't be opened and processed.
The extra overhead of page-by-page indexing (of opening the pagemeta.iff file) will only occur when indexing an INDIRECT document page-by-page.
Methods of Search-Indexing DjVu Files with External Metadata
The methods are the same as for Embedded Metadata, since if Embedded Metadata is not found, the search indexer looks for the external form of metadata storage in text files, as described above.
How to Index DjVu files in dtindexer.exe (the search-indexing application for DjVuSearch)
Assign the DjVu Viewer for use with DjVu files:
For all types of DjVu files, the DjVu Viewer must be defined as the external viewer for the DjVu file type. Otherwise, the search application will display a DjVu file as extracted text in HTML instead of as DjVu in the DjVu Viewer.
Indexing DjVu-Bundled files:
To build a search-index for DjVu-Bundled files, set up the index with an Include filter for *.djvu, as follows:
Indexing DjVu-Indirect files at the Document Level:
To index DjVu - Indirect files at the document level, we need to open and index the index.djvu file of each Indirect DjVu document, while excluding all of the page-level DjVu files that are exposed in the Indirect format.
The way to do this is to provide an Include filter for index.djvu, while also providing an Exclude filter for *.djvu, as follows:
Indexing DjVu-Indirect files at the Page Level:
To index DjVu - Indirect files at the page-level, we need to open and index the page-level DjVu files that are exposed in the DjVu-Indirect format, , while excluding the index.djvu files..
The way to do this is to provide an Include filter for *.djvu, while also providing an Exclude filter for index.djvu, as follows:
In DjVuSearch, there are two methods to open page-level DjVu files. The first is by creating a simple hyperlink directly to the DjVu page. In this method, the DjVu is treated as a stand-alone page and the internal page navigation buttons of the DjVu Viewer will be grayed-out.
The more advanced method is to create a hyperlink that opens the index.djvu file instead for the page, and then goto the page using a CGI page-open argument. In this method, the DjVu page is opened with internal Viewer page navigation enabled.
Note: for this later method to work, the names of the page-level DjVu files must end with a number that corresponds to the page number within the multi-page DjVu document. Here is an example:
The page number will be parsed from the filename, and will be used as the goto page argument in DjVuSearch.