Field guide — a token on every line

The DII load file format — Summation's token grammar

A DII carries a token at the start of every line. It is the least tabular of the four load files and the easiest to break by concatenating two volumes.

What it is

DII comes from the Summation lineage and abandons columns entirely. Every line is either a token beginning with @ or a bare continuation line, and a record ends where the next one starts — there is no terminator to leave off. @T opens a document and carries its ID. @C NAME value sets one named metadata field, one field per line. @D gives the image directory, and the bare lines that follow it are that document's page files, in page order. Lines beginning with a semicolon are comments, and any @-token appearing before the first @T is a file-level directive rather than part of a record.

@T ACME0000001
@C PGCOUNT 3
@C CUSTODIAN Rivera, Dana
@D @I\IMAGES\001\
ACME0000001.tif
ACME0000002.tif
ACME0000003.tif

The @I\ prefix on the directory is a placeholder that the loading platform substitutes with the volume root, which is why a DII can be moved between machines without rewriting every path. One trap worth knowing: a second, unrelated grammar also ships with the .dii extension, in which a bare @D opens a record, @T names a field whose value runs over as many lines as it needs until a bare @E closes it, and @EDOC closes the record. Same extension, different language. We identify which one a file speaks before parsing a single record, and handle both.

Structural errors, and why they cascade

Three DII findings are reported, and all three come from the same property: because a record is delimited by tokens rather than by lines, there is nothing in the file that says where a record must end.

Content before the first record — text that appears above any opening token — usually means two files were concatenated, or a header was pasted in. A field left open at end of file, and a record not closed before the next one opens, are the two shapes of the same underlying problem: the file relies on the next token to imply the end of the previous thing, so a missing token does not produce an error at the point of the mistake. It produces a record that quietly absorbs everything after it.

That is why a DII problem tends to be reported far from its cause. The checker reports the line where the unterminated structure started, not where the parser eventually noticed, because the first is the line you have to edit.

Counted against the other files

DII records join the cross-file count comparison: DAT rows, OPT document breaks, LFP document and child records, DII records, all placed side by side. When those numbers disagree, at least one file is describing a different set of documents than the others. A DII that absorbed a record through a missing token will show a count one lower than the DAT, which is often the first visible symptom of a structural error that is otherwise invisible.

Encoding

Checked first, as on every file: byte-order marks honored, UTF-8 verified rather than assumed, anything failing that check read as Windows-1252 and labeled. Summation-era files predate UTF-8 being a safe default, so this is the format where an unlabeled Windows code page turns up most often.