Field guide — the metadata table

The DAT load file format — delimiters and encoding

A DAT is the metadata table of an e-discovery production. It looks like line noise in a text editor for a good reason, and that reason is also why a well-meant edit can destroy it.

What it is

A DAT is a delimited table: one header line naming the columns, then one line per document. If a vendor hands you a Concordance DAT load file, this is what is inside it — custodian, dates, file names, subject lines, the Bates range each document occupies, and whatever else the producing party agreed to give you.

What it is not is a CSV. Real document metadata contains commas constantly (Rivera, Dana), quotation marks inside subject lines, tabs pasted in from spreadsheets, and line breaks inside comment and body fields. Every ordinary delimiter is already in the data. So the convention went the other way: delimit with characters that a business record will never contain. The usual set is ASCII 20 between columns, thorn (þ, ASCII 254) wrapped around every value, and the registered sign (®, ASCII 174) standing in for a line break that occurred inside a value. This is the pairing Concordance-style exports use by default, and it is what most platforms expect when someone asks you for "a DAT".

þBegBatesþ␔þEndBatesþ␔þCustodianþ␔þCommentsþ
þACME0000001þ␔þACME0000004þ␔þRivera, Danaþ␔þProduced 2026-03-11.®Redacted per privilege log entry 44.þ
þACME0000005þ␔þACME0000005þ␔þChen, Marcusþ␔þþ

þ

Text qualifier. Wraps every value, including empty ones — that is why the last field of the second row reads þþ. Because values are wrapped, a value may contain the column separator without breaking the row.

␔

Column separator, ASCII 20. It is a control character with no glyph of its own, so it is invisible in a text editor; we print it as ␔ here and in the tool so you can see where it sits — directly between the closing þ of one value and the opening þ of the next, with no space.

®

A line break that was inside a value. One document is one physical line, so a two-paragraph comment is stored with its newlines swapped for this character and restored on import. One thing to know before you export: we pass this character straight through into the CSV rather than turning it back into a line break, so a value that used it arrives in your spreadsheet with the ® still in it.

The cost of choosing unprintable delimiters is that the file looks like line noise in Notepad, Excel will not open it usefully, and a well-meaning edit in the wrong editor can destroy it. The benefit is that no value ever has to be escaped, which is the failure mode CSV never fully solved.

Not every producer ships that set, so we detect the dialect instead of assuming it. The tool samples the first 40 lines, tries each known delimiter/qualifier pairing, and keeps the one that yields a consistent column count across those lines — then tells you which one it found rather than quietly proceeding. Five families turn up in our corpus, and all five are covered by tests against files in it: ASCII 20 with þ; the inverted convention with þ between columns and ® as the qualifier; pipe with a caret qualifier; pipe with double quotes and CSV-style doubled-quote escaping; and tab-delimited with no qualifier at all. Header names vary as much as delimiters do — BegBates, BEGDOC, ProdBeg, Control Number and a dozen more — so the Bates columns are matched by normalized name rather than by position.

What breaks, and what the checker reports

The parsers never throw. A malformed row is marked and parsing continues, so you get a report on the whole file rather than an error message about line 12. Findings are grouped by kind with the first line numbers they occurred on, so a systematic problem reads as one finding covering 3,000 lines instead of 3,000 separate findings.

In a DAT specifically, four structural shapes are reported. Rows whose field count does not match the header — the usual sign that an unescaped delimiter slipped into a value, or that two volumes with different column sets were concatenated. Values left unwrapped where the rest of the file wraps them. A text qualifier opened and never closed, which is the worst of the four because it swallows everything after it and can turn the remainder of a large file into one enormous value. And the whole-file convention of a trailing delimiter on every line, which is legal but changes the field count and is worth knowing before you hand the file on.

Character encoding is checked first, on every file, because it decides how every later byte is read. Byte-order marks are honored, UTF-8 is verified rather than assumed, and a file that is not valid UTF-8 is read as Windows-1252 and labeled as such — because that is what an unlabeled load file from a Windows tool almost always is. Guessing silently here is how accented custodian names turn into mojibake three steps downstream.

Bates continuity

The Bates columns are found by header name, tolerating the usual spelling variations, and used as a begin/end pair where both exist or as a single column where they do not. Each value is split into a prefix and its trailing digits, and the values are walked in file order. In range mode the comparison that matters is the next document's begin against the previous document's end, which is what makes a genuine gap distinguishable from an ordinary multi-page document.

Four findings come out of that walk: gaps, duplicates checked at both ends of every range, values that do not advance on the one before them, and padding drift — the digit width changing mid-prefix, as in ACME000045 arriving after ACME0000044, which is rarely how software numbers a volume and usually means a value was typed by hand.

A gap is reported as a warning, never an error. Documents pulled for privilege after numbering leave exactly this trace, so the finding is the beginning of a comparison against the privilege log, not a verdict.

Converting a DAT to CSV

The CSV export is written to RFC 4180: comma-separated, CRLF line endings, values quoted only when they contain a comma, a double quote or a line break, and embedded quotes doubled. It begins with a UTF-8 byte-order mark, which is what stops Excel guessing a local code page. Multiline values survive — a comment field that held a line break keeps it, quoted, so it stays in one cell instead of spilling into the rows underneath. The header row is written exactly as it was read: no renaming, no reordering, no dropped columns.

One honest caveat, because it is a real limitation rather than a rounding error: the thorn qualifier has no escape sequence, so if a source value literally contains a þ, that one character is removed and the number of affected rows is reported. The alternative would be writing a file that misparses on import without saying anything.