v0.7.0 Diff-Aware Verification: design, and what Phase 1 implements

Part of the Cobalt Cockpit documentation. The milestone: roadmap-v0.7.0-diff-aware-verification.md.

Status: Phase 1 is implemented as four pure modules that nothing registers or calls. Phase 1.5 revised, in the same modules, the report a person reads: its layout, its width in terminal columns, and files whose text is not UTF-8. Cockpit's behaviour is unchanged: no option, hook, command, tool or capability was added, and the published version is still 0.6.5. Everything this page describes beyond those modules is design for later phases, each of which needs its own review and the owner's approval.

What it is, and what it is not#

The six gates say that checks ran and what they reported. They do not say whether the lines a change touched were among the lines its tests ran. This layer answers that narrower question for TypeScript and JavaScript, from an LCOV report, and only when Cockpit watched the test run that wrote the report and can show that the source it reports on is the source that run measured.

It is evidence of a limited kind: a coverage tool's claim about which lines ran, checked against what Cockpit observed, never against what the tests assert. A line that ran is not a line that was checked. It is advisory and opt-in, and it neither blocks nor permits DONE.

Decisions approved by the owner#

The options (planned for Phase 3, not in the manifest)#

The archived design named them diffCoverage*; they follow the approved command word now. Like the Stop guard's options they would be settable only by a person.

Facts about the host the design now rests on#

Measured on Claude Code 2.1.301 (Linux) with a throw-away probe plugin, before any implementation.

What the coverage tools write#

Measured on one synthetic TypeScript fixture with pinned versions in disposable directories: Node 22.23.3's test runner, c8 12.0.0, Jest 30.5.2 (babel and v8 providers, Babel 7.29.7) and Vitest 5.0.3 (v8 and istanbul providers, Vite 8.3.1). The fixtures used by the tests (tests/diffcov-fixtures.ts) are those reports.

So an entry means "the tool reported this line", and a missing entry proves nothing: under the statement tools, lines of code that never ran have no entry. Phase 1 therefore never reads a missing entry as "not executable".

Phase 1 modules#

diffcov-lcov: the parser#

A report is untrusted in every byte. It is refused whole on any damage or bound, with a code and a line: an empty or oversized report, invalid UTF-8 (checked by Cockpit, not by a replacing decoder), a byte-order mark anywhere but the start, a control character in a line, a malformed record, a record that does not end, a stray end, more than 8 KiB on a line, 50,000 records, too many entries, or 2 s of parsing. Nothing returns a partial result: a truncated record would turn lines that did not run into lines with no entry.

Only the changed lines it is asked about are kept, merged across records of the same path (a line ran if any record says so); a path named two ways, or only by different letter case, is marked and keeps nothing. The parser hands the event loop back every MiB through an injected pause, and reads its clock from an injected now.

reportReadVerdict states what a reader must have established before the bytes may be parsed: every parent a directory, the report an ordinary file with one hard link, not empty, at most 4 MiB; then exactly as many bytes as the stat gave, and the same stat again afterwards.

diffcov-paths: paths are keys#

A path a report names is never opened, stat'ed or resolved. Each SF: value is normalised by its text and refused if it is a URL of another host or with an encoded separator, a Windows or network path, another scheme, holds a backslash, a C0 or C1 control or a line or paragraph separator, leaves the repository root, or enters .git. Spellings of the root nested in one another resolve nothing. What remains is matched exactly against the paths Git listed as changed. The same module validates the person's path options (no leading -, among the rest) and escapes every repository-chosen text before it is shown: controls, format and invisible characters, bidirectional controls, lone surrogates, the backslash, and right-to-left letters, which would reorder the digits beside them, become \u{…}. Text wider than its room is shortened in the middle, by terminal columns, never inside an escape or a surrogate pair.

It also counts terminal columns, with no dependency (charColumns, textColumns): two for East Asian Wide and Fullwidth characters in the BMP (Unicode 15.1, with the unassigned gaps inside the CJK blocks filled, so a character a later Unicode assigns there counts wide too) and for every code point above the BMP; one for anything else, a combining mark included, so a line within its width also holds no more code points than columns. It counts some characters wider than they are drawn (an astral letter, a combining mark, the two halves of a flag), which only wraps a line early; escaping leaves no zero-width joiner or variation selector to build a wider sequence. One residual remains: ambiguous-width characters (the middle dot and ellipsis the report uses, Greek, Cyrillic) count one, as most terminals draw them, and a terminal set to draw them wide (a CJK locale option) can draw a line wider than it was counted. The tests check the table against the runtime's own Unicode data (every Emoji_Presentation, Han, Hiragana, Katakana and Hangul syllable character in the BMP counts two; nothing in ASCII or Latin-1 does); during development it was also checked against Python's unicodedata 15.1, which it covers whole.

diffcov-classify: one category per changed line#

Before any rule, a changed file's text must be readable: a NUL byte in its first 8,000 bytes is binary (BINARY, Git's own test), and bytes that are not UTF-8 are text in another encoding Cockpit cannot read (SOURCE_NOT_UTF8; Phase 1 called these binary too, and left them out of the header's Uncounted line). A NUL byte in a tracked file Git compared as text anyway (an attribute told it to) and named changed lines in is SOURCE_HAS_NUL. An untracked binary file, and one whose only change is removed lines, have no changed line to count and stay BINARY.

File rules first: not JS/TS, excluded by the option, generated (attribute or marker in the first five lines), or a line table that differs between Git and JavaScript. Then the line rules, in order:

Lines cost far more than bytes: a 4 MiB file of line feeds holds 4,194,304 lines, and classifying one such untracked file made as many verdicts, took about 4.8 s and peaked at 1.27 GiB (measured, before these limits). So the work is bounded by lines and paths, each checked before anything is allocated per line, and refused whole past a limit, never cut short:

Facts about a file's lines (blank, ignore comment) are kept as two bit sets, one bit per line each, read in one pass over the bytes, so together they take at most a quarter of the file's size. Changed line numbers that do not fit the file's text give the file a row (CHANGE_OUTSIDE_FILE) and no verdict. A review entry lists at most 40 ranges of lines and counts the rest, and long text is cut by index, so drawing the report is linear in what it shows. Measured in the host's runtime, inside a memory-capped scope whose idle test runner peaks at about 128 MiB: the 4 MiB file of line feeds is refused in about 10 ms (peak 141 MiB); a file at the per-file limit is classified in about 170 ms (246 MiB); a change at the total limit, 200,000 verdicts, takes about 285 ms with the report drawn (293 MiB); one line more is refused before a verdict exists; an entry of 50,000 ranges is drawn in 1 ms (it took 3.1 s before the 40-range cap). The tests assert the time and the structure (refusal before any verdict, bit-set sizes); peak memory comes from these measurements, not from an assertion.

The last line rule replaces the archived design's "no entry is NON-EXECUTABLE", which the producer measurement showed to be wrong. Every changed path has a file row; deleted lines are counted and have nothing to measure. A file excluded as generated is also listed for review, so a marker in a hand-written file cannot hide it quietly; an ignore comment the change adds is listed for classified files. The tests hold, over seeded random changes and reports that reach every file rule:

• exactly one category per changed line, and every path a row;
• no COVERED or UNCOVERED without bound evidence, and no NON-EXECUTABLE line but a blank one;
• taking any one record out never gives COVERED or UNCOVERED to a line that had neither, except a line that record made UNTRUSTED (with two conflicting records, the one left is used: the conflict was the only doubt), and never changes a blank or excluded line;
• a damaged report is refused, and then every non-blank changed line of a classified file is UNAVAILABLE;
• the same value and the same text for the same input.

The report a person reads#

The report is rendered in counts, never percentages, and its wording rules are tests: it opens with advisory: approves nothing and closes with ADVISORY ONLY. Execution coverage does not prove assertions or correctness. and Nothing here changes a gate, DONE, a review or a release. (word for word; one line each from 76 columns, wrapped below); no verdict word, gate name or ratio appears in its own text; the MATCHED words appear only for bound evidence, and no figure from the report appears otherwise; every reason is in words, and the advice for a refused report follows its reason; repository text and every header value are escaped; no line is wider than the width from 50 to 120 columns, counted in terminal columns as above, and so in code points too; a width or page that is not a finite number draws at 80 columns, page 1.

Its sections, in order:

• What was measured. Source: the working tree on HEAD (12 hex digits), the tracked and untracked paths changed, and the source identity (16 hex digits) or that it is unknown. Diff base: 12 hex digits, and whether it is the task's start or was given (the digits a person gives begin the id, so they are not repeated); a base that is not an ancestor of HEAD says so on a line of its own. Commit ids are never longer than a line, so a wrap cannot split one. Coverage: the report's path and hash, the observed run's closed label and, on a line of its own, its time. Evidence: MATCHED, or NOT USED with the reason, the words No changed line is counted as run or not run., and what to do where there is something to do.
• How much changed. Added, modified and deleted lines, and Uncounted: the files whose changed lines exist but could not be classified (a source the reader refused or could not decode, a NUL byte in a text diff, a binary attribute on text, changed lines outside the text read), each marked NOT COUNTED in its row.
• Changed lines by category. Always the same rows: COVERED and UNCOVERED with their counts when the evidence is MATCHED, and - with not measured when it is not, so that no zero stands for "nothing ran"; then EXCLUDED / NON-EXECUTABLE and UNAVAILABLE / UNTRUSTED, each with its two parts counted on their own rows, and under each part every reason that applies, on a line of its own with its count, most first. A reason belongs to one category only (categoryOf), so the reasons add up to their part and the parts to their group; the tests hold this for every verdict and for the rows as drawn.
• The page. Page n of m, one count for the whole report: each list shows its own twenty entries of that page and says which (21 to 40 on this page), all of itself (when it fits on one page), that it has ended on an earlier page, or none. A page past the end is drawn as the last one and says which page was asked for (one too large to write in digits, that a later page was); a page before the last ends with Continued on page n+1 of m.
• REVIEW REQUIRED. Above the entries, one line for each category on the page saying what it asks of a person. Entries are numbered across pages; each gives the path, its line ranges (at most 40, then how many ranges and lines there are), which break onto a new line only after a comma, and the category with the reason and the number of lines.
• Files. Every changed path, numbered, with its file rule in words; then the records the report named that were not used, the note when a tool lists lines that cannot run, and whose line attribution it is.

When no change could be reported at all, the report gives the reason, says that no changed line is counted or listed, and, where its words do not already say it, what a person can do (a later diff base or fewer untracked files for a change past the limits; the option to set for an unset or refused path).

Eleven golden renderings pin the whole text, line for line, of the states a person meets: MATCHED at 80 and 50 columns, NOT USED with a given base off the branch and an unknown identity, a refused report, an untrusted record, a tool that lists blank lines, a second page and a page past the end, wide characters at 50 columns, a report that could not be made, and the layer off. Each was generated from the renderer and read before it was pinned, and is kept as escaped ASCII.

diffcov-capture: what may be recorded, and when a report binds#

A command is recorded only if its raw text is plain words (no ;, &&, |, redirection, &, quotes, $, backticks, parentheses, globs or ~), from a closed list (node --test, vitest, jest, deno test, bun test, npx/bunx/pnpm exec of vitest or jest, npm/pnpm/yarn test or run of a script), with no flag that moves the tool's root (an abbreviation of one included: npm ran npm test --prefi=sub in sub, measured) and, for npm, pnpm and yarn, no option of the manager's own before --; read by Cockpit's own classifier as a test; on the main session, in the foreground, from the repository root; and only if the host's answer shows success field by field, holding no field the host does not declare. A script's name is the repository's text and is never stored. A record's time must lie between 2020 and 2100 (a safe integer past 8.64e15 is not a date in JavaScript, and formatting one throws), when it is made and again when it is read back; the report shows any other time as unknown and cannot throw on one.

A report binds to a record only when, in order: the reader accepted it; a record exists for this project and this report path; a record holds exactly these bytes; that run changed the report; the source now is the source that run measured; and it did not change while the report was being made. A rerun that left identical bytes, and bytes identical to an earlier run on another source, are notes beside MATCHED, not refusals: genuine reruns produce both. Eight records are kept per project; older ones fall out, and binding then fails closed.

Later phases, and what must come first#

Open for later: several reports in a monorepo; showing branch data; a model-readable form of the report (tool schemas stay unchanged in v0.7.0); capturing cd <dir> && <command> or commands with an environment prefix; coverage from CI.