See the PR for the why/what/proof. Contributors no longer touch scripts/convert-outputs.sha256; drift is advisory on pull requests and strict on main.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#803 changed scripts/build-hermes-plugin.py, so the generated Hermes plugin legitimately
changed and the drift manifest's hermes line went stale — main's "Validate converted outputs"
step failed on drift alone (25/26 passed; every invariant held). Regenerated with
scripts/test-convert-outputs.sh --update on a clean checkout of main; the suite is 26/26.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds scripts/test-convert-outputs.sh, the output half of the regression eval
(the install half landed in #828), and wires it plus the previously
un-wired test-agent-selection.sh (#779) into CI.
Why an eval at all: every converter bug so far passed lint and the existing
tests while the installed product was broken. #778 shipped a double-wrapped
description that was valid YAML, so a wrapper check passed; #817 dropped a
whole tool from --parallel and every remaining tool looked fine. Those are
invariant violations, not syntax errors.
Layer A (no history needed), for every agent x every converted tool:
round-trip parsed(generated).description == source description
strict-parse every generated frontmatter / TOML / YAML parses with a real
parser (kimi/vibe carry only an identifier: id == slug and the
prose file exists; aider/windsurf: "## Name" + description line)
count every tool emits exactly one output per roster agent
source every SOURCE frontmatter strict-parses and carries no leaked
quote — the desktop app reads sources with js-yaml (#473)
Layer B: scripts/convert-outputs.sha256, one aggregate hash per tool plus
divisions.json / tools.json / runbooks.json. A flipped line means outputs or a
contract changed; --update regenerates deliberately so review sees the blast
radius. Date-stable (no generated file embeds a date).
The expected side is derived by an INDEPENDENT strict parse of each source,
never by lib.sh's get_field: the generator uses get_field, so an expected
value derived the same way would move with a get_field bug and hide it —
which is exactly how #778 stayed invisible. That independence found two
shipping defects on the first green run:
- get_field returned only the first line of a multi-line plain scalar.
Three healthcare agents write their description as an indented
continuation; every generated output for them shipped it truncated
mid-sentence while the app showed the whole thing. get_field now folds
continuation lines the way YAML does (newline -> single space).
- get_field stripped only "field: " (one space). The same three files use
column-aligned frontmatter (name: X), so their generated names
carried leading whitespace in every tool's output. Plain-scalar padding
is now trimmed.
After both fixes get_field agrees with PyYAML on name and description for
all 273 sources. Acceptance: re-introducing #778's double-wrap, a dropped
tool, a divisions.json change, and an unquoted source each fail the eval
(the double-wrap via Layer A round-trip, not only the manifest).
Refs #778#817#473#810#826#828#779
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
