From de88e37de52c67175360215cdbdeb507be844ac9 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 10 Aug 2026 23:18:54 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20ADR-288/289/290=20=E2=80=94=20benchmark?= =?UTF-8?q?=20harness,=20wideband=20CSI=20ingest,=20vitals=20ground-truth?= =?UTF-8?q?=20rig?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three ADRs closing the highest-impact gaps from the 2026 SOTA research sweep: public-benchmark comparability (Widar3.0 ingest + leakage-guarded split protocols), wideband 802.11ax CSI ingest (FeitCSI/AX210), and a vitals ground-truth rig (reference ingest, alignment, Bland-Altman agreement, evidence grading). Implementations follow in this branch. Co-Authored-By: claude-flow Claude-Session: https://claude.ai/code/session_015TcKegTS7QqhWPC2L2SzaS --- ...288-public-benchmark-evaluation-harness.md | 106 ++++++++++++++++++ .../ADR-289-wideband-80211ax-csi-ingest.md | 91 +++++++++++++++ docs/adr/ADR-290-vitals-ground-truth-rig.md | 93 +++++++++++++++ docs/adr/README.md | 3 + 4 files changed, 293 insertions(+) create mode 100644 docs/adr/ADR-288-public-benchmark-evaluation-harness.md create mode 100644 docs/adr/ADR-289-wideband-80211ax-csi-ingest.md create mode 100644 docs/adr/ADR-290-vitals-ground-truth-rig.md diff --git a/docs/adr/ADR-288-public-benchmark-evaluation-harness.md b/docs/adr/ADR-288-public-benchmark-evaluation-harness.md new file mode 100644 index 00000000..2d9dcfc5 --- /dev/null +++ b/docs/adr/ADR-288-public-benchmark-evaluation-harness.md @@ -0,0 +1,106 @@ +# ADR-288: Public-benchmark evaluation harness — Widar3.0 ingest, standard split protocols, leakage guards + +- **Status**: Accepted — initial implementation (this PR) +- **Date**: 2026-08-10 +- **Deciders**: ruv +- **Tags**: training, evaluation, benchmarks, widar, mm-fi, leakage, honesty + +## Context + +RuView implements the field's key techniques (CSI ratio, BVP features, MAE +pretraining, rapid adaptation) but reports results only on self-collected data +with self-defined metrics (e.g. the README's held-out temporal-triplet +accuracy). A 2026 deep-research sweep of the WiFi-sensing literature found: + +1. Cross-domain generalization is the field's central unsolved problem; the + only widely reproduced cross-domain result is Widar3.0's BVP benchmark. +2. MM-Fi (NeurIPS 2023) is the standard WiFi-pose benchmark, with defined + cross-subject and cross-environment protocols. +3. The field had a documented leakage reckoning in 2024–2025: window-level + random splits on continuous recordings inflate accuracy (one dataset's F1 + collapsed from ~90% to ~22% under subject-disjoint splits — Sensors + 24(10):3159; Signals 6(4):59). + +`wifi-densepose-train` already has an `MmFiDataset` NPY loader and a +deterministic `SyntheticCsiDataset`, but no Widar3.0 ingest, no standard split +protocols, and no structural leakage guard. CLAUDE.md already requires +mean-pose baselines and leakage-free held-out splits for pose PCK; nothing in +the code enforces this. + +Without leaderboard-comparable numbers, RuView's claims cannot be ranked +against published systems, which blocks both scientific credibility and +commercial (OEM licensing) conversations. + +## Options considered + +1. **Do nothing; keep self-collected metrics.** Rejected: perpetuates the + comparability gap. +2. **Port a Python eval stack (SenseFi) alongside the Rust pipeline.** + Rejected: violates the v2 Rust-workspace direction and adds an unreviewed + dependency surface. +3. **Extend `wifi-densepose-train` with native loaders + protocol machinery.** + Chosen. + +## Decision + +Extend `v2/crates/wifi-densepose-train` with three additions: + +### 1. Widar3.0 ingest (`dataset::widar`) + +- A parser for the Intel 5300 `.dat` CSI log format ("bfee" records) used by + the Widar3.0 raw distribution: framed records with a 3-byte header + (2-byte little-endian length + 1-byte code 0xBB), a 20-byte bfee header + (timestamp_low, bfee_count, Nrx, Ntx, RSSI a/b/c, noise, agc, antenna_sel, + len, rate), and a packed 10-bit-per-component complex CSI payload of + 30 subcarrier groups. Invalid records are skipped with a warning, not a + panic — untrusted file input is validated at the boundary per CLAUDE.md. +- A `WidarDataset` implementing the existing `CsiDataset` trait, mapping + Widar's `Nrx × Ntx × 30` CSI into windowed `CsiSample`s via the existing + subcarrier interpolation, with domain metadata (user, room, orientation, + gesture) parsed from Widar's documented directory/file naming convention. +- No network access: the loader reads a local dataset root. Dataset download + remains a documented manual step. + +### 2. Split protocols (`protocols`) + +- A `SplitProtocol` type expressing the standard evaluations: cross-subject + (MM-Fi style), cross-environment/room, cross-orientation (Widar style), and + random-baseline (explicitly labelled as leakage-prone, for comparison only). +- Split assignment is a pure function of sample metadata + a seed — fully + deterministic, no RNG state. + +### 3. Leakage guards (`protocols::leakage`) + +- A structural `LeakageAudit` that, given a proposed train/test split, + verifies: (a) subject-disjointness, (b) environment-disjointness where the + protocol claims it, (c) no two windows from the same continuous recording + span both sides of the split. A failed audit is an `Err`, not a warning. +- PCK/accuracy reporting requires a `MeanPoseBaseline` computed from the + training split only, and reports model-vs-baseline together, enforcing the + CLAUDE.md rule in the type system rather than by convention. +- Evaluation output is an evidence-tagged report (`MEASURED` requires a + reproducer command line embedded in the report; anything else is emitted as + `SYNTHETIC` or `CLAIMED`). + +## Consequences + +- RuView results become comparable to published numbers (Widar3.0 cross-domain + gesture; MM-Fi cross-subject pose) for the first time. +- The leakage audit will make some existing internal numbers look worse. That + is the point. +- Parsing a legacy binary format adds maintenance surface; mitigated by + fixture-based tests with synthetic, deterministically generated `.dat` + bytes (no dataset redistribution). +- Widar's raw distribution is Intel 5300-specific; ESP32-captured data + continues through existing loaders. The protocols/leakage machinery is + loader-agnostic. + +## Validation + +- `cargo test -p wifi-densepose-train` — unit tests for the bfee parser + (truncated, corrupt, and valid synthetic fixtures), split determinism, + leakage-audit rejection cases, and mean-pose baseline math. +- `cargo bench -p wifi-densepose-train` — criterion benchmark for parser + throughput and split assignment on synthetic corpora. +- No accuracy numbers are claimed by this ADR; it delivers the machinery to + produce MEASURED ones. diff --git a/docs/adr/ADR-289-wideband-80211ax-csi-ingest.md b/docs/adr/ADR-289-wideband-80211ax-csi-ingest.md new file mode 100644 index 00000000..9eaa5024 --- /dev/null +++ b/docs/adr/ADR-289-wideband-80211ax-csi-ingest.md @@ -0,0 +1,91 @@ +# ADR-289: Wideband 802.11ax CSI ingest — FeitCSI/AX210 adapter and subcarrier-agnostic plumbing + +- **Status**: Accepted — initial implementation (this PR) +- **Date**: 2026-08-10 +- **Deciders**: ruv +- **Tags**: hardware, csi, 80211ax, ax210, feitcsi, ingest, mat + +## Context + +RuView's CSI ingest (`wifi-densepose-mat/src/integration/hardware_adapter.rs`) +supports ESP32 serial streams, the legacy Intel 5300 tool, and Atheros/Nexmon +paths. All of these are 802.11n-class: ≤40 MHz bandwidth, ≤114 subcarriers, +2.4/5 GHz. + +The 2026 research sweep found the field's center of gravity has moved to +Intel AX200/AX210 NICs via PicoScenes (closed-source core) and FeitCSI +(open-source, GPL): 802.11ax CSI at up to 160 MHz / 1992 subcarriers, +including the 6 GHz band. This is both the research-grade tier today and the +shape of the data 802.11bf silicon will deliver from ~2026 onward. RuView's +`wifi-densepose-hardware` crate already models 802.11bf session types, but no +ingest path can carry wideband CSI into the pipeline. + +Without a wideband path, RuView cannot develop against the best available +signal, cannot compare ESP32-grade results to wideband upper bounds, and will +meet 802.11bf silicon with no tested plumbing for >114-subcarrier frames. + +## Options considered + +1. **PicoScenes `.csi` ingest.** Rejected for now: the format is produced by a + closed-source core and is versioned/complex; parsing it without a + maintained spec invites silent corruption. +2. **Raw pcap + radiotap parsing.** Rejected: duplicates what FeitCSI already + does on-device, and pulls a packet-capture dependency into the pipeline. +3. **FeitCSI file/stream ingest.** Chosen: FeitCSI is open-source (its header + layout is auditable against the source), targets AX200/AX210, covers + 20–160 MHz including 6 GHz, and emits a compact binary record per frame. + +## Decision + +Extend `v2/crates/wifi-densepose-mat/src/integration` with: + +### 1. `feitcsi` record parser + +- A validated parser for FeitCSI's binary CSI record layout (header with + CSI buffer length, rate/bandwidth/channel metadata, antenna counts, RSSI, + timestamp, followed by interleaved complex CSI). The parser is written + against the documented layout, is version-checked, and rejects + records whose declared dimensions disagree with the buffer length — + untrusted file/stream input is validated at the boundary. +- Bounded allocation: a hard cap on subcarrier count (4096) and antenna + count (8) so a corrupt length field cannot cause unbounded allocation. + +### 2. `DeviceType::FeitCsi` in the hardware adapter + +- File-replay mode (read a recorded FeitCSI capture deterministically) and a + streaming mode fed by an external process writing to a path/pipe. No + privileged operations inside the crate: RuView does not configure the NIC; + FeitCSI's own tooling owns that, per least-authority. + +### 3. Subcarrier-agnostic plumbing + +- Ingest carries native subcarrier dimensionality end-to-end and converts to + pipeline width explicitly via the existing interpolation/decimation stage, + recording the native → pipeline mapping in frame metadata so downstream + consumers know the true spectral resolution. Bandwidth (20–160 MHz) and + band (2.4/5/6 GHz) become first-class frame metadata. + +## Consequences + +- RuView gains a research-grade wideband development path and a tested + ingest shape for future 802.11bf reporting (truncated CIR is a natural + extension of the same plumbing). +- GPL FeitCSI is used as an external tool, never linked: only its output + format is parsed. No licensing contamination of the MIT workspace. +- The parser tracks an external project's format; version checks fail loudly + on mismatch rather than misparse. +- ESP32 remains the deployed sensor tier; wideband is a development/ + validation tier. Accuracy claims from wideband captures must be tagged with + the capture hardware. + +## Validation + +- `cargo test -p wifi-densepose-mat` — parser tests over synthetic fixtures: + valid records at 20/80/160 MHz shapes, truncated buffer, dimension + mismatch, version mismatch, allocation-cap enforcement; adapter replay + determinism. +- `cargo bench -p wifi-densepose-mat` — criterion benchmark for record parse + throughput at 1992-subcarrier frames. +- Hardware validation on real AX210 silicon is explicitly out of scope for + this PR and remains required (per CLAUDE.md) before any capture-path + hardware claim; the file-replay path is testable without silicon. diff --git a/docs/adr/ADR-290-vitals-ground-truth-rig.md b/docs/adr/ADR-290-vitals-ground-truth-rig.md new file mode 100644 index 00000000..428c5f6b --- /dev/null +++ b/docs/adr/ADR-290-vitals-ground-truth-rig.md @@ -0,0 +1,93 @@ +# ADR-290: Vitals ground-truth rig — reference ingest, time alignment, and agreement metrics + +- **Status**: Accepted — initial implementation (this PR) +- **Date**: 2026-08-10 +- **Deciders**: ruv +- **Tags**: vitals, validation, ground-truth, bland-altman, evidence, honesty + +## Context + +`wifi-densepose-vitals` (ADR-021) extracts breathing (0.1–0.5 Hz) and heart +rate (0.8–2.0 Hz) from CSI. The 2026 research sweep found that every credible +vitals result in the literature ships with reference-sensor ground truth +(chest strap, pulse oximeter, ECG, or PSG), and that WiFi heart-rate numbers +without stated scope (single person, static, line-of-sight, short range) are +systematically misleading. RuView currently has no way to produce a MEASURED +vitals number: there is no reference-signal ingest, no time alignment between +CSI-derived estimates and a reference device, and no agreement statistics. + +CLAUDE.md requires accuracy statements to be tagged MEASURED (with a +reproducer), CLAIMED, or SYNTHETIC. For vitals, MEASURED is currently +unreachable. + +## Options considered + +1. **Live BLE/ANT+ integration with reference devices.** Rejected for now: + drivers and pairing are a hardware/product concern; the blocking gap is + the evaluation math, not the radio link. +2. **File-based reference ingest + offline agreement analysis.** Chosen: + every consumer reference device (Polar, Garmin, oximeters) exports + timestamped series; a file boundary keeps the crate dependency-free and + the pipeline deterministic. + +## Decision + +Add a `groundtruth` module to `v2/crates/wifi-densepose-vitals`: + +### 1. Reference series ingest + +- `ReferenceSeries`: timestamped samples (unix millis + value) for one + measurand (`HeartRateBpm` or `BreathingRateBrpm`), with device metadata + (make/model, measurement principle). Parsed from CSV (`timestamp_ms,value` + with a header line); malformed rows are rejected with row-numbered errors — + untrusted file input validated at the boundary. Non-monotonic timestamps + are an error, not silently sorted. + +### 2. Time alignment + +- Constant-offset estimation by maximizing normalized cross-correlation of + the estimate series against the reference over a bounded lag window + (default ±30 s), on a common resampled grid (nearest-sample, no + interpolation of physiological values across gaps larger than a + configurable limit). +- Optional linear clock-drift fit (offset + rate) for long sessions. + Alignment parameters are reported, never silently applied. + +### 3. Agreement metrics + +- `AgreementReport`: n paired samples, coverage fraction (time where both + series had valid samples), MAE, RMSE, mean error (bias), Bland–Altman + 95% limits of agreement, and percentage-within-tolerance (configurable, + default ±2 bpm HR / ±1 brpm breathing). +- Session scope is mandatory metadata: subject count, motion state + (static/moving), line-of-sight (LOS/NLOS/through-wall), distance band. + A report without scope cannot be constructed. + +### 4. Evidence tagging + +- `EvidenceGrade::Measured` is only constructible when the report carries a + reference device, non-zero paired samples, minimum coverage, and a + reproducer command string; otherwise the report grades as `Claimed` (real + data, no reference) or `Synthetic` (generated input). This mirrors + ADR-288's enforcement-in-types approach and the CLAUDE.md tagging rule. + +## Consequences + +- RuView can convert vitals claims from CLAIMED to MEASURED with a + reproducible offline analysis, session by session, scope by scope. +- Honest reporting will likely show heart-rate performance below marketing + intuition, especially NLOS/moving — that is the purpose. +- CSV ingest means a manual export step per session; acceptable at current + scale, and the format is the de-facto export of consumer reference gear. +- No clinical claim is implied: agreement statistics against consumer + reference devices are engineering evidence, not medical validation. + +## Validation + +- `cargo test -p wifi-densepose-vitals` — CSV rejection cases, alignment + recovery of known synthetic offsets/drifts, agreement metrics against + hand-computed fixtures, evidence-grade constructibility rules. +- `cargo bench -p wifi-densepose-vitals` — criterion benchmark for alignment + over hour-scale synthetic sessions. +- Real-session validation (ESP32 capture + chest strap) remains a follow-up + requiring hardware evidence per CLAUDE.md. diff --git a/docs/adr/README.md b/docs/adr/README.md index 11ca228c..c88f08dc 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -145,6 +145,9 @@ Statuses: **Proposed** (under discussion), **Accepted** (approved and/or impleme | [ADR-287](ADR-287-coherent-wideband-rf-tomography-crate.md) | `wifi-densepose-sar` — coherent wideband RF tomography research crate | Accepted (implemented, published) | | [ADR-285](ADR-285-homecore-wasm-first-metaharness.md) | WASM-first Homecore developer metaharness via `npx homecore` | Accepted (implemented and validated) | | [ADR-286](ADR-286-wifi-densepose-sar-harness-via-metaharness.md) | `wifi-densepose-sar-harness` — MetaHarness with darwin/router/flywheel | Accepted (implemented, published) | +| [ADR-288](ADR-288-public-benchmark-evaluation-harness.md) | Public-benchmark evaluation harness — Widar3.0 ingest, split protocols, leakage guards | Accepted (initial implementation) | +| [ADR-289](ADR-289-wideband-80211ax-csi-ingest.md) | Wideband 802.11ax CSI ingest — FeitCSI/AX210 adapter, subcarrier-agnostic plumbing | Accepted (initial implementation) | +| [ADR-290](ADR-290-vitals-ground-truth-rig.md) | Vitals ground-truth rig — reference ingest, alignment, agreement metrics | Accepted (initial implementation) | ---