smolkik_adm/RuView
1
0
Fork 0
mirror of https://github.com/ruvnet/RuView.git synced 2026-08-29 03:26:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(nlos): add consumer transient sensing pipeline

Browse Source
This commit is contained in:
rUv
2026-08-22 20:46:26 -04:00
parent 0df48df7b2
commit a373ff666b
100 changed files with 16218 additions and 6698 deletions
Download Patch File Download Diff File Expand all files Collapse all files

251
docs/adr/ADR-328-consumer-nlos-raw-transient-reproduction-pipeline.md Normal file
View File

@@ -0,0 +1,251 @@
# ADR-328: Consumer NLOS raw-transient pipeline and upstream reproduction boundary
| Field | Decision |
|---|---|
| **Status** | Accepted for staged implementation; software contracts may be validated in CI, live-hardware reproduction remains pending until a witness capture passes the protocol |
| **Date** | 2026-08-22 |
| **Owners** | RuView Labs maintainers and sensing research reviewers |
| **Scope** | Commodity optical transient acquisition, calibration, normalization, provenance, upstream reproduction |
| **Extends** | ADR-295, ADR-303, ADR-305, ADR-319, ADR-320 |
| **Related** | ADR-329, ADR-330, ADR-331 |
| **Primary implementation** | `v2/crates/ruview-nlos`, upstream `sidsoma/consumer-nlos`, `harness/ruview` advisory verification |
## Context
Somasundaram et al. introduce motion-induced aperture sampling (MAS) for
consumer time-of-flight LiDAR. Their measurement is not an ordinary depth map.
Each sensor zone records light intensity over time. The strong direct relay-wall
return is followed by much weaker multipath returns whose path lengths constrain
hidden geometry. Multiple frames supply redundant and spatially diverse samples
that improve signal-to-noise ratio and synthesize a larger virtual aperture.
The [Nature paper](https://doi.org/10.1038/s41586-026-10502-x),
[author manuscript](https://arxiv.org/html/2605.17865v1), and
[MIT project page](https://cornar.media.mit.edu/) report 3D reconstruction,
single and multi-object tracking, camera localization, and real-time tracking at
30 Hz. These are upstream `CLAIMED` results until RuView reproduces them with a
named live capture. They are not evidence that Apple exposes the required
measurement to an ordinary iOS application.
The upstream [consumer-nlos implementation](https://github.com/sidsoma/consumer-nlos)
targets ST's P-NUCLEO-53L8A1 research path and captures per-zone histograms. ST
documents the [VL53L8CH](https://www.st.com/en/imaging-and-photonics-solutions/vl53l8ch.html)
compact-normalized-histogram interface and configuration-dependent zone/bin/rate
limits. The exact silicon, expansion board, firmware/API and transient format
must be read from and bound to each capture rather than inferred from the kit
name. The
[P-NUCLEO-53L8A1](https://www.st.com/en/evaluation-tools/p-nucleo-53l8a1.html)
combines the expansion board with an STM32 Nucleo host.
RuView needs an explicit boundary because flattening the input to point clouds,
`Observation.value`, ARKit depth, or CSI destroys the delayed transient that the
inverse problem needs. A successful build or replay also cannot validate the
photon path, relay geometry, ambient-light behavior, or target reflectivity.
## Decision
### 1. Reproduce upstream before modifying its inference state
Phase 1 uses a pinned commit of the upstream implementation and a documented,
upstream-compatible ST assembly with verified raw or compact-normalized
histogram access. The current reference adapter targets VL53L8CH framing, but
the acceptance record stores the observed board/silicon identity and never
promotes a model name inferred from packaging. RuView records the exact upstream
commit, firmware/API digest, scoped enrollment/certificate reference, configuration, calibration
digest, and capture-manifest digest. Upstream code runs as an isolated research
sidecar or separately reviewed tool; it is not silently vendored into the
production Rust dependency graph.
The first geometry follows the upstream plug-and-play arrangement:
1. a planar relay surface fills the sensor field of view;
2. an opaque occluder prevents direct line of sight to the target;
3. a known rigid retroreflective target supplies the initial high-SNR case;
4. an independent ground-truth system measures the hidden trajectory; and
5. an empty-scene background is captured before each experimental block.
Diffuse rigid objects and people are later strata. They cannot be pooled into
the retroreflective acceptance result. The manuscript notes weaker diffuse
returns and an approximately fourth-power distance falloff in its diffuse model;
the project therefore reports performance by target material and range.
### 2. Preserve a first-class transient frame
`v2/crates/ruview-nlos` owns the initial versioned transient scaffold. The table
below is the capture-manifest/next-contract requirement for live promotion, not
a claim that every field already exists in the current v1 Rust/track schema.
The current scaffold implements bounded histogram/provenance/calibration pieces;
missing clock-domain, certificate, world-frame, configuration, and capture
bindings must land in a reviewed schema before L2. The eventual wire/storage
representation retains, at minimum:
| Field group | Required content | Rejection rule |
|---|---|---|
| Identity | schema version, sensor ID or certificate reference, session ID, sequence | unknown schema, unauthenticated identity, duplicate or regressing sequence |
| Time | sensor monotonic timestamp, host receive timestamp, clock domain and uncertainty | future, stale, non-finite, or unbounded clock error |
| Histogram | zone layout, temporal bin count and width, signed/unsigned count encoding, ambient estimate | zero/oversized dimensions, non-finite values, count overflow |
| Geometry | per-zone ray or relay-wall point, sensor intrinsics, world transform, coordinate-frame ID | non-invertible transform, unit mismatch, incompatible frame |
| Calibration | direct-return peak, mask, background reference, calibration and configuration digests | absent, expired, mismatched, or out-of-distribution calibration |
| Provenance | source `LIVE_HARDWARE`/`REPLAY`/`SYNTHETIC`, firmware and capture digests, evidence level | unknown never becomes live; replay and synthetic are visibly distinct |
The HAL may wrap the frame as a modality-specific payload, but it must not
reduce the histogram to a scalar before NLOS preprocessing. Raw transient input
has its own size/rate limits and parser fuzz surface.
### 3. Deterministic normalization stages
The reference normalization pipeline is ordered and individually testable:
1. verify identity, schema, bounds, sequence, timestamps, and calibration;
2. subtract a compatible empty-scene background;
3. locate and mask the strong one-bounce relay-surface return per zone;
4. align each zone's direct peak to the agreed temporal origin;
5. reject saturated, underexposed, or calibration-incompatible zones;
6. transform time/depth into the light-cone coordinate used by MAS; and
7. emit a normalized transient plus quality flags, never an unconditional track.
Calibration is an explicit state machine: `UNAVAILABLE`, `CAPTURING`, `VALID`,
`DEGRADED`, `EXPIRED`, or `REJECTED`. Only `VALID` calibration can produce a
live optical likelihood. Loss of calibration produces `unknown`, not a cached
or synthetic fallback.
### 4. Separate acquisition evidence from inference evidence
The capture manifest is append-only and content-addressed. It binds the raw
stream, exclusions, firmware, configuration, calibration, clock synchronization,
target/relay-surface stratum, and ground-truth source. Track records reference
the capture and model digests. RuVector or a particle filter may improve
temporal inference, but neither can upgrade acquisition provenance or fabricate
unobserved photons.
Raw captures are local research data by default and are not committed to Git.
Only schemas, small non-person fixtures, checksums, and aggregate metrics may be
reviewed in the repository under the data policy.
## Performance decision
The first live reproduction targets end-to-end track updates at **at least 27
Hz**, a preregistered operational definition of “roughly 30 fps.” The rate is
measured from accepted sensor frame through emitted track, not the configured
sensor clock. Dropped, duplicated, replayed, or late frames are not counted.
The frozen zone/bin/integration configuration must itself be documented and
demonstrated capable of this rate; ST configurations documented at 25 Hz cannot
pass merely because another configuration has a 30 Hz maximum.
Implementation budgets are:
| Stage | Budget and behavior |
|---|---|
| Parser and provenance | bounded allocation; reject before copying oversized dimensions |
| Normalization | one pass over zones × bins; reusable buffers; no unbounded capture queue |
| Tracking | bounded particle count and search volume; overload drops stale work rather than growing latency |
| End-to-end | report p50/p95 latency, update rate, frame loss, CPU and memory with named reproducer |
The paper describes 1,000 particles and a 5 cm proximity prior at 30 Hz. Those
are starting parameters, not RuView guarantees. Optimization must preserve a
golden likelihood/track tolerance and may not hide quality loss behind pooled
throughput.
## Security and privacy
1. USB/serial is the initial acquisition transport. A future network bridge
requires authenticated sensor identity, encryption, replay protection,
explicit bind configuration, and a separate threat review. No new
unauthenticated UDP listener is introduced.
2. Firmware and upstream code are supply-chain inputs. Pin commits and toolchain
versions, review licenses, scan dependencies, and verify build/capture
digests. Flashing remains an explicit, confirmed hardware mutation.
3. Malformed frames, NaN/Inf values, integer products, timestamp wrap,
decompression bombs, calibration substitution, and coordinate transforms are
fail-closed parser cases.
4. NLOS presence and trajectory data are sensitive even without imagery. The
approved operator/controller and every required participant receive purpose/
space/time notice and consent controls, pause/withdrawal and a persistent
indicator; raw retention is bounded and session track IDs never claim identity.
5. Safety-critical actuation is outside this ADR. A hypothesis never directly
drives a lock, vehicle, medical device, or emergency decision.
The full attacker/asset analysis is in
`docs/security/consumer-nlos-threat-model.md`.
## Alternatives considered
### Feed ARKit scene depth directly into the MIT algorithm
Rejected. Apple's documented scene-depth surface is processed distance data,
not the per-zone photon-arrival histogram required by the image formation
model. ARKit remains useful for pose and visible geometry under ADR-330.
### Treat CSI as a synthetic optical transient
Rejected. CSI and optical time-of-flight measure different physical channels.
CSI can contribute an independently calibrated likelihood under ADR-329; it
cannot replace the optical measurement or supervise itself.
### Port upstream code to Rust before reproducing it
Rejected for Phase 1. Simultaneously changing hardware, forward model, state
estimator, and language makes failures uninterpretable. The Rust port follows a
pinned upstream baseline and golden captures.
### Vendor all upstream firmware, Python, and data into RuView
Rejected. It increases supply-chain, license, binary-size, and data-governance
risk. The reproduction boundary uses pins, manifests, adapters, and small legal
fixtures instead.
## Consequences
### Positive
1. RuView gains an honest optical-transient modality without corrupting CSI or
scalar HAL semantics.
2. Upstream reproduction and RuView extensions remain experimentally separable.
3. Provenance, calibration, and failure states survive into every downstream
hypothesis.
4. The same contract can later support another histogram-capable ToF sensor.
### Costs and limitations
1. Initial use requires an external ST board and ground-truth rig.
2. Raw histograms and CSI increase sensitive data volume and governance burden.
3. Calibration, reflectivity, relay geometry, ambient light, and motion can
dominate algorithm changes.
4. This decision does not establish through-wall optical sensing, unrestricted
human reconstruction, identity, or production safety.
## Rollout and rollback
| Phase | Enable condition | Rollback trigger | Rollback action |
|---|---|---|---|
| R0 fixtures | schema/parser/property tests pass | parser panic, unbounded allocation, provenance ambiguity | disable crate feature and retain fixtures only |
| R1 upstream live reproduction | approved protocol, verified histogram interface, external ground truth, and `MEASURED` >=27 Hz accepted-update witness | calibration drift, direct line of sight, saturation, provenance gap | invalidate capture; return to controlled geometry |
| R2 Rust shadow | R1 remains valid and golden-capture agreement is within frozen tolerance | likelihood/track divergence or latency regression | keep upstream sidecar authoritative; disable Rust output |
| R3 candidate live | independent witness plus ADR-331 confidence/security/privacy/guardrail gate | research endpoint fails or privacy controls regress | emit `unknown`; withdraw capability certificate |
Rollback never relabels a failed live capture as a passing replay. Existing CSI,
RuVector, WorldGraph, and other RuView runtime paths remain independently usable.
## Objective acceptance mapping
| ID | Requirement | Evidence |
|---|---|---|
| NLOS-328-01 | Preserve bounded per-zone timing histograms and calibration provenance | Rust round-trip, boundary, fuzz/property, golden-vector tests |
| NLOS-328-02 | Reject stale, duplicate, oversized, unauthenticated, non-finite, and calibration-mismatched input | Negative parser and state-machine tests |
| NLOS-328-03 | Keep `LIVE_HARDWARE`, `REPLAY`, and `SYNTHETIC` mutually exclusive | provenance transition tests and capture manifest review |
| NLOS-328-04 | Reproduce hidden-target tracking at roughly 30 fps | `MEASURED` live capture, external ground truth, >=27 accepted updates/s |
| NLOS-328-05 | Prevent synthetic-only acceptance | `ruview nlos verify --require-research-pass` rejection tests |
| NLOS-328-06 | Make upstream/Rust comparison reproducible | pinned commit, firmware/config/calibration/capture SHA-256 digests |
| NLOS-328-07 | Preserve normal RuView operation when NLOS is absent | workspace tests with crate/sensor disabled; advisory harness reports `ABSENT` |
The authoritative experimental steps and statistical gate are in
`docs/research/consumer-nlos-acceptance-protocol.md`.
## References
1. Somasundaram et al., [“Imaging Hidden Objects with Consumer LiDAR via Motion Induced Sampling”](https://doi.org/10.1038/s41586-026-10502-x), Nature 653, 693–699 (2026).
2. [Author manuscript and methods](https://arxiv.org/html/2605.17865v1).
3. [MIT Consumer NLOS project](https://cornar.media.mit.edu/).
4. [Upstream implementation](https://github.com/sidsoma/consumer-nlos).
5. STMicroelectronics, [VL53L8CH product specification](https://www.st.com/en/imaging-and-photonics-solutions/vl53l8ch.html).
6. STMicroelectronics, [P-NUCLEO-53L8A1 evaluation kit](https://www.st.com/en/evaluation-tools/p-nucleo-53l8a1.html).

285
docs/adr/ADR-329-motion-induced-aperture-csi-fusion-world-model.md Normal file
View File

@@ -0,0 +1,285 @@
# ADR-329: Motion-induced aperture inference, CSI fusion, and RuView world-state integration
| Field | Decision |
|---|---|
| **Status** | Accepted for staged implementation; software fusion is testable with replay, retained performance claims require the live paired protocol |
| **Date** | 2026-08-22 |
| **Owners** | RuView Labs perception, RuVector, RuField, and WorldGraph maintainers |
| **Scope** | MAS forward model, temporal inference, calibrated RF/optical fusion, persistent identity-free state |
| **Depends on** | ADR-273, ADR-295, ADR-301 through ADR-305, ADR-311, ADR-319, ADR-328 |
| **Related** | ADR-330, ADR-331 |
| **Primary implementation** | `v2/crates/ruview-nlos`, `ruview.nlos.track.v1` |
## Context
The consumer-NLOS model converts weak, time-resolved optical multipath into a
posterior over hidden geometry or motion. Its strength is localized geometry
around a relay surface; its weaknesses include low signal-to-noise ratio,
reflectivity, range, aperture coverage, and temporal assumptions. RuView CSI
offers a different failure profile: RF can persist through walls and clutter,
but current commodity CSI generally provides coarser, environment-dependent
spatial evidence.
These modalities should not be concatenated merely because both are
interesting. Fusion is justified only when an independently evaluated RF
likelihood improves an objective endpoint over optical NLOS alone. It is also
unsafe to let a learned temporal store or graph overwrite current sensing
quality. RuVector, RuField, and WorldGraph must preserve uncertainty, freshness,
calibration, and provenance rather than manufacture confidence.
The [consumer-NLOS paper](https://arxiv.org/html/2605.17865v1) expresses tracking
as a sequential posterior and uses a particle filter to carry uncertainty and a
motion prior. This aligns with RuView's temporal and spatial primitives, but the
physics likelihood remains the authority for the optical observation.
## Decision
### 1. Use a typed latent state and explicit likelihood factors
For each session-scoped hidden target hypothesis, the latent state is
\[
X_t = (p_t, v_t, \Sigma_t, q_t, a_t)
\]
where position \(p_t\), velocity \(v_t\), covariance or particle distribution
\(\Sigma_t\), quality state \(q_t\), and bounded attributes \(a_t\) are expressed
in a versioned world coordinate frame. No identity field is inferred.
The fused filter computes
\[
P(X_t \mid L_{1:t}, R_{1:t}) \propto
P(L_t \mid X_t, C_L)^{w_L}
P(R_t \mid X_t, C_R)^{w_R}
P(X_t \mid X_{t-1}),
\]
where \(L_t\) is the normalized optical transient, \(R_t\) is independently
captured CSI evidence, and \(C_L,C_R\) are modality-specific calibration states.
The exponents are bounded reliability weights derived from current quality,
not user-facing confidence decoration.
Missing, stale, rejected or invalid modalities are omitted before factor
evaluation and contribute the multiplicative identity. They are never evaluated
as a zero likelihood raised to a zero weight (`0^0`).
Conditional independence is an approximation. Correlated errors such as motion,
clock drift, shared ground-truth leakage, and environmental change are measured
and documented. If correlation cannot be bounded, the implementation uses a
conservative mixture, covariance intersection, or gating rule instead of
multiplying overconfident factors.
### 2. Keep the MAS physics path inspectable
The optical likelihood follows the upstream structure:
1. transform the normalized transient into light-cone coordinates;
2. precompute a canonical space-time impulse response for the known target
shape or reconstruction basis;
3. propagate a bounded particle set using the frozen motion model;
4. render each particle by indexing the canonical response at current relay-wall
samples and pose;
5. score rendered and observed transients with a normalized, bounded likelihood;
6. normalize, calculate degeneracy diagnostics, and residual-resample; and
7. emit the posterior, entropy/effective-particle count, covariance, and quality.
Optimization may vectorize or cache rendering, but every optimized kernel is
compared against a scalar reference over golden and property-generated inputs.
NaN/Inf, all-zero likelihood, underflow, particle collapse, and out-of-volume
states produce a typed degradation or `unknown` result.
### 3. Calibrate CSI as a likelihood, not a centimetre claim
The initial RF factor is deliberately modest. It may provide:
1. hidden-region presence or absence likelihood;
2. coarse zone occupancy;
3. motion onset/cessation likelihood;
4. a broad position prior with measured covariance; or
5. an empty-volume vote for guarded optical background updates.
RF weights are zero when the CSI sensor identity, room calibration, coordinate
transform, freshness, or out-of-distribution gate is invalid. CSI never sharpens
an optical posterior beyond the calibration evidence that justifies it. A model
trained with LiDAR pseudo-labels is evaluated on a held-out partition with
external ground truth; it may not be scored against its own optical teacher.
### 4. Synchronize before fusion
Both inputs enter a bounded temporal join keyed by tenant, workspace, site,
world-frame ID, and capture session. The join records sensor timestamp, receive
timestamp, clock uncertainty, calibration digest, and maximum pairing skew.
Frames outside the preregistered skew are not interpolated into apparent
coherence. The system may make an optical-only or RF-only observation with the
missing modality named, but it may label output `fused` only when both factors
pass identity, freshness, calibration, and time/space alignment checks.
### 5. Divide responsibilities across RuView primitives
| Component | Responsibility | Explicit non-responsibility |
|---|---|---|
| MAS/particle filter | current optical likelihood and posterior update | long-term identity, authorization, hardware provenance creation |
| CSI adapter/model | current calibrated RF likelihood | optical histogram synthesis, centimetre precision without evidence |
| RuVector | embeddings for trajectory/canonical-response similarity and bounded temporal retrieval | replacing the live Bayesian update or upgrading evidence level |
| RuField | observation confidence, covariance, quality, provenance, calibration, and expiration | presenting unknown/stale data as ground truth |
| WorldGraph | session-scoped hidden-object hypothesis nodes and spatial/temporal relations | identity inference or permanent person graph by default |
| Evidence/witness layer | content digests, acceptance result, lineage and receipts | sensing, fusion, or actuation |
`ruview.nlos.track.v1` is the current shared output contract. It carries
track/session IDs, one source classification, position/velocity, covariance,
freshness/expiry, quality state, modality contribution weights, calibration
hash, evidence level, algorithm revision, and one provenance record. It does
not carry two authenticated modality lineages, a world-frame ID, coordinate
transform digest, or capture-manifest digest. Consequently v1 rejects measured
CSI fusion and `l3_corroborated`; the current CSI path is scope-bound synthetic
L0 regression only. A future v2 contract must add those bindings before F1/F2
measured fusion can be enabled. Consumers reject unknown schema versions and
stale tracks now; coordinate compatibility remains a future contract gate.
### 6. Cross-modal training follows, and cannot contaminate, evaluation
After a frozen rules/physics fusion baseline, LiDAR may supervise an RF model.
Dataset partitions are grouped by capture session, subject/target, room, sensor
configuration, and time block to prevent adjacent-frame and environment
leakage. Optical targets for RF training are probabilistic distributions with
quality masks, not hard ground truth. The test endpoint always uses independent
external ground truth and includes LiDAR-only, CSI-only, and fused arms.
The system retains a non-learned fallback. A learned score or fusion policy must
show stratified improvement and calibration before promotion. Model absence or
error degrades to the last independently validated factor, never direct action.
## Architecture and data flow
```mermaid
flowchart TD
A["Transient frame + optical calibration"] --> B["MAS optical likelihood"]
C["CSI frame + RF calibration"] --> D["RF likelihood"]
B --> E["Bounded temporal Bayesian join"]
D --> E
E --> F["ruview.nlos.track.v1"]
F --> G["RuField + RuVector + WorldGraph"]
```
Only the join creates a fused track. RuVector and WorldGraph are downstream
state consumers, not a shortcut around missing, stale, or rejected modalities.
## Performance decision
1. The live tracking loop is bounded to the newest accepted frame. Backpressure
drops obsolete queued work and records loss; it does not accumulate latency.
2. The particle count, canonical volume, search volume, history length, graph
nodes per session, and temporal-join window are configuration-bounded.
3. The fusion budget is measured independently from optical tracking. Report
p50/p95 sensor-to-track latency and update rate for LiDAR-only and fused arms.
4. Optimization is retained only when numerical agreement stays within the
frozen tolerance and the paired research endpoint does not regress.
5. At least 27 accepted LiDAR-only track updates per second is required before
the fusion endpoint is interpreted. A fast fused path cannot rescue a failed
reproduction.
## Security and privacy
1. The join accepts only authenticated sensor/session identities and compatible
tenant/workspace/world-frame bindings. Cross-tenant or cross-session joins
are impossible by type and policy.
2. Sequence and timestamp replay defenses are per modality. Reusing an old CSI
frame to make an optical track look persistent is rejected and audited.
3. A malicious modality may inject extreme likelihoods. Inputs and weights are
bounded, posterior influence is observable, and single-modality ablations are
recorded for forensic review.
4. Track IDs are random and session-scoped. Raw optical/CSI data is local by
default; downstream stores receive bounded hypotheses and provenance unless
an approved research protocol explicitly retains raw data.
5. Long-term memory uses TTL, purpose limitation, deletion, and tenant isolation.
Similarity is not identity. The graph must not create a biometric profile.
6. No fusion output grants actuation. Governed actions require the independent
ADR-321/327 policy, approval, freshness, and receipt path.
## Alternatives considered
### Concatenate optical and CSI tensors into one end-to-end network
Deferred. It obscures failure attribution, calibration, and missing-modality
behavior before a trustworthy baseline exists. A learned likelihood may be
added after the factorized paired benchmark.
### Use CSI only as a binary veto
Useful as an initial safety rule, but insufficient as the final design because
it discards calibrated spatial/motion information. The typed factor supports a
binary likelihood without fixing the architecture to it.
### Store only the posterior mean
Rejected. NLOS can be multimodal and ambiguous. Covariance, particles or a
bounded distribution summary, posterior entropy, and quality are essential for
honest downstream behavior.
### Let RuVector replace the particle filter
Rejected. Vector memory can retrieve similar histories or canonical responses,
but similarity alone is not the current physics likelihood and cannot enforce
frame-level calibration/provenance.
### Promote fusion if any pooled metric improves
Rejected. The objective endpoint, pairing, strata, confidence interval, and
non-success metrics are frozen before capture. Synthetic-only and pooled-only
gains do not pass.
## Consequences
### Positive
1. Optical localization evidence and RF persistence can complement one another without
conflating their measurement physics.
2. Every downstream state retains uncertainty, freshness, modality, and lineage.
3. The factorized baseline makes ablation, failure analysis, and rollback clear.
4. Cross-modal training has a leakage-resistant target and external evaluation.
### Costs and limitations
1. Coordinate/clock calibration and paired ground truth add operational burden.
2. Conditional-independence violations can make naive multiplication
overconfident; conservative fusion may sacrifice apparent sharpness.
3. Current CSI may not provide enough independent spatial information to meet
the 25 percent endpoint. A negative result is acceptable and stops rollout.
4. The architecture tracks hypotheses; it does not establish identity, intent,
photographic reconstruction, or safety certification.
## Rollout and rollback
| Phase | Behavior | Promotion | Rollback |
|---|---|---|---|
| F0 | optical-only reference plus recorded CSI | deterministic replay and calibration tests | disable RF factor |
| F1 | shadow fusion, output not consumed | ADR-328 live LiDAR-only reproduction is `MEASURED` at >=27 Hz in its capture-manifest witness report and paired live capture is complete | discard shadow output |
| F2 | research-visible fused track with explicit evidence | ADR-328 prerequisite remains valid; `MEASURED` >=25% endpoint has adjusted confidence support plus frozen-protocol, independent-CSI, security/privacy, guardrail and witness review | return to optical-only; invalidate certificate |
| F3 | cross-modal learned likelihood in shadow | held-out external-ground-truth gain and calibration | remove model artifact; retain factorized baseline |
Rollback is a configuration/capability-certificate change. Existing optical,
CSI, RuVector, RuField, and WorldGraph services continue independently.
## Objective acceptance mapping
| ID | Requirement | Evidence |
|---|---|---|
| NLOS-329-01 | Reference MAS likelihood is deterministic and bounded | scalar/golden/property tests, all-zero, omitted-factor/`0^0`, and numeric-extreme tests |
| NLOS-329-02 | Fused output requires valid identity, calibration, freshness, clocks, and coordinate frame for both factors | negative temporal/spatial join matrix plus missing/stale-factor omission tests |
| NLOS-329-03 | Missing/rejected modality cannot be mislabeled fused | modality/provenance state tests |
| NLOS-329-04 | RuVector/RuField/WorldGraph preserve uncertainty, TTL, lineage, and session scope | contract and integration tests |
| NLOS-329-05 | Fusion improves a preregistered objective endpoint | F2/ADR-331 gate under one frozen protocol over >=100 paired `LIVE_HARDWARE` sequences: `MEASURED` >=25% mean-position-error **or** lost-track-rate reduction, successful multiplicity-adjusted interval excludes zero, shared endpoint-pairing digest and full position coverage, all frozen guardrails pass, independent CSI plus privacy/security/witness reviews pass |
| NLOS-329-06 | Evaluation is leakage resistant | grouped partitions, frozen protocol, external ground truth, LiDAR/CSI/fused ablation |
| NLOS-329-07 | Optimization preserves correctness and bounded latency | reference equivalence, benchmark deltas, overload/backpressure tests |
| NLOS-329-08 | Fusion cannot actuate directly | policy/API tests proving hypothesis-only output |
## References
1. Somasundaram et al., [Nature article](https://doi.org/10.1038/s41586-026-10502-x).
2. [Motion-induced aperture model and particle-filter methods](https://arxiv.org/html/2605.17865v1).
3. ADR-273: Unified RF spatial world model.
4. ADR-295: Source provenance state machine.
5. ADR-301/302/303/304/305: calibration, OOD, ground truth, evidence, and authenticated sensor identity.
6. ADR-311: Real sensor fusion.
7. ADR-319: Witness chain.

288
docs/adr/ADR-330-native-and-web-ios-nlos-adapter-boundary.md Normal file
View File

@@ -0,0 +1,288 @@
# ADR-330: Native and web iOS NLOS adapters and Apple API boundary
| Field | Decision |
|---|---|
| **Status** | Accepted; native and web software surfaces are implementable now, direct built-in iPhone transient-NLOS remains unsupported until a documented Apple API and live device evidence exist |
| **Date** | 2026-08-22 |
| **Owners** | RuView Labs iOS, mobile web, API, security, and sensing maintainers |
| **Scope** | Swift package/app, ARKit context adapter, authenticated track transport, web replay/live UI, capability claims |
| **Depends on** | ADR-295, ADR-305, ADR-319, ADR-328, ADR-329 |
| **Related** | ADR-034, ADR-035, ADR-331 |
| **Implementation** | `ui/ios-nlos`, `ui/mobile`, `ruview.nlos.track.v1` |
## Context
The phrase “smartphone-grade LiDAR” describes a performance/cost class; it does
not guarantee that an App Store process can read every internal sensor signal.
The MIT technique needs each SPAD zone's photon-arrival histogram. Apple
documents ARKit APIs for processed
[`sceneDepth`](https://developer.apple.com/documentation/arkit/arframe/scenedepth),
[`smoothedSceneDepth`](https://developer.apple.com/documentation/arkit/arframe/smoothedscenedepth),
world tracking, and
[`sceneReconstruction`](https://developer.apple.com/documentation/arkit/arworldtrackingconfiguration/scenereconstruction).
Apple's [scene-depth point-cloud sample](https://developer.apple.com/documentation/arkit/displaying-a-point-cloud-using-scene-depth)
shows how applications request and unproject processed depth. These public
surfaces do not document the per-zone transient histogram used by the MAS
measurement model.
This is an API assessment, not a claim about undisclosed Apple hardware or
future operating systems. It must be reviewed against official documentation
for each supported iOS/Xcode release. Until the needed signal is documented and
validated on a physical device, RuView must not present ARKit depth as an MIT
NLOS reproduction.
The user still needs two useful iOS surfaces:
1. a native app that compiles a shared contract, uses public ARKit depth/pose for
visible context, and consumes external RuView NLOS tracks; and
2. a web-capable mobile UI that receives authenticated tracks or deterministic
replay without pretending to capture the phone's LiDAR.
## Decision
### 1. Publish one contract with explicit capability levels
All iOS surfaces consume `ruview.nlos.track.v1`. Capability is an enum, not
inferred from device marketing:
| Capability | Meaning | Permitted label |
|---|---|---|
| `unavailable` | no valid track source | unavailable/unknown |
| `replay` | deterministic fixture or recorded stream | `SYNTHETIC` or `REPLAY`, persistently watermarked |
| `arkit_context` | public ARKit pose/depth/mesh only | line-of-sight context; never NLOS |
| `external_live` | authenticated live track from histogram-capable external pipeline | live external NLOS, subject to evidence/certificate |
| `apple_transient_live` | future documented raw-transient Apple adapter | disabled until separate ADR, API proof, device witness, and ADR-331 gate |
Unknown values fail closed. UI copy names the actual source and evidence level.
It never shortens `external_live` to “iPhone sees around corners.”
### 2. Keep the Swift core platform-neutral
`ui/ios-nlos/Package.swift` defines:
1. `RuViewNLOSCore`, a pure Swift contract/validation library that can be unit
tested without ARKit; and
2. `RuViewNLOSApple`, an Apple-only adapter behind `canImport(ARKit)` and runtime
availability/capability checks.
The direct iOS app and shared `RuViewNLOS` scheme use the same validated model.
The current v1 core decoder rejects unknown/excess JSON shape, non-finite or
bounded-range numeric values, invalid covariance diagonals, expired/future
tracks, excessive arrays, duplicate IDs, and illegal provenance/evidence
transitions. It has no world-frame or capture-manifest field yet; those are
live-promotion contract requirements, not claims about this G0 scaffold.
The current Apple adapter is deliberately a static capability probe: it does
not start an `ARSession`, request camera permission, or capture/export any depth.
It documents the public-API boundary and supports the authenticated external
track client. A later, separately reviewed line-of-sight context adapter may
export camera pose, intrinsics, visible scene depth/confidence, smoothed depth,
and mesh metadata with explicit `arkit_context` labeling. Neither implementation
synthesizes photon histograms or changes context into an NLOS evidence source.
### 3. Treat the native app as a client of the external NLOS pipeline
The production data path is intended to be an authenticated, versioned RuView
track endpoint, not direct access to the ST board from UI code. The current v1
client verifies:
1. TLS and the configured RuView service identity;
2. a scoped, revocable pairing token or short-lived session authorization;
3. authenticated server session, envelope session, and schema bindings;
4. monotonic sequence and bounded clock skew;
5. track expiry, covariance, modality contributions, and calibration hash.
L2/live promotion additionally requires tenant/workspace authorization,
coordinate-frame and capture-manifest bindings, scoped enrolled sensor identity,
and a valid witness/capability certificate. Those fields are not silently
inferred from v1.
Network loss, app suspension, sensor unavailability, decode error, and stale
data immediately degrade the capability
and clear or visually expire the live track. Cached data is never silently live.
The initial Swift client further pins concrete bounds: `wss` only; no embedded
URL credentials or fragments; redirects refused; ephemeral URLSession state; a
256 KiB maximum message; an exact bounded JSON model; sequence no larger than
the JavaScript safe integer and strictly increasing within a bound session; at
most a 5 second track lifetime; and pairing tokens of 32–512 visible ASCII bytes
stored as `WhenUnlockedThisDeviceOnly` Keychain data. `SYNTHETIC` frames require
replay transport and the reserved zero calibration hash. Live provenance must
retain a raw/CNH transient kind; a nonzero calibration hash is enforced at L2
calibrated and above, not for every L1 live envelope. These are wire/client
safety rules, not proof that the stream is physically honest.
### 4. Web iOS consumes tickets and replay; it does not capture transients
The NLOS surface in `ui/mobile` implements the same schema and freshness rules.
Live mode first obtains an authenticated, short-lived, single-purpose ticket
from `/api/v1/nlos/ws-ticket`, then connects to the NLOS stream. Long-lived
OAuth tokens and credentials are not placed in query strings, logs, local
storage, or replay files. The current server binds the ticket to its server
session, a 30-second expiry, and one use; the client pins the returned WSS URL
to the configured same authority. Tenant/workspace/audience/origin claims are
required before L2 deployment but are not present in the v1 ticket schema.
Deterministic replay is a first-class developer/demo mode. It is visibly marked
`SYNTHETIC` throughout the view, uses fixed seeds/fixtures, cannot update live
spatial memory, and cannot satisfy the research gate. Bounded exact-key
validation and same-authority ticket checks are implemented. Deployment CSP,
server-enforced origin policy, and reconnect backoff remain required hardening
before live promotion.
The web view visualizes plan/perspective geometry, covariance/quality,
freshness/expiry, modality/provenance, and disconnected/degraded state. It does
not call an undocumented WebKit or ARKit bridge.
Both clients accept at most 1,000 ms of future clock skew and a 5,000 ms
envelope lifetime. The web profile is intentionally stricter under receive
silence: it clears a frame after 1,500 ms without a replacement even when the
publisher supplied a longer TTL. Native clears at the signed envelope expiry.
This conservative display-liveness difference does not alter wire acceptance,
evidence level, or research metrics.
### 5. Keep pose/context fusion separate from evidence fusion
ARKit pose may help align the phone display or a separately calibrated external
sensor. That transform is valid only when extrinsic calibration, timestamps,
coordinate conventions, and uncertainty pass. Scene depth may display the
relay wall or visible geometry. It is not added to the optical NLOS likelihood
unless a future reviewed model defines and evaluates that factor.
If the phone and external sensor are not rigidly mounted, phone pose cannot be
treated as sensor pose without an independently measured time-varying transform.
The UI may still display both frames separately.
### 6. Re-evaluate Apple support through a documented gate
At each major iOS/Xcode intake, a reviewer searches Apple's official SDK headers,
documentation, entitlements, privacy manifest requirements, and App Store rules.
Direct Apple transient support advances only if all are true:
1. a public supported API exposes timing histogram/count data with documented
units, dimensions, timestamps, and device support;
2. use requires no private symbols, jailbreak, reverse engineering, or hidden
entitlement;
3. a physical-device capture proves that the data includes usable multipath;
4. privacy/security review and user disclosure pass; and
5. the same live reproduction/fusion protocol passes under a new adapter ADR.
No marketing article, simulator API, depth-map correlation, or successful
compile satisfies this gate.
## Performance decision
| Surface | Software target | Measurement note |
|---|---|---|
| Swift core | decode/validate without blocking the main actor; bounded memory | benchmark representative max-size track frames |
| Native rendering | newest-frame policy; 30 fps-capable presentation where device allows | rendering rate is not sensing rate |
| Web validation/store | bounded per-message parsing and no unbounded history | reject oversized messages before state update |
| Web rendering | responsive recent-track visualization and backoff under loss | requestAnimationFrame rate is not NLOS update rate |
| Transport | p50/p95 server-to-view latency, reconnects and dropped/stale counts | report separately from sensor-to-track latency |
UI optimization may decimate display history, but it cannot decimate or reorder
the evidence record. Performance tests use `SYNTHETIC` fixtures and are labeled
software evidence only.
## Security and privacy
1. NLOS tracking can reveal a person outside direct view. Native and web apps
require explicit consent, purpose text, a persistent indicator, pause/stop,
and clear source/provenance. Background capture is disabled by default.
2. App Transport Security/TLS and short-lived scoped authorization are required.
Debug cleartext/local exceptions are not release defaults.
3. WebSocket messages are untrusted. Enforce maximum message size, schema,
numeric bounds, sequence, freshness, origin, rate, tenant, and coordinate
frame before rendering or storage.
4. Replay files contain no credentials and no raw person/CSI/transient data by
default. Fixtures are synthetic or approved/de-identified and immutable.
5. Native and web telemetry excludes positions, raw sensor frames, tokens,
calibration secrets, and stable person identifiers. Diagnostics use bounded
counters and redacted errors.
6. The display is advisory. It cannot directly actuate a device or certify that
a hidden region is safe.
## Alternatives considered
### Make the first milestone a direct iPhone NLOS app
Rejected. It makes the research depend on an undocumented measurement. The
external sensor proves the architecture independently while iOS remains a
transport, context, and presentation adapter.
### Use only a native app
Rejected. A web/mobile view lowers review and demo friction, exercises the
versioned protocol, and can run deterministic fixtures. It still must not claim
direct sensor access.
### Use only a web app and skip native Swift
Rejected. ARKit pose/depth and physical-device capability checks require the
native SDK. The pure Swift core also gives an independent decoder implementation.
### Embed a permanent bearer token in the app or WebSocket URL
Rejected. Tokens leak through logs, browser history, proxies, crash reports,
and screenshots. Use short-lived, scoped, one-use tickets.
### Treat replay as a transparent fallback during disconnect
Rejected. It would misrepresent stale/synthetic state as live. Replay is an
explicit operator mode with persistent labeling and separate state.
## Consequences
### Positive
1. Native and web iOS deliver useful live/replay experiences without blocking
the core research on Apple's API choices.
2. The same schema, freshness, provenance, and coordinate rules apply across
Rust, Swift, and TypeScript.
3. Public ARKit pose and visible geometry remain valuable but honestly scoped.
4. Future Apple transient access has a precise, reviewable activation gate.
### Costs and limitations
1. The first live iOS experience needs an external histogram sensor and RuView
host; the phone alone is not the sensing system.
2. Cross-language contract tests and release CI add maintenance.
3. Web presentation depends on an authenticated RuView backend; offline mode is
replay only.
4. Simulator and Linux Swift tests cannot validate ARKit, LiDAR hardware,
App Store behavior, or a physical iOS build.
## Rollout and rollback
| Phase | Enablement | Rollback trigger | Action |
|---|---|---|---|
| I0 | pure Swift/TypeScript contract and deterministic fixtures | decoder disagreement, unbounded input, provenance drift | disable view; fix contract/golden vectors |
| I1 | future separately reviewed ARKit context on supported device | permission/session/transform failure | capability `unavailable`; clear context |
| I2 | authenticated external live tracks with valid, unexpired evidence/capability certificate and privacy approval | stale/replay/auth/tenant/evidence mismatch or approval withdrawal | disconnect; clear live state; retain explicit replay option |
| I3 | future Apple transient adapter | any of five activation gates absent/regressed | remove capability certificate and adapter flag |
The NLOS tab/app can be removed or disabled without affecting core RuView CSI,
Rust sensing, memory, MCP, or orchestration.
## Objective acceptance mapping
| ID | Requirement | Evidence |
|---|---|---|
| NLOS-330-01 | Swift core builds/tests without ARKit | `cd ui/ios-nlos && swift test` on supported Swift host |
| NLOS-330-02 | Native iOS scheme builds with public APIs | macOS `xcodebuild` simulator gate and physical-device smoke evidence, separately labeled |
| NLOS-330-03 | Current Apple capability probe, absent a new adapter ADR, never emits raw-transient/live-NLOS provenance | capability/provenance unit tests and source review |
| NLOS-330-04 | Web contract accepts valid `ruview.nlos.track.v1` and rejects malformed/stale/oversized input | Jest/TypeScript boundary tests |
| NLOS-330-05 | Live web transport uses short-lived authenticated tickets | client/server integration and replay/origin/expiry negative tests |
| NLOS-330-06 | Replay remains persistently `SYNTHETIC` and cannot pass live gate | UI/store tests plus harness research rejection |
| NLOS-330-07 | Missing Xcode/hardware or NLOS backend degrades honestly | advisory verifier skips and offline/disconnect UI tests |
| NLOS-330-08 | Web and native builds remain separable from core runtime | build matrix with NLOS surfaces disabled/absent |
| NLOS-330-09 | Future Apple activation uses only documented public access | official SDK/header diff, entitlement scan, privacy/App Store review, device witness, and separate adapter ADR |
## References
1. Apple, [`ARFrame.sceneDepth`](https://developer.apple.com/documentation/arkit/arframe/scenedepth).
2. Apple, [`ARFrame.smoothedSceneDepth`](https://developer.apple.com/documentation/arkit/arframe/smoothedscenedepth).
3. Apple, [`ARWorldTrackingConfiguration.sceneReconstruction`](https://developer.apple.com/documentation/arkit/arworldtrackingconfiguration/scenereconstruction).
4. Apple, [Displaying a point cloud using scene depth](https://developer.apple.com/documentation/arkit/displaying-a-point-cloud-using-scene-depth).
5. Somasundaram et al., [consumer-NLOS measurement model](https://arxiv.org/html/2605.17865v1).
6. STMicroelectronics, [VL53L8CH raw compact normalized histogram interface](https://www.st.com/en/imaging-and-photonics-solutions/vl53l8ch.html).

331
docs/adr/ADR-331-consumer-nlos-evidence-privacy-benchmark-metaharness.md Normal file
View File

@@ -0,0 +1,331 @@
# ADR-331: Consumer NLOS evidence levels, privacy/security, benchmarks, and optional MetaHarness governance
| Field | Decision |
|---|---|
| **Status** | Accepted; software gates are implemented independently of live-hardware evidence, research promotion remains blocked until the preregistered capture passes |
| **Date** | 2026-08-22 |
| **Owners** | RuView Labs maintainers, security/privacy reviewers, benchmark owners |
| **Scope** | Claim taxonomy, acceptance records, benchmark governance, contributor harness, release/promotion and rollback |
| **Depends on** | ADR-166, ADR-168, ADR-282, ADR-295, ADR-299, ADR-303 through ADR-305, ADR-318/319, ADR-328 through ADR-330 |
| **Implementation** | `harness/ruview`, `docs/security/consumer-nlos-threat-model.md`, `docs/research/consumer-nlos-acceptance-protocol.md` |
## Context
Consumer NLOS is unusually easy to overstate. A point-cloud animation can look
plausible even when driven by replay, stale state, calibration leakage, direct
line of sight, or a reflectivity regime unlike the deployment target. Native and
web builds can validate contracts without measuring one multipath photon. CSI
and LiDAR may also share temporal or labeling leakage, producing an apparent
fusion gain that contains no independent RF information.
The capability reveals presence and trajectories outside direct view, which
raises privacy and misuse risks despite not producing a conventional photograph.
It also has no present safety case for autonomous actuation. The project needs a
promotion gate that separates source integrity, software validation, controlled
laboratory evidence, field generalization, and production evidence.
The existing RuView contributor harness is dependency-free at runtime and uses
MetaHarness/Flywheel/Darwin only as development aids. NLOS verification should
reuse that posture: helpful, deterministic, and removable, never a requirement
for sensing, fusion, memory, routing, MCP, or normal runtime behavior.
## Decision
The `@ruvnet/ruview` packed-size ceiling increases from 160 KiB to 220 KiB to
carry the bounded NLOS contract verifier, research-gate evaluator, and operator
skill. The existing no-source-map, no-runtime-dependency, tarball smoke, and
claim-honesty gates remain mandatory. This is a reviewed capability budget, not
an unbounded exemption; the current dry-run package is approximately 203 KiB.
### 1. Keep three orthogonal labels
Every NLOS result records:
1. **Source provenance**: exactly one logical value for live hardware, replay,
or synthetic input; unknown is never coerced to live. The acceptance record
encodes these as `LIVE_HARDWARE`/`REPLAY`/`SYNTHETIC`, while the
`ruview.nlos.track.v1` wire contract uses `live`/`replay`/`synthetic`.
2. **Claim tag**: `MEASURED` with a named reproducer/manifest, `CLAIMED` with a
primary external source, or `SYNTHETIC` for simulation/fixtures. A build is
“validated software,” not a measured sensing result.
3. **ADR-282 evidence level**:
| Level | Meaning for NLOS |
|---|---|
| L0 | simulation or generated transient/track fixtures only |
| L1 | captured replay; deterministic pipeline behavior, not a fresh live result |
| L2 | controlled laboratory capture with external ground truth and frozen protocol |
| L3 | held-out room plus target/subject validation with leakage-resistant splits |
| L4 | multi-site field pilot under approved privacy/safety operations |
| L5 | production operational evidence, incident monitoring, drift and rollback history |
The upstream paper is cited as `CLAIMED` in RuView until reproduced. The first
RuView live acceptance can reach L2 only. No amount of L0/L1 replay volume
upgrades a capability to L2.
This ADR-282 maturity level is not the similarly named wire
`evidenceLevel` in `ruview.nlos.track.v1`. V1 accepts
`l0_synthetic`/`l1_measured`/`l2_calibrated`; `l3_corroborated` is reserved for
a future contract that can retain authenticated dual-modality lineage. These
values describe one envelope's source/calibration ceiling. They never self-promote
research maturity: a synthetic envelope is ADR L0, a captured replay remains
ADR L1 regardless of its historical wire label, and a live
`l2_calibrated` envelope reaches ADR L2 only through this
frozen external-ground-truth witness protocol. The acceptance JSON therefore
uses separate `claim_tag: MEASURED` and ADR maturity `evidence_level: L2`.
### 2. Separate software, research, and release gates
| Gate | What can pass it | What it proves | What it does not prove |
|---|---|---|---|
| Software | Rust/Swift/TypeScript unit, property, contract, build, replay and security tests | implementations compile and enforce declared invariants | photon capture, NLOS accuracy, physical iPhone support |
| Research | preregistered `LIVE_HARDWARE` capture with external ground truth | objective reproduction and fusion endpoints for named strata | field generalization, safety, identity, production readiness |
| Release/promotion | software + security + privacy + required evidence/certificate + human review | capability may be exposed at its exact evidence level | authority to actuate or claim a higher level |
An unavailable Xcode/hardware toolchain is an explicit `SKIPPED`, not a pass or
failure. A present partial/incompatible surface fails shallow discovery. If no
toolchain executes, the verifier reports `NO_BUILD_TOOLCHAINS_AVAILABLE`, never
a pass. Available-build success is only a subset of Gate A and is forbidden in
a hardware or release claim.
### 3. Preregister the reproduction and fusion endpoint
Before opening the test partition, freeze:
1. upstream commit and firmware/configuration;
2. sensor/CSI identity, calibration and coordinate transforms;
3. scene/target strata, exclusion rules and direct-line-of-sight checks;
4. randomized sequence order and grouped split manifest;
5. background capture and OOD/freshness thresholds;
6. track initialization, lost-track definition and maximum association gap;
7. primary metrics and bootstrap confidence interval procedure; and
8. all model/weight/threshold versions for LiDAR-only and fused arms.
The hard gate requested for this program is:
1. **Reproduction:** LiDAR-only produces at least 27 accepted end-to-end track
updates per second on live hardware, the objective definition of roughly
30 fps.
2. **Fusion:** over at least 100 paired live sequences, fusion achieves
\[
G_e = \frac{E_L-E_F}{E_L} \ge 0.25
\quad\text{or}\quad
G_\ell = \frac{\ell_L-\ell_F}{\ell_L} \ge 0.25,
\]
where \(E\) is the preregistered mean target-position error and \(\ell\) is the
preregistered lost-track rate. Both metrics are reported even if only one is
the success endpoint. Zero denominators do not count as improvement.
The protocol also reports confidence intervals, p95 position error, time to
first lock, false tracks in an empty hidden volume, update/latency distribution,
frame loss, calibration/OOD rejection, and performance per reflectivity, range,
motion, relay surface, room, and RF geometry. These secondary metrics prevent a
single successful aggregate from hiding unacceptable behavior.
### 4. Use a bounded, reviewable acceptance record
`ruview.nlos.acceptance.v1` is repository-contained JSON that references, but
does not embed, sensitive captures. Required fields include:
1. exact schema, `LIVE_HARDWARE` provenance, `MEASURED` claim tag, L2 evidence
level, and `EXTERNAL` ground truth;
2. protocol-frozen-before-capture flag;
3. enrolled external ST VL53L8-series sensor-model label for v1, verified raw/CNH transient kind, full
upstream SHA-1, and SHA-256 digests for protocol, firmware/API, combined and
CSI capture manifests, scoped enrolled identities, calibration and analysis;
4. zero synthetic frames and zero replay frames in the scored set;
5. independently verified CSI, a CSI-only ablation, at least one CSI source and
at least 100 paired sequences;
6. raw aggregate counts/durations/sums from which each arm's update rate, mean
position error, lost-track rate, offered optical rate and fused optical-frame
loss are recomputed; full paired-sequence position coverage and shared
endpoint denominators are mandatory; and
7. preregistered bootstrap seed digest, at least 10,000 resamples,
multiplicity-adjusted intervals, witness review, LOS exclusion, and passed
privacy/security review.
The advisory verifier validates an exact-key record, bounds, provenance,
digests, arithmetic, and thresholds; unknown fields fail so secrets or raw data
cannot silently hitchhike in the acceptance artifact. It cannot independently
prove that a digest corresponds to an honest physical experiment. Human reviewers inspect the immutable manifest,
external ground-truth synchronization, raw-capture access controls, exclusions,
and analysis reproducer before promotion.
### 5. Extend, but do not require, the RuView contributor harness
The dependency-free `@ruvnet/ruview` CLI/MCP registry adds governed advisory
surfaces. MCP/static inspection remains read-only. Local `--run-builds` is an
explicit execution mode, must target a trusted checkout, uses an allowlisted
child environment and redacted bounded tails, and is not a sandbox:
| Surface | Behavior |
|---|---|
| `ruview nlos plan` / `ruview_nlos_plan` | returns four staged phases, measurement invariant, expected surfaces and exit gates |
| `ruview nlos verify` | shallowly discovers expected Rust/native/web manifests/contracts, can explicitly run available local builds in a trusted checkout, and evaluates an optional live acceptance record |
| `ruview_nlos_verify` | read-only MCP inspection of the auto-detected repository and optional repository-confined evidence; rejects repository selection and build execution |
| `consumer-nlos` skill | contributor playbook for measurement boundary, reproduction, fusion, evidence, security, and iOS limitations |
The verifier discovers `v2/crates/ruview-nlos`, `ui/ios-nlos`, and `ui/mobile`.
If an optional surface is absent, it reports `ABSENT`. Once its feature marker
exists, missing, escaping, oversized or incompatible required artifacts are
`MALFORMED` and fail. String/manifest discovery does not replace contract tests.
Evidence files are regular, bounded, repository-confined JSON; path escape,
symlink escape, oversize, malformed JSON, replay, or synthetic input fails.
MetaHarness, Ruflo, Darwin, and Flywheel remain contributor tooling. Direct
crate/app builds without the harness are the evidence that runtime does not
require it; the verifier's status field is not proof by itself. Harness
proposals cannot modify sensing evidence, promote a
model, publish a package, merge code, or authorize hardware. Promotion retains
human review.
For future harness-policy evolution, a proposal is retained only if a frozen
holdout shows more than 2 percent quality lift, contributor cost regression is
below 1 percent, p95 latency regression is below 5 percent, security/legacy
tests do not regress, provenance is verified, and a human approves it. These
thresholds govern contributor tooling only; they do not replace the 25 percent
sensor-fusion endpoint.
### 6. Apply privacy-by-default controls
1. A named controller/operator records purpose, lawful/organizational basis,
approved spaces, experiment window, access list, retention, and deletion.
2. Visible notices and an active sensing indicator are required. Participants
can pause/withdraw where applicable. No hidden deployment is allowed.
3. Raw transient, CSI, ground-truth video/tag data, and joined trajectories are
P0/P1 research data: encrypted locally, access logged, minimized, separated
by tenant/experiment, and deleted on the frozen schedule. They are never
committed to Git or emitted in harness logs.
4. Default outputs are bounded session-scoped tracks with covariance,
provenance and expiry. No face image is needed, but “camera-free” is not
“privacy-free.” No biometric identity or cross-session re-identification is
claimed or enabled by default.
5. Dataset/model publication needs a separate disclosure/re-identification
review, consent/license check, and removal request path.
6. Safety and high-consequence use are out of scope. Tracks are advisory and
never direct actuation authority.
### 7. Make security a release blocker
The threat model in `docs/security/consumer-nlos-threat-model.md` is mandatory.
Release requires no confirmed high/critical finding across:
1. untrusted transient/track/parser boundaries and numeric/resource exhaustion;
2. sensor, session, calibration and capture identity/provenance;
3. replay/stale/future/duplicate and cross-tenant/world-frame attacks;
4. transport authorization, short-lived web tickets, TLS/origin and secret
handling;
5. firmware/upstream/npm/Cargo/Swift supply chain and license review;
6. raw research-data retention, logs, fixtures, telemetry and repository policy;
7. denial/degradation behavior and lack of direct actuation.
Findings are confirmed with focused tests before remediation claims. Automated
scanner output alone is not a confirmed vulnerability or a cleared release.
## Performance and optimization governance
Performance reports name hardware, OS/toolchain, configuration, capture digest,
commit, warmup, sample count, and statistics. Report sensor capture rate,
accepted frame rate, track update rate, sensor-to-track latency, service-to-view
latency, CPU, peak memory, queue drops, and output quality separately.
Optimization sequence:
1. profile parser, normalization, rendering/likelihood, resampling, temporal
join, serialization, and UI independently;
2. add scalar golden vectors and property tests before vectorization/caching;
3. bound buffers/particles/history and prefer newest-frame backpressure;
4. compare LiDAR-only and fused quality after every material optimization; and
5. roll back any change that violates numerical tolerance, security, privacy,
freshness, or the preregistered endpoint.
Configured 30 Hz, display 30/60 fps, and replay throughput are not equivalent
to live end-to-end tracking. Reports must name which one was measured.
## Alternatives considered
### Allow synthetic evidence to pass when hardware is unavailable
Rejected. Synthetic and replay are essential for software QA but cannot measure
the required optical path or independent RF gain.
### Treat the published MIT result as RuView's baseline evidence
Rejected. It is a primary `CLAIMED` reference with different hardware/scenes and
does not validate RuView's integration, privacy, security, or fusion.
### Make MetaHarness a required runtime coordinator
Rejected. Core sensing, fusion, memory, routing, MCP and UI must operate without
development orchestration. An advisory tool cannot be a physical trust anchor.
### Store all raw captures indefinitely for reproducibility
Rejected. Reproducibility uses controlled access, immutable digests, frozen
manifests and a retention schedule. Indefinite person/space/RF data creates
disproportionate risk.
### Promote on the 25 percent point estimate alone
Rejected. Pairing, confidence intervals, strata, secondary harms, calibration,
provenance, security, privacy and human review remain required.
## Consequences
### Positive
1. Reviewers can distinguish compile/replay success from physical NLOS evidence.
2. The 30 fps and 25 percent claims become executable, preregistered gates.
3. Privacy and security are part of experiment design, not a post hoc checklist.
4. The optional harness makes missing toolchains/surfaces explicit without
weakening the independent runtime.
### Costs and limitations
1. Controlled capture, external ground truth, review and data governance take
more time than a visual demo.
2. The acceptance JSON verifies integrity/arithmetic, not physical truth by
itself; witness review remains necessary.
3. L2 success does not establish L3–L5 generalization or safety.
4. A negative fusion result blocks promotion even if each modality is
individually interesting.
## Rollout and rollback
| Stage | Published capability | Required evidence | Rollback |
|---|---|---|---|
| G0 | development preview | L0 synthetic software fixtures, explicitly labeled | disable flag/view/tool; retain tests |
| G1 | captured replay demo | L1 capture manifest and privacy approval | revoke fixture/access; return to synthetic |
| G2 | controlled research result | L2 live protocol + acceptance + security/privacy review | invalidate certificate; remove live label; optical-only or unknown |
| G3+ | held-out/pilot/production | level-specific ADR-282 artifacts and monitoring | level downgrade, stop capture, delete per schedule, incident review |
Rollback preserves audit/witness records and never rewrites provenance. It may
disable NLOS/fusion independently while CSI and the rest of RuView continue.
## Objective acceptance mapping
| ID | Requirement | Evidence |
|---|---|---|
| NLOS-331-01 | Claim tag, provenance and L0–L5 level never alias | type/state tests, UI label tests, claim checker |
| NLOS-331-02 | Synthetic/replay cannot pass research | `evaluateResearchEvidence` negative tests and CLI nonzero with `--require-research-pass` |
| NLOS-331-03 | Reproduction is roughly 30 fps | live accepted update rate >=27 Hz, external-ground-truth manifest |
| NLOS-331-04 | Fusion has objective value | `MEASURED` in the capture-manifest witness report under the frozen protocol over >=100 paired sequences; >=25% mean-error **or** lost-track reduction with adjusted interval excluding zero, shared pairing digest, and all frozen guardrails passing |
| NLOS-331-05 | Harness remains optional | core/app tests without harness dependencies; absent-surface test |
| NLOS-331-06 | Present malformed surface/evidence fails closed | path/bounds/schema/digest/provenance and partial-surface tests |
| NLOS-331-07 | Privacy controls cover collection through deletion | approved experiment record, access/retention/deletion audit |
| NLOS-331-08 | No unaccepted high/critical security finding at release | threat-model review, focused regression tests and dependency/secret scans; `NLOS-SEC-EX-001` was closed by lockfile remediation without an exception, while future findings still require correction or an exact signed, unexpired exception record |
| NLOS-331-09 | Optimization does not trade away quality or freshness | golden/reference equivalence and named benchmark report |
| NLOS-331-10 | No hardware/App Store claim from CI alone | PR/release claim check and explicit physical-device witness field |
| NLOS-331-11 | Optional harness-policy evolution cannot self-promote or regress its frozen gates | `MEASURED` frozen held-out/anchor report proving >2% quality lift, <1% cost regression, <5% p95 latency regression, unchanged security/legacy gates, verified provenance and human approval; otherwise discard proposal |
## References
1. ADR-282: mandatory L0–L5 evidence ladder.
2. ADR-295: source provenance state machine.
3. ADR-303/304/305: ground-truth synchronization, evidence engine, authenticated sensor identity.
4. ADR-318/319: capability certificates and witness chain.
5. MIT, [Consumer NLOS project and 30 Hz demonstration](https://cornar.media.mit.edu/).
6. Somasundaram et al., [Nature paper](https://doi.org/10.1038/s41586-026-10502-x).
7. STMicroelectronics, [VL53L8CH raw histogram interface](https://www.st.com/en/imaging-and-photonics-solutions/vl53l8ch.html).

545
docs/research/consumer-nlos-acceptance-protocol.md Normal file
View File

@@ -0,0 +1,545 @@
# Preregistered RuView consumer NLOS reproduction and CSI-fusion protocol
**Protocol ID:** `ruview-consumer-nlos-v1`
**Status:** Template to freeze before the first scored capture
**Governed by:** ADR-328, ADR-329, ADR-330, ADR-331
**Primary claim scope:** controlled, identity-free tracking of a known hidden
rigid target using an externally enrolled, histogram-capable ST transient sensor
whose exact silicon/firmware/API are recorded; not built-in iPhone
NLOS, through-wall optical sensing, unrestricted people tracking, or safety use
## 1. Research question and decision rule
This protocol answers two ordered questions:
1. Can the pinned consumer-NLOS baseline track a hidden target from live
commodity transient LiDAR at roughly 30 frames per second?
2. After that is established, does independently captured RuView CSI improve
target-position error or lost-track rate by enough to justify fusion?
The program passes only when all provenance/privacy/security gates pass and:
1. the LiDAR-only arm emits at least **27 accepted end-to-end track updates per
second** on live hardware; and
2. over at least **100 paired live sequences**, the fused arm reduces either
mean target-position error or lost-track rate by **at least 25 percent**
relative to LiDAR-only, with its adjusted interval excluding zero and all
frozen guardrails passing. Fused update rate remains a reported guardrail,
not an added substitute for the requested improvement endpoint.
`SYNTHETIC`, simulator, generated, duplicated, or captured replay frames cannot
pass. They may be used for software QA and pilot power planning only. A build,
UI frame rate, configured sensor frequency, or upstream paper result is not the
measured endpoint.
## 2. Hypotheses
### H1: live reproduction
For the primary controlled stratum, the LiDAR-only pipeline's accepted
sensor-to-track update rate is at least 27 Hz. Failure stops confirmatory fusion
interpretation. Diagnostics may continue but are labeled exploratory.
### H2a: position-error improvement
\[
G_e = \frac{\bar E_L - \bar E_F}{\bar E_L} \ge 0.25,
\]
where \(\bar E_L\) and \(\bar E_F\) are sequence-weighted mean Euclidean target
position errors for LiDAR-only and fused arms, using an external ground-truth
coordinate frame.
### H2b: lost-track improvement
\[
G_\ell = \frac{\ell_L - \ell_F}{\ell_L} \ge 0.25,
\]
where \(\ell\) is the fraction of evaluable time that satisfies the frozen
lost-track definition.
Fusion succeeds if H2a **or** H2b meets its magnitude and multiplicity-adjusted
uncertainty gate. Both metrics and all guardrails are reported. A zero baseline
denominator cannot establish gain.
## 3. Roles and separation of duties
| Role | Responsibility | Must not do |
|---|---|---|
| Protocol owner | freeze protocol, strata, splits and decision rule | inspect sealed confirmatory results before freeze |
| Capture operator | approved setup, consent, identity/calibration, run manifest | tune model/thresholds during scored capture |
| Ground-truth owner | independent system, clock/transform checks, sealed labels | feed labels into online LiDAR or fusion arms |
| Model owner | freeze upstream/Rust/CSI/fusion artifacts | alter artifacts after test partition opens |
| Analyst | run committed reproducer and report all endpoints/strata | delete trials or change exclusions post hoc |
| Security/privacy reviewer | approve collection, access, retention, threats | waive live provenance or identity/actuation boundaries |
| Witness reviewer | verify digests, randomization, exclusions and analysis | equate acceptance JSON arithmetic with physical audit |
One person may hold multiple roles in a pilot, but protocol/model ownership and
ground-truth/confirmatory analysis should be independently reviewed.
## 4. Hardware and software freeze
Complete and sign this table before scored capture:
| Item | Frozen value |
|---|---|
| ST kit and sensor | exact board and silicon read from hardware; raw/CNH API compatibility witnessed; scoped enrollment/certificate reference stored in the restricted manifest rather than a guessable raw-serial hash |
| Firmware | source/release, compiler/toolchain, binary SHA-256 |
| Sensor configuration | zones, bins, bin width, requested rate, integration/subsampling, ambient settings |
| Upstream baseline | `sidsoma/consumer-nlos` full commit SHA and clean/patch manifest |
| RuView | full Git commit SHA; `ruview-nlos` crate feature/config digest |
| CSI nodes | authenticated IDs, hardware/firmware, channel/bandwidth/subcarrier configuration |
| Optical calibration | wall points/plane, direct-return masks/peaks, background, timestamps, SHA-256, expiry |
| RF calibration | room/link fingerprint, coordinate transform, timestamp, SHA-256, expiry/OOD threshold |
| Ground truth | device/camera/tag firmware/software, calibration digest and measured clock uncertainty |
| Models | canonical target response, particle count/motion prior/score, CSI model, fusion weights and digests |
| Hosts | CPU/GPU/RAM/OS, power mode, process priority, compiler/runtime versions |
| Analysis | script/lockfile/container digest, bootstrap seed list and report template |
Changing any frozen item starts a new protocol version or invalidates the
affected block. No silent patch is permitted.
## 5. Physical setup and primary stratum
### 5.1 Geometry
1. Mount the transient sensor so a matte, light-colored planar relay surface
fills its field of view.
2. Place an opaque occluder so no sensor zone, phone camera, or operator-facing
optical path directly sees the scored target. Record a setup photograph/mesh
for review; do not publish participant imagery by default.
3. Start with the upstream-friendly geometry: sensor-to-wall less than 1 m and
wall-to-target approximately 1 to 1.5 m, then record exact distances.
4. Define a right-handed world coordinate frame, units in metres, transform
chain, uncertainties, and a hidden-region boundary before capture.
5. Place CSI nodes/APs in a fixed documented configuration. Confirm CSI is not
derived from the LiDAR, target-control signal, or ground-truth system.
### 5.2 Primary target
The confirmatory reproduction target is a known rigid approximately 25 cm
retroreflective patch or the exact upstream canonical target. The target shape
and response are frozen before the confirmatory split. This supports a scoped
known-shape tracking claim only.
Diffuse rigid objects, hands/people, multiple objects, other sizes, longer
ranges, sunlight, non-planar relay surfaces and moving sensors are separate
exploratory or later confirmatory strata. Never pool them to imply generalized
human/scene reconstruction.
### 5.3 Ground truth
Use an independent externally calibrated system, for example an overhead camera
with a rigid AprilTag/active marker or a surveyed motion stage. The system must
observe the target while remaining unavailable to the online algorithms. Measure
spatial transform error and clock offset/jitter before and after each block.
Ground-truth labels remain sealed until capture and all three online arms are frozen.
A label interpolated beyond the frozen maximum gap makes that time point
unevaluable; it is not imputed from the NLOS output.
## 6. Calibration and negative controls
For every block:
1. enroll/verify sensor and CSI identities;
2. fit the relay wall and verify per-zone direct-return distance against a
physical measurement;
3. capture the frozen-duration empty-scene optical background with no person or
target in the hidden region;
4. capture RF empty-room calibration under the approved protocol;
5. verify clocks, transform chain and calibration digests;
6. run an empty hidden-region negative sequence;
7. run a direct-line-of-sight exclusion check with the occluder; and
8. mark calibration `VALID` only after all bounds pass.
Negative controls include sensor disconnected, CSI disconnected, stale/replayed
frame injection in a non-scored software run, target absent, static distractor,
and calibration mismatch. The live confirmatory stream contains no injected
synthetic/replay frames.
## 7. Trial unit, sample size and randomization
### 7.1 Paired sequence
A paired sequence is one continuous, live, preregistered target trajectory whose
accepted transient-histogram and CSI frames are delivered simultaneously to frozen
online arms:
1. **L:** LiDAR-only MAS tracker; CSI is unavailable to every decision in this
arm; and
2. **C:** CSI-only ablation, reported to establish independent RF information;
it need not satisfy a centimetre-localization promotion threshold; and
3. **F:** the same optical inputs and initialization plus the frozen calibrated
CSI likelihood.
All three arms use the same synchronized live interval; L/F share the offered
optical fan-out and C/F share the offered CSI fan-out. They do not take turns on
different captures. One-arm drops remain outcomes, not exclusions. If resource contention is material, run them
on matched isolated hosts fed by the authenticated live fan-out and record fan-
out latency/drop parity. Replaying a recording later does not satisfy the live
gate.
### 7.2 Minimum and power
Capture at least 100 valid paired sequences in the primary stratum. Before
confirmatory capture, use a disjoint pilot or synthetic/replay data to estimate
cluster variance and document power for detecting a 25 percent gain at family-
wise alpha 0.05. If the calculated requirement exceeds 100, use the larger
number. Pilot sequences, rooms and target paths do not enter confirmatory
metrics.
Each sequence duration, initialization window and trajectory family is frozen.
Include translation directions, speeds and positions across the hidden volume,
not 100 copies of one favorable path. Randomize trajectory order and block order
with a committed seed. Counterbalance any host assignment.
### 7.3 Grouping and splits
Calibration/training, pilot and confirmatory partitions are grouped by capture
session, time block, path family, target instance, room and sensor configuration.
Adjacent frames from one sequence cannot cross partitions. If a learned CSI
model uses LiDAR supervision, no optical target/embedding from the confirmatory
partition is used for training, thresholding, normalization or early stopping.
## 8. Online quality and exclusion rules
Freeze numeric values for each placeholder before capture:
| Rule | Frozen value |
|---|---|
| Valid calibration age and OOD bounds | `<fill before capture>` |
| Maximum optical/CSI/ground-truth clock uncertainty | `<fill>` ms |
| Maximum optical-to-CSI pairing skew | `<fill>` ms |
| Maximum ground-truth interpolation gap | `<fill>` ms |
| Track association radius | `<fill>` m |
| Lost-track consecutive interval | `<fill>` frames or ms |
| Posterior quality/entropy/effective-particle threshold | `<fill>` |
| Saturation/underexposure and minimum valid zones | `<fill>` |
| Maximum sequence frame-loss fraction | `<fill>` |
| Warmup/initialization exclusion | `<fill>` frames, applied to all three arms |
Pre-capture exclusions only:
1. consent/safety/indicator failure;
2. sensor/CSI/ground-truth identity or clock failure;
3. calibration invalid before sequence start;
4. direct line of sight or physical setup outside tolerance;
5. target controller/ground truth did not execute the randomized trajectory;
6. raw capture corruption affecting all affected arms; or
7. host failure prevents paired operation.
Algorithm failure, low signal, lost track, high error, drift, overload, one-arm
drop, poor target position, unfavorable reflectivity or unexpected but in-scope
motion are outcomes, not exclusions. Report all excluded sequences with reason
and arm-independent timing.
## 9. Endpoint definitions
### 9.1 Accepted update rate
For each sequence and arm, use the entire frozen evaluable wall-time window:
\[
f = \frac{N_{valid,new}}{T_{evaluable}},
\]
where `valid,new` means a newly computed, schema-valid, calibrated, fresh track
from a unique live sensor frame after the symmetric warmup. Duplicates, replay,
late/stale frames, renderer frames and cached outputs do not count. Report
the accepted sensor-frame count as well and require track updates not to exceed
it. If an implementation instead uses first-to-last span, its estimator is
`(N-1)/(t_last-t_first)`, not `N/span`. Report sequence distribution and overall accepted updates divided by evaluable wall
time. H1 and the fused performance guard use the lower preregistered aggregate
definition, not the maximum instantaneous rate.
### 9.2 Position error
At each evaluable matched timestamp:
\[
e_{a,t}=\|\hat p_{a,t}-p^{GT}_t\|_2.
\]
Compute a mean within each sequence first, then the equally weighted mean across
sequences so long sequences do not dominate. Report median, p95 and axis-wise
error as secondary metrics. Invalid/missing intervals contribute to lost-track
rate and cannot simply disappear from the report. For the confirmatory position
endpoint, every paired sequence receives a score in both arms. A sequence with
no valid position receives the preregistered worst-case/censoring penalty; it is
not dropped. Consequently each arm's `position_error_sample_count` must equal
`paired_sequences`, and the shared scoring-mask/penalty rules are bound by
`endpoint_pairing_sha256`.
### 9.3 Lost-track rate
A track is lost when the arm has no `VALID` matched hypothesis within the frozen
association radius for at least the frozen consecutive interval after warmup.
Lost-track rate is lost evaluable time divided by total evaluable time. Report
number/duration of episodes and reacquisition time. Track-ID changes without
position loss are reported separately.
### 9.4 Empty-region false tracks and safety guardrails
Report confident-track time and event count during preregistered empty-region
sequences. Also report p95 latency, frame loss, calibration/OOD rejection and
non-winning primary metric. A fusion gain accompanied by materially worse empty-
region false tracks, severe latency/update-rate loss, provenance failure or a
privacy/security failure is not promoted even if the 25 percent arithmetic
passes. `offered_optical_frame_count` is the number of eligible optical frames
offered to both paired arms. Its rate is recomputed against the shared
LiDAR/fused evaluable duration and cannot exceed `sensor_configured_max_hz`.
Fused frame loss is recomputed as
`(offered_optical_frame_count - fused.accepted_sensor_frame_count) /
offered_optical_frame_count`; a supplied rate that differs fails closed.
## 10. Statistical analysis
1. Calculate paired per-sequence deltas and relative gains. Do not treat frames
within a sequence as independent samples.
2. Use a paired cluster bootstrap over sequences with at least 10,000 resamples
and committed seeds. Report point estimate and two-sided interval for every
metric.
3. Because H2a/H2b are alternative success endpoints, control family-wise error
with Bonferroni-adjusted 97.5 percent confidence intervals or a frozen
equivalent procedure. The successful endpoint needs point gain at least 25
percent and its adjusted interval must exclude zero improvement.
4. Report all primary/secondary/stratified results, exclusions and missingness.
No optional stopping; capture count is frozen by power/minimum before the
confirmatory set opens.
5. Sensitivity analyses vary the frozen association/lost-track thresholds only
as clearly labeled exploratory analysis after the primary result.
If only one endpoint passes, state exactly which one. Do not summarize it as
“25 percent more accurate” when the passing endpoint was lost-track rate.
## 11. Execution sequence
### Gate A: software before participants/live capture
1. Run the NLOS Rust unit/property/golden tests and workspace gate.
2. Run Swift core tests and macOS iOS simulator build where available.
3. Run mobile web tests, typecheck, lint and web export.
4. Run harness tests/security/brain/flywheel/manifest/package gates.
5. Run secret/raw-data/dependency/license scans and close confirmed high/critical
findings.
6. Exercise deterministic `SYNTHETIC` replay, stale/disconnect/oversize/replay
rejection and empty fixture. Label all outputs software/L0.
Convenience check:
```bash
cd harness/ruview
node bin/cli.js nlos verify --repo ../.. --run-builds
```
This reports only the available build/discovery subset, with explicit skips. It
cannot pass full Gate A, Gate B, or a release gate by itself; steps 1–6 remain
required.
### Gate B: upstream live reproduction
1. Freeze/sign protocol, privacy approval, identities, software/hardware table,
randomization and analysis container.
2. Inspect geometry and direct-line-of-sight exclusion.
3. Calibrate optical/RF/ground truth and run negative control.
4. Execute live primary-stratum sequences with LiDAR-only online output and
ground-truth labels sealed.
5. Verify accepted end-to-end update rate >=27 Hz and report tracking/error/
empty-region diagnostics.
6. If Gate B fails, stop confirmatory fusion interpretation; fix path under a
new protocol version.
### Gate C: paired live fusion
1. Freeze the CSI likelihood, fusion weights, arm fan-out and host assignment.
2. Execute at least the powered minimum of paired randomized live sequences.
3. Open ground truth once capture/artifacts/exclusions are immutable.
4. Run the committed analysis, bootstrap and stratified report.
5. Retain fusion only if update-rate, 25 percent endpoint, provenance,
privacy/security and guardrail review pass.
### Gate D: Apple/iOS claim
Native/web software may ship as external-track clients after Gate A. A claim that
built-in Apple LiDAR performs transient NLOS needs a separate adapter ADR,
documented public histogram API, physical-device capture, privacy/App Store
review and fresh Gates B/C. ARKit scene depth alone cannot enter Gate B.
## 12. Required artifacts
| Artifact | Contains | Excludes |
|---|---|---|
| Frozen protocol | signed/versioned text, hypotheses, thresholds, seeds | post-result edits |
| Capture manifest | content digests, identities, configurations, timing, strata, exclusions | raw credentials/private keys |
| Raw store | encrypted transients/CSI/ground truth under access/retention policy | Git/package/harness inclusion |
| Split manifest | grouped calibration/pilot/confirmatory IDs and hash | participant identity in public artifact |
| Analysis reproducer | locked dependencies, script, seeds, exact tables/plots | manual spreadsheet-only results |
| Security/privacy record | approvals, threat verification, retention/deletion | blanket “camera-free is safe” claim |
| Acceptance JSON | bounded aggregate fields and SHA-256 references | raw trajectories/sensor data/tokens |
| Witness report | reviewer checks and exact evidence level | higher-level field/production implication |
## 13. Acceptance JSON
The repository-contained record consumed by `ruview nlos verify` has this exact
top-level and per-arm key set. Unknown fields are rejected; values below are
placeholders, not results:
```json
{
"schema": "ruview.nlos.acceptance.v1",
"source": "LIVE_HARDWARE",
"claim_tag": "MEASURED",
"evidence_level": "L2",
"ground_truth": "EXTERNAL",
"protocol_frozen_before_capture": true,
"witness_reviewed": true,
"privacy_review_passed": true,
"security_review_passed": true,
"los_exclusion_verified": true,
"independent_csi_verified": true,
"sensor_model": "<actual-enrolled-external-ST-VL53L8-series-model>",
"transient_kind": "COMPACT_NORMALIZED_HISTOGRAM",
"sensor_configured_max_hz": "<frozen-number>",
"upstream_commit": "<full-40-hex-sha1>",
"protocol_sha256": "<nonzero-64-hex>",
"capture_manifest_sha256": "<nonzero-64-hex>",
"sensor_identity_sha256": "<nonzero-64-hex-scoped-enrollment-reference>",
"calibration_sha256": "<nonzero-64-hex>",
"firmware_sha256": "<nonzero-64-hex>",
"analysis_sha256": "<nonzero-64-hex>",
"endpoint_pairing_sha256": "<nonzero-64-hex-shared-position/lost-track-evaluable-mask-manifest>",
"sensor_configuration_sha256": "<nonzero-64-hex>",
"witness_report_sha256": "<nonzero-64-hex>",
"privacy_review_sha256": "<nonzero-64-hex>",
"security_review_sha256": "<nonzero-64-hex>",
"guardrail_report_sha256": "<nonzero-64-hex>",
"csi_capture_manifest_sha256": "<nonzero-64-hex>",
"csi_sensor_identity_sha256": "<nonzero-64-hex-scoped-enrollment-reference>",
"csi_calibration_sha256": "<nonzero-64-hex>",
"synthetic_frames": 0,
"replay_frames": 0,
"paired_sequences": "<integer-at-least-100>",
"csi_source_count": "<positive-integer>",
"offered_optical_frame_count": "<positive-integer>",
"lidar_only": {
"sequence_count": "<equals-paired-sequences>",
"accepted_sensor_frame_count": "<integer>",
"accepted_update_count": "<integer>",
"evaluable_duration_s": "<number>",
"update_hz": "<accepted-update-count/evaluable-duration>",
"position_error_sum_m": "<sum-of-evaluable-sequence-means>",
"position_error_sample_count": "<equals-paired-sequences-after-frozen-missing-output-penalty>",
"position_error_m": "<sum/sample-count>",
"lost_track_duration_s": "<number>",
"evaluable_track_duration_s": "<number>",
"lost_track_rate": "<lost/evaluable-duration>"
},
"csi_only": {
"sequence_count": "<equals-paired-sequences>",
"accepted_sensor_frame_count": "<integer>",
"accepted_update_count": "<integer>",
"evaluable_duration_s": "<number>",
"update_hz": "<number>",
"lost_track_duration_s": "<number>",
"evaluable_track_duration_s": "<number>",
"lost_track_rate": "<number>"
},
"fused": {
"sequence_count": "<equals-paired-sequences>",
"accepted_sensor_frame_count": "<integer>",
"accepted_update_count": "<integer>",
"evaluable_duration_s": "<number>",
"update_hz": "<accepted-update-count/evaluable-duration>",
"position_error_sum_m": "<sum-of-evaluable-sequence-means>",
"position_error_sample_count": "<equals-paired-sequences-after-frozen-missing-output-penalty>",
"position_error_m": "<sum/sample-count>",
"lost_track_duration_s": "<number>",
"evaluable_track_duration_s": "<number>",
"lost_track_rate": "<lost/evaluable-duration>"
},
"confidence": {
"bootstrap_resamples": "<integer-at-least-10000>",
"bootstrap_seed_list_sha256": "<nonzero-64-hex>",
"familywise_confidence_level": 0.975,
"position_error_reduction_lower": "<number>",
"position_error_reduction_upper": "<number>",
"lost_track_reduction_lower": "<number>",
"lost_track_reduction_upper": "<number>"
},
"guardrails": {
"empty_false_track_rate": "<number>",
"empty_false_track_rate_max": "<frozen-number>",
"fused_p95_latency_ms": "<number>",
"fused_p95_latency_max_ms": "<frozen-number>",
"fused_frame_loss_rate": "<(offered-fused-accepted)/offered>",
"fused_frame_loss_rate_max": "<frozen-number>",
"exclusion_fraction": "<number>",
"exclusion_fraction_max": "<frozen-number>",
"fused_update_rate_ratio_min": "<frozen-number>",
"nonwinning_position_error_regression_max": "<frozen-number>",
"nonwinning_lost_track_regression_max": "<frozen-number>"
}
}
```
Run the hard arithmetic/provenance gate only after witness review:
```bash
cd harness/ruview
node bin/cli.js nlos verify --repo ../.. \
--evidence-file evidence/nlos/acceptance.json \
--require-research-pass
```
Do not commit a fabricated placeholder file merely to exercise this command;
unit tests already cover synthetic fixtures.
## 14. Reporting language
Permitted after software only:
> `SYNTHETIC`: the Rust, Swift, and web contracts pass deterministic replay and
> boundary tests. No live NLOS accuracy or iPhone hardware claim was evaluated.
Permitted after a passing L2 capture:
> `MEASURED` on capture `<digest>` under protocol `ruview-consumer-nlos-v1`, the
> external `<sensor_model>/<transient_kind>` LiDAR-only arm emitted `<rate>` accepted updates/s. Fusion
> reduced `<exact endpoint>` by `<gain>` relative to LiDAR-only over `<n>` paired
> controlled sequences. This is L2 controlled-laboratory evidence for the named
> target/geometry, not built-in iPhone, diffuse-human, field, or safety evidence.
Forbidden:
1. “iPhone sees around corners” from ARKit depth, native/web build or external
track display;
2. “30 fps” from requested sensor/display rate rather than accepted end-to-end
live tracks;
3. “25 percent more accurate” when only lost-track rate passed;
4. “hardware validated” without sensor/firmware/capture/ground-truth witness;
5. pooled human/general-scene claims from the retroreflective target stratum; or
6. identity, intent, through-wall optical, collision-avoidance or safety claims.
## 15. Stop and rollback criteria
Stop capture immediately for consent/indicator failure, eye/electrical hazard,
credential compromise, raw-data leak, unapproved person entry, cross-tenant
join, provenance ambiguity, calibration/clock failure or confirmed high/critical
security issue. Quarantine the affected capture and never repair its label.
Stop promotion when H1 fails, neither H2 endpoint passes, fusion guardrails are
materially worse, exclusions exceed the frozen tolerance, confidence analysis
cannot be reproduced, or any required artifact is absent. Roll back to
LiDAR-only, explicit replay, or unavailable. Issue a new protocol version before
recapture; do not tune against and reuse the opened confirmatory set.
## 16. Primary sources
1. Somasundaram et al., [Nature paper](https://doi.org/10.1038/s41586-026-10502-x).
2. [Author manuscript and MAS/particle-filter methods](https://arxiv.org/html/2605.17865v1).
3. [MIT project and reported real-time demonstration](https://cornar.media.mit.edu/).
4. [Upstream implementation and hardware procedure](https://github.com/sidsoma/consumer-nlos).
5. STMicroelectronics, [VL53L8CH histogram interface](https://www.st.com/en/imaging-and-photonics-solutions/vl53l8ch.html).
6. STMicroelectronics, [P-NUCLEO-53L8A1](https://www.st.com/en/evaluation-tools/p-nucleo-53l8a1.html).

159
docs/schemas/ruview-nlos-track-v1.schema.json Normal file
View File

@@ -0,0 +1,159 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://ruview.ai/schemas/ruview-nlos-track-v1.schema.json",
"title": "RuView NLOS track envelope v1",
"type": "object",
"additionalProperties": false,
"required": [
"schema", "sessionId", "sequence", "capturedAtUnixMs", "expiresAtUnixMs",
"source", "evidenceLevel", "algorithmVersion", "calibrationHash", "provenance", "tracks"
],
"properties": {
"schema": { "const": "ruview.nlos.track.v1" },
"sessionId": { "$ref": "#/$defs/label" },
"sequence": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
"capturedAtUnixMs": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
"expiresAtUnixMs": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
"source": { "enum": ["live", "replay", "synthetic"] },
"evidenceLevel": { "enum": ["l0_synthetic", "l1_measured", "l2_calibrated"] },
"algorithmVersion": { "$ref": "#/$defs/label" },
"calibrationHash": { "$ref": "#/$defs/sha256" },
"provenance": { "$ref": "#/$defs/provenance" },
"tracks": {
"type": "array",
"maxItems": 16,
"items": { "$ref": "#/$defs/track" }
}
},
"allOf": [
{
"if": { "properties": { "source": { "const": "synthetic" } }, "required": ["source"] },
"then": {
"properties": {
"evidenceLevel": { "const": "l0_synthetic" },
"calibrationHash": { "const": "0000000000000000000000000000000000000000000000000000000000000000" },
"provenance": {
"properties": {
"transientKind": { "const": "replay" },
"transport": { "const": "replay" }
}
}
}
}
},
{
"if": { "properties": { "source": { "const": "live" } }, "required": ["source"] },
"then": {
"properties": {
"evidenceLevel": { "enum": ["l1_measured", "l2_calibrated"] },
"provenance": {
"properties": {
"transientKind": { "enum": ["raw_histogram", "compact_normalized_histogram"] },
"histogramPreserved": { "const": true },
"transport": { "enum": ["usb_serial", "ruview_server"] }
}
}
}
}
},
{
"if": { "properties": { "source": { "const": "replay" } }, "required": ["source"] },
"then": {
"properties": {
"provenance": {
"properties": {
"transientKind": { "const": "replay" },
"histogramPreserved": { "const": true },
"transport": { "const": "replay" }
}
}
}
}
},
{
"if": {
"properties": { "evidenceLevel": { "const": "l2_calibrated" } },
"required": ["evidenceLevel"]
},
"then": {
"properties": {
"calibrationHash": {
"not": { "const": "0000000000000000000000000000000000000000000000000000000000000000" }
}
}
}
}
],
"$defs": {
"label": { "type": "string", "minLength": 1, "maxLength": 64, "pattern": "^[A-Za-z0-9_.:-]+$" },
"sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
"unit": { "type": "number", "minimum": 0, "maximum": 1 },
"vector": {
"type": "object",
"additionalProperties": false,
"required": ["x", "y", "z"],
"properties": { "x": { "type": "number" }, "y": { "type": "number" }, "z": { "type": "number" } }
},
"positionVector": {
"type": "object", "additionalProperties": false, "required": ["x", "y", "z"],
"properties": {
"x": { "type": "number", "minimum": -100, "maximum": 100 },
"y": { "type": "number", "minimum": -100, "maximum": 100 },
"z": { "type": "number", "minimum": -100, "maximum": 100 }
}
},
"velocityVector": {
"type": "object", "additionalProperties": false, "required": ["x", "y", "z"],
"properties": {
"x": { "type": "number", "minimum": -20, "maximum": 20 },
"y": { "type": "number", "minimum": -20, "maximum": 20 },
"z": { "type": "number", "minimum": -20, "maximum": 20 }
}
},
"covarianceVector": {
"type": "object", "additionalProperties": false, "required": ["x", "y", "z"],
"properties": {
"x": { "type": "number", "minimum": 0, "maximum": 10 },
"y": { "type": "number", "minimum": 0, "maximum": 10 },
"z": { "type": "number", "minimum": 0, "maximum": 10 }
}
},
"provenance": {
"type": "object",
"additionalProperties": false,
"required": ["sensorId", "sensorModel", "firmwareVersion", "transientKind", "histogramPreserved", "transport"],
"properties": {
"sensorId": { "$ref": "#/$defs/label" },
"sensorModel": { "$ref": "#/$defs/label" },
"firmwareVersion": { "$ref": "#/$defs/label" },
"transientKind": { "enum": ["raw_histogram", "compact_normalized_histogram", "depth_only", "replay"] },
"histogramPreserved": { "type": "boolean" },
"transport": { "enum": ["usb_serial", "ruview_server", "replay"] }
}
},
"track": {
"type": "object",
"additionalProperties": false,
"required": [
"trackId", "state", "positionM", "velocityMps", "covarianceDiagonalM2",
"confidence", "posteriorEntropy", "signalQuality", "modalityContributions"
],
"properties": {
"trackId": { "$ref": "#/$defs/label" },
"state": { "enum": ["tracking", "degraded", "unknown"] },
"positionM": { "$ref": "#/$defs/positionVector" },
"velocityMps": { "$ref": "#/$defs/velocityVector" },
"covarianceDiagonalM2": { "$ref": "#/$defs/covarianceVector" },
"confidence": { "$ref": "#/$defs/unit" },
"posteriorEntropy": { "type": "number", "minimum": 0 },
"signalQuality": { "$ref": "#/$defs/unit" },
"modalityContributions": {
"type": "object",
"additionalProperties": false,
"required": ["lidar", "csi"],
"properties": { "lidar": { "$ref": "#/$defs/unit" }, "csi": { "$ref": "#/$defs/unit" } }
}
}
}
}
}

132
docs/schemas/ruview-nlos-transient-v1.schema.json Normal file
View File

@@ -0,0 +1,132 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://ruview.ai/schemas/ruview-nlos-transient-v1.schema.json",
"title": "RuView raw optical transient frame v1",
"type": "object",
"additionalProperties": false,
"required": [
"schema", "sessionId", "sequence", "capturedAtUnixMs", "monotonicNs", "source",
"evidenceLevel", "binWidthPs", "startBin", "sensorPose", "calibrationHash", "provenance", "zones"
],
"properties": {
"schema": { "const": "ruview.nlos.transient.v1" },
"sessionId": { "type": "string", "minLength": 1, "maxLength": 64, "pattern": "^[A-Za-z0-9_.:-]+$" },
"sequence": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
"capturedAtUnixMs": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
"monotonicNs": { "type": "integer", "minimum": 0 },
"source": { "enum": ["live", "replay", "synthetic"] },
"evidenceLevel": { "enum": ["l0_synthetic", "l1_measured", "l2_calibrated"] },
"binWidthPs": { "type": "number", "minimum": 1, "maximum": 10000 },
"startBin": { "type": "integer", "minimum": 0, "maximum": 65535 },
"sensorPose": {
"type": "object",
"additionalProperties": false,
"required": ["translationM", "quaternionXyzw"],
"properties": {
"translationM": { "$ref": "#/$defs/vector" },
"quaternionXyzw": {
"type": "array", "minItems": 4, "maxItems": 4,
"items": { "type": "number", "minimum": -1, "maximum": 1 }
}
}
},
"calibrationHash": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
"provenance": {
"type": "object",
"additionalProperties": false,
"required": ["sensorId", "sensorModel", "firmwareVersion", "transientKind", "histogramPreserved", "transport"],
"properties": {
"sensorId": { "$ref": "#/$defs/label" },
"sensorModel": { "$ref": "#/$defs/label" },
"firmwareVersion": { "$ref": "#/$defs/label" },
"transientKind": { "enum": ["raw_histogram", "compact_normalized_histogram", "depth_only", "replay"] },
"histogramPreserved": { "type": "boolean" },
"transport": { "enum": ["usb_serial", "ruview_server", "replay"] }
}
},
"zones": {
"type": "array", "minItems": 1, "maxItems": 64,
"items": {
"type": "object", "additionalProperties": false,
"required": ["zoneId", "wallPointM", "distanceM", "ambient", "histogram"],
"properties": {
"zoneId": { "type": "integer", "minimum": 0, "maximum": 63 },
"wallPointM": { "$ref": "#/$defs/vector" },
"distanceM": { "type": "number", "minimum": 0.01, "maximum": 10 },
"ambient": { "type": "integer", "minimum": 0, "maximum": 4294967295 },
"histogram": {
"type": "array", "minItems": 8, "maxItems": 128,
"items": { "type": "integer", "minimum": 0, "maximum": 65535 }
}
}
}
}
},
"allOf": [
{
"if": { "properties": { "source": { "const": "synthetic" } }, "required": ["source"] },
"then": {
"properties": {
"evidenceLevel": { "const": "l0_synthetic" },
"calibrationHash": { "const": "0000000000000000000000000000000000000000000000000000000000000000" },
"provenance": {
"properties": {
"transientKind": { "const": "replay" },
"transport": { "const": "replay" }
}
}
}
}
},
{
"if": { "properties": { "source": { "const": "live" } }, "required": ["source"] },
"then": {
"properties": {
"evidenceLevel": { "enum": ["l1_measured", "l2_calibrated"] },
"provenance": {
"properties": {
"transientKind": { "enum": ["raw_histogram", "compact_normalized_histogram"] },
"histogramPreserved": { "const": true },
"transport": { "enum": ["usb_serial", "ruview_server"] }
}
}
}
}
},
{
"if": { "properties": { "source": { "const": "replay" } }, "required": ["source"] },
"then": {
"properties": {
"provenance": {
"properties": {
"transientKind": { "const": "replay" },
"histogramPreserved": { "const": true },
"transport": { "const": "replay" }
}
}
}
}
},
{
"if": {
"properties": { "evidenceLevel": { "const": "l2_calibrated" } },
"required": ["evidenceLevel"]
},
"then": {
"properties": {
"calibrationHash": {
"not": { "const": "0000000000000000000000000000000000000000000000000000000000000000" }
}
}
}
}
],
"$defs": {
"label": { "type": "string", "minLength": 1, "maxLength": 64, "pattern": "^[A-Za-z0-9_.:-]+$" },
"vector": {
"type": "object", "additionalProperties": false,
"required": ["x", "y", "z"],
"properties": { "x": { "type": "number" }, "y": { "type": "number" }, "z": { "type": "number" } }
}
}
}

320
docs/security/consumer-nlos-threat-model.md Normal file
View File

@@ -0,0 +1,320 @@
# Consumer NLOS threat model
**Status:** Required design and release gate for ADR-328 through ADR-331
**Last reviewed:** 2026-08-22
**Scope:** enrolled histogram-capable ST transient acquisition with recorded
board/silicon/firmware/API identity, MAS tracking/reconstruction, CSI
fusion, RuVector/RuField/WorldGraph state, native iOS, web iOS, evidence and
contributor harness
**Security posture:** Research capability, local-first, identity-free,
fail-closed, no direct actuation
## 1. Executive security decision
Around-the-corner tracking changes the privacy boundary of a space: a person can
be observed without being in direct view of the operator or device. The output
is sparse probability/geometry rather than a photograph, but location,
trajectory, occupancy and joined RF evidence remain sensitive. “Camera-free”
must never be used as “privacy-free.”
The first RuView deployment is a controlled, consented research experiment with
an external, upstream-compatible ST histogram sensor over local USB. Raw transients, CSI and
ground truth remain local under an approved retention plan. Live APIs expose
only bounded, expiring, session-scoped hypotheses. No NLOS output directly
controls an actuator or certifies that a hidden region is safe.
This PR is software stage G0 only. Its local USB source label (`st-local`) is
not cryptographic sensor enrollment, and `ruview.nlos.track.v1` has no tenant,
workspace or world-frame field. The controls below that depend on those
bindings are mandatory live-promotion work; their documentation does not enable
L2 or make the current scaffold physically authenticated.
Release is blocked by a confirmed high or critical finding, missing consent,
unauthenticated sensor/transport, ambiguous live/replay provenance, unbounded
parser, cross-tenant/world-frame join, or absent stale/replay protection.
## 2. System and trust boundaries
```mermaid
flowchart TD
A["Physical scene + ground truth"] --> B["ST sensor + firmware"]
B --> C["USB acquisition + transient parser"]
D["Authenticated CSI sensors"] --> E["Calibration + temporal join"]
C --> F["MAS likelihood + tracker"]
F --> E
E --> G["RuField / RuVector / WorldGraph"]
G --> H["Authenticated API + short-lived ticket"]
H --> I["Native and web iOS clients"]
```
Trust boundaries are crossed at every arrow. Physical proximity does not imply
logical trust. USB is a device/input boundary; the ST firmware and upstream
Python are supply-chain inputs; CSI is an independent, potentially malicious
modality; memory is tenant/purpose separated; API clients are untrusted; and
the UI is not evidence authority.
The contributor harness reads repository metadata and optional bounded evidence
JSON. It does not receive raw captures, credentials, sensing authority, or a
runtime role.
## 3. Assets
| Asset | Security/privacy need | Default handling |
|---|---|---|
| Raw per-zone photon histograms | integrity, confidentiality, purpose/retention | local encrypted research store; never Git/logs |
| Raw CSI/CIR and RF calibration | confidentiality, tenant isolation, provenance | edge/local; no default cloud export |
| Ground-truth video/tags/trajectories | highest participant privacy and synchronization integrity | separated encrypted store, least access, scheduled deletion |
| Sensor identity and firmware/config digest | authenticity and chain of custody | certificate/key reference plus content digest; no private key export |
| Wall/background/extrinsic calibration | integrity, freshness, deployment binding | signed/content-addressed, expiring, invalidatable |
| Track posterior/covariance/history | confidentiality, freshness, non-reidentification | session ID, TTL, bounded history, purpose-limited |
| WorldGraph/RuVector state | tenant isolation, deletion, no identity escalation | tenant/workspace namespace, TTL, audited access |
| API credentials/tickets | confidentiality, replay resistance, least authority | native pairing secret in ThisDeviceOnly Keychain; web uses short-lived scoped ticket; never URL/log/local storage bearer |
| Capture/evidence manifest | integrity, reproducibility, nonrepudiation | canonical digest and witness record; no embedded raw data |
| Model/canonical response | integrity and license/provenance | pinned digest, reviewed source, immutable artifact |
| Availability/quality state | integrity and fail-closed semantics | explicit unavailable/degraded/unknown/expired states |
## 4. Adversaries and misuse
1. **Unauthorized operator** deploys or leaves sensing active without notice or
beyond the approved room/time/purpose.
2. **Nearby attacker** injects RF, optical, physical-motion or relay-surface
changes to produce, hide or move a track.
3. **Compromised sensor/firmware** fabricates histograms, sequence/timestamps,
calibration identity or firmware version.
4. **Compromised CSI node** contributes an extreme likelihood or replays a frame
to create false persistence.
5. **Malicious client** sends parser bombs, requests another tenant/session,
reuses a web ticket, scrapes track history or alters provenance labels.
6. **Supply-chain attacker** compromises upstream Git/Python/firmware, Cargo/npm/
Swift packages, build tools or model artifacts.
7. **Insider/researcher** copies raw captures, joins session tracks across time,
bypasses deletion or selectively reports favorable trials.
8. **Curious contributor agent/harness** attempts to read secrets/raw data,
execute untrusted evidence, mutate code/hardware or promote a claim.
9. **Accidental failure** includes clock/coordinate mismatch, stale calibration,
overflow, NaN, queue buildup, app suspension and simulator/replay confusion.
Out of scope does not mean acceptable: nation-state hardware implants and
physical destruction are not fully mitigated by this software, so high-assurance
or safety deployments require a separate hardware/security case.
## 5. Security invariants
1. `LIVE_HARDWARE`, `REPLAY`, and `SYNTHETIC` are mutually exclusive by type.
Unknown is never live.
2. A frame/track is usable only when schema, identity, session, sequence,
freshness, calibration, coordinate frame, numeric bounds and size pass.
3. Fused output requires two independently valid modalities within the frozen
time/space join bounds. Missing or rejected input cannot be labeled fused.
4. RuVector/WorldGraph similarity and persistence never upgrade acquisition
provenance, infer identity, or bypass freshness.
5. Raw sensor and ground-truth data is local/minimized by default and never
enters Git, harness manifests, package tarballs, logs or client telemetry.
6. Authorization is tenant/workspace/session/purpose scoped. A read scope is not
actuation authority.
7. Stale, contradictory, out-of-distribution or low-quality evidence produces
`unknown`/degraded, not a confident fallback.
8. No parser allocates from unchecked zone × bin × object × history dimensions.
9. No client rendering rate, simulator build or replay can satisfy the live
research gate.
10. A NLOS hypothesis has no direct actuator callback.
## 6. Threat register
Ratings describe the uncontrolled design. “Required control” is a release gate;
it is not a claim that every future implementation is automatically safe.
| ID | Threat | Initial risk | Required controls | Verification / residual risk |
|---|---|---:|---|---|
| T01 | Covert or overbroad sensing beyond direct view | Critical | explicit approved purpose/space/time, participant notice/consent, persistent indicator, pause/stop, background off, audit and deletion | privacy review plus physical walkthrough; residual High for misuse by an authorized operator, so no unconsented deployment |
| T02 | Unauthenticated sensor or CSI impersonation | Critical | ADR-305 identity, enrollment, tenant/session binding, TLS where networked, digest/certificate on frame/calibration | forged/unknown identity tests; residual Medium for stolen keys, mitigated by rotation/revocation |
| T03 | Replay/duplicate/future timestamp creates false track | High | monotonic sequence, nonce/session, bounded skew/TTL, duplicate cache, clock uncertainty, stale clear | exact/changed replay, wrap and clock-jump tests; residual Low/Medium under clock loss → unknown |
| T04 | Calibration/background substitution or poisoning | High | signed/content-addressed calibration bound to sensor/config/room, expiry/OOD, controlled empty capture, immutable audit | mismatched/expired/poisoned calibration tests; residual Medium for slow physical drift |
| T05 | Direct line of sight contaminates “hidden” result | High | opaque geometry check, independent scene inspection, registered camera/ground-truth exclusion, capture manifest | randomized occluder/negative trials; residual Medium for unnoticed reflections/view gaps |
| T06 | Optical/RF adversarial injection or physical spoof | High | per-modality quality/OOD, bounded influence, optical-only/CSI-only ablation, multi-view/temporal consistency, unknown on contradiction | extreme likelihood and conflicting-modality tests; residual High in adversarial environments, so no safety claim |
| T07 | Coordinate or time-frame mismatch fuses different targets | High | typed units/frame IDs, calibrated transform+uncertainty, bounded pairing skew, no cross-frame fallback | incompatible frame/unit/skew test matrix; residual Low after fail-closed controls |
| T08 | Parser overflow, NaN/Inf, oversized histogram/track | High | pre-allocation dimension/product bounds, finite checks, fixed max message/evidence size, fuzz/property tests, bounded tails | sanitizers/fuzz/boundary corpus; residual Low/Medium for third-party decoders |
| T09 | Queue/memory/CPU exhaustion hides freshness | High | bounded queues/particles/history, newest-frame policy, rate limits, timeouts, backpressure/drop counters | overload/slow-client tests; residual Medium under sustained physical/authorized load |
| T10 | Web ticket/token theft or cross-origin stream | High | ATS/TLS, short-lived one-use scoped ticket, fixed origin, no bearer in URL/log/local storage, CSP, reconnect bounds | expiry/reuse/origin/tenant negative integration tests; residual Medium for compromised client device |
| T11 | Native/web stale UI remains visually live | High | expiry timer independent of incoming messages, clear on suspend/disconnect/decode error, provenance/quality always visible | fake-clock/suspend/disconnect/replay UI tests; residual Low |
| T12 | Cross-tenant or cross-session memory leakage | Critical | typed namespace at ingest/join/store/query, authorization, TTL/deletion, no global person ID | tenant/session isolation tests and access audit; residual Medium for admin/backup controls |
| T13 | Re-identification from trajectory/embedding | High | random session track IDs, no identity training/labels, bounded TTL, purpose limitation, aggregation, access logging | privacy review and deletion tests; residual Medium/High in small populations, so identity use prohibited |
| T14 | Raw data, secrets or positions leak via Git/log/telemetry/package | High | repo incident controls, `.gitignore`, secret/data scans, redaction, synthetic fixtures, no raw telemetry, tarball review | repository-policy, package dry run, log tests; residual Low/Medium for human export |
| T15 | Upstream/firmware/dependency/model compromise | High | pin commits/versions/digests, license/source review, isolated sidecar, dependency audit, signed release/provenance, no auto-flash | SBOM/audit/reproducible hash and firmware review; residual Medium for build toolchain |
| T16 | Malicious evidence JSON/path escape/code execution | High | regular bounded repository-confined JSON, realpath containment, no eval/import, strict schema/digest/arithmetic | traversal/symlink/oversize/malformed tests; residual Low |
| T17 | Selective reporting, leakage or metric gaming | High | preregistration, immutable grouped split, external ground truth, paired arms, all strata/secondary metrics, witness review | independent analysis reproduction; residual Medium for undisclosed captures |
| T18 | Learned RF model copies LiDAR labels and appears independent | High | group splits, test on external ground truth, LiDAR/CSI/fused ablations, teacher quality masks, sealed holdout | leakage/adjoining-frame tests; residual Medium under environmental confounding |
| T19 | Harness/agent mutation gains sensing or release authority | High | MCP/static tools read-only and default-deny; opt-in CLI builds only in a trusted checkout with scrubbed environment/redacted tails; MetaHarness dev-only; human promotion | policy/tool/build-output tests and package manifest; residual Medium because build tools execute repository code |
| T20 | Track drives unsafe actuator/decision | Critical | hypothesis-only API, explicit no actuator, ADR-321/327 independent governed action and deployment safety case | interface/source tests; residual Critical if bypassed, therefore release blocker |
| T21 | Private/undocumented Apple API use | High | public SDK only, `canImport`/availability checks, official-doc review, no hidden entitlement/reverse engineering | Xcode source/entitlement review; residual Low with external-sensor-only claim |
| T22 | Eye/laser or electrical hazard from modified hardware | High | unmodified Class 1 sensor, manufacturer limits, approved power/enclosure, no emitter modification, trained operator | hardware checklist; residual Medium; any optical modification requires new safety review |
## 7. Input-boundary controls
### 7.1 Transient frame
Reject before allocating or mutating state when any of these is true:
1. schema/version is unknown;
2. zone/bin dimensions are zero, exceed configured maxima, or their product
overflows;
3. bin width, timestamps, pose, intrinsics, wall points, ambient level or
normalized counts are non-finite/out of physical configured range;
4. sensor/session identity is absent, revoked or mismatched;
5. sequence is duplicate/regressing or time is stale/future beyond uncertainty;
6. firmware/config/calibration digest differs from the session manifest;
7. transform is non-invertible, units differ, or coordinate frame is unknown;
8. calibration is not `VALID`; or
9. the source-provenance transition is illegal.
### 7.2 Track/API message
`ruview.nlos.track.v1` validation happens before store/render. Bound message
bytes, track count, history length, covariance/particle summary, strings and
metadata. Position/velocity/covariance are finite and covariance is valid under
the chosen representation. The client owns an independent expiry timer.
Tenant/workspace/world-frame, capture-certificate and audience/origin checks are
the required L2 contract. Current v1 validates session/schema/sequence/freshness,
bounded exact shape, calibration hash and generic transient provenance only; it
must remain G0 until the missing authorization/frame bindings are versioned end
to end.
### 7.3 Evidence record
The harness permits at most 1 MiB, regular JSON inside the canonical repository
root. `realpath` containment rejects `..` and symlink escapes. JSON is parsed as
data, never imported or executed. Research pass requires exact live provenance,
external ground truth, frozen protocol, digests, zero synthetic/replay frames,
minimum sample size, valid rates/errors and the frozen arithmetic.
## 8. Authentication, authorization and key handling
1. Before live promotion, sensor enrollment creates a non-secret, scoped sensor identity reference and protects
the private key outside captures/manifests. Revocation invalidates future
frames and live capability certificates.
2. Service authorization is least-privilege: read tracks for one tenant,
workspace, session and purpose. It grants no calibration write, firmware
flash, memory export, evidence promotion or actuation.
3. The current G0 browser exchanges an authenticated session for a 30-second,
one-use WebSocket ticket bound to the server session; the client requires a
server-session acknowledgement, pins the returned WSS URL to the configured
same authority, and the server uses exact CORS. It does **not** yet bind
tenant/workspace/audience/origin claims in the ticket. Those bindings and
their negative tests are mandatory before L2. Tickets are not stored
persistently. Native pairing tokens are scoped, revocable, validated before
use, and stored as `WhenUnlockedThisDeviceOnly` Keychain data.
4. TLS certificate verification is on in release builds. Debug/local exceptions
are explicit, non-exportable release configuration.
5. Secrets are never command-line arguments when avoidable and never printed in
error tails. Harness tools accept no raw token/API key field.
6. Research-store encryption keys are purpose/experiment scoped, kept outside
captures, manifests and backups, access-audited, rotated on compromise, and
destroyed at retention expiry. Backup retention cannot silently defeat
deletion or crypto-erasure evidence.
## 9. Privacy impact and data lifecycle
| Stage | Minimization | Access/retention | Deletion proof |
|---|---|---|---|
| Capture | record only approved zones/modalities/window; no audio; ground truth separated | named researchers, encrypted local store, frozen short schedule | manifest tombstone plus storage audit |
| Calibration | no person present; bind to room/sensor/config | operators and pipeline; expire on change | invalidation record and artifact deletion |
| Inference | process raw at edge; emit bounded position/covariance/provenance | live authorized clients only | TTL and session teardown tests |
| RuVector/WorldGraph | no stable identity; session scope; minimal embeddings/relations | tenant/purpose-scoped queries | namespace purge and index compaction evidence |
| Evidence | hashes and aggregate metrics, no raw person data | reviewers | retain per research governance without reconstructing raw capture |
| Logs/telemetry | counters, digests truncated where needed, redacted errors | operators/security | rotation verification |
Consent/notice must explain around-corner sensing in plain language. An optical
sensor without RGB imagery can still infer a hidden person's location and
movement. Withdrawal/deletion limitations for already aggregated, non-personal
published metrics are documented before participation.
## 10. Availability and safe degradation
Every dependency has an explicit safe state:
| Failure | Required behavior |
|---|---|
| sensor disconnect or malformed frame | stop live optical updates; emit unavailable/unknown |
| calibration/OOD failure | invalidate likelihood; request recalibration; no cached live fallback |
| CSI missing | optical-only label if optical remains valid; never fused |
| optical missing | RF-only/coarse label if separately permitted; never NLOS/fused |
| clock/coordinate disagreement | no join; record diagnostic counter |
| tracker numeric collapse | reset bounded filter and emit unknown during reacquisition |
| API auth/ticket expiry | disconnect and clear live state |
| native suspension/web background | expire display independently of server |
| RuVector/WorldGraph unavailable | current local track may continue without persistence; no memory fabrication |
| harness/MetaHarness unavailable | no runtime impact; run direct crate/app tests |
## 11. Security verification plan
Minimum software evidence is staged: current G0 must pass the checks applicable
to its implemented schema; every listed check is mandatory before live L2
promotion. A test cannot stand in for a field the protocol does not yet carry.
1. Rust unit/property/fuzz or boundary tests for transient/track decode,
dimension-product overflow, numeric extremes, calibration/provenance state,
temporal/coordinate join and bounded filter behavior.
2. Swift and TypeScript cross-language golden vectors plus malformed/stale/
oversized/provenance tests.
3. Authentication integration tests for invalid/expired/reused/cross-origin/
cross-tenant tickets and disconnect state.
4. Empty/rejected modality, conflicting modality, replay, clock jump, sequence
wrap, overload and slow-client tests.
5. Repository secret/data incident scan, Cargo/npm/Swift dependency review,
license/SBOM review and `npm pack --dry-run` inspection.
6. Harness policy, evidence traversal/oversize/synthetic rejection, manifest,
brain/flywheel replay and full legacy tests.
7. Manual review for private Apple API/entitlement use, raw data/credentials in
fixtures/logs, direct actuator callbacks and overclaimed metrics.
### Closed dependency exception record `NLOS-SEC-EX-001`
| Field | Record |
|---|---|
| Status | **CLOSED BY REMEDIATION**; no risk exception was approved or consumed |
| Original finding | high-severity `image-size@1.2.1` advisories [GHSA-w3rx-r6r6-pgpr](https://github.com/advisories/GHSA-w3rx-r6r6-pgpr) and [GHSA-5p2g-fcmc-qvqq](https://github.com/advisories/GHSA-5p2g-fcmc-qvqq) through the prior Metro lock |
| Remediation | aligned the Expo SDK 55 dependency set and regenerated `ui/mobile/package-lock.json`; the resulting 2026-08-22 audit reports zero high and zero critical findings |
| Verification | `npm ci --ignore-scripts` plus `npm audit --json`; local audit digest `b6e5236f2b07dec4f714d60e7c8a5683348464455392d636fc77b9c591bb8dcb` |
| Remaining findings | ten moderate transitive build-tool findings, including `uuid` through `xcode`; tracked normally and not covered by an exception |
| CI policy | fail every high or critical dependency finding; no NLOS allowlist exists |
| Promotion mapping | the former dependency block is removed; all other G0→G1 physical, privacy, authorization, provenance and witness gates remain closed until independently satisfied |
The closed record is retained so reviewers can see why the lockfile changed and
verify that the project remediated rather than silently accepted the finding.
Live L2 evidence additionally needs a physical setup inspection, firmware and
sensor identity witness, direct-line-of-sight exclusion, external ground-truth
clock test, capture access/retention review, and independent reproduction of the
analysis from the immutable manifest.
## 12. Incident response and rollback
On suspected unauthorized sensing, provenance failure, key theft, raw-data leak,
metric manipulation or unsafe downstream use:
1. stop capture and live stream; revoke sensor/client credentials and tickets;
2. freeze bounded logs/manifests without copying unnecessary raw participant
data;
3. disable the NLOS/fusion capability certificate and UI live flag;
4. notify the privacy/security owner and affected participants/organizations as
required;
5. determine affected tenants/sessions/captures/models and delete/quarantine per
policy;
6. patch and add a focused regression test; rerun the complete gate;
7. re-enable only after independent review and, where evidence was affected, a
fresh preregistered capture.
Rollback may return to optical-only, RF-only, replay-only or fully unavailable.
It never changes old provenance or represents replay as a live substitute.
## 13. Residual-risk decision
Even with controls, authorized misuse, physical spoofing, environment shift,
re-identification from trajectories and supply-chain compromise retain material
risk. Therefore the accepted scope is controlled, consented RuView Labs
research and advisory visualization. Public-space surveillance, covert sensing,
biometric identity, through-wall safety guarantees, vehicle collision avoidance,
medical monitoring and autonomous actuation are not approved by these ADRs.
Any expansion requires a new threat model, evidence level, deployment/privacy
review, operational monitoring, incident/rollback plan and accountable owner.