feat(mobile): add gated LiDAR point cloud
6
.github/workflows/consumer-nlos-ci.yml
vendored
@@ -210,6 +210,7 @@ jobs:
|
||||
"under 1.5 seconds",
|
||||
"under 100 milliseconds",
|
||||
"zero runtime dependency",
|
||||
"Three.js",
|
||||
"physical iPhone validation pending",
|
||||
)
|
||||
for phrase in required_adr_contract:
|
||||
@@ -225,6 +226,8 @@ jobs:
|
||||
"nlos-evidence-state",
|
||||
"nlos-synthetic-watermark",
|
||||
"nlos-beta-setup",
|
||||
"nlos-lidar-point-cloud",
|
||||
"nlos-cloud-target-count",
|
||||
)
|
||||
for token in required_flow_contract:
|
||||
assert token in flow, f"missing Maestro UI contract token: {token}"
|
||||
@@ -276,6 +279,7 @@ jobs:
|
||||
"overview-390x844.png",
|
||||
"synthetic-390x844.png",
|
||||
"setup-390x844.png",
|
||||
"point-cloud-390x844.png",
|
||||
)
|
||||
for name in expected_screenshots:
|
||||
path = screenshot_dir / name
|
||||
@@ -286,7 +290,7 @@ jobs:
|
||||
|
||||
print(
|
||||
"Validated ADR 342, Maestro UI contract, five evidence states, "
|
||||
"and three 390x844 PNG baselines"
|
||||
"and four 390x844 PNG baselines"
|
||||
)
|
||||
PY
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
| **Status** | Proposed; software implementation and deterministic review evidence are part of a stacked UI-only pull request, while physical-device validation remains operator-gated |
|
||||
| **Date** | 2026-08-23 |
|
||||
| **Owners** | RuView Labs mobile, iOS, design, accessibility, privacy, security, and research maintainers |
|
||||
| **Scope** | Presentation, responsive layout, evidence-state projection, accessibility semantics, mobile end-to-end review flow, and screenshot baselines for the Expo and native SwiftUI NLOS clients |
|
||||
| **Scope** | Presentation, responsive layout, evidence-state projection, accessible target and LiDAR point-cloud views, mobile end-to-end review flow, and screenshot baselines for the Expo and native SwiftUI NLOS clients |
|
||||
| **Depends on** | ADR-295, ADR-318, ADR-319, ADR-330, ADR-340, ADR-341 |
|
||||
| **Stacked pull request base** | `feat/consumer-nlos-ruview`, the head branch for pull request 1687; retarget to `main` after the dependency merges |
|
||||
| **Implementation boundary** | `ui/mobile` presentation and tests, `ui/ios-nlos/App` presentation, the mobile UI screenshot baselines, and the UI validation portion of `.github/workflows/consumer-nlos-ci.yml` |
|
||||
@@ -73,7 +73,8 @@ design reviewers, privacy and security reviewers, and the release operator.
|
||||
existing validation and freshness boundary.
|
||||
3. Setup, provenance, privacy, and interpretation guidance in a stable reading
|
||||
order.
|
||||
4. Reviewable screenshot baselines for overview, synthetic, and setup states.
|
||||
4. Reviewable screenshot baselines for overview, synthetic, setup, and Three.js
|
||||
point-cloud states.
|
||||
5. A production-browser Playwright end-to-end suite and a mobile Maestro flow
|
||||
that verify navigation, state visibility, synthetic watermarking, setup
|
||||
affordances, and the primary local action.
|
||||
@@ -115,6 +116,11 @@ UNVERIFIED` only when an authenticated live attempt is active. It resolves to
|
||||
setup without collecting credentials or diagnostics.
|
||||
7. Motion is decorative and bounded. Reduced-motion mode removes continuous
|
||||
orbit, sweep, pulse, and parallax effects without removing information.
|
||||
8. The web point-cloud mode uses the already installed Three.js runtime. It
|
||||
renders only deterministic relay samples and target returns derived from
|
||||
tracks that have already passed the display gate. The native surface uses an
|
||||
equivalent SwiftUI Canvas projection. Both identify the view as a rendered
|
||||
reconstruction rather than raw iPhone LiDAR output.
|
||||
|
||||
### Performance budgets
|
||||
|
||||
@@ -129,6 +135,10 @@ UNVERIFIED` only when an authenticated live attempt is active. It resolves to
|
||||
client, sensor subscription, or background timer.
|
||||
5. Screenshot rendering is deterministic: fixed viewport, fixed fixtures,
|
||||
reduced motion, local assets, and no dependency on a live endpoint.
|
||||
6. Point-cloud geometry is bounded to 288 schematic relay points plus 96
|
||||
target-return points for each of at most 16 validated tracks, or 1,824 total
|
||||
points. Browser device pixel ratio is capped at 1.5. GPU resources, animation
|
||||
frames, resize observers, and pointer listeners are released on unmount.
|
||||
|
||||
The timing budgets are targets until a physical iPhone report records device,
|
||||
OS, build commit, build mode, browser, network conditioning, repetition count,
|
||||
@@ -188,6 +198,12 @@ onRender:
|
||||
render no hidden-target geometry
|
||||
render safe recovery guidance for the current state
|
||||
|
||||
if selectedView == pointCloud:
|
||||
build bounded deterministic cloud from the same displayable tracks only
|
||||
render schematic relay points as non-evidence context
|
||||
label the surface as a reconstruction view, not raw LiDAR
|
||||
use Three.js WebGL on web and SwiftUI Canvas on native
|
||||
|
||||
preserve setup, privacy, interpretation, and feedback paths
|
||||
disable decorative motion when reduced motion is requested
|
||||
```
|
||||
@@ -245,7 +261,9 @@ hold.
|
||||
| Component | Responsibility | May change in this decision | Must not change |
|
||||
|---|---|---|---|
|
||||
| Expo NLOS screen and presentation components | Responsive hierarchy, tokens, evidence-state label, accessible controls, instrument composition | Yes | Store semantics, service requests, credential lifecycle |
|
||||
| Expo Three.js point-cloud adapter | Deterministic bounded BufferGeometry, local WebGL rendering, orbit input, reduced motion, resize, and disposal | Yes | Raw sensor access, network fetches, provenance promotion, persistence |
|
||||
| Native SwiftUI shell and presentation helpers | Equivalent hierarchy, tokens, state labels, Dynamic Type, reduced motion, hit targets | Yes | `AppModel`, Apple capability probe, stream guard, Keychain, diagnostics |
|
||||
| Native SwiftUI point-cloud projection | Accessible Canvas rendering from the same displayable track list | Yes | Three.js embedding, WebView, raw ARKit depth, or new entitlement |
|
||||
| Existing Expo store and hooks | Authoritative source, freshness, frame, rejection, and connection inputs | No | Any sensing or transport behavior |
|
||||
| Existing native model and core packages | Authoritative connection, frame, track, diagnostic, and capability inputs | No | Any sensing, validation, persistence, or permission behavior |
|
||||
| Playwright production-browser suite | Black-box responsive navigation, overflow assertions, local state actions, and screenshot reproduction | Yes | Live endpoint use, credentials, or physical-device claims |
|
||||
@@ -275,7 +293,8 @@ pixel identity between React Native Web and SwiftUI is not.
|
||||
```text
|
||||
existing validated domain state
|
||||
-> pure evidence-state projection
|
||||
-> local presentation tree
|
||||
-> displayable track allowlist
|
||||
-> local target view or deterministic point-cloud projection
|
||||
-> pixels and accessibility semantics
|
||||
```
|
||||
|
||||
@@ -311,6 +330,13 @@ screenshots as cross-platform review references, not as proof of native pixel
|
||||
output. Require an Xcode 26 build gate and a named physical-device review before
|
||||
accepting native behavior or performance claims.
|
||||
|
||||
For the point-cloud surface, use Three.js directly in the Expo web build because
|
||||
it is already a locked runtime dependency and can render bounded BufferGeometry
|
||||
without a remote asset. Use SwiftUI Canvas for the native counterpart. Both
|
||||
consume only the existing fail-closed displayable track list. Do not add
|
||||
`expo-gl`, React Three Fiber, a WebView, a raw point-cloud schema, or a new sensor
|
||||
adapter in this UI-only decision.
|
||||
|
||||
## Alternatives with quantified tradeoffs
|
||||
|
||||
Alternatives are scored from 1, poor, to 5, strong. Weighted score is out of
|
||||
@@ -332,6 +358,17 @@ zero runtime dependency. The cost is duplicated token maintenance. The control
|
||||
is a screenshot review plus identical state names and accessibility assertions
|
||||
on both platforms.
|
||||
|
||||
### Point-cloud renderer alternatives
|
||||
|
||||
The same weights apply to the narrower renderer choice.
|
||||
|
||||
| Alternative | Clarity 25% | Epistemic safety 25% | Delivery cost 15% | Native accessibility 20% | Maintainability 15% | Weighted score | Decision |
|
||||
|---|---:|---:|---:|---:|---:|---:|---|
|
||||
| Three.js web plus SwiftUI Canvas native | 5 | 5 | 4 | 4 | 4 | 4.50 | Selected because it satisfies the requested Three.js view while preserving platform-native accessibility and adding zero runtime dependency |
|
||||
| Three.js inside a native WebView | 5 | 4 | 3 | 2 | 4 | 3.70 | Rejected because focus, Dynamic Type, startup, memory, and lifecycle behavior are weaker |
|
||||
| Add `expo-gl` and React Three Fiber for native Expo | 5 | 4 | 2 | 3 | 3 | 3.60 | Rejected because new native dependencies and build risk are disproportionate to a UI-only change |
|
||||
| Static SVG or Canvas cloud on every platform | 3 | 5 | 5 | 4 | 5 | 4.30 | Rejected as the sole implementation because it does not provide the requested interactive Three.js web experience |
|
||||
|
||||
## Accessibility
|
||||
|
||||
1. Every interactive element has a descriptive label, role or trait, and a hit
|
||||
@@ -350,6 +387,10 @@ on both platforms.
|
||||
the accessibility tree.
|
||||
8. The synthetic watermark has a programmatic label in addition to its visual
|
||||
rendering.
|
||||
9. The WebGL canvas is hidden from the accessibility tree and its containing
|
||||
view exposes one concise label with evidence freshness, hypothesis count,
|
||||
and gated target-return count. Orbit gestures are optional and convey no
|
||||
exclusive information.
|
||||
|
||||
## Security and privacy
|
||||
|
||||
@@ -360,9 +401,11 @@ on both platforms.
|
||||
ephemeral credential in memory, while the approved native client uses its
|
||||
existing Keychain boundary. UI code cannot log, render, snapshot, export, or
|
||||
persist the credential.
|
||||
3. Screenshot and E2E fixtures are synthetic and contain no CSI, image, depth,
|
||||
point-cloud, location, person, device identifier, token, endpoint, or private
|
||||
diagnostic data.
|
||||
3. Screenshot and E2E fixtures are synthetic and contain no CSI, image, raw
|
||||
depth, captured point cloud, location, person, device identifier, token,
|
||||
endpoint, or private diagnostic data. The displayed point cloud is generated
|
||||
locally from the deterministic synthetic track fixture and schematic relay
|
||||
geometry.
|
||||
4. Stale, malformed, unauthenticated, replayed, unknown, or contradictory input
|
||||
fails closed through the existing validation layer and the state projection.
|
||||
5. The synthetic watermark remains visible in every synthetic visualization
|
||||
@@ -374,6 +417,9 @@ on both platforms.
|
||||
credentials, diagnostics, or scene context.
|
||||
8. CI uses read-only repository permission and receives no signing, App Store
|
||||
Connect, endpoint, or test-user secret for the UI contract.
|
||||
9. Three.js receives no URL, shader text, texture, model, worker, or external
|
||||
input. It allocates local typed arrays from already gated numeric track data
|
||||
and releases WebGL resources when the view changes or unmounts.
|
||||
|
||||
## Performance budgets
|
||||
|
||||
@@ -400,12 +446,15 @@ claim. A threshold without a completed physical run remains `TARGET`.
|
||||
stale, disconnected, contradiction, and synthetic watermark cases.
|
||||
3. Recompose the Expo NLOS screen at the reference viewport and add the
|
||||
production-browser Playwright suite plus Maestro mobile flow.
|
||||
4. Recompose the native SwiftUI surface using equivalent semantics, Dynamic
|
||||
4. Add the bounded Three.js point-cloud adapter, deterministic generator,
|
||||
accessible fallback, and fail-closed zero-return tests.
|
||||
5. Recompose the native SwiftUI surface using equivalent semantics, Dynamic
|
||||
Type, and reduced-motion behavior.
|
||||
5. Capture deterministic Expo overview, synthetic, and setup baselines.
|
||||
6. Run type, lint, unit, bundle, Swift package, Xcode 26, contract, accessibility,
|
||||
6. Capture deterministic Expo overview, synthetic, setup, and point-cloud
|
||||
baselines.
|
||||
7. Run type, lint, unit, bundle, Swift package, Xcode 26, contract, accessibility,
|
||||
security, and diff-scope review gates.
|
||||
7. Complete the named physical iPhone gate before calling native behavior or
|
||||
8. Complete the named physical iPhone gate before calling native behavior or
|
||||
mobile performance validated.
|
||||
|
||||
### Failure handling
|
||||
@@ -416,7 +465,8 @@ claim. A threshold without a completed physical run remains `TARGET`.
|
||||
2. If a frame becomes stale, remove its geometry on the same committed state
|
||||
transition. Reconnection alone does not restore it.
|
||||
3. If decorative rendering fails, retain plain text state, controls, privacy,
|
||||
and provenance. Decoration cannot block operation.
|
||||
and provenance. Decoration cannot block operation. If WebGL creation or its
|
||||
context fails, identify the static fallback without promoting evidence.
|
||||
4. If the viewport overflows, stack secondary content and remove nonessential
|
||||
decoration before reducing essential text or hit areas.
|
||||
5. If the performance target fails, profile render count, SVG complexity,
|
||||
@@ -436,15 +486,16 @@ claim. A threshold without a completed physical run remains `TARGET`.
|
||||
| Requirement | Automated evidence | Human or physical evidence | Pass condition |
|
||||
|---|---|---|---|
|
||||
| UI-only scope | Pull-request diff allowlist and service or model tests | Reviewer checks no sensing, transport, permissions, persistence, or retention diff | Zero out-of-scope behavior changes |
|
||||
| Cognitum-inspired, RuView-owned language | Three screenshot baselines and token review | Product reviewer compares hierarchy and general visual language | Requested characteristics present; zero Cognitum branding or private assets |
|
||||
| Cognitum-inspired, RuView-owned language | Four screenshot baselines and token review | Product reviewer compares hierarchy and general visual language | Requested characteristics present; zero Cognitum branding or private assets |
|
||||
| Five honest states | Unit tests plus `nlos-evidence-state` Playwright and Maestro selectors | Reviewer verifies wording and fail-closed hierarchy | All five exact labels are representable; unknown never becomes verified |
|
||||
| Synthetic safety | Unit and E2E assertions for `nlos-synthetic-watermark` | Screenshot review of synthetic baseline | Label and watermark both visible |
|
||||
| LiDAR point-cloud UI | Deterministic generator tests, exact point budgets, Three.js canvas readiness, reduced-motion capture, and zero-return fail-closed assertion | Browser and native visual review plus physical iPhone GPU check | WebGL renders 288 schematic relay points plus 96 returns per gated track; unverified states render zero target returns; native equivalent is labeled as a projection |
|
||||
| 390 by 844 layout | PNG dimension contract and browser overflow assertion | Screenshot review | Exact dimensions and no horizontal overflow |
|
||||
| 44 point targets | Style and interaction assertions where supported | iPhone accessibility inspector and manual target review | Every interactive hit rectangle is at least 44 by 44 points |
|
||||
| Accessibility | Semantic tests, lint, and reduced-motion fixture | VoiceOver, Dynamic Type at 200 percent, contrast, and reduced-motion review | State and primary actions remain understandable and operable |
|
||||
| Initial usable render under 1.5 seconds | Production build and instrumentation contract | Named modern iPhone run with recorded conditions | Measured usable render is below 1.5 seconds |
|
||||
| Local interaction p95 under 100 milliseconds | Deterministic action loop instrumentation | Named iPhone run over at least 30 repetitions | Recorded p95 is below 100 milliseconds |
|
||||
| Screenshot baselines | CI validates PNG format, names, dimensions, and synthetic marker contract | Reviewer approves overview, synthetic, and setup compositions | All three approved and traceable to the commit |
|
||||
| Screenshot baselines | CI validates PNG format, names, dimensions, and synthetic marker contract | Reviewer approves overview, synthetic, setup, and point-cloud compositions | All four approved and traceable to the commit |
|
||||
| No security or privacy expansion | Unit suite, dependency audit, secret scan, workflow permission review | Privacy and security diff review | No new collection, permission, storage, network, secret, or retention path |
|
||||
| Native equivalence | Swift tests and Xcode 26 unsigned simulator build | Named physical iPhone inspection | Equivalent hierarchy and state semantics; no pixel-equivalence claim |
|
||||
|
||||
@@ -486,14 +537,18 @@ full iPhone validation.
|
||||
motion and deterministic fixtures. Assert `scrollWidth <= clientWidth`.
|
||||
3. Capture and review:
|
||||
`docs/screenshots/consumer-nlos-mobile-ui/overview-390x844.png`,
|
||||
`docs/screenshots/consumer-nlos-mobile-ui/synthetic-390x844.png`, and
|
||||
`docs/screenshots/consumer-nlos-mobile-ui/setup-390x844.png`.
|
||||
`docs/screenshots/consumer-nlos-mobile-ui/synthetic-390x844.png`,
|
||||
`docs/screenshots/consumer-nlos-mobile-ui/setup-390x844.png`, and
|
||||
`docs/screenshots/consumer-nlos-mobile-ui/point-cloud-390x844.png`.
|
||||
4. Assert each PNG is exactly 390 by 844. Assert the synthetic baseline and E2E
|
||||
flow preserve the `SYNTHETIC` label and watermark.
|
||||
5. Exercise `SYNTHETIC`, `LIVE VERIFIED`, `LIVE UNVERIFIED`, `STALE`, and
|
||||
5. Select LiDAR cloud, assert the Three.js canvas reports ready, assert the
|
||||
deterministic fixture renders 96 gated target returns, and confirm the
|
||||
390-wide cloud has no horizontal overflow.
|
||||
6. Exercise `SYNTHETIC`, `LIVE VERIFIED`, `LIVE UNVERIFIED`, `STALE`, and
|
||||
`DISCONNECTED` projections. Assert stale, unverified, and disconnected states
|
||||
render no hidden-target geometry.
|
||||
6. Run Swift package tests and the Xcode 26 unsigned simulator build. Confirm
|
||||
render no hidden-target geometry or point-cloud target returns.
|
||||
7. Run Swift package tests and the Xcode 26 unsigned simulator build. Confirm
|
||||
zero changes to core packages, sensing, transport, credentials, permissions,
|
||||
diagnostic schema, or retention.
|
||||
|
||||
|
||||
@@ -3,12 +3,15 @@
|
||||
These images are deterministic review captures from the production Expo web
|
||||
export at a 390 by 844 viewport. The Playwright flow navigates the real mobile
|
||||
application, exercises the synthetic replay control, verifies provenance and
|
||||
watermark requirements, and writes the PNG files in this directory.
|
||||
watermark requirements, opens the deterministic Three.js LiDAR point cloud,
|
||||
and writes the PNG files in this directory.
|
||||
|
||||
The captures contain only the disconnected state, governed setup copy, and the
|
||||
built in synthetic fixture. They use no live endpoint, credential, sensor data,
|
||||
person data, or remote asset. They are not native iPhone screenshots, physical
|
||||
LiDAR evidence, performance evidence, or proof of optical NLOS capability.
|
||||
built in synthetic fixture. The point cloud is generated locally from gated
|
||||
track hypotheses and schematic relay geometry. It is not a raw sensor point
|
||||
cloud. The captures use no live endpoint, credential, sensor data, person data,
|
||||
or remote asset. They are not native iPhone screenshots, physical LiDAR
|
||||
evidence, performance evidence, or proof of optical NLOS capability.
|
||||
|
||||
Regenerate them from `ui/mobile` with `npm run e2e:web`. The executable design
|
||||
and evidence contract is documented in
|
||||
|
||||
|
Before Width: | Height: | Size: 80 KiB After Width: | Height: | Size: 79 KiB |
BIN
docs/screenshots/consumer-nlos-mobile-ui/point-cloud-390x844.png
Normal file
|
After Width: | Height: | Size: 69 KiB |
|
Before Width: | Height: | Size: 83 KiB After Width: | Height: | Size: 83 KiB |
|
Before Width: | Height: | Size: 52 KiB After Width: | Height: | Size: 53 KiB |
@@ -10,6 +10,7 @@ struct ContentView: View {
|
||||
@State private var exportConsent = false
|
||||
@State private var diagnosticURL: URL?
|
||||
@State private var exportError: String?
|
||||
@State private var spatialMode: TrackCanvasMode = .targets
|
||||
|
||||
var body: some View {
|
||||
NavigationStack {
|
||||
@@ -186,13 +187,24 @@ struct ContentView: View {
|
||||
|
||||
private var visualizationCard: some View {
|
||||
instrumentCard(eyebrow: "02 / SPATIAL", title: "Hidden target hypotheses", accent: RuViewTheme.cyan) {
|
||||
Text("Only validated, fresh tracks are shown. Uncertainty rings represent the reported position covariance.")
|
||||
Picker("Spatial visualization", selection: $spatialMode) {
|
||||
Text("TRACKS").tag(TrackCanvasMode.targets)
|
||||
Text("POINT CLOUD").tag(TrackCanvasMode.pointCloud)
|
||||
}
|
||||
.pickerStyle(.segmented)
|
||||
.accessibilityIdentifier("nlos-spatial-mode")
|
||||
|
||||
Text(
|
||||
spatialMode == .pointCloud
|
||||
? "The point cloud is a deterministic rendering of gated reconstruction tracks and relay geometry. It is not raw iPhone LiDAR output."
|
||||
: "Only validated, fresh tracks are shown. Uncertainty rings represent the reported position covariance."
|
||||
)
|
||||
.font(.caption)
|
||||
.foregroundStyle(RuViewTheme.textSecondary)
|
||||
.fixedSize(horizontal: false, vertical: true)
|
||||
|
||||
ZStack {
|
||||
TrackCanvas(tracks: displayableTracks)
|
||||
TrackCanvas(tracks: displayableTracks, mode: spatialMode)
|
||||
.frame(height: 310)
|
||||
.privacySensitive()
|
||||
|
||||
|
||||
@@ -2,8 +2,16 @@ import Foundation
|
||||
import RuViewNLOSCore
|
||||
import SwiftUI
|
||||
|
||||
enum TrackCanvasMode: String, CaseIterable, Identifiable {
|
||||
case targets
|
||||
case pointCloud
|
||||
|
||||
var id: String { rawValue }
|
||||
}
|
||||
|
||||
struct TrackCanvas: View {
|
||||
let tracks: [NLOSTrack]
|
||||
let mode: TrackCanvasMode
|
||||
|
||||
var body: some View {
|
||||
Canvas { context, size in
|
||||
@@ -14,20 +22,24 @@ struct TrackCanvas: View {
|
||||
with: .color(Color(red: 0.014, green: 0.029, blue: 0.043))
|
||||
)
|
||||
drawGrid(context: &context, size: size)
|
||||
drawRadar(context: &context, size: size)
|
||||
if mode == .pointCloud {
|
||||
drawPointCloud(context: &context, size: size)
|
||||
} else {
|
||||
drawRadar(context: &context, size: size)
|
||||
|
||||
let radiusMeters = max(
|
||||
5,
|
||||
min(100, tracks.flatMap { [abs($0.positionM.x), abs($0.positionM.z)] }.max() ?? 5)
|
||||
)
|
||||
|
||||
for track in tracks {
|
||||
draw(
|
||||
track: track,
|
||||
context: &context,
|
||||
size: size,
|
||||
radiusMeters: radiusMeters
|
||||
let radiusMeters = max(
|
||||
5,
|
||||
min(100, tracks.flatMap { [abs($0.positionM.x), abs($0.positionM.z)] }.max() ?? 5)
|
||||
)
|
||||
|
||||
for track in tracks {
|
||||
draw(
|
||||
track: track,
|
||||
context: &context,
|
||||
size: size,
|
||||
radiusMeters: radiusMeters
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
.clipShape(RoundedRectangle(cornerRadius: 16))
|
||||
@@ -36,7 +48,84 @@ struct TrackCanvas: View {
|
||||
.stroke(Color.cyan.opacity(0.24), lineWidth: 1)
|
||||
}
|
||||
.shadow(color: Color.cyan.opacity(0.08), radius: 16)
|
||||
.accessibilityHidden(true)
|
||||
.accessibilityElement(children: .ignore)
|
||||
.accessibilityLabel(
|
||||
mode == .pointCloud
|
||||
? "Projected LiDAR reconstruction cloud with \(tracks.count) gated hidden target hypotheses"
|
||||
: "Plan view with \(tracks.count) gated hidden target hypotheses"
|
||||
)
|
||||
}
|
||||
|
||||
private func drawPointCloud(context: inout GraphicsContext, size: CGSize) {
|
||||
let cyan = Color(red: 0.129, green: 0.831, blue: 0.906)
|
||||
let green = Color(red: 0.345, green: 0.949, blue: 0.545)
|
||||
let orange = Color(red: 1.000, green: 0.714, blue: 0.361)
|
||||
|
||||
for row in 0..<10 {
|
||||
for column in 0..<16 {
|
||||
let x = size.width * (0.08 + CGFloat(column) / 17)
|
||||
let floorY = size.height * (0.58 + CGFloat(row) * 0.032)
|
||||
let wallY = size.height * (0.12 + CGFloat(row) * 0.039)
|
||||
let wallX = x + CGFloat(row - 5) * 1.8
|
||||
let floorOpacity = 0.28 + Double(row) * 0.025
|
||||
let wallColor = row > 7 ? orange : cyan
|
||||
context.fill(
|
||||
Path(ellipseIn: CGRect(x: x - 1.1, y: floorY - 1.1, width: 2.2, height: 2.2)),
|
||||
with: .color(cyan.opacity(floorOpacity))
|
||||
)
|
||||
context.fill(
|
||||
Path(ellipseIn: CGRect(x: wallX - 1.15, y: wallY - 1.15, width: 2.3, height: 2.3)),
|
||||
with: .color(wallColor.opacity(0.52))
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
let goldenAngle = Double.pi * (3 - sqrt(5.0))
|
||||
for (trackIndex, track) in tracks.prefix(16).enumerated() {
|
||||
let centerX = size.width / 2
|
||||
+ CGFloat(max(-6, min(6, track.positionM.x))) * size.width * 0.055
|
||||
- CGFloat(max(0, min(8, track.positionM.z))) * size.width * 0.012
|
||||
let centerY = size.height * 0.63
|
||||
- CGFloat(max(0, min(4, track.positionM.y))) * size.height * 0.10
|
||||
+ CGFloat(max(0, min(8, track.positionM.z))) * size.height * 0.018
|
||||
let radiusX = max(8, min(34, CGFloat(sqrt(track.covarianceDiagonalM2.x)) * 28))
|
||||
let radiusY = max(8, min(38, CGFloat(sqrt(track.covarianceDiagonalM2.y)) * 30))
|
||||
let color = track.state == .degraded ? orange : green
|
||||
|
||||
for sample in 0..<72 {
|
||||
let normalizedY = 1 - 2 * ((Double(sample) + 0.5) / 72)
|
||||
let radial = sqrt(max(0, 1 - normalizedY * normalizedY))
|
||||
let theta = Double(sample) * goldenAngle + Double(trackIndex) * 0.73
|
||||
let shell = 0.5 + Double((sample * 37 + trackIndex * 17) % 47) / 94
|
||||
let point = CGPoint(
|
||||
x: centerX + CGFloat(cos(theta) * radial * shell) * radiusX,
|
||||
y: centerY + CGFloat(normalizedY * shell) * radiusY
|
||||
)
|
||||
let diameter: CGFloat = sample.isMultiple(of: 5) ? 3.2 : 2.1
|
||||
context.fill(
|
||||
Path(ellipseIn: CGRect(
|
||||
x: point.x - diameter / 2,
|
||||
y: point.y - diameter / 2,
|
||||
width: diameter,
|
||||
height: diameter
|
||||
)),
|
||||
with: .color(color.opacity(0.82))
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
context.draw(
|
||||
Text("LIDAR POINT CLOUD / NATIVE SWIFTUI CANVAS")
|
||||
.font(.caption2.bold().monospaced())
|
||||
.foregroundColor(cyan),
|
||||
at: CGPoint(x: size.width / 2, y: 18)
|
||||
)
|
||||
context.draw(
|
||||
Text("\(tracks.count * 72) GATED TARGET RETURNS")
|
||||
.font(.caption2.bold().monospaced())
|
||||
.foregroundColor(Color.white.opacity(0.64)),
|
||||
at: CGPoint(x: size.width / 2, y: size.height - 16)
|
||||
)
|
||||
}
|
||||
|
||||
private func draw(
|
||||
|
||||
@@ -21,6 +21,22 @@ name: RuView NLOS instrument UI evidence flow
|
||||
id: "nlos-synthetic-watermark"
|
||||
- assertVisible:
|
||||
id: "nlos-evidence-state"
|
||||
- scrollUntilVisible:
|
||||
element:
|
||||
id: "nlos-view-cloud"
|
||||
direction: DOWN
|
||||
timeout: 10000
|
||||
- tapOn:
|
||||
id: "nlos-view-cloud"
|
||||
- scrollUntilVisible:
|
||||
element:
|
||||
id: "nlos-lidar-point-cloud"
|
||||
direction: DOWN
|
||||
timeout: 10000
|
||||
- assertVisible:
|
||||
id: "nlos-lidar-point-cloud"
|
||||
- assertVisible:
|
||||
id: "nlos-cloud-target-count"
|
||||
- scrollUntilVisible:
|
||||
element:
|
||||
id: "nlos-beta-setup"
|
||||
|
||||
@@ -106,4 +106,28 @@ test.describe('RuView NLOS mobile instrument UI', () => {
|
||||
|
||||
await capture(page, 'synthetic-390x844.png');
|
||||
});
|
||||
|
||||
test('renders a gated Three.js LiDAR point cloud without horizontal overflow', async ({ page }) => {
|
||||
await openNlos(page);
|
||||
const replay = page.getByRole('button', { name: 'USE SYNTHETIC REPLAY' });
|
||||
await replay.scrollIntoViewIfNeeded();
|
||||
await replay.click();
|
||||
|
||||
await page.getByTestId('nlos-view-cloud').click();
|
||||
const cloud = page.getByTestId('nlos-lidar-point-cloud');
|
||||
await cloud.scrollIntoViewIfNeeded();
|
||||
await expect(cloud).toBeVisible();
|
||||
await expect(page.getByTestId('nlos-lidar-point-cloud-canvas')).toHaveAttribute('data-ready', 'true');
|
||||
await expect(page.getByTestId('nlos-cloud-target-count')).toHaveText('96');
|
||||
await expect(page.getByText('THREE.JS / WEBGL')).toBeVisible();
|
||||
await expect(page.getByTestId('nlos-synthetic-watermark')).toBeVisible();
|
||||
|
||||
const dimensions = await cloud.evaluate((element) => ({
|
||||
viewport: element.clientWidth,
|
||||
content: element.scrollWidth,
|
||||
}));
|
||||
expect(dimensions.content).toBeLessThanOrEqual(dimensions.viewport + 1);
|
||||
|
||||
await capture(page, 'point-cloud-390x844.png');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -23,7 +23,13 @@ export default defineConfig({
|
||||
launchOptions: process.env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH
|
||||
? {
|
||||
executablePath: process.env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH,
|
||||
args: ['--no-sandbox', '--disable-dev-shm-usage'],
|
||||
args: [
|
||||
'--no-sandbox',
|
||||
'--disable-dev-shm-usage',
|
||||
'--use-gl=angle',
|
||||
'--use-angle=swiftshader-webgl',
|
||||
'--enable-unsafe-swiftshader',
|
||||
],
|
||||
}
|
||||
: undefined,
|
||||
},
|
||||
|
||||
@@ -10,6 +10,11 @@ import {
|
||||
NLOS_FEEDBACK_URL,
|
||||
} from '@/screens/NLOSScreen/BetaSetupCard';
|
||||
import { resolveNlosEvidenceState } from '@/screens/NLOSScreen/ProvenancePanel';
|
||||
import {
|
||||
buildLidarPointCloud,
|
||||
LIDAR_POINTS_PER_TRACK,
|
||||
LIDAR_RELAY_POINT_COUNT,
|
||||
} from '@/screens/NLOSScreen/lidarPointCloud';
|
||||
|
||||
const mockSafeAreaInsets = { top: 0, right: 0, bottom: 0, left: 0 };
|
||||
|
||||
@@ -241,6 +246,8 @@ describe('NLOSScreen', () => {
|
||||
expect(screen.getByTestId('nlos-evidence-state').props.children).toBe('LIVE UNVERIFIED');
|
||||
expect(screen.getByTestId('nlos-track-count').props.children).toBe(0);
|
||||
expect(screen.queryByText(live.tracks[0].trackId)).toBeNull();
|
||||
fireEvent.press(screen.getByTestId('nlos-view-cloud'));
|
||||
expect(screen.getByTestId('nlos-cloud-target-count').props.children).toBe(0);
|
||||
|
||||
Object.assign(mockNlosResult, {
|
||||
frame: { ...live, source: 'replay' },
|
||||
@@ -293,6 +300,18 @@ describe('NLOSScreen', () => {
|
||||
expect(resolveNlosEvidenceState(usbFrame, 'fresh', 'live')).toBe('LIVE VERIFIED');
|
||||
});
|
||||
|
||||
it('builds a deterministic bounded LiDAR reconstruction cloud from gated tracks', () => {
|
||||
const first = buildLidarPointCloud(syntheticFrame.tracks);
|
||||
const second = buildLidarPointCloud(syntheticFrame.tracks);
|
||||
|
||||
expect(first.relayPointCount).toBe(LIDAR_RELAY_POINT_COUNT);
|
||||
expect(first.targetPointCount).toBe(syntheticFrame.tracks.length * LIDAR_POINTS_PER_TRACK);
|
||||
expect(first.totalPointCount).toBe(first.relayPointCount + first.targetPointCount);
|
||||
expect(Array.from(first.positions)).toEqual(Array.from(second.positions));
|
||||
expect(Array.from(first.colors)).toEqual(Array.from(second.colors));
|
||||
expect(Array.from(first.positions).every(Number.isFinite)).toBe(true);
|
||||
});
|
||||
|
||||
it('keeps privacy, setup, explainer, and feedback controls visible in the screen tree', () => {
|
||||
const { NLOSScreen } = require('@/screens/NLOSScreen');
|
||||
render(<ThemeProvider><NLOSScreen /></ThemeProvider>);
|
||||
@@ -311,5 +330,11 @@ describe('NLOSScreen', () => {
|
||||
expect(screen.getByTestId('nlos-view-plan').props.accessibilityState.selected).toBe(true);
|
||||
fireEvent.press(screen.getByTestId('nlos-view-perspective'));
|
||||
expect(screen.getByTestId('nlos-view-perspective').props.accessibilityState.selected).toBe(true);
|
||||
fireEvent.press(screen.getByTestId('nlos-view-cloud'));
|
||||
expect(screen.getByTestId('nlos-view-cloud').props.accessibilityState.selected).toBe(true);
|
||||
expect(screen.getByTestId('nlos-lidar-point-cloud')).toBeTruthy();
|
||||
expect(screen.getByTestId('nlos-cloud-target-count').props.children).toBe(
|
||||
syntheticFrame.tracks.length * LIDAR_POINTS_PER_TRACK,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -3,8 +3,9 @@ import { View } from 'react-native';
|
||||
import Svg, { Circle, Ellipse, Line, Polygon, Rect, Text as SvgText } from 'react-native-svg';
|
||||
import { instrumentColors } from '@/components/InstrumentPanel';
|
||||
import type { NlosFreshness, NlosTrack } from '@/types/nlos';
|
||||
import { LidarPointCloud } from './LidarPointCloud';
|
||||
|
||||
export type NlosViewMode = 'plan' | 'perspective';
|
||||
export type NlosViewMode = 'plan' | 'perspective' | 'cloud';
|
||||
|
||||
interface HiddenTargetVisualizationProps {
|
||||
tracks: NlosTrack[];
|
||||
@@ -119,6 +120,10 @@ export const HiddenTargetVisualization = memo(({
|
||||
);
|
||||
const displayWidth = Math.max(260, Math.min(width, 560));
|
||||
|
||||
if (mode === 'cloud') {
|
||||
return <LidarPointCloud tracks={tracks} freshness={freshness} width={displayWidth} />;
|
||||
}
|
||||
|
||||
return (
|
||||
<View
|
||||
testID="nlos-target-visualization"
|
||||
|
||||
98
ui/mobile/src/screens/NLOSScreen/LidarPointCloud.tsx
Normal file
@@ -0,0 +1,98 @@
|
||||
import { memo, useMemo } from 'react';
|
||||
import { StyleSheet, View } from 'react-native';
|
||||
import Svg, { Circle, Line, Rect, Text as SvgText } from 'react-native-svg';
|
||||
import { instrumentColors } from '@/components/InstrumentPanel';
|
||||
import { ThemedText } from '@/components/ThemedText';
|
||||
import type { NlosFreshness, NlosTrack } from '@/types/nlos';
|
||||
import { buildLidarPointCloud, LIDAR_RELAY_POINT_COUNT } from './lidarPointCloud';
|
||||
|
||||
interface LidarPointCloudProps {
|
||||
tracks: NlosTrack[];
|
||||
freshness: NlosFreshness;
|
||||
width: number;
|
||||
}
|
||||
|
||||
interface ProjectedPoint {
|
||||
color: string;
|
||||
radius: number;
|
||||
x: number;
|
||||
y: number;
|
||||
}
|
||||
|
||||
const CANVAS_WIDTH = 360;
|
||||
const CANVAS_HEIGHT = 260;
|
||||
|
||||
const rgb = (red: number, green: number, blue: number) => (
|
||||
`rgb(${Math.round(red * 255)}, ${Math.round(green * 255)}, ${Math.round(blue * 255)})`
|
||||
);
|
||||
|
||||
export const LidarPointCloud = memo(({ tracks, freshness, width }: LidarPointCloudProps) => {
|
||||
const cloud = useMemo(() => buildLidarPointCloud(tracks), [tracks]);
|
||||
const points = useMemo(() => {
|
||||
const projected: ProjectedPoint[] = [];
|
||||
for (let index = 0; index < cloud.totalPointCount; index += 1) {
|
||||
const isRelay = index < LIDAR_RELAY_POINT_COUNT;
|
||||
if ((isRelay && index % 3 !== 0) || (!isRelay && index % 2 !== 0)) continue;
|
||||
const offset = index * 3;
|
||||
const x = cloud.positions[offset];
|
||||
const y = cloud.positions[offset + 1];
|
||||
const z = cloud.positions[offset + 2];
|
||||
projected.push({
|
||||
x: 180 + x * 34 - z * 8,
|
||||
y: 220 - y * 46 + z * 7,
|
||||
radius: isRelay ? 1.15 : 1.8,
|
||||
color: rgb(cloud.colors[offset], cloud.colors[offset + 1], cloud.colors[offset + 2]),
|
||||
});
|
||||
}
|
||||
return projected;
|
||||
}, [cloud]);
|
||||
const displayWidth = Math.max(260, Math.min(width, 560));
|
||||
|
||||
return (
|
||||
<View
|
||||
testID="nlos-lidar-point-cloud"
|
||||
accessibilityRole="image"
|
||||
accessibilityLabel={`Projected LiDAR reconstruction cloud with ${cloud.targetPointCount} gated target returns from ${tracks.length} hidden target hypotheses. ${freshness} evidence.`}
|
||||
style={{ alignSelf: 'center', width: displayWidth, aspectRatio: CANVAS_WIDTH / CANVAS_HEIGHT }}
|
||||
>
|
||||
<Svg width="100%" height="100%" viewBox={`0 0 ${CANVAS_WIDTH} ${CANVAS_HEIGHT}`}>
|
||||
<Rect x={12} y={12} width={336} height={236} rx={14} fill={instrumentColors.panelRaised} stroke={instrumentColors.border} />
|
||||
<Line x1={36} y1={220} x2={328} y2={220} stroke={instrumentColors.borderStrong} />
|
||||
<Line x1={180} y1={220} x2={78} y2={176} stroke={instrumentColors.cyanDim} />
|
||||
<Line x1={180} y1={220} x2={286} y2={174} stroke={instrumentColors.cyanDim} />
|
||||
{points.map((point, index) => (
|
||||
<Circle
|
||||
key={`cloud-point-${index}`}
|
||||
cx={point.x}
|
||||
cy={point.y}
|
||||
r={point.radius}
|
||||
fill={point.color}
|
||||
opacity={freshness === 'fresh' ? 0.88 : 0.35}
|
||||
/>
|
||||
))}
|
||||
<SvgText x={24} y={34} fill={instrumentColors.cyan} fontSize={9} letterSpacing={1.1}>LIDAR CLOUD / NATIVE PROJECTION</SvgText>
|
||||
</Svg>
|
||||
<View pointerEvents="none" style={styles.metricRow}>
|
||||
<ThemedText testID="nlos-cloud-target-count" preset="labelLg" style={styles.metricValue}>
|
||||
{cloud.targetPointCount}
|
||||
</ThemedText>
|
||||
<ThemedText preset="mono" style={styles.metricLabel}>GATED TARGET RETURNS</ThemedText>
|
||||
</View>
|
||||
</View>
|
||||
);
|
||||
});
|
||||
|
||||
LidarPointCloud.displayName = 'LidarPointCloud';
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
metricRow: {
|
||||
position: 'absolute',
|
||||
left: 24,
|
||||
bottom: 24,
|
||||
flexDirection: 'row',
|
||||
alignItems: 'baseline',
|
||||
gap: 8,
|
||||
},
|
||||
metricValue: { color: instrumentColors.green, fontSize: 14, lineHeight: 16 },
|
||||
metricLabel: { color: instrumentColors.textSecondary, fontSize: 8, letterSpacing: 0.8 },
|
||||
});
|
||||
393
ui/mobile/src/screens/NLOSScreen/LidarPointCloud.web.tsx
Normal file
@@ -0,0 +1,393 @@
|
||||
import { memo, useCallback, useEffect, useMemo, useRef, useState } from 'react';
|
||||
import { StyleSheet, View } from 'react-native';
|
||||
import * as THREE from 'three';
|
||||
import { instrumentColors } from '@/components/InstrumentPanel';
|
||||
import { ThemedText } from '@/components/ThemedText';
|
||||
import type { NlosFreshness, NlosTrack } from '@/types/nlos';
|
||||
import { buildLidarPointCloud } from './lidarPointCloud';
|
||||
|
||||
interface LidarPointCloudProps {
|
||||
tracks: NlosTrack[];
|
||||
freshness: NlosFreshness;
|
||||
width: number;
|
||||
}
|
||||
|
||||
type RendererState = 'initializing' | 'ready' | 'fallback';
|
||||
|
||||
const CANVAS_ASPECT_RATIO = 360 / 260;
|
||||
|
||||
interface WebSceneState {
|
||||
geometry: THREE.BufferGeometry;
|
||||
material: THREE.PointsMaterial;
|
||||
points: THREE.Points;
|
||||
render: () => void;
|
||||
}
|
||||
|
||||
const disposeMaterial = (material: THREE.Material | THREE.Material[]) => {
|
||||
if (Array.isArray(material)) material.forEach((entry) => entry.dispose());
|
||||
else material.dispose();
|
||||
};
|
||||
|
||||
const rgb = (red: number, green: number, blue: number) => (
|
||||
`rgb(${Math.round(red * 255)}, ${Math.round(green * 255)}, ${Math.round(blue * 255)})`
|
||||
);
|
||||
|
||||
export const LidarPointCloud = memo(({ tracks, freshness, width }: LidarPointCloudProps) => {
|
||||
const hostRef = useRef<HTMLDivElement | null>(null);
|
||||
const sceneRef = useRef<WebSceneState | null>(null);
|
||||
const [rendererState, setRendererState] = useState<RendererState>('initializing');
|
||||
const cloud = useMemo(() => buildLidarPointCloud(tracks), [tracks]);
|
||||
const displayWidth = Math.max(260, Math.min(width, 560));
|
||||
const fallbackPoints = useMemo(() => {
|
||||
const height = displayWidth / CANVAS_ASPECT_RATIO;
|
||||
const points: Array<{ color: string; left: number; top: number }> = [];
|
||||
for (let index = 0; index < cloud.totalPointCount; index += 1) {
|
||||
const isRelay = index < cloud.relayPointCount;
|
||||
if ((isRelay && index % 3 !== 0) || (!isRelay && index % 2 !== 0)) continue;
|
||||
const offset = index * 3;
|
||||
const x = cloud.positions[offset];
|
||||
const y = cloud.positions[offset + 1];
|
||||
const z = cloud.positions[offset + 2];
|
||||
points.push({
|
||||
left: ((180 + x * 34 - z * 8) / 360) * displayWidth,
|
||||
top: ((220 - y * 46 + z * 7) / 260) * height,
|
||||
color: rgb(cloud.colors[offset], cloud.colors[offset + 1], cloud.colors[offset + 2]),
|
||||
});
|
||||
}
|
||||
return points;
|
||||
}, [cloud, displayWidth]);
|
||||
const assignHost = useCallback((node: unknown) => {
|
||||
hostRef.current = node as HTMLDivElement | null;
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
const host = hostRef.current;
|
||||
if (!host) return undefined;
|
||||
|
||||
let animationFrame = 0;
|
||||
let resizeObserver: ResizeObserver | null = null;
|
||||
let disposed = false;
|
||||
let renderer: THREE.WebGLRenderer | null = null;
|
||||
const listeners: Array<() => void> = [];
|
||||
|
||||
try {
|
||||
renderer = new THREE.WebGLRenderer({
|
||||
alpha: false,
|
||||
antialias: false,
|
||||
powerPreference: 'high-performance',
|
||||
});
|
||||
renderer.setClearColor(0x071017, 1);
|
||||
renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 1.5));
|
||||
renderer.outputColorSpace = THREE.SRGBColorSpace;
|
||||
renderer.domElement.dataset.testid = 'nlos-lidar-point-cloud-canvas';
|
||||
renderer.domElement.dataset.ready = 'false';
|
||||
renderer.domElement.setAttribute('aria-hidden', 'true');
|
||||
renderer.domElement.style.position = 'absolute';
|
||||
renderer.domElement.style.inset = '0';
|
||||
renderer.domElement.style.zIndex = '0';
|
||||
renderer.domElement.style.touchAction = 'none';
|
||||
host.appendChild(renderer.domElement);
|
||||
|
||||
const scene = new THREE.Scene();
|
||||
scene.background = new THREE.Color(0x071017);
|
||||
scene.fog = new THREE.FogExp2(0x071017, 0.055);
|
||||
const camera = new THREE.PerspectiveCamera(42, CANVAS_ASPECT_RATIO, 0.1, 40);
|
||||
let yaw = 0.55;
|
||||
let pitch = 0.34;
|
||||
let radius = 8.6;
|
||||
let dragging = false;
|
||||
let pointerX = 0;
|
||||
let pointerY = 0;
|
||||
|
||||
const geometry = new THREE.BufferGeometry();
|
||||
geometry.setAttribute('position', new THREE.BufferAttribute(new Float32Array(0), 3));
|
||||
geometry.setAttribute('color', new THREE.BufferAttribute(new Float32Array(0), 3));
|
||||
const material = new THREE.PointsMaterial({
|
||||
blending: THREE.AdditiveBlending,
|
||||
depthWrite: false,
|
||||
opacity: 0.92,
|
||||
size: 0.065,
|
||||
sizeAttenuation: true,
|
||||
transparent: true,
|
||||
vertexColors: true,
|
||||
});
|
||||
const points = new THREE.Points(geometry, material);
|
||||
scene.add(points);
|
||||
|
||||
const floorGrid = new THREE.GridHelper(8, 16, 0x174957, 0x102934);
|
||||
floorGrid.position.y = -0.04;
|
||||
floorGrid.position.z = -1.25;
|
||||
scene.add(floorGrid);
|
||||
|
||||
const relayPlane = new THREE.GridHelper(8, 16, 0xffb65c, 0x16414b);
|
||||
relayPlane.rotation.x = Math.PI / 2;
|
||||
relayPlane.position.set(0, 1.55, 0.72);
|
||||
scene.add(relayPlane);
|
||||
|
||||
const sensor = new THREE.Mesh(
|
||||
new THREE.OctahedronGeometry(0.11, 0),
|
||||
new THREE.MeshBasicMaterial({ color: 0x24d3e5, wireframe: true }),
|
||||
);
|
||||
sensor.position.set(0, 0.18, 2.45);
|
||||
scene.add(sensor);
|
||||
|
||||
const rayGeometry = new THREE.BufferGeometry().setFromPoints([
|
||||
sensor.position,
|
||||
new THREE.Vector3(0, 1.2, 0.72),
|
||||
new THREE.Vector3(0.9, 1.1, -2.1),
|
||||
]);
|
||||
const rayMaterial = new THREE.LineBasicMaterial({ color: 0x58f28b, opacity: 0.42, transparent: true });
|
||||
const ray = new THREE.Line(rayGeometry, rayMaterial);
|
||||
scene.add(ray);
|
||||
|
||||
const sizeRenderer = () => {
|
||||
if (!renderer || disposed) return;
|
||||
const nextWidth = Math.max(1, host.clientWidth || 360);
|
||||
const nextHeight = Math.max(1, host.clientHeight || 260);
|
||||
camera.aspect = nextWidth / nextHeight;
|
||||
camera.updateProjectionMatrix();
|
||||
renderer.setSize(nextWidth, nextHeight, false);
|
||||
};
|
||||
const renderScene = () => {
|
||||
if (!renderer || disposed) return;
|
||||
const horizontalRadius = radius * Math.cos(pitch);
|
||||
camera.position.set(
|
||||
Math.sin(yaw) * horizontalRadius,
|
||||
1.25 + Math.sin(pitch) * radius,
|
||||
Math.cos(yaw) * horizontalRadius - 0.65,
|
||||
);
|
||||
camera.lookAt(0, 1.15, -1.05);
|
||||
renderer.render(scene, camera);
|
||||
};
|
||||
|
||||
sceneRef.current = { geometry, material, points, render: renderScene };
|
||||
|
||||
const onPointerDown = (event: PointerEvent) => {
|
||||
dragging = true;
|
||||
pointerX = event.clientX;
|
||||
pointerY = event.clientY;
|
||||
renderer?.domElement.setPointerCapture?.(event.pointerId);
|
||||
};
|
||||
const onPointerMove = (event: PointerEvent) => {
|
||||
if (!dragging) return;
|
||||
yaw += (event.clientX - pointerX) * 0.008;
|
||||
pitch = THREE.MathUtils.clamp(pitch + (event.clientY - pointerY) * 0.006, -0.1, 0.85);
|
||||
pointerX = event.clientX;
|
||||
pointerY = event.clientY;
|
||||
renderScene();
|
||||
};
|
||||
const onPointerUp = (event: PointerEvent) => {
|
||||
dragging = false;
|
||||
renderer?.domElement.releasePointerCapture?.(event.pointerId);
|
||||
};
|
||||
const onWheel = (event: WheelEvent) => {
|
||||
event.preventDefault();
|
||||
radius = THREE.MathUtils.clamp(radius + event.deltaY * 0.008, 4.8, 13);
|
||||
renderScene();
|
||||
};
|
||||
const onContextLost = (event: Event) => {
|
||||
event.preventDefault();
|
||||
setRendererState('fallback');
|
||||
};
|
||||
const onContextRestored = () => {
|
||||
setRendererState('ready');
|
||||
renderScene();
|
||||
};
|
||||
|
||||
const canvas = renderer.domElement;
|
||||
canvas.addEventListener('pointerdown', onPointerDown);
|
||||
canvas.addEventListener('pointermove', onPointerMove);
|
||||
canvas.addEventListener('pointerup', onPointerUp);
|
||||
canvas.addEventListener('pointercancel', onPointerUp);
|
||||
canvas.addEventListener('wheel', onWheel, { passive: false });
|
||||
canvas.addEventListener('webglcontextlost', onContextLost);
|
||||
canvas.addEventListener('webglcontextrestored', onContextRestored);
|
||||
listeners.push(
|
||||
() => canvas.removeEventListener('pointerdown', onPointerDown),
|
||||
() => canvas.removeEventListener('pointermove', onPointerMove),
|
||||
() => canvas.removeEventListener('pointerup', onPointerUp),
|
||||
() => canvas.removeEventListener('pointercancel', onPointerUp),
|
||||
() => canvas.removeEventListener('wheel', onWheel),
|
||||
() => canvas.removeEventListener('webglcontextlost', onContextLost),
|
||||
() => canvas.removeEventListener('webglcontextrestored', onContextRestored),
|
||||
);
|
||||
|
||||
sizeRenderer();
|
||||
renderScene();
|
||||
canvas.dataset.ready = 'true';
|
||||
setRendererState('ready');
|
||||
|
||||
if (typeof ResizeObserver !== 'undefined') {
|
||||
resizeObserver = new ResizeObserver(() => {
|
||||
sizeRenderer();
|
||||
renderScene();
|
||||
});
|
||||
resizeObserver.observe(host);
|
||||
} else {
|
||||
window.addEventListener('resize', sizeRenderer);
|
||||
listeners.push(() => window.removeEventListener('resize', sizeRenderer));
|
||||
}
|
||||
|
||||
const reduceMotion = window.matchMedia?.('(prefers-reduced-motion: reduce)').matches ?? false;
|
||||
if (!reduceMotion) {
|
||||
const animate = () => {
|
||||
if (disposed) return;
|
||||
if (!dragging) yaw += 0.0009;
|
||||
renderScene();
|
||||
animationFrame = window.requestAnimationFrame(animate);
|
||||
};
|
||||
animationFrame = window.requestAnimationFrame(animate);
|
||||
}
|
||||
|
||||
return () => {
|
||||
disposed = true;
|
||||
sceneRef.current = null;
|
||||
window.cancelAnimationFrame(animationFrame);
|
||||
resizeObserver?.disconnect();
|
||||
listeners.forEach((remove) => remove());
|
||||
scene.remove(points, floorGrid, relayPlane, sensor, ray);
|
||||
geometry.dispose();
|
||||
material.dispose();
|
||||
floorGrid.geometry.dispose();
|
||||
disposeMaterial(floorGrid.material);
|
||||
relayPlane.geometry.dispose();
|
||||
disposeMaterial(relayPlane.material);
|
||||
sensor.geometry.dispose();
|
||||
disposeMaterial(sensor.material);
|
||||
rayGeometry.dispose();
|
||||
rayMaterial.dispose();
|
||||
renderer?.dispose();
|
||||
if (canvas.parentNode === host) host.removeChild(canvas);
|
||||
};
|
||||
} catch {
|
||||
sceneRef.current = null;
|
||||
setRendererState('fallback');
|
||||
return () => {
|
||||
disposed = true;
|
||||
window.cancelAnimationFrame(animationFrame);
|
||||
resizeObserver?.disconnect();
|
||||
listeners.forEach((remove) => remove());
|
||||
renderer?.dispose();
|
||||
const canvas = renderer?.domElement;
|
||||
if (canvas?.parentNode === host) host.removeChild(canvas);
|
||||
};
|
||||
}
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
const state = sceneRef.current;
|
||||
if (!state) return;
|
||||
|
||||
const currentPosition = state.geometry.getAttribute('position') as THREE.BufferAttribute;
|
||||
if (currentPosition.count === cloud.totalPointCount) {
|
||||
(currentPosition.array as Float32Array).set(cloud.positions);
|
||||
currentPosition.needsUpdate = true;
|
||||
const currentColor = state.geometry.getAttribute('color') as THREE.BufferAttribute;
|
||||
(currentColor.array as Float32Array).set(cloud.colors);
|
||||
currentColor.needsUpdate = true;
|
||||
} else {
|
||||
const nextGeometry = new THREE.BufferGeometry();
|
||||
nextGeometry.setAttribute('position', new THREE.BufferAttribute(cloud.positions.slice(), 3));
|
||||
nextGeometry.setAttribute('color', new THREE.BufferAttribute(cloud.colors.slice(), 3));
|
||||
state.geometry.dispose();
|
||||
state.geometry = nextGeometry;
|
||||
state.points.geometry = nextGeometry;
|
||||
}
|
||||
state.geometry.computeBoundingSphere();
|
||||
state.material.opacity = freshness === 'fresh' ? 0.92 : 0.34;
|
||||
state.material.needsUpdate = true;
|
||||
state.render();
|
||||
}, [cloud, freshness]);
|
||||
|
||||
return (
|
||||
<View
|
||||
ref={assignHost}
|
||||
testID="nlos-lidar-point-cloud"
|
||||
accessibilityRole="image"
|
||||
accessibilityLabel={`Interactive Three.js LiDAR reconstruction cloud with ${cloud.targetPointCount} gated target returns from ${tracks.length} hidden target hypotheses. ${freshness} evidence.`}
|
||||
style={[styles.host, { width: displayWidth, aspectRatio: CANVAS_ASPECT_RATIO }]}
|
||||
>
|
||||
{rendererState === 'fallback' ? (
|
||||
<View pointerEvents="none" style={styles.fallbackLayer}>
|
||||
{fallbackPoints.map((point, index) => (
|
||||
<View
|
||||
key={`${index}-${point.left}-${point.top}`}
|
||||
style={[
|
||||
styles.fallbackPoint,
|
||||
{ backgroundColor: point.color, left: point.left, top: point.top },
|
||||
]}
|
||||
/>
|
||||
))}
|
||||
</View>
|
||||
) : null}
|
||||
<View pointerEvents="none" style={styles.topHud}>
|
||||
<ThemedText preset="mono" style={styles.hudTitle}>LIDAR POINT CLOUD</ThemedText>
|
||||
<ThemedText preset="mono" style={styles.rendererLabel}>
|
||||
{rendererState === 'fallback' ? 'STATIC FALLBACK' : 'THREE.JS / WEBGL'}
|
||||
</ThemedText>
|
||||
</View>
|
||||
<View pointerEvents="none" style={styles.bottomHud}>
|
||||
<View>
|
||||
<ThemedText testID="nlos-cloud-target-count" preset="labelLg" style={styles.metricValue}>
|
||||
{cloud.targetPointCount}
|
||||
</ThemedText>
|
||||
<ThemedText preset="mono" style={styles.metricLabel}>GATED TARGET RETURNS</ThemedText>
|
||||
</View>
|
||||
<ThemedText preset="mono" style={styles.gestureHint}>
|
||||
{rendererState === 'ready' ? 'DRAG TO ORBIT' : 'WEBGL UNAVAILABLE'}
|
||||
</ThemedText>
|
||||
</View>
|
||||
</View>
|
||||
);
|
||||
});
|
||||
|
||||
LidarPointCloud.displayName = 'LidarPointCloud';
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
host: {
|
||||
position: 'relative',
|
||||
alignSelf: 'center',
|
||||
overflow: 'hidden',
|
||||
backgroundColor: '#071017',
|
||||
borderColor: instrumentColors.border,
|
||||
borderWidth: 1,
|
||||
borderRadius: 14,
|
||||
},
|
||||
fallbackLayer: {
|
||||
...StyleSheet.absoluteFillObject,
|
||||
zIndex: 1,
|
||||
},
|
||||
fallbackPoint: {
|
||||
position: 'absolute',
|
||||
width: 2,
|
||||
height: 2,
|
||||
borderRadius: 1,
|
||||
},
|
||||
topHud: {
|
||||
position: 'absolute',
|
||||
top: 12,
|
||||
left: 12,
|
||||
right: 12,
|
||||
zIndex: 2,
|
||||
flexDirection: 'row',
|
||||
alignItems: 'center',
|
||||
justifyContent: 'space-between',
|
||||
gap: 8,
|
||||
},
|
||||
hudTitle: { color: instrumentColors.cyan, fontSize: 9, letterSpacing: 1.15 },
|
||||
rendererLabel: { color: instrumentColors.textSecondary, fontSize: 8, letterSpacing: 0.8 },
|
||||
bottomHud: {
|
||||
position: 'absolute',
|
||||
left: 12,
|
||||
right: 12,
|
||||
bottom: 10,
|
||||
zIndex: 2,
|
||||
flexDirection: 'row',
|
||||
alignItems: 'flex-end',
|
||||
justifyContent: 'space-between',
|
||||
gap: 8,
|
||||
},
|
||||
metricValue: { color: instrumentColors.green, fontSize: 16, lineHeight: 18 },
|
||||
metricLabel: { color: instrumentColors.textSecondary, fontSize: 7, letterSpacing: 0.7 },
|
||||
gestureHint: { color: instrumentColors.textSecondary, fontSize: 8, letterSpacing: 0.8 },
|
||||
});
|
||||
@@ -29,7 +29,11 @@ const ViewModePicker = ({
|
||||
onChange: (value: NlosViewMode) => void;
|
||||
}) => (
|
||||
<View accessibilityRole="tablist" style={styles.picker}>
|
||||
{(['plan', 'perspective'] as const).map((option) => {
|
||||
{([
|
||||
['plan', '2D PLAN'],
|
||||
['perspective', '3D VIEW'],
|
||||
['cloud', 'LIDAR CLOUD'],
|
||||
] as const).map(([option, label]) => {
|
||||
const selected = option === value;
|
||||
return (
|
||||
<Pressable
|
||||
@@ -42,9 +46,12 @@ const ViewModePicker = ({
|
||||
>
|
||||
<ThemedText
|
||||
preset="mono"
|
||||
style={{ color: selected ? instrumentColors.cyan : instrumentColors.textSecondary }}
|
||||
style={[
|
||||
styles.pickerLabel,
|
||||
{ color: selected ? instrumentColors.cyan : instrumentColors.textSecondary },
|
||||
]}
|
||||
>
|
||||
{option === 'plan' ? '2D PLAN' : '3D VIEW'}
|
||||
{label}
|
||||
</ThemedText>
|
||||
</Pressable>
|
||||
);
|
||||
@@ -448,6 +455,7 @@ const styles = StyleSheet.create({
|
||||
borderColor: instrumentColors.cyanDim,
|
||||
borderWidth: 1,
|
||||
},
|
||||
pickerLabel: { fontSize: 9, letterSpacing: 0.55, textAlign: 'center' },
|
||||
visualizationStage: { position: 'relative', overflow: 'hidden' },
|
||||
watermark: {
|
||||
...StyleSheet.absoluteFillObject,
|
||||
|
||||
120
ui/mobile/src/screens/NLOSScreen/lidarPointCloud.ts
Normal file
@@ -0,0 +1,120 @@
|
||||
import type { NlosTrack } from '@/types/nlos';
|
||||
|
||||
export const LIDAR_RELAY_POINT_COUNT = 288;
|
||||
export const LIDAR_POINTS_PER_TRACK = 96;
|
||||
export const LIDAR_MAX_TRACKS = 16;
|
||||
|
||||
export interface LidarPointCloudData {
|
||||
positions: Float32Array;
|
||||
colors: Float32Array;
|
||||
relayPointCount: number;
|
||||
targetPointCount: number;
|
||||
totalPointCount: number;
|
||||
}
|
||||
|
||||
const GOLDEN_ANGLE = Math.PI * (3 - Math.sqrt(5));
|
||||
|
||||
const clamp = (value: number, minimum: number, maximum: number) => (
|
||||
Math.max(minimum, Math.min(maximum, value))
|
||||
);
|
||||
|
||||
const writePoint = (
|
||||
positions: Float32Array,
|
||||
colors: Float32Array,
|
||||
index: number,
|
||||
point: readonly [number, number, number],
|
||||
color: readonly [number, number, number],
|
||||
) => {
|
||||
const offset = index * 3;
|
||||
positions[offset] = point[0];
|
||||
positions[offset + 1] = point[1];
|
||||
positions[offset + 2] = point[2];
|
||||
colors[offset] = color[0];
|
||||
colors[offset + 1] = color[1];
|
||||
colors[offset + 2] = color[2];
|
||||
};
|
||||
|
||||
export const buildLidarPointCloud = (tracks: readonly NlosTrack[]): LidarPointCloudData => {
|
||||
const visibleTracks = tracks.slice(0, LIDAR_MAX_TRACKS);
|
||||
const targetPointCount = visibleTracks.length * LIDAR_POINTS_PER_TRACK;
|
||||
const totalPointCount = LIDAR_RELAY_POINT_COUNT + targetPointCount;
|
||||
const positions = new Float32Array(totalPointCount * 3);
|
||||
const colors = new Float32Array(totalPointCount * 3);
|
||||
let pointIndex = 0;
|
||||
|
||||
for (let row = 0; row < 12; row += 1) {
|
||||
for (let column = 0; column < 12; column += 1) {
|
||||
const x = -3.85 + column * 0.7;
|
||||
const z = 2.2 - row * 0.55;
|
||||
const scanRipple = Math.sin(column * 1.7 + row * 0.8) * 0.025;
|
||||
writePoint(
|
||||
positions,
|
||||
colors,
|
||||
pointIndex,
|
||||
[x, scanRipple, z],
|
||||
[0.08, 0.52 + row * 0.008, 0.61 + column * 0.006],
|
||||
);
|
||||
pointIndex += 1;
|
||||
}
|
||||
}
|
||||
|
||||
for (let row = 0; row < 12; row += 1) {
|
||||
for (let column = 0; column < 12; column += 1) {
|
||||
const x = -3.85 + column * 0.7;
|
||||
const y = 0.15 + row * 0.25;
|
||||
const relayRipple = Math.cos(column * 1.3 + row * 0.9) * 0.018;
|
||||
const edgeMix = row > 9 ? 0.42 : 0;
|
||||
writePoint(
|
||||
positions,
|
||||
colors,
|
||||
pointIndex,
|
||||
[x, y, 0.72 + relayRipple],
|
||||
[0.12 + edgeMix, 0.58 + edgeMix * 0.42, 0.66 - edgeMix * 0.3],
|
||||
);
|
||||
pointIndex += 1;
|
||||
}
|
||||
}
|
||||
|
||||
visibleTracks.forEach((track, trackIndex) => {
|
||||
const uncertaintyX = clamp(Math.sqrt(track.covarianceDiagonalM2.x), 0.08, 0.85);
|
||||
const uncertaintyY = clamp(Math.sqrt(track.covarianceDiagonalM2.y), 0.08, 0.85);
|
||||
const uncertaintyZ = clamp(Math.sqrt(track.covarianceDiagonalM2.z), 0.08, 0.85);
|
||||
const centerX = clamp(track.positionM.x, -6, 6) * 0.55;
|
||||
const centerY = 0.45 + clamp(track.positionM.y, 0, 4) * 0.5;
|
||||
const centerZ = 0.35 - clamp(track.positionM.z, 0, 8) * 0.58;
|
||||
const degraded = track.state === 'degraded';
|
||||
const intensity = 0.58 + clamp(track.confidence, 0, 1) * 0.42;
|
||||
|
||||
for (let sample = 0; sample < LIDAR_POINTS_PER_TRACK; sample += 1) {
|
||||
const normalizedY = 1 - 2 * ((sample + 0.5) / LIDAR_POINTS_PER_TRACK);
|
||||
const radial = Math.sqrt(Math.max(0, 1 - normalizedY * normalizedY));
|
||||
const theta = sample * GOLDEN_ANGLE + trackIndex * 0.73;
|
||||
const shell = 0.5 + ((sample * 37 + trackIndex * 17) % 47) / 94;
|
||||
const directionX = Math.cos(theta) * radial;
|
||||
const directionZ = Math.sin(theta) * radial;
|
||||
|
||||
writePoint(
|
||||
positions,
|
||||
colors,
|
||||
pointIndex,
|
||||
[
|
||||
centerX + directionX * uncertaintyX * shell,
|
||||
centerY + normalizedY * uncertaintyY * shell,
|
||||
centerZ + directionZ * uncertaintyZ * shell,
|
||||
],
|
||||
degraded
|
||||
? [intensity, 0.52 * intensity, 0.16]
|
||||
: [0.22 * intensity, intensity, 0.48 * intensity],
|
||||
);
|
||||
pointIndex += 1;
|
||||
}
|
||||
});
|
||||
|
||||
return {
|
||||
positions,
|
||||
colors,
|
||||
relayPointCount: LIDAR_RELAY_POINT_COUNT,
|
||||
targetPointCount,
|
||||
totalPointCount,
|
||||
};
|
||||
};
|
||||