diff --git a/.github/workflows/consumer-nlos-ci.yml b/.github/workflows/consumer-nlos-ci.yml index 63b5506e..93e17c5b 100644 --- a/.github/workflows/consumer-nlos-ci.yml +++ b/.github/workflows/consumer-nlos-ci.yml @@ -210,6 +210,7 @@ jobs: "under 1.5 seconds", "under 100 milliseconds", "zero runtime dependency", + "Three.js", "physical iPhone validation pending", ) for phrase in required_adr_contract: @@ -225,6 +226,8 @@ jobs: "nlos-evidence-state", "nlos-synthetic-watermark", "nlos-beta-setup", + "nlos-lidar-point-cloud", + "nlos-cloud-target-count", ) for token in required_flow_contract: assert token in flow, f"missing Maestro UI contract token: {token}" @@ -276,6 +279,7 @@ jobs: "overview-390x844.png", "synthetic-390x844.png", "setup-390x844.png", + "point-cloud-390x844.png", ) for name in expected_screenshots: path = screenshot_dir / name @@ -286,7 +290,7 @@ jobs: print( "Validated ADR 342, Maestro UI contract, five evidence states, " - "and three 390x844 PNG baselines" + "and four 390x844 PNG baselines" ) PY diff --git a/docs/adr/ADR-342-cognitum-inspired-mobile-instrument-ui.md b/docs/adr/ADR-342-cognitum-inspired-mobile-instrument-ui.md index 51636d8d..a527e4af 100644 --- a/docs/adr/ADR-342-cognitum-inspired-mobile-instrument-ui.md +++ b/docs/adr/ADR-342-cognitum-inspired-mobile-instrument-ui.md @@ -5,7 +5,7 @@ | **Status** | Proposed; software implementation and deterministic review evidence are part of a stacked UI-only pull request, while physical-device validation remains operator-gated | | **Date** | 2026-08-23 | | **Owners** | RuView Labs mobile, iOS, design, accessibility, privacy, security, and research maintainers | -| **Scope** | Presentation, responsive layout, evidence-state projection, accessibility semantics, mobile end-to-end review flow, and screenshot baselines for the Expo and native SwiftUI NLOS clients | +| **Scope** | Presentation, responsive layout, evidence-state projection, accessible target and LiDAR point-cloud views, mobile end-to-end review flow, and screenshot baselines for the Expo and native SwiftUI NLOS clients | | **Depends on** | ADR-295, ADR-318, ADR-319, ADR-330, ADR-340, ADR-341 | | **Stacked pull request base** | `feat/consumer-nlos-ruview`, the head branch for pull request 1687; retarget to `main` after the dependency merges | | **Implementation boundary** | `ui/mobile` presentation and tests, `ui/ios-nlos/App` presentation, the mobile UI screenshot baselines, and the UI validation portion of `.github/workflows/consumer-nlos-ci.yml` | @@ -73,7 +73,8 @@ design reviewers, privacy and security reviewers, and the release operator. existing validation and freshness boundary. 3. Setup, provenance, privacy, and interpretation guidance in a stable reading order. -4. Reviewable screenshot baselines for overview, synthetic, and setup states. +4. Reviewable screenshot baselines for overview, synthetic, setup, and Three.js + point-cloud states. 5. A production-browser Playwright end-to-end suite and a mobile Maestro flow that verify navigation, state visibility, synthetic watermarking, setup affordances, and the primary local action. @@ -115,6 +116,11 @@ UNVERIFIED` only when an authenticated live attempt is active. It resolves to setup without collecting credentials or diagnostics. 7. Motion is decorative and bounded. Reduced-motion mode removes continuous orbit, sweep, pulse, and parallax effects without removing information. +8. The web point-cloud mode uses the already installed Three.js runtime. It + renders only deterministic relay samples and target returns derived from + tracks that have already passed the display gate. The native surface uses an + equivalent SwiftUI Canvas projection. Both identify the view as a rendered + reconstruction rather than raw iPhone LiDAR output. ### Performance budgets @@ -129,6 +135,10 @@ UNVERIFIED` only when an authenticated live attempt is active. It resolves to client, sensor subscription, or background timer. 5. Screenshot rendering is deterministic: fixed viewport, fixed fixtures, reduced motion, local assets, and no dependency on a live endpoint. +6. Point-cloud geometry is bounded to 288 schematic relay points plus 96 + target-return points for each of at most 16 validated tracks, or 1,824 total + points. Browser device pixel ratio is capped at 1.5. GPU resources, animation + frames, resize observers, and pointer listeners are released on unmount. The timing budgets are targets until a physical iPhone report records device, OS, build commit, build mode, browser, network conditioning, repetition count, @@ -188,6 +198,12 @@ onRender: render no hidden-target geometry render safe recovery guidance for the current state + if selectedView == pointCloud: + build bounded deterministic cloud from the same displayable tracks only + render schematic relay points as non-evidence context + label the surface as a reconstruction view, not raw LiDAR + use Three.js WebGL on web and SwiftUI Canvas on native + preserve setup, privacy, interpretation, and feedback paths disable decorative motion when reduced motion is requested ``` @@ -245,7 +261,9 @@ hold. | Component | Responsibility | May change in this decision | Must not change | |---|---|---|---| | Expo NLOS screen and presentation components | Responsive hierarchy, tokens, evidence-state label, accessible controls, instrument composition | Yes | Store semantics, service requests, credential lifecycle | +| Expo Three.js point-cloud adapter | Deterministic bounded BufferGeometry, local WebGL rendering, orbit input, reduced motion, resize, and disposal | Yes | Raw sensor access, network fetches, provenance promotion, persistence | | Native SwiftUI shell and presentation helpers | Equivalent hierarchy, tokens, state labels, Dynamic Type, reduced motion, hit targets | Yes | `AppModel`, Apple capability probe, stream guard, Keychain, diagnostics | +| Native SwiftUI point-cloud projection | Accessible Canvas rendering from the same displayable track list | Yes | Three.js embedding, WebView, raw ARKit depth, or new entitlement | | Existing Expo store and hooks | Authoritative source, freshness, frame, rejection, and connection inputs | No | Any sensing or transport behavior | | Existing native model and core packages | Authoritative connection, frame, track, diagnostic, and capability inputs | No | Any sensing, validation, persistence, or permission behavior | | Playwright production-browser suite | Black-box responsive navigation, overflow assertions, local state actions, and screenshot reproduction | Yes | Live endpoint use, credentials, or physical-device claims | @@ -275,7 +293,8 @@ pixel identity between React Native Web and SwiftUI is not. ```text existing validated domain state -> pure evidence-state projection - -> local presentation tree + -> displayable track allowlist + -> local target view or deterministic point-cloud projection -> pixels and accessibility semantics ``` @@ -311,6 +330,13 @@ screenshots as cross-platform review references, not as proof of native pixel output. Require an Xcode 26 build gate and a named physical-device review before accepting native behavior or performance claims. +For the point-cloud surface, use Three.js directly in the Expo web build because +it is already a locked runtime dependency and can render bounded BufferGeometry +without a remote asset. Use SwiftUI Canvas for the native counterpart. Both +consume only the existing fail-closed displayable track list. Do not add +`expo-gl`, React Three Fiber, a WebView, a raw point-cloud schema, or a new sensor +adapter in this UI-only decision. + ## Alternatives with quantified tradeoffs Alternatives are scored from 1, poor, to 5, strong. Weighted score is out of @@ -332,6 +358,17 @@ zero runtime dependency. The cost is duplicated token maintenance. The control is a screenshot review plus identical state names and accessibility assertions on both platforms. +### Point-cloud renderer alternatives + +The same weights apply to the narrower renderer choice. + +| Alternative | Clarity 25% | Epistemic safety 25% | Delivery cost 15% | Native accessibility 20% | Maintainability 15% | Weighted score | Decision | +|---|---:|---:|---:|---:|---:|---:|---| +| Three.js web plus SwiftUI Canvas native | 5 | 5 | 4 | 4 | 4 | 4.50 | Selected because it satisfies the requested Three.js view while preserving platform-native accessibility and adding zero runtime dependency | +| Three.js inside a native WebView | 5 | 4 | 3 | 2 | 4 | 3.70 | Rejected because focus, Dynamic Type, startup, memory, and lifecycle behavior are weaker | +| Add `expo-gl` and React Three Fiber for native Expo | 5 | 4 | 2 | 3 | 3 | 3.60 | Rejected because new native dependencies and build risk are disproportionate to a UI-only change | +| Static SVG or Canvas cloud on every platform | 3 | 5 | 5 | 4 | 5 | 4.30 | Rejected as the sole implementation because it does not provide the requested interactive Three.js web experience | + ## Accessibility 1. Every interactive element has a descriptive label, role or trait, and a hit @@ -350,6 +387,10 @@ on both platforms. the accessibility tree. 8. The synthetic watermark has a programmatic label in addition to its visual rendering. +9. The WebGL canvas is hidden from the accessibility tree and its containing + view exposes one concise label with evidence freshness, hypothesis count, + and gated target-return count. Orbit gestures are optional and convey no + exclusive information. ## Security and privacy @@ -360,9 +401,11 @@ on both platforms. ephemeral credential in memory, while the approved native client uses its existing Keychain boundary. UI code cannot log, render, snapshot, export, or persist the credential. -3. Screenshot and E2E fixtures are synthetic and contain no CSI, image, depth, - point-cloud, location, person, device identifier, token, endpoint, or private - diagnostic data. +3. Screenshot and E2E fixtures are synthetic and contain no CSI, image, raw + depth, captured point cloud, location, person, device identifier, token, + endpoint, or private diagnostic data. The displayed point cloud is generated + locally from the deterministic synthetic track fixture and schematic relay + geometry. 4. Stale, malformed, unauthenticated, replayed, unknown, or contradictory input fails closed through the existing validation layer and the state projection. 5. The synthetic watermark remains visible in every synthetic visualization @@ -374,6 +417,9 @@ on both platforms. credentials, diagnostics, or scene context. 8. CI uses read-only repository permission and receives no signing, App Store Connect, endpoint, or test-user secret for the UI contract. +9. Three.js receives no URL, shader text, texture, model, worker, or external + input. It allocates local typed arrays from already gated numeric track data + and releases WebGL resources when the view changes or unmounts. ## Performance budgets @@ -400,12 +446,15 @@ claim. A threshold without a completed physical run remains `TARGET`. stale, disconnected, contradiction, and synthetic watermark cases. 3. Recompose the Expo NLOS screen at the reference viewport and add the production-browser Playwright suite plus Maestro mobile flow. -4. Recompose the native SwiftUI surface using equivalent semantics, Dynamic +4. Add the bounded Three.js point-cloud adapter, deterministic generator, + accessible fallback, and fail-closed zero-return tests. +5. Recompose the native SwiftUI surface using equivalent semantics, Dynamic Type, and reduced-motion behavior. -5. Capture deterministic Expo overview, synthetic, and setup baselines. -6. Run type, lint, unit, bundle, Swift package, Xcode 26, contract, accessibility, +6. Capture deterministic Expo overview, synthetic, setup, and point-cloud + baselines. +7. Run type, lint, unit, bundle, Swift package, Xcode 26, contract, accessibility, security, and diff-scope review gates. -7. Complete the named physical iPhone gate before calling native behavior or +8. Complete the named physical iPhone gate before calling native behavior or mobile performance validated. ### Failure handling @@ -416,7 +465,8 @@ claim. A threshold without a completed physical run remains `TARGET`. 2. If a frame becomes stale, remove its geometry on the same committed state transition. Reconnection alone does not restore it. 3. If decorative rendering fails, retain plain text state, controls, privacy, - and provenance. Decoration cannot block operation. + and provenance. Decoration cannot block operation. If WebGL creation or its + context fails, identify the static fallback without promoting evidence. 4. If the viewport overflows, stack secondary content and remove nonessential decoration before reducing essential text or hit areas. 5. If the performance target fails, profile render count, SVG complexity, @@ -436,15 +486,16 @@ claim. A threshold without a completed physical run remains `TARGET`. | Requirement | Automated evidence | Human or physical evidence | Pass condition | |---|---|---|---| | UI-only scope | Pull-request diff allowlist and service or model tests | Reviewer checks no sensing, transport, permissions, persistence, or retention diff | Zero out-of-scope behavior changes | -| Cognitum-inspired, RuView-owned language | Three screenshot baselines and token review | Product reviewer compares hierarchy and general visual language | Requested characteristics present; zero Cognitum branding or private assets | +| Cognitum-inspired, RuView-owned language | Four screenshot baselines and token review | Product reviewer compares hierarchy and general visual language | Requested characteristics present; zero Cognitum branding or private assets | | Five honest states | Unit tests plus `nlos-evidence-state` Playwright and Maestro selectors | Reviewer verifies wording and fail-closed hierarchy | All five exact labels are representable; unknown never becomes verified | | Synthetic safety | Unit and E2E assertions for `nlos-synthetic-watermark` | Screenshot review of synthetic baseline | Label and watermark both visible | +| LiDAR point-cloud UI | Deterministic generator tests, exact point budgets, Three.js canvas readiness, reduced-motion capture, and zero-return fail-closed assertion | Browser and native visual review plus physical iPhone GPU check | WebGL renders 288 schematic relay points plus 96 returns per gated track; unverified states render zero target returns; native equivalent is labeled as a projection | | 390 by 844 layout | PNG dimension contract and browser overflow assertion | Screenshot review | Exact dimensions and no horizontal overflow | | 44 point targets | Style and interaction assertions where supported | iPhone accessibility inspector and manual target review | Every interactive hit rectangle is at least 44 by 44 points | | Accessibility | Semantic tests, lint, and reduced-motion fixture | VoiceOver, Dynamic Type at 200 percent, contrast, and reduced-motion review | State and primary actions remain understandable and operable | | Initial usable render under 1.5 seconds | Production build and instrumentation contract | Named modern iPhone run with recorded conditions | Measured usable render is below 1.5 seconds | | Local interaction p95 under 100 milliseconds | Deterministic action loop instrumentation | Named iPhone run over at least 30 repetitions | Recorded p95 is below 100 milliseconds | -| Screenshot baselines | CI validates PNG format, names, dimensions, and synthetic marker contract | Reviewer approves overview, synthetic, and setup compositions | All three approved and traceable to the commit | +| Screenshot baselines | CI validates PNG format, names, dimensions, and synthetic marker contract | Reviewer approves overview, synthetic, setup, and point-cloud compositions | All four approved and traceable to the commit | | No security or privacy expansion | Unit suite, dependency audit, secret scan, workflow permission review | Privacy and security diff review | No new collection, permission, storage, network, secret, or retention path | | Native equivalence | Swift tests and Xcode 26 unsigned simulator build | Named physical iPhone inspection | Equivalent hierarchy and state semantics; no pixel-equivalence claim | @@ -486,14 +537,18 @@ full iPhone validation. motion and deterministic fixtures. Assert `scrollWidth <= clientWidth`. 3. Capture and review: `docs/screenshots/consumer-nlos-mobile-ui/overview-390x844.png`, - `docs/screenshots/consumer-nlos-mobile-ui/synthetic-390x844.png`, and - `docs/screenshots/consumer-nlos-mobile-ui/setup-390x844.png`. + `docs/screenshots/consumer-nlos-mobile-ui/synthetic-390x844.png`, + `docs/screenshots/consumer-nlos-mobile-ui/setup-390x844.png`, and + `docs/screenshots/consumer-nlos-mobile-ui/point-cloud-390x844.png`. 4. Assert each PNG is exactly 390 by 844. Assert the synthetic baseline and E2E flow preserve the `SYNTHETIC` label and watermark. -5. Exercise `SYNTHETIC`, `LIVE VERIFIED`, `LIVE UNVERIFIED`, `STALE`, and +5. Select LiDAR cloud, assert the Three.js canvas reports ready, assert the + deterministic fixture renders 96 gated target returns, and confirm the + 390-wide cloud has no horizontal overflow. +6. Exercise `SYNTHETIC`, `LIVE VERIFIED`, `LIVE UNVERIFIED`, `STALE`, and `DISCONNECTED` projections. Assert stale, unverified, and disconnected states - render no hidden-target geometry. -6. Run Swift package tests and the Xcode 26 unsigned simulator build. Confirm + render no hidden-target geometry or point-cloud target returns. +7. Run Swift package tests and the Xcode 26 unsigned simulator build. Confirm zero changes to core packages, sensing, transport, credentials, permissions, diagnostic schema, or retention. diff --git a/docs/screenshots/consumer-nlos-mobile-ui/README.md b/docs/screenshots/consumer-nlos-mobile-ui/README.md index 3aff93b2..4ce42ddd 100644 --- a/docs/screenshots/consumer-nlos-mobile-ui/README.md +++ b/docs/screenshots/consumer-nlos-mobile-ui/README.md @@ -3,12 +3,15 @@ These images are deterministic review captures from the production Expo web export at a 390 by 844 viewport. The Playwright flow navigates the real mobile application, exercises the synthetic replay control, verifies provenance and -watermark requirements, and writes the PNG files in this directory. +watermark requirements, opens the deterministic Three.js LiDAR point cloud, +and writes the PNG files in this directory. The captures contain only the disconnected state, governed setup copy, and the -built in synthetic fixture. They use no live endpoint, credential, sensor data, -person data, or remote asset. They are not native iPhone screenshots, physical -LiDAR evidence, performance evidence, or proof of optical NLOS capability. +built in synthetic fixture. The point cloud is generated locally from gated +track hypotheses and schematic relay geometry. It is not a raw sensor point +cloud. The captures use no live endpoint, credential, sensor data, person data, +or remote asset. They are not native iPhone screenshots, physical LiDAR +evidence, performance evidence, or proof of optical NLOS capability. Regenerate them from `ui/mobile` with `npm run e2e:web`. The executable design and evidence contract is documented in diff --git a/docs/screenshots/consumer-nlos-mobile-ui/overview-390x844.png b/docs/screenshots/consumer-nlos-mobile-ui/overview-390x844.png index 7483a005..4c7f5dfe 100644 Binary files a/docs/screenshots/consumer-nlos-mobile-ui/overview-390x844.png and b/docs/screenshots/consumer-nlos-mobile-ui/overview-390x844.png differ diff --git a/docs/screenshots/consumer-nlos-mobile-ui/point-cloud-390x844.png b/docs/screenshots/consumer-nlos-mobile-ui/point-cloud-390x844.png new file mode 100644 index 00000000..e46ea355 Binary files /dev/null and b/docs/screenshots/consumer-nlos-mobile-ui/point-cloud-390x844.png differ diff --git a/docs/screenshots/consumer-nlos-mobile-ui/setup-390x844.png b/docs/screenshots/consumer-nlos-mobile-ui/setup-390x844.png index 32aa9dba..9136c1e1 100644 Binary files a/docs/screenshots/consumer-nlos-mobile-ui/setup-390x844.png and b/docs/screenshots/consumer-nlos-mobile-ui/setup-390x844.png differ diff --git a/docs/screenshots/consumer-nlos-mobile-ui/synthetic-390x844.png b/docs/screenshots/consumer-nlos-mobile-ui/synthetic-390x844.png index 277462ac..76454274 100644 Binary files a/docs/screenshots/consumer-nlos-mobile-ui/synthetic-390x844.png and b/docs/screenshots/consumer-nlos-mobile-ui/synthetic-390x844.png differ diff --git a/ui/ios-nlos/App/ContentView.swift b/ui/ios-nlos/App/ContentView.swift index 684cf1d8..e9664afa 100644 --- a/ui/ios-nlos/App/ContentView.swift +++ b/ui/ios-nlos/App/ContentView.swift @@ -10,6 +10,7 @@ struct ContentView: View { @State private var exportConsent = false @State private var diagnosticURL: URL? @State private var exportError: String? + @State private var spatialMode: TrackCanvasMode = .targets var body: some View { NavigationStack { @@ -186,13 +187,24 @@ struct ContentView: View { private var visualizationCard: some View { instrumentCard(eyebrow: "02 / SPATIAL", title: "Hidden target hypotheses", accent: RuViewTheme.cyan) { - Text("Only validated, fresh tracks are shown. Uncertainty rings represent the reported position covariance.") + Picker("Spatial visualization", selection: $spatialMode) { + Text("TRACKS").tag(TrackCanvasMode.targets) + Text("POINT CLOUD").tag(TrackCanvasMode.pointCloud) + } + .pickerStyle(.segmented) + .accessibilityIdentifier("nlos-spatial-mode") + + Text( + spatialMode == .pointCloud + ? "The point cloud is a deterministic rendering of gated reconstruction tracks and relay geometry. It is not raw iPhone LiDAR output." + : "Only validated, fresh tracks are shown. Uncertainty rings represent the reported position covariance." + ) .font(.caption) .foregroundStyle(RuViewTheme.textSecondary) .fixedSize(horizontal: false, vertical: true) ZStack { - TrackCanvas(tracks: displayableTracks) + TrackCanvas(tracks: displayableTracks, mode: spatialMode) .frame(height: 310) .privacySensitive() diff --git a/ui/ios-nlos/App/TrackCanvas.swift b/ui/ios-nlos/App/TrackCanvas.swift index d7f7560e..7f6a26c1 100644 --- a/ui/ios-nlos/App/TrackCanvas.swift +++ b/ui/ios-nlos/App/TrackCanvas.swift @@ -2,8 +2,16 @@ import Foundation import RuViewNLOSCore import SwiftUI +enum TrackCanvasMode: String, CaseIterable, Identifiable { + case targets + case pointCloud + + var id: String { rawValue } +} + struct TrackCanvas: View { let tracks: [NLOSTrack] + let mode: TrackCanvasMode var body: some View { Canvas { context, size in @@ -14,20 +22,24 @@ struct TrackCanvas: View { with: .color(Color(red: 0.014, green: 0.029, blue: 0.043)) ) drawGrid(context: &context, size: size) - drawRadar(context: &context, size: size) + if mode == .pointCloud { + drawPointCloud(context: &context, size: size) + } else { + drawRadar(context: &context, size: size) - let radiusMeters = max( - 5, - min(100, tracks.flatMap { [abs($0.positionM.x), abs($0.positionM.z)] }.max() ?? 5) - ) - - for track in tracks { - draw( - track: track, - context: &context, - size: size, - radiusMeters: radiusMeters + let radiusMeters = max( + 5, + min(100, tracks.flatMap { [abs($0.positionM.x), abs($0.positionM.z)] }.max() ?? 5) ) + + for track in tracks { + draw( + track: track, + context: &context, + size: size, + radiusMeters: radiusMeters + ) + } } } .clipShape(RoundedRectangle(cornerRadius: 16)) @@ -36,7 +48,84 @@ struct TrackCanvas: View { .stroke(Color.cyan.opacity(0.24), lineWidth: 1) } .shadow(color: Color.cyan.opacity(0.08), radius: 16) - .accessibilityHidden(true) + .accessibilityElement(children: .ignore) + .accessibilityLabel( + mode == .pointCloud + ? "Projected LiDAR reconstruction cloud with \(tracks.count) gated hidden target hypotheses" + : "Plan view with \(tracks.count) gated hidden target hypotheses" + ) + } + + private func drawPointCloud(context: inout GraphicsContext, size: CGSize) { + let cyan = Color(red: 0.129, green: 0.831, blue: 0.906) + let green = Color(red: 0.345, green: 0.949, blue: 0.545) + let orange = Color(red: 1.000, green: 0.714, blue: 0.361) + + for row in 0..<10 { + for column in 0..<16 { + let x = size.width * (0.08 + CGFloat(column) / 17) + let floorY = size.height * (0.58 + CGFloat(row) * 0.032) + let wallY = size.height * (0.12 + CGFloat(row) * 0.039) + let wallX = x + CGFloat(row - 5) * 1.8 + let floorOpacity = 0.28 + Double(row) * 0.025 + let wallColor = row > 7 ? orange : cyan + context.fill( + Path(ellipseIn: CGRect(x: x - 1.1, y: floorY - 1.1, width: 2.2, height: 2.2)), + with: .color(cyan.opacity(floorOpacity)) + ) + context.fill( + Path(ellipseIn: CGRect(x: wallX - 1.15, y: wallY - 1.15, width: 2.3, height: 2.3)), + with: .color(wallColor.opacity(0.52)) + ) + } + } + + let goldenAngle = Double.pi * (3 - sqrt(5.0)) + for (trackIndex, track) in tracks.prefix(16).enumerated() { + let centerX = size.width / 2 + + CGFloat(max(-6, min(6, track.positionM.x))) * size.width * 0.055 + - CGFloat(max(0, min(8, track.positionM.z))) * size.width * 0.012 + let centerY = size.height * 0.63 + - CGFloat(max(0, min(4, track.positionM.y))) * size.height * 0.10 + + CGFloat(max(0, min(8, track.positionM.z))) * size.height * 0.018 + let radiusX = max(8, min(34, CGFloat(sqrt(track.covarianceDiagonalM2.x)) * 28)) + let radiusY = max(8, min(38, CGFloat(sqrt(track.covarianceDiagonalM2.y)) * 30)) + let color = track.state == .degraded ? orange : green + + for sample in 0..<72 { + let normalizedY = 1 - 2 * ((Double(sample) + 0.5) / 72) + let radial = sqrt(max(0, 1 - normalizedY * normalizedY)) + let theta = Double(sample) * goldenAngle + Double(trackIndex) * 0.73 + let shell = 0.5 + Double((sample * 37 + trackIndex * 17) % 47) / 94 + let point = CGPoint( + x: centerX + CGFloat(cos(theta) * radial * shell) * radiusX, + y: centerY + CGFloat(normalizedY * shell) * radiusY + ) + let diameter: CGFloat = sample.isMultiple(of: 5) ? 3.2 : 2.1 + context.fill( + Path(ellipseIn: CGRect( + x: point.x - diameter / 2, + y: point.y - diameter / 2, + width: diameter, + height: diameter + )), + with: .color(color.opacity(0.82)) + ) + } + } + + context.draw( + Text("LIDAR POINT CLOUD / NATIVE SWIFTUI CANVAS") + .font(.caption2.bold().monospaced()) + .foregroundColor(cyan), + at: CGPoint(x: size.width / 2, y: 18) + ) + context.draw( + Text("\(tracks.count * 72) GATED TARGET RETURNS") + .font(.caption2.bold().monospaced()) + .foregroundColor(Color.white.opacity(0.64)), + at: CGPoint(x: size.width / 2, y: size.height - 16) + ) } private func draw( diff --git a/ui/mobile/e2e/nlos_mobile_ui.yaml b/ui/mobile/e2e/nlos_mobile_ui.yaml index b858be63..5964bc07 100644 --- a/ui/mobile/e2e/nlos_mobile_ui.yaml +++ b/ui/mobile/e2e/nlos_mobile_ui.yaml @@ -21,6 +21,22 @@ name: RuView NLOS instrument UI evidence flow id: "nlos-synthetic-watermark" - assertVisible: id: "nlos-evidence-state" +- scrollUntilVisible: + element: + id: "nlos-view-cloud" + direction: DOWN + timeout: 10000 +- tapOn: + id: "nlos-view-cloud" +- scrollUntilVisible: + element: + id: "nlos-lidar-point-cloud" + direction: DOWN + timeout: 10000 +- assertVisible: + id: "nlos-lidar-point-cloud" +- assertVisible: + id: "nlos-cloud-target-count" - scrollUntilVisible: element: id: "nlos-beta-setup" diff --git a/ui/mobile/e2e/web/nlos.mobile.spec.ts b/ui/mobile/e2e/web/nlos.mobile.spec.ts index 32c90b97..4aa4a039 100644 --- a/ui/mobile/e2e/web/nlos.mobile.spec.ts +++ b/ui/mobile/e2e/web/nlos.mobile.spec.ts @@ -106,4 +106,28 @@ test.describe('RuView NLOS mobile instrument UI', () => { await capture(page, 'synthetic-390x844.png'); }); + + test('renders a gated Three.js LiDAR point cloud without horizontal overflow', async ({ page }) => { + await openNlos(page); + const replay = page.getByRole('button', { name: 'USE SYNTHETIC REPLAY' }); + await replay.scrollIntoViewIfNeeded(); + await replay.click(); + + await page.getByTestId('nlos-view-cloud').click(); + const cloud = page.getByTestId('nlos-lidar-point-cloud'); + await cloud.scrollIntoViewIfNeeded(); + await expect(cloud).toBeVisible(); + await expect(page.getByTestId('nlos-lidar-point-cloud-canvas')).toHaveAttribute('data-ready', 'true'); + await expect(page.getByTestId('nlos-cloud-target-count')).toHaveText('96'); + await expect(page.getByText('THREE.JS / WEBGL')).toBeVisible(); + await expect(page.getByTestId('nlos-synthetic-watermark')).toBeVisible(); + + const dimensions = await cloud.evaluate((element) => ({ + viewport: element.clientWidth, + content: element.scrollWidth, + })); + expect(dimensions.content).toBeLessThanOrEqual(dimensions.viewport + 1); + + await capture(page, 'point-cloud-390x844.png'); + }); }); diff --git a/ui/mobile/playwright.config.ts b/ui/mobile/playwright.config.ts index 1a67131a..cc8e1ea6 100644 --- a/ui/mobile/playwright.config.ts +++ b/ui/mobile/playwright.config.ts @@ -23,7 +23,13 @@ export default defineConfig({ launchOptions: process.env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH ? { executablePath: process.env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH, - args: ['--no-sandbox', '--disable-dev-shm-usage'], + args: [ + '--no-sandbox', + '--disable-dev-shm-usage', + '--use-gl=angle', + '--use-angle=swiftshader-webgl', + '--enable-unsafe-swiftshader', + ], } : undefined, }, diff --git a/ui/mobile/src/__tests__/screens/NLOSScreen.test.tsx b/ui/mobile/src/__tests__/screens/NLOSScreen.test.tsx index 27001a1a..071b67b0 100644 --- a/ui/mobile/src/__tests__/screens/NLOSScreen.test.tsx +++ b/ui/mobile/src/__tests__/screens/NLOSScreen.test.tsx @@ -10,6 +10,11 @@ import { NLOS_FEEDBACK_URL, } from '@/screens/NLOSScreen/BetaSetupCard'; import { resolveNlosEvidenceState } from '@/screens/NLOSScreen/ProvenancePanel'; +import { + buildLidarPointCloud, + LIDAR_POINTS_PER_TRACK, + LIDAR_RELAY_POINT_COUNT, +} from '@/screens/NLOSScreen/lidarPointCloud'; const mockSafeAreaInsets = { top: 0, right: 0, bottom: 0, left: 0 }; @@ -241,6 +246,8 @@ describe('NLOSScreen', () => { expect(screen.getByTestId('nlos-evidence-state').props.children).toBe('LIVE UNVERIFIED'); expect(screen.getByTestId('nlos-track-count').props.children).toBe(0); expect(screen.queryByText(live.tracks[0].trackId)).toBeNull(); + fireEvent.press(screen.getByTestId('nlos-view-cloud')); + expect(screen.getByTestId('nlos-cloud-target-count').props.children).toBe(0); Object.assign(mockNlosResult, { frame: { ...live, source: 'replay' }, @@ -293,6 +300,18 @@ describe('NLOSScreen', () => { expect(resolveNlosEvidenceState(usbFrame, 'fresh', 'live')).toBe('LIVE VERIFIED'); }); + it('builds a deterministic bounded LiDAR reconstruction cloud from gated tracks', () => { + const first = buildLidarPointCloud(syntheticFrame.tracks); + const second = buildLidarPointCloud(syntheticFrame.tracks); + + expect(first.relayPointCount).toBe(LIDAR_RELAY_POINT_COUNT); + expect(first.targetPointCount).toBe(syntheticFrame.tracks.length * LIDAR_POINTS_PER_TRACK); + expect(first.totalPointCount).toBe(first.relayPointCount + first.targetPointCount); + expect(Array.from(first.positions)).toEqual(Array.from(second.positions)); + expect(Array.from(first.colors)).toEqual(Array.from(second.colors)); + expect(Array.from(first.positions).every(Number.isFinite)).toBe(true); + }); + it('keeps privacy, setup, explainer, and feedback controls visible in the screen tree', () => { const { NLOSScreen } = require('@/screens/NLOSScreen'); render(); @@ -311,5 +330,11 @@ describe('NLOSScreen', () => { expect(screen.getByTestId('nlos-view-plan').props.accessibilityState.selected).toBe(true); fireEvent.press(screen.getByTestId('nlos-view-perspective')); expect(screen.getByTestId('nlos-view-perspective').props.accessibilityState.selected).toBe(true); + fireEvent.press(screen.getByTestId('nlos-view-cloud')); + expect(screen.getByTestId('nlos-view-cloud').props.accessibilityState.selected).toBe(true); + expect(screen.getByTestId('nlos-lidar-point-cloud')).toBeTruthy(); + expect(screen.getByTestId('nlos-cloud-target-count').props.children).toBe( + syntheticFrame.tracks.length * LIDAR_POINTS_PER_TRACK, + ); }); }); diff --git a/ui/mobile/src/screens/NLOSScreen/HiddenTargetVisualization.tsx b/ui/mobile/src/screens/NLOSScreen/HiddenTargetVisualization.tsx index 33ef64a2..e6ece677 100644 --- a/ui/mobile/src/screens/NLOSScreen/HiddenTargetVisualization.tsx +++ b/ui/mobile/src/screens/NLOSScreen/HiddenTargetVisualization.tsx @@ -3,8 +3,9 @@ import { View } from 'react-native'; import Svg, { Circle, Ellipse, Line, Polygon, Rect, Text as SvgText } from 'react-native-svg'; import { instrumentColors } from '@/components/InstrumentPanel'; import type { NlosFreshness, NlosTrack } from '@/types/nlos'; +import { LidarPointCloud } from './LidarPointCloud'; -export type NlosViewMode = 'plan' | 'perspective'; +export type NlosViewMode = 'plan' | 'perspective' | 'cloud'; interface HiddenTargetVisualizationProps { tracks: NlosTrack[]; @@ -119,6 +120,10 @@ export const HiddenTargetVisualization = memo(({ ); const displayWidth = Math.max(260, Math.min(width, 560)); + if (mode === 'cloud') { + return ; + } + return ( ( + `rgb(${Math.round(red * 255)}, ${Math.round(green * 255)}, ${Math.round(blue * 255)})` +); + +export const LidarPointCloud = memo(({ tracks, freshness, width }: LidarPointCloudProps) => { + const cloud = useMemo(() => buildLidarPointCloud(tracks), [tracks]); + const points = useMemo(() => { + const projected: ProjectedPoint[] = []; + for (let index = 0; index < cloud.totalPointCount; index += 1) { + const isRelay = index < LIDAR_RELAY_POINT_COUNT; + if ((isRelay && index % 3 !== 0) || (!isRelay && index % 2 !== 0)) continue; + const offset = index * 3; + const x = cloud.positions[offset]; + const y = cloud.positions[offset + 1]; + const z = cloud.positions[offset + 2]; + projected.push({ + x: 180 + x * 34 - z * 8, + y: 220 - y * 46 + z * 7, + radius: isRelay ? 1.15 : 1.8, + color: rgb(cloud.colors[offset], cloud.colors[offset + 1], cloud.colors[offset + 2]), + }); + } + return projected; + }, [cloud]); + const displayWidth = Math.max(260, Math.min(width, 560)); + + return ( + + ); +}); + +LidarPointCloud.displayName = 'LidarPointCloud'; + +const styles = StyleSheet.create({ + metricRow: { + position: 'absolute', + left: 24, + bottom: 24, + flexDirection: 'row', + alignItems: 'baseline', + gap: 8, + }, + metricValue: { color: instrumentColors.green, fontSize: 14, lineHeight: 16 }, + metricLabel: { color: instrumentColors.textSecondary, fontSize: 8, letterSpacing: 0.8 }, +}); diff --git a/ui/mobile/src/screens/NLOSScreen/LidarPointCloud.web.tsx b/ui/mobile/src/screens/NLOSScreen/LidarPointCloud.web.tsx new file mode 100644 index 00000000..599ce4fc --- /dev/null +++ b/ui/mobile/src/screens/NLOSScreen/LidarPointCloud.web.tsx @@ -0,0 +1,393 @@ +import { memo, useCallback, useEffect, useMemo, useRef, useState } from 'react'; +import { StyleSheet, View } from 'react-native'; +import * as THREE from 'three'; +import { instrumentColors } from '@/components/InstrumentPanel'; +import { ThemedText } from '@/components/ThemedText'; +import type { NlosFreshness, NlosTrack } from '@/types/nlos'; +import { buildLidarPointCloud } from './lidarPointCloud'; + +interface LidarPointCloudProps { + tracks: NlosTrack[]; + freshness: NlosFreshness; + width: number; +} + +type RendererState = 'initializing' | 'ready' | 'fallback'; + +const CANVAS_ASPECT_RATIO = 360 / 260; + +interface WebSceneState { + geometry: THREE.BufferGeometry; + material: THREE.PointsMaterial; + points: THREE.Points; + render: () => void; +} + +const disposeMaterial = (material: THREE.Material | THREE.Material[]) => { + if (Array.isArray(material)) material.forEach((entry) => entry.dispose()); + else material.dispose(); +}; + +const rgb = (red: number, green: number, blue: number) => ( + `rgb(${Math.round(red * 255)}, ${Math.round(green * 255)}, ${Math.round(blue * 255)})` +); + +export const LidarPointCloud = memo(({ tracks, freshness, width }: LidarPointCloudProps) => { + const hostRef = useRef(null); + const sceneRef = useRef(null); + const [rendererState, setRendererState] = useState('initializing'); + const cloud = useMemo(() => buildLidarPointCloud(tracks), [tracks]); + const displayWidth = Math.max(260, Math.min(width, 560)); + const fallbackPoints = useMemo(() => { + const height = displayWidth / CANVAS_ASPECT_RATIO; + const points: Array<{ color: string; left: number; top: number }> = []; + for (let index = 0; index < cloud.totalPointCount; index += 1) { + const isRelay = index < cloud.relayPointCount; + if ((isRelay && index % 3 !== 0) || (!isRelay && index % 2 !== 0)) continue; + const offset = index * 3; + const x = cloud.positions[offset]; + const y = cloud.positions[offset + 1]; + const z = cloud.positions[offset + 2]; + points.push({ + left: ((180 + x * 34 - z * 8) / 360) * displayWidth, + top: ((220 - y * 46 + z * 7) / 260) * height, + color: rgb(cloud.colors[offset], cloud.colors[offset + 1], cloud.colors[offset + 2]), + }); + } + return points; + }, [cloud, displayWidth]); + const assignHost = useCallback((node: unknown) => { + hostRef.current = node as HTMLDivElement | null; + }, []); + + useEffect(() => { + const host = hostRef.current; + if (!host) return undefined; + + let animationFrame = 0; + let resizeObserver: ResizeObserver | null = null; + let disposed = false; + let renderer: THREE.WebGLRenderer | null = null; + const listeners: Array<() => void> = []; + + try { + renderer = new THREE.WebGLRenderer({ + alpha: false, + antialias: false, + powerPreference: 'high-performance', + }); + renderer.setClearColor(0x071017, 1); + renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 1.5)); + renderer.outputColorSpace = THREE.SRGBColorSpace; + renderer.domElement.dataset.testid = 'nlos-lidar-point-cloud-canvas'; + renderer.domElement.dataset.ready = 'false'; + renderer.domElement.setAttribute('aria-hidden', 'true'); + renderer.domElement.style.position = 'absolute'; + renderer.domElement.style.inset = '0'; + renderer.domElement.style.zIndex = '0'; + renderer.domElement.style.touchAction = 'none'; + host.appendChild(renderer.domElement); + + const scene = new THREE.Scene(); + scene.background = new THREE.Color(0x071017); + scene.fog = new THREE.FogExp2(0x071017, 0.055); + const camera = new THREE.PerspectiveCamera(42, CANVAS_ASPECT_RATIO, 0.1, 40); + let yaw = 0.55; + let pitch = 0.34; + let radius = 8.6; + let dragging = false; + let pointerX = 0; + let pointerY = 0; + + const geometry = new THREE.BufferGeometry(); + geometry.setAttribute('position', new THREE.BufferAttribute(new Float32Array(0), 3)); + geometry.setAttribute('color', new THREE.BufferAttribute(new Float32Array(0), 3)); + const material = new THREE.PointsMaterial({ + blending: THREE.AdditiveBlending, + depthWrite: false, + opacity: 0.92, + size: 0.065, + sizeAttenuation: true, + transparent: true, + vertexColors: true, + }); + const points = new THREE.Points(geometry, material); + scene.add(points); + + const floorGrid = new THREE.GridHelper(8, 16, 0x174957, 0x102934); + floorGrid.position.y = -0.04; + floorGrid.position.z = -1.25; + scene.add(floorGrid); + + const relayPlane = new THREE.GridHelper(8, 16, 0xffb65c, 0x16414b); + relayPlane.rotation.x = Math.PI / 2; + relayPlane.position.set(0, 1.55, 0.72); + scene.add(relayPlane); + + const sensor = new THREE.Mesh( + new THREE.OctahedronGeometry(0.11, 0), + new THREE.MeshBasicMaterial({ color: 0x24d3e5, wireframe: true }), + ); + sensor.position.set(0, 0.18, 2.45); + scene.add(sensor); + + const rayGeometry = new THREE.BufferGeometry().setFromPoints([ + sensor.position, + new THREE.Vector3(0, 1.2, 0.72), + new THREE.Vector3(0.9, 1.1, -2.1), + ]); + const rayMaterial = new THREE.LineBasicMaterial({ color: 0x58f28b, opacity: 0.42, transparent: true }); + const ray = new THREE.Line(rayGeometry, rayMaterial); + scene.add(ray); + + const sizeRenderer = () => { + if (!renderer || disposed) return; + const nextWidth = Math.max(1, host.clientWidth || 360); + const nextHeight = Math.max(1, host.clientHeight || 260); + camera.aspect = nextWidth / nextHeight; + camera.updateProjectionMatrix(); + renderer.setSize(nextWidth, nextHeight, false); + }; + const renderScene = () => { + if (!renderer || disposed) return; + const horizontalRadius = radius * Math.cos(pitch); + camera.position.set( + Math.sin(yaw) * horizontalRadius, + 1.25 + Math.sin(pitch) * radius, + Math.cos(yaw) * horizontalRadius - 0.65, + ); + camera.lookAt(0, 1.15, -1.05); + renderer.render(scene, camera); + }; + + sceneRef.current = { geometry, material, points, render: renderScene }; + + const onPointerDown = (event: PointerEvent) => { + dragging = true; + pointerX = event.clientX; + pointerY = event.clientY; + renderer?.domElement.setPointerCapture?.(event.pointerId); + }; + const onPointerMove = (event: PointerEvent) => { + if (!dragging) return; + yaw += (event.clientX - pointerX) * 0.008; + pitch = THREE.MathUtils.clamp(pitch + (event.clientY - pointerY) * 0.006, -0.1, 0.85); + pointerX = event.clientX; + pointerY = event.clientY; + renderScene(); + }; + const onPointerUp = (event: PointerEvent) => { + dragging = false; + renderer?.domElement.releasePointerCapture?.(event.pointerId); + }; + const onWheel = (event: WheelEvent) => { + event.preventDefault(); + radius = THREE.MathUtils.clamp(radius + event.deltaY * 0.008, 4.8, 13); + renderScene(); + }; + const onContextLost = (event: Event) => { + event.preventDefault(); + setRendererState('fallback'); + }; + const onContextRestored = () => { + setRendererState('ready'); + renderScene(); + }; + + const canvas = renderer.domElement; + canvas.addEventListener('pointerdown', onPointerDown); + canvas.addEventListener('pointermove', onPointerMove); + canvas.addEventListener('pointerup', onPointerUp); + canvas.addEventListener('pointercancel', onPointerUp); + canvas.addEventListener('wheel', onWheel, { passive: false }); + canvas.addEventListener('webglcontextlost', onContextLost); + canvas.addEventListener('webglcontextrestored', onContextRestored); + listeners.push( + () => canvas.removeEventListener('pointerdown', onPointerDown), + () => canvas.removeEventListener('pointermove', onPointerMove), + () => canvas.removeEventListener('pointerup', onPointerUp), + () => canvas.removeEventListener('pointercancel', onPointerUp), + () => canvas.removeEventListener('wheel', onWheel), + () => canvas.removeEventListener('webglcontextlost', onContextLost), + () => canvas.removeEventListener('webglcontextrestored', onContextRestored), + ); + + sizeRenderer(); + renderScene(); + canvas.dataset.ready = 'true'; + setRendererState('ready'); + + if (typeof ResizeObserver !== 'undefined') { + resizeObserver = new ResizeObserver(() => { + sizeRenderer(); + renderScene(); + }); + resizeObserver.observe(host); + } else { + window.addEventListener('resize', sizeRenderer); + listeners.push(() => window.removeEventListener('resize', sizeRenderer)); + } + + const reduceMotion = window.matchMedia?.('(prefers-reduced-motion: reduce)').matches ?? false; + if (!reduceMotion) { + const animate = () => { + if (disposed) return; + if (!dragging) yaw += 0.0009; + renderScene(); + animationFrame = window.requestAnimationFrame(animate); + }; + animationFrame = window.requestAnimationFrame(animate); + } + + return () => { + disposed = true; + sceneRef.current = null; + window.cancelAnimationFrame(animationFrame); + resizeObserver?.disconnect(); + listeners.forEach((remove) => remove()); + scene.remove(points, floorGrid, relayPlane, sensor, ray); + geometry.dispose(); + material.dispose(); + floorGrid.geometry.dispose(); + disposeMaterial(floorGrid.material); + relayPlane.geometry.dispose(); + disposeMaterial(relayPlane.material); + sensor.geometry.dispose(); + disposeMaterial(sensor.material); + rayGeometry.dispose(); + rayMaterial.dispose(); + renderer?.dispose(); + if (canvas.parentNode === host) host.removeChild(canvas); + }; + } catch { + sceneRef.current = null; + setRendererState('fallback'); + return () => { + disposed = true; + window.cancelAnimationFrame(animationFrame); + resizeObserver?.disconnect(); + listeners.forEach((remove) => remove()); + renderer?.dispose(); + const canvas = renderer?.domElement; + if (canvas?.parentNode === host) host.removeChild(canvas); + }; + } + }, []); + + useEffect(() => { + const state = sceneRef.current; + if (!state) return; + + const currentPosition = state.geometry.getAttribute('position') as THREE.BufferAttribute; + if (currentPosition.count === cloud.totalPointCount) { + (currentPosition.array as Float32Array).set(cloud.positions); + currentPosition.needsUpdate = true; + const currentColor = state.geometry.getAttribute('color') as THREE.BufferAttribute; + (currentColor.array as Float32Array).set(cloud.colors); + currentColor.needsUpdate = true; + } else { + const nextGeometry = new THREE.BufferGeometry(); + nextGeometry.setAttribute('position', new THREE.BufferAttribute(cloud.positions.slice(), 3)); + nextGeometry.setAttribute('color', new THREE.BufferAttribute(cloud.colors.slice(), 3)); + state.geometry.dispose(); + state.geometry = nextGeometry; + state.points.geometry = nextGeometry; + } + state.geometry.computeBoundingSphere(); + state.material.opacity = freshness === 'fresh' ? 0.92 : 0.34; + state.material.needsUpdate = true; + state.render(); + }, [cloud, freshness]); + + return ( +