mirror of
https://github.com/ruvnet/RuView.git
synced 2026-08-26 02:04:55 +00:00
feat: add native iPhone LiDAR sensor and web viewer (#1684)
* feat(ios): add RuView LiDAR frame protocol * feat(ios): capture ARKit scene depth for RuView * feat(ios): stream compact LiDAR frames over websocket * feat(ios): add native LiDAR capture UI * feat(ios): add RuView LiDAR app entrypoint * feat(web): add LiDAR bridge web package * feat(web): decode RuView LiDAR wire frames * feat(web): add local LiDAR websocket relay * feat(web): add LiDAR browser viewer * feat(web): render live LiDAR point cloud * fix(ios): use wall clock time for LiDAR provenance * feat(web): style LiDAR viewer * test(web): add LiDAR codec tests * docs: add iPhone LiDAR integration guide * docs(adr): define iPhone LiDAR sensor bridge * fix(ios): harden and validate LiDAR bridge * fix(ios): qualify LiDAR wire depth type
This commit is contained in:
70
.github/workflows/iphone-lidar.yml
vendored
Normal file
70
.github/workflows/iphone-lidar.yml
vendored
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
name: iPhone LiDAR integration
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
paths:
|
||||||
|
- 'integrations/iphone-lidar/**'
|
||||||
|
- 'docs/adr/ADR-340-iphone-lidar-sensor-bridge.md'
|
||||||
|
- '.github/workflows/iphone-lidar.yml'
|
||||||
|
pull_request:
|
||||||
|
paths:
|
||||||
|
- 'integrations/iphone-lidar/**'
|
||||||
|
- 'docs/adr/ADR-340-iphone-lidar-sensor-bridge.md'
|
||||||
|
- '.github/workflows/iphone-lidar.yml'
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
web:
|
||||||
|
name: Node relay and codec
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
defaults:
|
||||||
|
run:
|
||||||
|
working-directory: integrations/iphone-lidar/web
|
||||||
|
steps:
|
||||||
|
- name: Checkout code
|
||||||
|
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
|
||||||
|
|
||||||
|
- name: Set up Node
|
||||||
|
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
|
||||||
|
with:
|
||||||
|
node-version: '22'
|
||||||
|
cache: npm
|
||||||
|
cache-dependency-path: integrations/iphone-lidar/web/package-lock.json
|
||||||
|
|
||||||
|
- name: Install dependencies
|
||||||
|
run: npm ci --ignore-scripts
|
||||||
|
|
||||||
|
- name: Run tests
|
||||||
|
run: npm test
|
||||||
|
|
||||||
|
- name: Audit runtime dependencies
|
||||||
|
run: npm audit --omit=optional --audit-level=high
|
||||||
|
|
||||||
|
ios:
|
||||||
|
name: iOS 17 compile
|
||||||
|
runs-on: macos-15
|
||||||
|
steps:
|
||||||
|
- name: Checkout code
|
||||||
|
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
|
||||||
|
|
||||||
|
- name: Compile native sources with strict concurrency
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
sdk="$(xcrun --sdk iphoneos --show-sdk-path)"
|
||||||
|
build_dir="$RUNNER_TEMP/ruview-lidar-build"
|
||||||
|
mkdir -p "$build_dir"
|
||||||
|
cd "$build_dir"
|
||||||
|
xcrun swiftc \
|
||||||
|
-parse-as-library \
|
||||||
|
-target arm64-apple-ios17.0 \
|
||||||
|
-sdk "$sdk" \
|
||||||
|
-module-name RuViewLiDAR \
|
||||||
|
-strict-concurrency=complete \
|
||||||
|
-warnings-as-errors \
|
||||||
|
-emit-module \
|
||||||
|
-emit-module-path "$build_dir/RuViewLiDAR.swiftmodule" \
|
||||||
|
-c "$GITHUB_WORKSPACE"/integrations/iphone-lidar/native/RuViewLiDAR/*.swift
|
||||||
59
docs/adr/ADR-340-iphone-lidar-sensor-bridge.md
Normal file
59
docs/adr/ADR-340-iphone-lidar-sensor-bridge.md
Normal file
@@ -0,0 +1,59 @@
|
|||||||
|
# ADR 340: iPhone LiDAR Sensor Bridge
|
||||||
|
|
||||||
|
Status: Proposed
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
RuView needs a low cost mobile geometry sensor that can contribute calibrated spatial observations without coupling the perception substrate to Apple frameworks.
|
||||||
|
|
||||||
|
ARKit exposes rear LiDAR scene depth through `ARFrame.sceneDepth` and `smoothedSceneDepth` on supported devices. Ordinary mobile web pages do not receive this ARKit depth surface directly, so native capture and web visualization must be separated.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Use a two layer architecture.
|
||||||
|
|
||||||
|
1. Native Swift and ARKit perform acquisition.
|
||||||
|
2. A modality neutral wire frame transports geometry into browser tools and, next, the RuView HAL.
|
||||||
|
|
||||||
|
The native client will capture depth, confidence, camera intrinsics, and world tracking pose. RGB imagery is excluded from the default transport.
|
||||||
|
|
||||||
|
The protocol identifier is `ruview.lidar.depth.v1`.
|
||||||
|
|
||||||
|
Depth samples are quantized to UInt16 millimeters for transport. Confidence remains UInt8. The default sender downsamples by two spatially and caps transmission at 15 FPS. Full fidelity depth remains available locally for future on device inference.
|
||||||
|
|
||||||
|
## RuView integration boundary
|
||||||
|
|
||||||
|
The transport must not become a second world model. The production receiver converts each packet into the canonical `ruview-hal::Observation`, then passes it through authenticated sensor identity, provenance, OOD gating, uncertainty aware fusion, spatial memory, and WorldGraph adapters.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
1. `source=live` is valid only for frames produced by an active ARKit session.
|
||||||
|
2. Sequence numbers are monotonic per sensor session.
|
||||||
|
3. Wall clock timestamp is separate from ARKit monotonic frame timing.
|
||||||
|
4. RGB is off by default and requires an explicit higher privacy capability.
|
||||||
|
5. Browser clients consume geometry but are not treated as authoritative sensors.
|
||||||
|
6. Unsupported devices fail closed rather than substituting simulated depth.
|
||||||
|
|
||||||
|
## Performance target
|
||||||
|
|
||||||
|
`[SYNTHETIC]` A 256 x 192 Float32 depth map is about 196 KB before confidence and metadata. Downsampling to 128 x 96 and encoding each sample as two byte depth plus one byte confidence yields about 36.9 KB raw. At 15 FPS the raw sensor payload is about 553 KB/s. Base64 raises this to roughly 737 KB/s before JSON metadata. These values are arithmetic sizing estimates, not device measurements.
|
||||||
|
|
||||||
|
The `[CLAIMED target]` for local network latency is below 150 ms p95. A later binary WebSocket or QUIC transport can remove base64 overhead; the exact end-to-end reduction must be measured before it is claimed.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
The development relay is LAN-facing, requires a random per-run bearer token, bounds message size, and restricts the files it serves. Its default `ws://` transport is not encrypted, so it is not a production trust boundary.
|
||||||
|
|
||||||
|
Production requires WSS, authenticated sensor identity, replay protection, message size limits, per tenant authorization, provenance receipts, and explicit retention policy before persistence.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Benefits include commodity hardware, metric depth, tracked camera pose, rapid room scanning, calibration support for RF sensing, and a practical ground truth source for RuView experiments.
|
||||||
|
|
||||||
|
The main limitation is that Apple provides processed scene depth rather than the underlying raw transient LiDAR waveform. Therefore this implementation supports direct geometry and sensor fusion now, but does not reproduce research systems that require raw multipath time of flight transients for non line of sight reconstruction.
|
||||||
|
|
||||||
|
## Acceptance criteria
|
||||||
|
|
||||||
|
A physical LiDAR capable iPhone must stream live geometry to the browser viewer with monotonically increasing sequence numbers, no RGB payload, valid confidence maps, and below 150 ms p95 local network latency over a 60 second run.
|
||||||
|
|
||||||
|
CI type-checking and simulator runs do not satisfy this criterion. Until a captured physical-device run records the environment and results, the hardware behavior and latency remain unverified.
|
||||||
78
integrations/iphone-lidar/README.md
Normal file
78
integrations/iphone-lidar/README.md
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
# RuView iPhone LiDAR
|
||||||
|
|
||||||
|
This experimental integration provides the native and browser components needed to use a LiDAR-capable iPhone as a RuView geometry sensor. The native source is type-checked against the iOS SDK in CI; physical-device validation is tracked separately below.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```text
|
||||||
|
iPhone LiDAR
|
||||||
|
-> ARKit sceneDepth
|
||||||
|
-> depth + confidence + camera intrinsics + device pose
|
||||||
|
-> compact u16 millimeter wire frame
|
||||||
|
-> WebSocket relay
|
||||||
|
-> browser point cloud
|
||||||
|
-> future RuView HAL / fusion ingest
|
||||||
|
```
|
||||||
|
|
||||||
|
The native path is the sensor. The web path is a receiver and visualization surface. Mobile Safari does not expose ARKit scene depth directly to ordinary web pages, so the browser cannot replace the native capture layer on iPhone today.
|
||||||
|
|
||||||
|
## Native iPhone path
|
||||||
|
|
||||||
|
Create an iOS SwiftUI app target in Xcode, deployment target iOS 17 or newer, then add the files under `native/RuViewLiDAR/` to the target.
|
||||||
|
|
||||||
|
Add this Info.plist value:
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<key>NSCameraUsageDescription</key>
|
||||||
|
<string>RuView uses the camera and LiDAR scanner to capture local depth geometry.</string>
|
||||||
|
```
|
||||||
|
|
||||||
|
Run on a physical LiDAR capable iPhone or iPad. The simulator does not provide LiDAR scene depth.
|
||||||
|
|
||||||
|
The app requests `ARWorldTrackingConfiguration` with `.sceneDepth`, checks `supportsFrameSemantics`, extracts `ARDepthData.depthMap` and `confidenceMap`, and never transmits RGB camera frames.
|
||||||
|
|
||||||
|
## Browser path
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd integrations/iphone-lidar/web
|
||||||
|
npm ci
|
||||||
|
npm test
|
||||||
|
npm start
|
||||||
|
```
|
||||||
|
|
||||||
|
The relay prints a random per-run access token. Open the printed browser URL and set the iPhone endpoint to the printed native URL. They have this form:
|
||||||
|
|
||||||
|
```text
|
||||||
|
http://HOST:8787/?token=TOKEN
|
||||||
|
ws://HOST:8787/ws/lidar?token=TOKEN
|
||||||
|
```
|
||||||
|
|
||||||
|
Set `RUVIEW_LIDAR_TOKEN` to supply the token explicitly. The token only prevents unauthenticated peers from joining the development relay; because `ws://` does not encrypt it, production use requires TLS and `wss://`.
|
||||||
|
|
||||||
|
## Wire format
|
||||||
|
|
||||||
|
Schema: `ruview.lidar.depth.v1`
|
||||||
|
|
||||||
|
Depth is downsampled by 2 in each dimension by default and streamed at a maximum of 15 FPS. Each depth sample is encoded as little endian UInt16 millimeters plus one UInt8 confidence value. `[SYNTHETIC]` Arithmetic sizing reduces the depth payload from roughly 196 KB per 256 x 192 Float32 frame to roughly 37 KB per 128 x 96 frame before base64 and JSON overhead.
|
||||||
|
|
||||||
|
`[SYNTHETIC]` At 15 FPS that is approximately 0.75 MB/s after base64 overhead, versus roughly 8 MB/s for uncompressed Float32 JSON at full resolution. These are sizing estimates, not device or network measurements.
|
||||||
|
|
||||||
|
## Privacy and governance
|
||||||
|
|
||||||
|
The initial implementation labels provenance as `source=live` and `privacyClass=geometry-only`. It sends depth geometry, confidence, camera intrinsics, pose, sequence, and wall clock timestamp. It does not send RGB imagery.
|
||||||
|
|
||||||
|
The development relay requires an ephemeral token and bounds each WebSocket message, but it is not a production trust boundary. Production integration should terminate the WebSocket inside RuView, authenticate the device using the existing sensor identity path, convert each frame into `ruview-hal::Observation`, and attach witness receipts before fusion or persistence.
|
||||||
|
|
||||||
|
## Validation status
|
||||||
|
|
||||||
|
- `[MEASURED]` The committed Node tests cover wire decoding, malformed inputs, relay authentication, static-file restrictions, and live WebSocket forwarding.
|
||||||
|
- `[MEASURED]` GitHub Actions type-checks the native sources with strict concurrency against the iOS 17 SDK.
|
||||||
|
- Physical iPhone capture, end-to-end rendering, confidence-map behavior, and the latency target are not yet measured. A simulator or CI compile does not satisfy the hardware acceptance test.
|
||||||
|
|
||||||
|
## Acceptance test
|
||||||
|
|
||||||
|
1. Run the relay and browser viewer.
|
||||||
|
2. Run the native app on a LiDAR capable iPhone.
|
||||||
|
3. Start LiDAR capture and enable streaming.
|
||||||
|
4. Move the phone through a room.
|
||||||
|
5. Verify the browser shows a changing point cloud, sequence increases monotonically, latency stays below the `[CLAIMED target]` of 150 ms p95 on a local WiFi network, and no RGB payload is present in captured WebSocket frames.
|
||||||
121
integrations/iphone-lidar/native/RuViewLiDAR/ContentView.swift
Normal file
121
integrations/iphone-lidar/native/RuViewLiDAR/ContentView.swift
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
import SwiftUI
|
||||||
|
|
||||||
|
struct ContentView: View {
|
||||||
|
@StateObject private var capture = LiDARCaptureManager()
|
||||||
|
@State private var endpoint = "ws://HOST:8787/ws/lidar?token=TOKEN"
|
||||||
|
@State private var streaming = false
|
||||||
|
@State private var status = "Idle"
|
||||||
|
|
||||||
|
private let streamer = WebSocketStreamer()
|
||||||
|
|
||||||
|
var body: some View {
|
||||||
|
NavigationStack {
|
||||||
|
Form {
|
||||||
|
Section("Sensor") {
|
||||||
|
HStack {
|
||||||
|
Text("State")
|
||||||
|
Spacer()
|
||||||
|
Text(stateText)
|
||||||
|
.foregroundStyle(stateColor)
|
||||||
|
}
|
||||||
|
HStack {
|
||||||
|
Text("Capture FPS")
|
||||||
|
Spacer()
|
||||||
|
Text(capture.framesPerSecond.formatted(.number.precision(.fractionLength(1))))
|
||||||
|
}
|
||||||
|
if let frame = capture.lastFrame {
|
||||||
|
HStack {
|
||||||
|
Text("Depth")
|
||||||
|
Spacer()
|
||||||
|
Text("\(frame.depth.width) x \(frame.depth.height)")
|
||||||
|
}
|
||||||
|
HStack {
|
||||||
|
Text("Sequence")
|
||||||
|
Spacer()
|
||||||
|
Text("\(frame.provenance.sequence)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Button("Start LiDAR") {
|
||||||
|
capture.start(smoothed: false)
|
||||||
|
}
|
||||||
|
.disabled(capture.state == .running)
|
||||||
|
|
||||||
|
Button("Stop") {
|
||||||
|
capture.stop()
|
||||||
|
}
|
||||||
|
.disabled(capture.state != .running)
|
||||||
|
}
|
||||||
|
|
||||||
|
Section("RuView Stream") {
|
||||||
|
TextField("ws://host:port/ws/lidar", text: $endpoint)
|
||||||
|
.textInputAutocapitalization(.never)
|
||||||
|
.autocorrectionDisabled()
|
||||||
|
|
||||||
|
Toggle("Stream geometry", isOn: $streaming)
|
||||||
|
.onChange(of: streaming) { _, enabled in
|
||||||
|
Task {
|
||||||
|
if enabled {
|
||||||
|
do {
|
||||||
|
try await streamer.connect(to: endpoint)
|
||||||
|
status = "Connected"
|
||||||
|
} catch {
|
||||||
|
streaming = false
|
||||||
|
status = error.localizedDescription
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
await streamer.disconnect()
|
||||||
|
status = "Disconnected"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Text(status)
|
||||||
|
.font(.caption)
|
||||||
|
.foregroundStyle(.secondary)
|
||||||
|
}
|
||||||
|
|
||||||
|
Section("Privacy") {
|
||||||
|
Text("This implementation transmits depth geometry, confidence, camera intrinsics, and device pose. RGB camera frames are not transmitted.")
|
||||||
|
.font(.footnote)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
.navigationTitle("RuView LiDAR")
|
||||||
|
.onAppear {
|
||||||
|
capture.onFrame = { frame in
|
||||||
|
guard streaming else { return }
|
||||||
|
Task {
|
||||||
|
do {
|
||||||
|
try await streamer.send(frame, maxFPS: 15, sampleStep: 2)
|
||||||
|
} catch {
|
||||||
|
await MainActor.run {
|
||||||
|
status = error.localizedDescription
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
.onDisappear {
|
||||||
|
capture.stop()
|
||||||
|
Task { await streamer.disconnect() }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private var stateText: String {
|
||||||
|
switch capture.state {
|
||||||
|
case .idle: return "Idle"
|
||||||
|
case .unsupported: return "No LiDAR"
|
||||||
|
case .running: return "Live"
|
||||||
|
case .failed(let message): return "Error: \(message)"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private var stateColor: Color {
|
||||||
|
switch capture.state {
|
||||||
|
case .running: return .green
|
||||||
|
case .failed, .unsupported: return .red
|
||||||
|
case .idle: return .secondary
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,138 @@
|
|||||||
|
import ARKit
|
||||||
|
import CoreVideo
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
@MainActor
|
||||||
|
final class LiDARCaptureManager: NSObject, ObservableObject {
|
||||||
|
enum State: Equatable {
|
||||||
|
case idle
|
||||||
|
case unsupported
|
||||||
|
case running
|
||||||
|
case failed(String)
|
||||||
|
}
|
||||||
|
|
||||||
|
@Published private(set) var state: State = .idle
|
||||||
|
@Published private(set) var lastFrame: RuViewLiDARFrame?
|
||||||
|
@Published private(set) var framesPerSecond: Double = 0
|
||||||
|
|
||||||
|
let session = ARSession()
|
||||||
|
var onFrame: (@MainActor @Sendable (RuViewLiDARFrame) -> Void)?
|
||||||
|
|
||||||
|
private var sequence: UInt64 = 0
|
||||||
|
private var lastTimestamp: TimeInterval?
|
||||||
|
private let processingQueue = DispatchQueue(label: "one.ruv.lidar.capture", qos: .userInitiated)
|
||||||
|
|
||||||
|
override init() {
|
||||||
|
super.init()
|
||||||
|
session.delegate = self
|
||||||
|
session.delegateQueue = processingQueue
|
||||||
|
}
|
||||||
|
|
||||||
|
func start(smoothed: Bool = false) {
|
||||||
|
let configuration = ARWorldTrackingConfiguration()
|
||||||
|
let semantic: ARConfiguration.FrameSemantics = smoothed ? .smoothedSceneDepth : .sceneDepth
|
||||||
|
|
||||||
|
guard ARWorldTrackingConfiguration.supportsFrameSemantics(semantic) else {
|
||||||
|
state = .unsupported
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
configuration.frameSemantics.insert(semantic)
|
||||||
|
configuration.worldAlignment = .gravity
|
||||||
|
session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
|
||||||
|
state = .running
|
||||||
|
}
|
||||||
|
|
||||||
|
func stop() {
|
||||||
|
session.pause()
|
||||||
|
state = .idle
|
||||||
|
}
|
||||||
|
|
||||||
|
nonisolated private func makeFrame(from frame: ARFrame) -> RuViewLiDARFrame? {
|
||||||
|
guard let sceneDepth = frame.sceneDepth ?? frame.smoothedSceneDepth else { return nil }
|
||||||
|
let depthMap = sceneDepth.depthMap
|
||||||
|
let confidenceMap = sceneDepth.confidenceMap
|
||||||
|
|
||||||
|
guard CVPixelBufferLockBaseAddress(depthMap, .readOnly) == kCVReturnSuccess else {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
defer { CVPixelBufferUnlockBaseAddress(depthMap, .readOnly) }
|
||||||
|
|
||||||
|
guard CVPixelBufferGetPixelFormatType(depthMap) == kCVPixelFormatType_DepthFloat32,
|
||||||
|
let depthBase = CVPixelBufferGetBaseAddress(depthMap) else {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
let width = CVPixelBufferGetWidth(depthMap)
|
||||||
|
let height = CVPixelBufferGetHeight(depthMap)
|
||||||
|
let stride = CVPixelBufferGetBytesPerRow(depthMap) / MemoryLayout<Float>.size
|
||||||
|
let pointer = depthBase.assumingMemoryBound(to: Float.self)
|
||||||
|
|
||||||
|
var meters = [Float]()
|
||||||
|
meters.reserveCapacity(width * height)
|
||||||
|
for y in 0..<height {
|
||||||
|
let row = pointer.advanced(by: y * stride)
|
||||||
|
for x in 0..<width {
|
||||||
|
let value = row[x]
|
||||||
|
meters.append(value.isFinite && value > 0 ? value : 0)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var confidence = [UInt8](repeating: 0, count: width * height)
|
||||||
|
if let confidenceMap,
|
||||||
|
CVPixelBufferGetPixelFormatType(confidenceMap) == kCVPixelFormatType_OneComponent8,
|
||||||
|
CVPixelBufferGetWidth(confidenceMap) == width,
|
||||||
|
CVPixelBufferGetHeight(confidenceMap) == height,
|
||||||
|
CVPixelBufferLockBaseAddress(confidenceMap, .readOnly) == kCVReturnSuccess {
|
||||||
|
defer { CVPixelBufferUnlockBaseAddress(confidenceMap, .readOnly) }
|
||||||
|
if let confidenceBase = CVPixelBufferGetBaseAddress(confidenceMap) {
|
||||||
|
let confidenceStride = CVPixelBufferGetBytesPerRow(confidenceMap)
|
||||||
|
let confidencePointer = confidenceBase.assumingMemoryBound(to: UInt8.self)
|
||||||
|
for y in 0..<height {
|
||||||
|
let row = confidencePointer.advanced(by: y * confidenceStride)
|
||||||
|
for x in 0..<width {
|
||||||
|
confidence[y * width + x] = row[x]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return RuViewLiDARFrame(
|
||||||
|
intrinsics: frame.camera.intrinsics,
|
||||||
|
imageResolution: frame.camera.imageResolution,
|
||||||
|
cameraTransform: frame.camera.transform,
|
||||||
|
depthWidth: width,
|
||||||
|
depthHeight: height,
|
||||||
|
depthMeters: meters,
|
||||||
|
confidence: confidence,
|
||||||
|
sequence: 0,
|
||||||
|
timestamp: Date().timeIntervalSince1970
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
extension LiDARCaptureManager: ARSessionDelegate {
|
||||||
|
nonisolated func session(_ session: ARSession, didUpdate frame: ARFrame) {
|
||||||
|
guard let base = makeFrame(from: frame) else { return }
|
||||||
|
let frameTimestamp = frame.timestamp
|
||||||
|
|
||||||
|
Task { @MainActor in
|
||||||
|
sequence &+= 1
|
||||||
|
let corrected = base.assigningSequence(sequence)
|
||||||
|
|
||||||
|
if let previous = lastTimestamp {
|
||||||
|
let delta = frameTimestamp - previous
|
||||||
|
if delta > 0 { framesPerSecond = 1.0 / delta }
|
||||||
|
}
|
||||||
|
lastTimestamp = frameTimestamp
|
||||||
|
lastFrame = corrected
|
||||||
|
onFrame?(corrected)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
nonisolated func session(_ session: ARSession, didFailWithError error: Error) {
|
||||||
|
Task { @MainActor in
|
||||||
|
state = .failed(error.localizedDescription)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
118
integrations/iphone-lidar/native/RuViewLiDAR/RuViewFrame.swift
Normal file
118
integrations/iphone-lidar/native/RuViewLiDAR/RuViewFrame.swift
Normal file
@@ -0,0 +1,118 @@
|
|||||||
|
import Foundation
|
||||||
|
import simd
|
||||||
|
|
||||||
|
struct RuViewLiDARFrame: Codable, Sendable {
|
||||||
|
struct Intrinsics: Codable, Sendable {
|
||||||
|
let fx: Float
|
||||||
|
let fy: Float
|
||||||
|
let cx: Float
|
||||||
|
let cy: Float
|
||||||
|
let imageWidth: Int
|
||||||
|
let imageHeight: Int
|
||||||
|
}
|
||||||
|
|
||||||
|
struct Pose: Codable, Sendable {
|
||||||
|
let matrix: [Float]
|
||||||
|
}
|
||||||
|
|
||||||
|
struct Depth: Codable, Sendable {
|
||||||
|
let width: Int
|
||||||
|
let height: Int
|
||||||
|
let meters: [Float]
|
||||||
|
let confidence: [UInt8]
|
||||||
|
}
|
||||||
|
|
||||||
|
struct Provenance: Codable, Sendable {
|
||||||
|
let sensor: String
|
||||||
|
let source: String
|
||||||
|
let privacyClass: String
|
||||||
|
let sequence: UInt64
|
||||||
|
let timestampNs: UInt64
|
||||||
|
let schema: String
|
||||||
|
}
|
||||||
|
|
||||||
|
let type: String
|
||||||
|
let intrinsics: Intrinsics
|
||||||
|
let pose: Pose
|
||||||
|
let depth: Depth
|
||||||
|
let provenance: Provenance
|
||||||
|
|
||||||
|
init(
|
||||||
|
intrinsics: simd_float3x3,
|
||||||
|
imageResolution: CGSize,
|
||||||
|
cameraTransform: simd_float4x4,
|
||||||
|
depthWidth: Int,
|
||||||
|
depthHeight: Int,
|
||||||
|
depthMeters: [Float],
|
||||||
|
confidence: [UInt8],
|
||||||
|
sequence: UInt64,
|
||||||
|
timestamp: TimeInterval
|
||||||
|
) {
|
||||||
|
self.type = "ruview.lidar.depth.v1"
|
||||||
|
self.intrinsics = Intrinsics(
|
||||||
|
fx: intrinsics.columns.0.x,
|
||||||
|
fy: intrinsics.columns.1.y,
|
||||||
|
cx: intrinsics.columns.2.x,
|
||||||
|
cy: intrinsics.columns.2.y,
|
||||||
|
imageWidth: Int(imageResolution.width),
|
||||||
|
imageHeight: Int(imageResolution.height)
|
||||||
|
)
|
||||||
|
self.pose = Pose(matrix: cameraTransform.columnMajorArray)
|
||||||
|
self.depth = Depth(
|
||||||
|
width: depthWidth,
|
||||||
|
height: depthHeight,
|
||||||
|
meters: depthMeters,
|
||||||
|
confidence: confidence
|
||||||
|
)
|
||||||
|
self.provenance = Provenance(
|
||||||
|
sensor: "apple-arkit-scene-depth",
|
||||||
|
source: "live",
|
||||||
|
privacyClass: "geometry-only",
|
||||||
|
sequence: sequence,
|
||||||
|
timestampNs: UInt64(max(0, timestamp) * 1_000_000_000),
|
||||||
|
schema: "ruview.lidar.depth.v1"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func assigningSequence(_ sequence: UInt64) -> RuViewLiDARFrame {
|
||||||
|
RuViewLiDARFrame(
|
||||||
|
type: type,
|
||||||
|
intrinsics: intrinsics,
|
||||||
|
pose: pose,
|
||||||
|
depth: depth,
|
||||||
|
provenance: Provenance(
|
||||||
|
sensor: provenance.sensor,
|
||||||
|
source: provenance.source,
|
||||||
|
privacyClass: provenance.privacyClass,
|
||||||
|
sequence: sequence,
|
||||||
|
timestampNs: provenance.timestampNs,
|
||||||
|
schema: provenance.schema
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
private init(
|
||||||
|
type: String,
|
||||||
|
intrinsics: Intrinsics,
|
||||||
|
pose: Pose,
|
||||||
|
depth: Depth,
|
||||||
|
provenance: Provenance
|
||||||
|
) {
|
||||||
|
self.type = type
|
||||||
|
self.intrinsics = intrinsics
|
||||||
|
self.pose = pose
|
||||||
|
self.depth = depth
|
||||||
|
self.provenance = provenance
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private extension simd_float4x4 {
|
||||||
|
var columnMajorArray: [Float] {
|
||||||
|
[
|
||||||
|
columns.0.x, columns.0.y, columns.0.z, columns.0.w,
|
||||||
|
columns.1.x, columns.1.y, columns.1.z, columns.1.w,
|
||||||
|
columns.2.x, columns.2.y, columns.2.z, columns.2.w,
|
||||||
|
columns.3.x, columns.3.y, columns.3.z, columns.3.w
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
import SwiftUI
|
||||||
|
|
||||||
|
@main
|
||||||
|
struct RuViewLiDARApp: App {
|
||||||
|
var body: some Scene {
|
||||||
|
WindowGroup {
|
||||||
|
ContentView()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
import Foundation
|
||||||
|
|
||||||
|
actor WebSocketStreamer {
|
||||||
|
enum StreamError: Error {
|
||||||
|
case invalidURL
|
||||||
|
}
|
||||||
|
|
||||||
|
struct WirePacket: Codable {
|
||||||
|
struct Depth: Codable {
|
||||||
|
let width: Int
|
||||||
|
let height: Int
|
||||||
|
let encoding: String
|
||||||
|
let millimetersBase64: String
|
||||||
|
let confidenceBase64: String
|
||||||
|
}
|
||||||
|
|
||||||
|
let type: String
|
||||||
|
let intrinsics: RuViewLiDARFrame.Intrinsics
|
||||||
|
let pose: RuViewLiDARFrame.Pose
|
||||||
|
let depth: Depth
|
||||||
|
let provenance: RuViewLiDARFrame.Provenance
|
||||||
|
}
|
||||||
|
|
||||||
|
private var task: URLSessionWebSocketTask?
|
||||||
|
private let encoder = JSONEncoder()
|
||||||
|
private var lastSentNs: UInt64 = 0
|
||||||
|
|
||||||
|
func connect(to endpoint: String) throws {
|
||||||
|
guard let url = URL(string: endpoint),
|
||||||
|
url.scheme == "ws" || url.scheme == "wss" else {
|
||||||
|
throw StreamError.invalidURL
|
||||||
|
}
|
||||||
|
task?.cancel(with: .goingAway, reason: nil)
|
||||||
|
let socket = URLSession.shared.webSocketTask(with: url)
|
||||||
|
socket.resume()
|
||||||
|
task = socket
|
||||||
|
}
|
||||||
|
|
||||||
|
func disconnect() {
|
||||||
|
task?.cancel(with: .goingAway, reason: nil)
|
||||||
|
task = nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func send(_ frame: RuViewLiDARFrame, maxFPS: UInt64 = 15, sampleStep: Int = 2) async throws {
|
||||||
|
guard let task else { return }
|
||||||
|
|
||||||
|
let timestamp = frame.provenance.timestampNs
|
||||||
|
let minDelta = 1_000_000_000 / max(1, maxFPS)
|
||||||
|
guard timestamp >= lastSentNs + minDelta else { return }
|
||||||
|
lastSentNs = timestamp
|
||||||
|
|
||||||
|
let packet = Self.makeWirePacket(frame, sampleStep: max(1, sampleStep))
|
||||||
|
let data = try encoder.encode(packet)
|
||||||
|
guard let string = String(data: data, encoding: .utf8) else { return }
|
||||||
|
try await task.send(.string(string))
|
||||||
|
}
|
||||||
|
|
||||||
|
static func makeWirePacket(_ frame: RuViewLiDARFrame, sampleStep: Int) -> WirePacket {
|
||||||
|
let step = max(1, sampleStep)
|
||||||
|
let sourceWidth = frame.depth.width
|
||||||
|
let sourceHeight = frame.depth.height
|
||||||
|
let width = (sourceWidth + step - 1) / step
|
||||||
|
let height = (sourceHeight + step - 1) / step
|
||||||
|
|
||||||
|
var millimeters = Data(capacity: width * height * 2)
|
||||||
|
var confidence = Data(capacity: width * height)
|
||||||
|
|
||||||
|
for y in stride(from: 0, to: sourceHeight, by: step) {
|
||||||
|
for x in stride(from: 0, to: sourceWidth, by: step) {
|
||||||
|
let index = y * sourceWidth + x
|
||||||
|
let meters = frame.depth.meters[index]
|
||||||
|
let mm = UInt16(clamping: Int((meters * 1000).rounded()))
|
||||||
|
var littleEndian = mm.littleEndian
|
||||||
|
withUnsafeBytes(of: &littleEndian) { millimeters.append(contentsOf: $0) }
|
||||||
|
confidence.append(frame.depth.confidence[index])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return WirePacket(
|
||||||
|
type: frame.type,
|
||||||
|
intrinsics: frame.intrinsics,
|
||||||
|
pose: frame.pose,
|
||||||
|
depth: WirePacket.Depth(
|
||||||
|
width: width,
|
||||||
|
height: height,
|
||||||
|
encoding: "u16le-mm+u8-confidence",
|
||||||
|
millimetersBase64: millimeters.base64EncodedString(),
|
||||||
|
confidenceBase64: confidence.base64EncodedString()
|
||||||
|
),
|
||||||
|
provenance: frame.provenance
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
108
integrations/iphone-lidar/web/app.mjs
Normal file
108
integrations/iphone-lidar/web/app.mjs
Normal file
@@ -0,0 +1,108 @@
|
|||||||
|
import { decodeLiDARPacket, depthToPointCloud } from './codec.mjs';
|
||||||
|
|
||||||
|
const canvas = document.querySelector('#view');
|
||||||
|
const ctx = canvas.getContext('2d');
|
||||||
|
const status = document.querySelector('#status');
|
||||||
|
const fpsEl = document.querySelector('#fps');
|
||||||
|
const pointsEl = document.querySelector('#points');
|
||||||
|
const seqEl = document.querySelector('#seq');
|
||||||
|
const latencyEl = document.querySelector('#latency');
|
||||||
|
const sensorEl = document.querySelector('#sensor');
|
||||||
|
|
||||||
|
let lastFrameAt = performance.now();
|
||||||
|
let yaw = 0.3;
|
||||||
|
let pitch = -0.15;
|
||||||
|
let scale = 120;
|
||||||
|
|
||||||
|
function connect() {
|
||||||
|
const token = new URLSearchParams(location.search).get('token');
|
||||||
|
if (!token) {
|
||||||
|
status.textContent = 'TOKEN REQUIRED';
|
||||||
|
status.dataset.state = 'warn';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const protocol = location.protocol === 'https:' ? 'wss:' : 'ws:';
|
||||||
|
const socket = new WebSocket(`${protocol}//${location.host}/ws/lidar?token=${encodeURIComponent(token)}`);
|
||||||
|
|
||||||
|
socket.addEventListener('open', () => {
|
||||||
|
status.textContent = 'LIVE';
|
||||||
|
status.dataset.state = 'live';
|
||||||
|
});
|
||||||
|
|
||||||
|
socket.addEventListener('close', () => {
|
||||||
|
status.textContent = 'RECONNECTING';
|
||||||
|
status.dataset.state = 'warn';
|
||||||
|
setTimeout(connect, 1000);
|
||||||
|
});
|
||||||
|
|
||||||
|
socket.addEventListener('message', (event) => {
|
||||||
|
try {
|
||||||
|
const raw = JSON.parse(event.data);
|
||||||
|
const frame = decodeLiDARPacket(raw);
|
||||||
|
const points = depthToPointCloud(frame, 1);
|
||||||
|
render(points);
|
||||||
|
|
||||||
|
const now = performance.now();
|
||||||
|
const delta = now - lastFrameAt;
|
||||||
|
lastFrameAt = now;
|
||||||
|
fpsEl.textContent = delta > 0 ? (1000 / delta).toFixed(1) : '0.0';
|
||||||
|
pointsEl.textContent = points.length.toLocaleString();
|
||||||
|
seqEl.textContent = frame.provenance.sequence;
|
||||||
|
latencyEl.textContent = Math.max(0, Date.now() - Number(frame.provenance.timestampNs / 1_000_000)).toFixed(0);
|
||||||
|
sensorEl.textContent = frame.provenance.sensor;
|
||||||
|
} catch (error) {
|
||||||
|
console.error(error);
|
||||||
|
status.textContent = 'FRAME ERROR';
|
||||||
|
status.dataset.state = 'warn';
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function resize() {
|
||||||
|
const dpr = Math.min(devicePixelRatio || 1, 2);
|
||||||
|
const rect = canvas.getBoundingClientRect();
|
||||||
|
canvas.width = Math.max(1, Math.floor(rect.width * dpr));
|
||||||
|
canvas.height = Math.max(1, Math.floor(rect.height * dpr));
|
||||||
|
ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
function render(points) {
|
||||||
|
resize();
|
||||||
|
const w = canvas.clientWidth;
|
||||||
|
const h = canvas.clientHeight;
|
||||||
|
ctx.clearRect(0, 0, w, h);
|
||||||
|
ctx.fillStyle = '#091018';
|
||||||
|
ctx.fillRect(0, 0, w, h);
|
||||||
|
|
||||||
|
const cy = Math.cos(yaw);
|
||||||
|
const sy = Math.sin(yaw);
|
||||||
|
const cp = Math.cos(pitch);
|
||||||
|
const sp = Math.sin(pitch);
|
||||||
|
|
||||||
|
ctx.fillStyle = '#58e0d2';
|
||||||
|
for (let i = 0; i < points.length; i += 1) {
|
||||||
|
let [x, y, z] = points[i];
|
||||||
|
const rx = x * cy - z * sy;
|
||||||
|
const rz = x * sy + z * cy;
|
||||||
|
const ry = y * cp - rz * sp;
|
||||||
|
const rz2 = y * sp + rz * cp;
|
||||||
|
const perspective = 1 / Math.max(0.35, 1.8 - rz2 * 0.12);
|
||||||
|
const px = w / 2 + rx * scale * perspective;
|
||||||
|
const py = h / 2 + ry * scale * perspective;
|
||||||
|
if (px >= 0 && px < w && py >= 0 && py < h) ctx.fillRect(px, py, 1.4, 1.4);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
canvas.addEventListener('pointermove', (event) => {
|
||||||
|
if (!event.buttons) return;
|
||||||
|
yaw += event.movementX * 0.006;
|
||||||
|
pitch += event.movementY * 0.006;
|
||||||
|
});
|
||||||
|
|
||||||
|
canvas.addEventListener('wheel', (event) => {
|
||||||
|
event.preventDefault();
|
||||||
|
scale = Math.max(40, Math.min(400, scale - event.deltaY * 0.2));
|
||||||
|
}, { passive: false });
|
||||||
|
|
||||||
|
connect();
|
||||||
121
integrations/iphone-lidar/web/codec.mjs
Normal file
121
integrations/iphone-lidar/web/codec.mjs
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
export function decodeLiDARPacket(packet) {
|
||||||
|
if (!packet || packet.type !== 'ruview.lidar.depth.v1') {
|
||||||
|
throw new Error('Unsupported LiDAR packet type');
|
||||||
|
}
|
||||||
|
|
||||||
|
const { depth } = packet;
|
||||||
|
if (!depth || depth.encoding !== 'u16le-mm+u8-confidence') {
|
||||||
|
throw new Error('Unsupported depth encoding');
|
||||||
|
}
|
||||||
|
|
||||||
|
assertPositiveInteger(depth.width, 'depth.width');
|
||||||
|
assertPositiveInteger(depth.height, 'depth.height');
|
||||||
|
assertIntrinsics(packet.intrinsics);
|
||||||
|
if (!packet.pose || !Array.isArray(packet.pose.matrix) || packet.pose.matrix.length !== 16
|
||||||
|
|| packet.pose.matrix.some((value) => !Number.isFinite(value))) {
|
||||||
|
throw new Error('Invalid camera pose');
|
||||||
|
}
|
||||||
|
|
||||||
|
const mmBytes = base64ToBytes(depth.millimetersBase64, 'millimetersBase64');
|
||||||
|
const confidence = base64ToBytes(depth.confidenceBase64, 'confidenceBase64');
|
||||||
|
const expectedPixels = depth.width * depth.height;
|
||||||
|
|
||||||
|
if (!Number.isSafeInteger(expectedPixels) || expectedPixels > 1_000_000) {
|
||||||
|
throw new Error('Depth dimensions exceed the supported pixel limit');
|
||||||
|
}
|
||||||
|
|
||||||
|
if (mmBytes.byteLength !== expectedPixels * 2) {
|
||||||
|
throw new Error(`Depth payload length mismatch: expected ${expectedPixels * 2}, got ${mmBytes.byteLength}`);
|
||||||
|
}
|
||||||
|
if (confidence.byteLength !== expectedPixels) {
|
||||||
|
throw new Error(`Confidence payload length mismatch: expected ${expectedPixels}, got ${confidence.byteLength}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const view = new DataView(mmBytes.buffer, mmBytes.byteOffset, mmBytes.byteLength);
|
||||||
|
const meters = new Float32Array(expectedPixels);
|
||||||
|
for (let i = 0; i < expectedPixels; i += 1) {
|
||||||
|
meters[i] = view.getUint16(i * 2, true) / 1000;
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
...packet,
|
||||||
|
depth: {
|
||||||
|
width: depth.width,
|
||||||
|
height: depth.height,
|
||||||
|
meters,
|
||||||
|
confidence,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function depthToPointCloud(frame, confidenceThreshold = 1) {
|
||||||
|
const { width, height, meters, confidence } = frame.depth;
|
||||||
|
const { fx, fy, cx, cy, imageWidth, imageHeight } = frame.intrinsics;
|
||||||
|
|
||||||
|
assertPositiveInteger(width, 'depth.width');
|
||||||
|
assertPositiveInteger(height, 'depth.height');
|
||||||
|
assertIntrinsics(frame.intrinsics);
|
||||||
|
if (meters.length !== width * height || confidence.length !== width * height) {
|
||||||
|
throw new Error('Decoded depth array length mismatch');
|
||||||
|
}
|
||||||
|
if (!Number.isFinite(confidenceThreshold) || confidenceThreshold < 0 || confidenceThreshold > 255) {
|
||||||
|
throw new Error('Invalid confidence threshold');
|
||||||
|
}
|
||||||
|
|
||||||
|
const sx = width / imageWidth;
|
||||||
|
const sy = height / imageHeight;
|
||||||
|
const scaledFx = fx * sx;
|
||||||
|
const scaledFy = fy * sy;
|
||||||
|
const scaledCx = cx * sx;
|
||||||
|
const scaledCy = cy * sy;
|
||||||
|
|
||||||
|
const points = [];
|
||||||
|
for (let v = 0; v < height; v += 1) {
|
||||||
|
for (let u = 0; u < width; u += 1) {
|
||||||
|
const index = v * width + u;
|
||||||
|
const z = meters[index];
|
||||||
|
if (!Number.isFinite(z) || z <= 0 || confidence[index] < confidenceThreshold) continue;
|
||||||
|
|
||||||
|
const x = ((u - scaledCx) / scaledFx) * z;
|
||||||
|
const y = ((v - scaledCy) / scaledFy) * z;
|
||||||
|
points.push([x, y === 0 ? 0 : -y, -z]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return points;
|
||||||
|
}
|
||||||
|
|
||||||
|
function assertPositiveInteger(value, name) {
|
||||||
|
if (!Number.isSafeInteger(value) || value <= 0) {
|
||||||
|
throw new Error(`${name} must be a positive integer`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function assertIntrinsics(intrinsics) {
|
||||||
|
if (!intrinsics
|
||||||
|
|| !Number.isFinite(intrinsics.fx) || intrinsics.fx <= 0
|
||||||
|
|| !Number.isFinite(intrinsics.fy) || intrinsics.fy <= 0
|
||||||
|
|| !Number.isFinite(intrinsics.cx)
|
||||||
|
|| !Number.isFinite(intrinsics.cy)) {
|
||||||
|
throw new Error('Invalid camera intrinsics');
|
||||||
|
}
|
||||||
|
assertPositiveInteger(intrinsics.imageWidth, 'intrinsics.imageWidth');
|
||||||
|
assertPositiveInteger(intrinsics.imageHeight, 'intrinsics.imageHeight');
|
||||||
|
}
|
||||||
|
|
||||||
|
function base64ToBytes(value, name) {
|
||||||
|
if (typeof value !== 'string' || value.length === 0 || value.length % 4 !== 0
|
||||||
|
|| !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(value)) {
|
||||||
|
throw new Error(`${name} must be canonical base64`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (typeof Buffer !== 'undefined') {
|
||||||
|
return Uint8Array.from(Buffer.from(value, 'base64'));
|
||||||
|
}
|
||||||
|
|
||||||
|
const binary = atob(value);
|
||||||
|
const bytes = new Uint8Array(binary.length);
|
||||||
|
for (let i = 0; i < binary.length; i += 1) {
|
||||||
|
bytes[i] = binary.charCodeAt(i);
|
||||||
|
}
|
||||||
|
return bytes;
|
||||||
|
}
|
||||||
69
integrations/iphone-lidar/web/codec.test.mjs
Normal file
69
integrations/iphone-lidar/web/codec.test.mjs
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
import test from 'node:test';
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { decodeLiDARPacket, depthToPointCloud } from './codec.mjs';
|
||||||
|
|
||||||
|
function b64(bytes) {
|
||||||
|
return Buffer.from(bytes).toString('base64');
|
||||||
|
}
|
||||||
|
|
||||||
|
test('decodes u16 millimeter depth and confidence', () => {
|
||||||
|
const packet = {
|
||||||
|
type: 'ruview.lidar.depth.v1',
|
||||||
|
intrinsics: { fx: 100, fy: 100, cx: 1, cy: 1, imageWidth: 2, imageHeight: 2 },
|
||||||
|
pose: { matrix: Array(16).fill(0) },
|
||||||
|
depth: {
|
||||||
|
width: 2,
|
||||||
|
height: 2,
|
||||||
|
encoding: 'u16le-mm+u8-confidence',
|
||||||
|
millimetersBase64: b64([0xe8,0x03,0xd0,0x07,0xb8,0x0b,0xa0,0x0f]),
|
||||||
|
confidenceBase64: b64([2,2,1,0]),
|
||||||
|
},
|
||||||
|
provenance: { sensor: 'test', source: 'live', privacyClass: 'geometry-only', sequence: 1, timestampNs: 1, schema: 'ruview.lidar.depth.v1' },
|
||||||
|
};
|
||||||
|
|
||||||
|
const frame = decodeLiDARPacket(packet);
|
||||||
|
assert.deepEqual(Array.from(frame.depth.meters), [1,2,3,4]);
|
||||||
|
assert.deepEqual(Array.from(frame.depth.confidence), [2,2,1,0]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('rejects malformed payload length', () => {
|
||||||
|
assert.throws(() => decodeLiDARPacket({
|
||||||
|
type: 'ruview.lidar.depth.v1',
|
||||||
|
intrinsics: { fx: 100, fy: 100, cx: 1, cy: 1, imageWidth: 2, imageHeight: 2 },
|
||||||
|
pose: { matrix: Array(16).fill(0) },
|
||||||
|
depth: { width: 2, height: 2, encoding: 'u16le-mm+u8-confidence', millimetersBase64: b64([1,2]), confidenceBase64: b64([1,1,1,1]) },
|
||||||
|
}), /length mismatch/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('rejects invalid dimensions, intrinsics, pose, and base64', () => {
|
||||||
|
const valid = {
|
||||||
|
type: 'ruview.lidar.depth.v1',
|
||||||
|
intrinsics: { fx: 100, fy: 100, cx: 0, cy: 0, imageWidth: 1, imageHeight: 1 },
|
||||||
|
pose: { matrix: Array(16).fill(0) },
|
||||||
|
depth: {
|
||||||
|
width: 1,
|
||||||
|
height: 1,
|
||||||
|
encoding: 'u16le-mm+u8-confidence',
|
||||||
|
millimetersBase64: b64([0xe8, 0x03]),
|
||||||
|
confidenceBase64: b64([2]),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
assert.throws(() => decodeLiDARPacket({ ...valid, depth: { ...valid.depth, width: 0 } }), /positive integer/);
|
||||||
|
assert.throws(() => decodeLiDARPacket({ ...valid, intrinsics: { ...valid.intrinsics, fx: 0 } }), /intrinsics/);
|
||||||
|
assert.throws(() => decodeLiDARPacket({ ...valid, pose: { matrix: [1] } }), /pose/);
|
||||||
|
assert.throws(() => decodeLiDARPacket({
|
||||||
|
...valid,
|
||||||
|
depth: { ...valid.depth, millimetersBase64: '!!!!' },
|
||||||
|
}), /canonical base64/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('projects depth into a point cloud and honors confidence', () => {
|
||||||
|
const frame = {
|
||||||
|
intrinsics: { fx: 100, fy: 100, cx: 0, cy: 0, imageWidth: 2, imageHeight: 2 },
|
||||||
|
depth: { width: 2, height: 2, meters: Float32Array.from([1,1,1,1]), confidence: Uint8Array.from([2,0,2,0]) },
|
||||||
|
};
|
||||||
|
const points = depthToPointCloud(frame, 1);
|
||||||
|
assert.equal(points.length, 2);
|
||||||
|
assert.deepEqual(points[0], [0, 0, -1]);
|
||||||
|
});
|
||||||
36
integrations/iphone-lidar/web/index.html
Normal file
36
integrations/iphone-lidar/web/index.html
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width,initial-scale=1,viewport-fit=cover" />
|
||||||
|
<title>RuView iPhone LiDAR</title>
|
||||||
|
<link rel="stylesheet" href="./styles.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<main>
|
||||||
|
<header>
|
||||||
|
<div>
|
||||||
|
<p class="eyebrow">RUVIEW SENSOR BRIDGE</p>
|
||||||
|
<h1>iPhone LiDAR</h1>
|
||||||
|
</div>
|
||||||
|
<span id="status">CONNECTING</span>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<section class="metrics">
|
||||||
|
<div><strong id="fps">0.0</strong><span>FPS</span></div>
|
||||||
|
<div><strong id="points">0</strong><span>POINTS</span></div>
|
||||||
|
<div><strong id="seq">0</strong><span>SEQ</span></div>
|
||||||
|
<div><strong id="latency">0</strong><span>MS</span></div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<canvas id="view"></canvas>
|
||||||
|
|
||||||
|
<footer>
|
||||||
|
<span>Geometry only</span>
|
||||||
|
<span>No RGB upload</span>
|
||||||
|
<span id="sensor">Waiting for sensor</span>
|
||||||
|
</footer>
|
||||||
|
</main>
|
||||||
|
<script type="module" src="./app.mjs"></script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
39
integrations/iphone-lidar/web/package-lock.json
generated
Normal file
39
integrations/iphone-lidar/web/package-lock.json
generated
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
{
|
||||||
|
"name": "@ruview/iphone-lidar-web",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"lockfileVersion": 3,
|
||||||
|
"requires": true,
|
||||||
|
"packages": {
|
||||||
|
"": {
|
||||||
|
"name": "@ruview/iphone-lidar-web",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"dependencies": {
|
||||||
|
"ws": "^8.18.3"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=20"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/ws": {
|
||||||
|
"version": "8.21.3",
|
||||||
|
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz",
|
||||||
|
"integrity": "sha512-201TZ/kPWxoPr/OKWjquZR1SWKXcvxdH+e1xrx89b3YbmzLMFCLfnaG1HFIgWzJOEWZ7MvpK++odZufgYR50Rw==",
|
||||||
|
"license": "MIT",
|
||||||
|
"engines": {
|
||||||
|
"node": ">=10.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"bufferutil": "^4.0.1",
|
||||||
|
"utf-8-validate": ">=5.0.2"
|
||||||
|
},
|
||||||
|
"peerDependenciesMeta": {
|
||||||
|
"bufferutil": {
|
||||||
|
"optional": true
|
||||||
|
},
|
||||||
|
"utf-8-validate": {
|
||||||
|
"optional": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
16
integrations/iphone-lidar/web/package.json
Normal file
16
integrations/iphone-lidar/web/package.json
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
{
|
||||||
|
"name": "@ruview/iphone-lidar-web",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.1.0",
|
||||||
|
"type": "module",
|
||||||
|
"scripts": {
|
||||||
|
"start": "node relay.mjs",
|
||||||
|
"test": "node --test"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=20"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"ws": "^8.18.3"
|
||||||
|
}
|
||||||
|
}
|
||||||
166
integrations/iphone-lidar/web/relay.mjs
Normal file
166
integrations/iphone-lidar/web/relay.mjs
Normal file
@@ -0,0 +1,166 @@
|
|||||||
|
import { randomBytes, timingSafeEqual } from 'node:crypto';
|
||||||
|
import http from 'node:http';
|
||||||
|
import { readFile } from 'node:fs/promises';
|
||||||
|
import { extname, join } from 'node:path';
|
||||||
|
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||||
|
import { WebSocket, WebSocketServer } from 'ws';
|
||||||
|
|
||||||
|
const root = fileURLToPath(new URL('.', import.meta.url));
|
||||||
|
const staticFiles = new Set(['index.html', 'app.mjs', 'codec.mjs', 'styles.css']);
|
||||||
|
const maxPayloadBytes = 2_000_000;
|
||||||
|
|
||||||
|
function constantTimeEqual(left, right) {
|
||||||
|
const leftBytes = Buffer.from(left, 'utf8');
|
||||||
|
const rightBytes = Buffer.from(right, 'utf8');
|
||||||
|
return leftBytes.length === rightBytes.length && timingSafeEqual(leftBytes, rightBytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
function rejectUpgrade(socket, status, message) {
|
||||||
|
const body = `${message}\n`;
|
||||||
|
socket.end(
|
||||||
|
`HTTP/1.1 ${status}\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\nContent-Length: ${Buffer.byteLength(body)}\r\n\r\n${body}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createLiDARRelay({ token, rootDirectory = root } = {}) {
|
||||||
|
const accessToken = token || randomBytes(24).toString('hex');
|
||||||
|
const clients = new Set();
|
||||||
|
|
||||||
|
const server = http.createServer(async (req, res) => {
|
||||||
|
if (req.method !== 'GET' && req.method !== 'HEAD') {
|
||||||
|
res.writeHead(405, { allow: 'GET, HEAD' }).end();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
let pathname;
|
||||||
|
try {
|
||||||
|
pathname = decodeURIComponent(new URL(req.url || '/', 'http://localhost').pathname);
|
||||||
|
} catch {
|
||||||
|
res.writeHead(400).end('bad path');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const filename = pathname === '/' ? 'index.html' : pathname.slice(1);
|
||||||
|
if (!staticFiles.has(filename)) {
|
||||||
|
res.writeHead(404).end('not found');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const data = await readFile(join(rootDirectory, filename));
|
||||||
|
const contentType = {
|
||||||
|
'.html': 'text/html; charset=utf-8',
|
||||||
|
'.mjs': 'text/javascript; charset=utf-8',
|
||||||
|
'.css': 'text/css; charset=utf-8',
|
||||||
|
}[extname(filename)] || 'application/octet-stream';
|
||||||
|
res.writeHead(200, {
|
||||||
|
'content-type': contentType,
|
||||||
|
'cache-control': 'no-store',
|
||||||
|
'content-security-policy': "default-src 'self'; connect-src 'self' ws: wss:; img-src 'self'; style-src 'self'; base-uri 'none'; frame-ancestors 'none'",
|
||||||
|
'referrer-policy': 'no-referrer',
|
||||||
|
'x-content-type-options': 'nosniff',
|
||||||
|
});
|
||||||
|
if (req.method === 'HEAD') res.end();
|
||||||
|
else res.end(data);
|
||||||
|
} catch {
|
||||||
|
res.writeHead(404).end('not found');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const wss = new WebSocketServer({ noServer: true, maxPayload: maxPayloadBytes });
|
||||||
|
|
||||||
|
server.on('upgrade', (req, socket, head) => {
|
||||||
|
let url;
|
||||||
|
try {
|
||||||
|
url = new URL(req.url || '/', 'http://localhost');
|
||||||
|
} catch {
|
||||||
|
rejectUpgrade(socket, '400 Bad Request', 'bad websocket URL');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (url.pathname !== '/ws/lidar') {
|
||||||
|
rejectUpgrade(socket, '404 Not Found', 'not found');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const suppliedToken = url.searchParams.get('token') || '';
|
||||||
|
if (!constantTimeEqual(suppliedToken, accessToken)) {
|
||||||
|
rejectUpgrade(socket, '401 Unauthorized', 'valid LiDAR relay token required');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
wss.handleUpgrade(req, socket, head, (websocket) => {
|
||||||
|
wss.emit('connection', websocket, req);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
wss.on('connection', (socket) => {
|
||||||
|
clients.add(socket);
|
||||||
|
socket.on('close', () => clients.delete(socket));
|
||||||
|
socket.on('error', () => socket.terminate());
|
||||||
|
socket.on('message', (data, isBinary) => {
|
||||||
|
if (isBinary) return;
|
||||||
|
|
||||||
|
let packet;
|
||||||
|
try {
|
||||||
|
packet = JSON.parse(data.toString());
|
||||||
|
} catch {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (packet?.type !== 'ruview.lidar.depth.v1') return;
|
||||||
|
|
||||||
|
for (const peer of clients) {
|
||||||
|
if (peer !== socket && peer.readyState === WebSocket.OPEN) {
|
||||||
|
peer.send(data.toString());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
accessToken,
|
||||||
|
server,
|
||||||
|
async listen(port = 0, host = '127.0.0.1') {
|
||||||
|
await new Promise((resolve, reject) => {
|
||||||
|
server.once('error', reject);
|
||||||
|
server.listen(port, host, () => {
|
||||||
|
server.off('error', reject);
|
||||||
|
resolve();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
return server.address();
|
||||||
|
},
|
||||||
|
async close() {
|
||||||
|
for (const peer of clients) peer.terminate();
|
||||||
|
await new Promise((resolve) => wss.close(resolve));
|
||||||
|
if (server.listening) {
|
||||||
|
await new Promise((resolve, reject) => {
|
||||||
|
server.close((error) => (error ? reject(error) : resolve()));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function configuredPort(value) {
|
||||||
|
const parsed = Number(value);
|
||||||
|
if (!Number.isSafeInteger(parsed) || parsed < 1 || parsed > 65_535) {
|
||||||
|
throw new Error(`PORT must be an integer from 1 to 65535; received ${value}`);
|
||||||
|
}
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
const directInvocation = process.argv[1]
|
||||||
|
&& pathToFileURL(process.argv[1]).href === import.meta.url;
|
||||||
|
|
||||||
|
if (directInvocation) {
|
||||||
|
const port = configuredPort(process.env.PORT || '8787');
|
||||||
|
const host = process.env.HOST || '0.0.0.0';
|
||||||
|
const relay = createLiDARRelay({ token: process.env.RUVIEW_LIDAR_TOKEN });
|
||||||
|
await relay.listen(port, host);
|
||||||
|
const encodedToken = encodeURIComponent(relay.accessToken);
|
||||||
|
console.log(`RuView iPhone LiDAR relay listening on ${host}:${port}`);
|
||||||
|
console.log(`Browser: http://<host>:${port}/?token=${encodedToken}`);
|
||||||
|
console.log(`Native endpoint: ws://<host>:${port}/ws/lidar?token=${encodedToken}`);
|
||||||
|
}
|
||||||
73
integrations/iphone-lidar/web/relay.test.mjs
Normal file
73
integrations/iphone-lidar/web/relay.test.mjs
Normal file
@@ -0,0 +1,73 @@
|
|||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import http from 'node:http';
|
||||||
|
import test from 'node:test';
|
||||||
|
import { WebSocket } from 'ws';
|
||||||
|
import { createLiDARRelay } from './relay.mjs';
|
||||||
|
|
||||||
|
function connect(url) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const socket = new WebSocket(url);
|
||||||
|
socket.once('open', () => resolve(socket));
|
||||||
|
socket.once('error', reject);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function request(url) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
http.get(url, (response) => {
|
||||||
|
response.resume();
|
||||||
|
response.once('end', () => resolve(response));
|
||||||
|
}).once('error', reject);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
test('relay requires a token, limits static files, and forwards LiDAR frames', async (t) => {
|
||||||
|
const token = 'test-token-for-relay';
|
||||||
|
const relay = createLiDARRelay({ token });
|
||||||
|
const address = await relay.listen(0, '127.0.0.1');
|
||||||
|
const httpBase = `http://127.0.0.1:${address.port}`;
|
||||||
|
const wsBase = `ws://127.0.0.1:${address.port}/ws/lidar`;
|
||||||
|
const sockets = [];
|
||||||
|
|
||||||
|
t.after(async () => {
|
||||||
|
for (const socket of sockets) socket.terminate();
|
||||||
|
await relay.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
const indexResponse = await request(`${httpBase}/?token=${token}`);
|
||||||
|
assert.equal(indexResponse.statusCode, 200);
|
||||||
|
assert.match(indexResponse.headers['content-security-policy'], /frame-ancestors 'none'/);
|
||||||
|
|
||||||
|
const sourceResponse = await request(`${httpBase}/relay.mjs`);
|
||||||
|
assert.equal(sourceResponse.statusCode, 404);
|
||||||
|
|
||||||
|
await new Promise((resolve, reject) => {
|
||||||
|
const unauthorized = new WebSocket(wsBase);
|
||||||
|
unauthorized.once('unexpected-response', (_request, response) => {
|
||||||
|
assert.equal(response.statusCode, 401);
|
||||||
|
response.resume();
|
||||||
|
resolve();
|
||||||
|
});
|
||||||
|
unauthorized.once('open', () => reject(new Error('unauthorized websocket opened')));
|
||||||
|
unauthorized.once('error', () => {});
|
||||||
|
});
|
||||||
|
|
||||||
|
const sender = await connect(`${wsBase}?token=${encodeURIComponent(token)}`);
|
||||||
|
const receiver = await connect(`${wsBase}?token=${encodeURIComponent(token)}`);
|
||||||
|
sockets.push(sender, receiver);
|
||||||
|
|
||||||
|
const received = new Promise((resolve, reject) => {
|
||||||
|
const timer = setTimeout(() => reject(new Error('timed out waiting for relayed frame')), 2_000);
|
||||||
|
receiver.once('message', (data, isBinary) => {
|
||||||
|
clearTimeout(timer);
|
||||||
|
resolve({ data, isBinary });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
const frame = { type: 'ruview.lidar.depth.v1', provenance: { sequence: 7 } };
|
||||||
|
sender.send(JSON.stringify(frame));
|
||||||
|
const message = await received;
|
||||||
|
|
||||||
|
assert.equal(message.isBinary, false);
|
||||||
|
assert.deepEqual(JSON.parse(message.data.toString()), frame);
|
||||||
|
});
|
||||||
1
integrations/iphone-lidar/web/styles.css
Normal file
1
integrations/iphone-lidar/web/styles.css
Normal file
@@ -0,0 +1 @@
|
|||||||
|
*{box-sizing:border-box}body{margin:0;background:#05080d;color:#e8f1f5;font-family:ui-monospace,SFMono-Regular,Menlo,monospace}main{min-height:100vh;padding:18px;display:grid;grid-template-rows:auto auto 1fr auto;gap:14px}header{display:flex;align-items:end;justify-content:space-between}h1{margin:0;font-size:clamp(28px,6vw,54px)}.eyebrow{margin:0 0 6px;color:#58e0d2;font-size:11px;letter-spacing:.16em}#status{border:1px solid #2a3946;border-radius:999px;padding:7px 11px;font-size:11px}#status[data-state=live]{color:#58e0d2;border-color:#58e0d2}.metrics{display:grid;grid-template-columns:repeat(4,1fr);gap:8px}.metrics div{background:#0b1219;border:1px solid #17222d;border-radius:10px;padding:10px}.metrics strong{display:block;font-size:18px}.metrics span,footer{font-size:10px;color:#8aa0af}canvas{width:100%;height:100%;min-height:55vh;border-radius:14px;border:1px solid #17222d;background:#091018;touch-action:none}footer{display:flex;gap:16px;flex-wrap:wrap}@media(max-width:640px){.metrics{grid-template-columns:repeat(2,1fr)}canvas{min-height:58vh}}
|
||||||
Reference in New Issue
Block a user