mirror of
https://github.com/ruvnet/RuView.git
synced 2026-08-26 10:05:54 +00:00
docs: deep research + ADR-324/325 on ESP32-S3 micro-LLM inference
Adds a source-cited research survey of the 2026 ESP32-S3 micro-LLM demonstrations (slvDev esp32-ai, DaveBben esp32-llm, llama2.c, Gemma-3n Per-Layer Embeddings) with all external numbers tagged CLAIMED, plus: - ADR-324: micro-LLM inference is a bounded research spike on dedicated 16MB-flash companion hardware and an explicit production non-goal for CSI node firmware (flash-impossible on the 8MB/4MB fleet, core/PSRAM contention with sensing, no task capability at TinyStories scale). - ADR-325: design investigation into flash-resident memory-mapped quantized weight streaming for RuView's own edge models, with a measured-benchmark-first gate and provenance requirements. No firmware, partition, or wire changes; documentation only. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01KCYFXcPDG6yTCxwefb6iy6
This commit is contained in:
Claude
111
docs/adr/ADR-324-esp32-micro-llm-research-spike.md
Normal file
111
docs/adr/ADR-324-esp32-micro-llm-research-spike.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# ADR-324: ESP32-S3 micro-LLM inference — bounded research spike, production non-goal
|
||||
|
||||
- **Status**: Proposed
|
||||
- **Date**: 2026-08-16
|
||||
- **Deciders**: ruv
|
||||
- **Owners**: RuView firmware and edge runtime maintainers
|
||||
- **Tags**: esp32, llm, edge, research, evidence-labeling, firmware
|
||||
- **Numbering note**: ADR-324 is the next free number in the authoring
|
||||
checkout (highest existing: ADR-323; ADR-322 is an observed gap). Re-run the
|
||||
ADR index/collision check immediately before merge and rename if needed.
|
||||
- **Extends**: ADR-028, ADR-039, ADR-040, ADR-045, ADR-110, ADR-175, ADR-304
|
||||
- **Supersedes**: None
|
||||
- **Companion research**: `docs/research/esp32-micro-llm-inference.md`
|
||||
- **Companion ADR**: ADR-325 (flash-resident weight streaming)
|
||||
|
||||
## Executive decision
|
||||
|
||||
RuView will treat on-device micro-LLM inference on ESP32-S3 as a **bounded,
|
||||
optional research spike on dedicated companion hardware**, and explicitly a
|
||||
**non-goal for production CSI node firmware**. No LLM code, model weights, or
|
||||
partition changes enter `firmware/esp32-csi-node/` under this ADR. Every
|
||||
externally sourced performance number is `CLAIMED` until reproduced on RuView
|
||||
silicon with a captured witness log.
|
||||
|
||||
## Context
|
||||
|
||||
Public demonstrations in 2026 (slvDev's `esp32-ai`, building on
|
||||
`karpathy/llama2.c` and `DaveBben/esp32-llm`) show a 28.9 M-parameter
|
||||
TinyStories transformer running at 9.5–9.88 tok/s (`CLAIMED`) on an ESP32-S3
|
||||
with 512 KB SRAM, 8 MB PSRAM, and 16 MB flash, using 4-bit quantization and
|
||||
Gemma-3n-style Per-Layer Embeddings memory-mapped from flash. The companion
|
||||
research document analyzes the techniques and coverage in depth.
|
||||
|
||||
The question for RuView is whether any of this belongs on our sensing nodes.
|
||||
The material facts are:
|
||||
|
||||
1. **It does not fit our fleet.** Production nodes use the ADR-045 8 MB
|
||||
partition map (2× 2 MB OTA + 1.875 MB SPIFFS) or the 4 MB variant. The
|
||||
14.9 MB model artifact cannot be stored on either (`SYNTHETIC`, partition
|
||||
arithmetic). Replication requires a 16 MB-flash S3 SKU we do not deploy.
|
||||
2. **It contends with the product.** The demos own both cores and run no
|
||||
radio traffic during inference. RuView nodes dedicate core 0 to
|
||||
WiFi + 20 Hz CSI capture and core 1 to the Tier 1–2 DSP pipeline and WASM
|
||||
modules (ADR-039). LLM inference would degrade the primary sensing path.
|
||||
3. **The model class has no task capability.** TinyStories-scale models
|
||||
generate short fiction only — no instruction following, no factual recall.
|
||||
Every on-node use case we examined is served better by deterministic
|
||||
templates or is blocked on datasets that do not exist.
|
||||
4. **The memory techniques are genuinely valuable** — for RuView's own edge
|
||||
models, which is split out as ADR-325 so its fate is independent of the
|
||||
LLM demo's.
|
||||
|
||||
The rejected premise is: "an LLM now runs on our chip, therefore our product
|
||||
should run an LLM." Feasibility of a demo is not fitness for a sensing
|
||||
appliance.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **Production non-goal.** `firmware/esp32-csi-node/` gains no LLM inference
|
||||
path, no model partitions, and no LLM-derived output fields. Any future
|
||||
reversal requires a new ADR with `MEASURED` evidence of zero CSI-path
|
||||
regression.
|
||||
2. **Bounded research spike (optional, unscheduled).** If funded, the spike
|
||||
runs on a dedicated ESP32-S3 N16R8-class devkit that performs no CSI
|
||||
capture, following the validation plan in the research document:
|
||||
unmodified `esp32-ai` reproduction with witness log; WiFi-coexistence
|
||||
penalty measurement; a ≤3.5 M-parameter fit-our-flash retrain solely to
|
||||
characterize the quality cliff.
|
||||
3. **Evidence labeling.** All numbers from external sources remain `CLAIMED`
|
||||
in every RuView document, README, and communication until step 2 produces
|
||||
`MEASURED` rows backed by a committed witness log (WITNESS-LOG discipline
|
||||
per ADR-028). Builds and simulators are not hardware evidence.
|
||||
4. **Trust boundaries.** Third-party inference code and weights undergo
|
||||
license review at intake; model weights are never committed to this
|
||||
repository; fetched artifacts are SHA-256 pinned. Micro-LLM output is
|
||||
never admitted into the evidence engine (ADR-304), never labeled as
|
||||
perception, and never enters a safety-relevant path (fall/presence
|
||||
alerts).
|
||||
5. **ESP32-C6 exclusion.** The spike targets S3 only; the single-core RISC-V
|
||||
C6 without PIE SIMD is out of scope (ADR-110 unaffected).
|
||||
|
||||
## Options considered
|
||||
|
||||
- **Do nothing.** Cheapest, but discards transferable memory techniques and
|
||||
leaves recurring "can we run an LLM on the nodes?" questions undocumented.
|
||||
Rejected in favor of a written boundary.
|
||||
- **Integrate into production firmware as a Tier 3.** Rejected: flash
|
||||
impossibility on deployed SKUs, core/PSRAM contention with sensing,
|
||||
no task capability at this model scale, safety-path hallucination risk.
|
||||
- **Companion-hardware research spike (chosen).** Isolates risk, produces
|
||||
`MEASURED` data, feeds ADR-325.
|
||||
- **Server-side LLM only.** Remains the correct place for any real language
|
||||
capability; this ADR does not change server architecture.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Contributors get a citable "no" for production LLM integration and a
|
||||
citable "yes, like this" for research.
|
||||
- The spike, if run, costs one devkit and bounded engineering time, and
|
||||
produces the repository's first `MEASURED` micro-LLM data.
|
||||
- Risk accepted: the field moves fast; this ADR may need revisiting if
|
||||
instruction-capable models reach ~10 M parameters. The evidence-labeling
|
||||
and non-contention requirements would still hold.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/esp32-micro-llm-inference.md` (sources, techniques, fit
|
||||
analysis)
|
||||
- ADR-028 (capability audit / witness discipline), ADR-039 (edge tiers),
|
||||
ADR-045 (partition map), ADR-175 (measured quantization precedent),
|
||||
ADR-304 (evidence engine), ADR-325 (companion decision)
|
||||
111
docs/adr/ADR-325-flash-resident-edge-model-weight-streaming.md
Normal file
111
docs/adr/ADR-325-flash-resident-edge-model-weight-streaming.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# ADR-325: Flash-resident weight streaming for RuView edge models
|
||||
|
||||
- **Status**: Proposed
|
||||
- **Date**: 2026-08-16
|
||||
- **Deciders**: ruv
|
||||
- **Owners**: RuView firmware and edge runtime maintainers
|
||||
- **Tags**: esp32, memory, quantization, edge, inference, firmware, wasm
|
||||
- **Numbering note**: ADR-325 follows ADR-324 in the authoring checkout.
|
||||
Re-run the ADR index/collision check immediately before merge and rename if
|
||||
needed.
|
||||
- **Extends**: ADR-028, ADR-039, ADR-040, ADR-045, ADR-102, ADR-163, ADR-175
|
||||
- **Supersedes**: None
|
||||
- **Companion research**: `docs/research/esp32-micro-llm-inference.md`
|
||||
- **Companion ADR**: ADR-324 (micro-LLM research spike)
|
||||
|
||||
## Executive decision
|
||||
|
||||
RuView adopts a **design investigation** into flash-resident, memory-mapped,
|
||||
quantized weight storage for its **own** edge inference models on ESP32-S3 —
|
||||
the durable technique behind the 2026 micro-LLM demos — decoupled from any
|
||||
language-model ambition. The target outcome is that future neural
|
||||
presence/pose/vitals edge models are bounded by *flash* capacity rather than
|
||||
by the ~7.36 MB of free PSRAM or ~280 KB of free SRAM, on the existing 8 MB
|
||||
fleet SKU. This ADR authorizes design and benchmarking only; any partition
|
||||
map or wire change requires its own follow-up ADR with `MEASURED` evidence.
|
||||
|
||||
## Context
|
||||
|
||||
Today RuView's on-node intelligence is DSP (ADR-039 Tiers 1–2) plus WASM3
|
||||
modules in 640 KB of PSRAM arenas (ADR-040/041), with neural inference done
|
||||
server-side. If RuView ever ships a neural edge model (e.g. an on-node
|
||||
presence classifier to cut the Tier 2 heuristic's false-positive rate), the
|
||||
naive approach loads all weights into PSRAM/SRAM, capping model size and
|
||||
competing with WASM arenas and CSI buffers.
|
||||
|
||||
The `esp32-ai` demonstration (`CLAIMED`, see companion research) validated a
|
||||
different memory contract on identical silicon: keep only activations and hot
|
||||
weights in SRAM, the compute-dense core in PSRAM, and the bulk of parameters
|
||||
4-bit-quantized in flash, accessed via the ESP32-S3 MMU's memory-mapped
|
||||
read path with sparse, cache-friendly gathers (~450 bytes/token in the demo).
|
||||
The same contract applies to non-LLM models: embedding/lookup layers,
|
||||
frozen feature banks, and per-domain calibration tables are all
|
||||
sparse-access, read-only structures.
|
||||
|
||||
RuView precedents: ADR-175 established measured INT8 quantization discipline;
|
||||
ADR-163 established edge latency measurement; ADR-028 established witness
|
||||
evidence. This ADR composes them.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **Adopt the memory contract as a design target** for future RuView edge
|
||||
models on ESP32-S3: read-only quantized weight banks live in a dedicated
|
||||
flash region and are memory-mapped, not copied; PSRAM holds compute-dense
|
||||
weights and scratch; SRAM holds activations. INT8 remains the default
|
||||
quantization (ADR-175 precedent); sub-byte formats require their own
|
||||
measured accuracy gate.
|
||||
2. **Fit the fleet, not the demo.** All designs must fit the ADR-045 8 MB
|
||||
partition map's current free space (1.875 MB SPIFFS region or a future
|
||||
dedicated `model` data partition of comparable size). Designs requiring a
|
||||
16 MB SKU are research-only under ADR-324.
|
||||
3. **Benchmark before build.** The first deliverable is a microbenchmark
|
||||
suite on real S3 silicon measuring: memory-mapped flash gather latency and
|
||||
bandwidth (sequential vs. strided), ESP-DSP SIMD matmul throughput
|
||||
against PSRAM- vs. flash-resident operands, and CSI-pipeline interference
|
||||
(CSI packets/s at 20 Hz with the benchmark running on core 1). Results
|
||||
are committed as `MEASURED` with witness logs; until then, all sizing in
|
||||
this ADR is `SYNTHETIC`/`CLAIMED`.
|
||||
4. **Integrity and provenance.** Any model bank flashed to a node carries a
|
||||
SHA-256 recorded in NVS and reported in the node hello; unsigned or
|
||||
mismatched banks are rejected at mount time. Model artifacts are never
|
||||
committed to this repository; distribution follows the edge module
|
||||
registry path (ADR-102) with the same review gates as WASM modules.
|
||||
5. **Non-goals.** No production partition change, no wire-format change, no
|
||||
model training, and no LLM integration under this ADR. WASM-visible
|
||||
weight access (a `model_read` host API) is deferred until the benchmark
|
||||
shows the flash path sustains it alongside CSI capture.
|
||||
|
||||
## Options considered
|
||||
|
||||
- **Status quo (all weights in RAM).** Simple, but caps any future edge model
|
||||
at well under the free PSRAM after WASM arenas, and couples model growth to
|
||||
RAM contention with the sensing pipeline. Rejected as the default.
|
||||
- **Flash-resident streaming (chosen for investigation).** Raises the
|
||||
capacity ceiling ~an order of magnitude on existing SKUs; costs MMU window
|
||||
management and benchmark-verified bandwidth budgeting.
|
||||
- **16 MB SKU migration.** Hardware cost and fleet churn for an unproven
|
||||
need; remains available later and is orthogonal to this technique.
|
||||
- **Keep all neural inference server-side.** Remains the default for heavy
|
||||
models; this ADR only lowers the barrier for small on-node models where
|
||||
latency/privacy justify them.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Positive: future edge models are flash-bounded (~1.8 MB usable today,
|
||||
`SYNTHETIC`) instead of RAM-bounded; techniques arrive pre-validated by a
|
||||
measured benchmark rather than by external claims; provenance gating
|
||||
extends the existing module-registry trust model to weights.
|
||||
- Negative/risks: flash read contention with OTA and SPIFFS logging must be
|
||||
measured, not assumed; MMU mapping bugs are a new failure class at the
|
||||
hardware boundary (input validation at the mount path is mandatory);
|
||||
benchmark effort is spent even if no neural edge model ever ships.
|
||||
- The ADR-324 spike and this benchmark share kernels; either can proceed
|
||||
without the other.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/esp32-micro-llm-inference.md` (technique analysis, sources)
|
||||
- ADR-028 (witness evidence), ADR-039 (edge tiers and core budget), ADR-040/
|
||||
ADR-041 (WASM sensing), ADR-045 (partition map), ADR-102 (edge module
|
||||
registry), ADR-163 (edge latency measurement), ADR-175 (measured INT8
|
||||
quantization)
|
||||
217
docs/research/esp32-micro-llm-inference.md
Normal file
217
docs/research/esp32-micro-llm-inference.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# Deep research: micro-LLM inference on ESP32-S3-class microcontrollers
|
||||
|
||||
- **Date**: 2026-08-16
|
||||
- **Status**: Research survey (no RuView hardware evidence yet)
|
||||
- **Related ADRs**: ADR-324 (micro-LLM research spike), ADR-325 (flash-resident
|
||||
weight streaming), ADR-028 (ESP32 capability audit), ADR-039 (edge
|
||||
intelligence tiers), ADR-040 (WASM programmable sensing), ADR-045 (8 MB
|
||||
partition table), ADR-110 (ESP32-C6 extension), ADR-175 (INT8 quantization,
|
||||
measured)
|
||||
- **Evidence policy**: Every performance or capability number in this document
|
||||
is tagged `MEASURED` (reproduced on RuView hardware with a witness log),
|
||||
`CLAIMED` (asserted by an external source, not reproduced by us), or
|
||||
`SYNTHETIC` (derived arithmetic, no hardware run). As of this writing there
|
||||
are **zero** `MEASURED` micro-LLM numbers in this repository.
|
||||
|
||||
## 1. Executive summary
|
||||
|
||||
In mid-2026 a developer known as slvDev (YouTube channel "The Stack") published
|
||||
`esp32-ai`, a demonstration that a 28.9 M-parameter transformer trained on
|
||||
TinyStories runs entirely on an ~$8 ESP32-S3 at roughly 9.5–9.88 tokens/s
|
||||
(`CLAIMED`). The result was widely covered (Tom's Hardware, The Register,
|
||||
CNX Software, Hackster, XDA, geeky-gadgets). The enabling ideas are:
|
||||
|
||||
1. **Per-Layer Embeddings (PLE)**, borrowed from Google's Gemma 3n: ~25 M of
|
||||
the 28.9 M parameters live in a memory-mapped lookup table in flash that is
|
||||
*read*, not computed on — only ~450 bytes of table data are touched per
|
||||
token (`CLAIMED`).
|
||||
2. **4-bit quantization**, compressing the model to 14.9 MB so it fits a
|
||||
16 MB flash part (`CLAIMED`).
|
||||
3. **Memory-hierarchy mapping**: activations and norm weights in ~512 KB SRAM,
|
||||
the ~3.9 M-parameter dense core and output head in 8 MB PSRAM, the PLE
|
||||
table memory-mapped in flash (`CLAIMED`).
|
||||
4. **A modified `llama2.c` inference loop** with ESP32-S3 hardware-aware
|
||||
optimizations, building on earlier work by DaveBben.
|
||||
|
||||
This is a genuine engineering milestone — roughly 100× more parameters than
|
||||
the prior ESP32 record (~260 K) — but the model is a toy: it generates short
|
||||
children's stories and, per its own author, "will not answer questions, follow
|
||||
instructions, write code, or know facts" (`CLAIMED`).
|
||||
|
||||
**Bottom line for RuView**: the demo does not run on RuView production nodes
|
||||
as-is (our fleet is 8 MB/4 MB flash; the demo needs 16 MB), it contends
|
||||
directly with the CSI sensing workload for both cores and PSRAM, and a
|
||||
TinyStories model has no useful task capability for RF perception. The
|
||||
*techniques*, however — flash-resident memory-mapped quantized weights, PLE-
|
||||
style table lookup, ESP-DSP SIMD kernels, dual-core scheduling — are directly
|
||||
transferable to RuView's own edge models and are the durable value of this
|
||||
research. ADR-324 and ADR-325 turn those two conclusions into decisions.
|
||||
|
||||
## 2. Landscape and timeline
|
||||
|
||||
| Year | Project | Model | Claimed throughput | Hardware | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| 2023 | `karpathy/llama2.c` | any Llama-2-architecture checkpoint | n/a (host) | PC | Single-file C inference engine; upstream of everything below |
|
||||
| 2024 | `DaveBben/esp32-llm` | 260 K params (tinyllamas, TinyStories) | 19.13 tok/s (`CLAIMED`) | ESP32-S3FH4R2, 2 MB PSRAM | First notable ESP32 LLM port; ESP-DSP SIMD dot products, both cores, 240 MHz CPU, 80 MHz PSRAM, enlarged instruction cache |
|
||||
| 2026 | `slvDev/esp32-ai` ("The Stack") | 28.9 M params, 4-bit, 14.9 MB, TinyStories | 9.5–9.88 tok/s end-to-end, ~94.9 ms/token compute (`CLAIMED`) | ESP32-S3, 512 KB SRAM, 8 MB PSRAM, **16 MB flash** | First known application of Gemma-3n-style PLE at this scale of hardware; MIT license; SHA-256-verified model fetch |
|
||||
| 2026 | `wladimiravila/esp32s3-distributed-ai` | 56 M params split across 3 boards | not verified by us | 3× ESP32-S3 over ESP-NOW | "Split-PLE + KV cache, fully offline" (`CLAIMED`); shows the scaling direction |
|
||||
|
||||
Context for scale: a 28.9 M-parameter model is ~4 orders of magnitude smaller
|
||||
than frontier LLMs and ~2 orders smaller than the smallest generally useful
|
||||
instruction-following models (~1–3 B). TinyStories (Eldan & Li, 2023,
|
||||
arXiv:2305.07759) exists precisely because sub-100 M models can produce
|
||||
coherent English only when the training distribution is radically constrained
|
||||
(vocabulary of a young child, short narrative form).
|
||||
|
||||
## 3. Technique deep dive
|
||||
|
||||
### 3.1 Per-Layer Embeddings (PLE) / flash-resident lookup tables
|
||||
|
||||
Gemma 3n introduced PLE to cut the *resident* memory footprint of a model:
|
||||
a large share of parameters is restructured into per-layer embedding tables
|
||||
that are gathered by token id rather than multiplied against activations.
|
||||
Gathers are sparse — a handful of rows per token — so the table can live in
|
||||
slow, cheap storage (here: memory-mapped SPI flash via the ESP32-S3 MMU)
|
||||
without putting flash bandwidth on the critical path. `esp32-ai` claims only
|
||||
~450 bytes of table reads per token (`CLAIMED`), which at ~10 tok/s is a
|
||||
trivial ~4.5 KB/s of flash read traffic.
|
||||
|
||||
Why this matters generally: it converts the dominant constraint from
|
||||
*RAM capacity* to *flash capacity*, and flash is the cheapest memory on these
|
||||
parts. The compute-active core shrinks to ~3.9 M parameters (~2 MB at 4-bit),
|
||||
which fits comfortably in PSRAM.
|
||||
|
||||
### 3.2 Quantization
|
||||
|
||||
- 4-bit weight quantization: 28.9 M params → 14.9 MB total artifact
|
||||
(`CLAIMED`). Sub-byte weights on Xtensa cost unpack instructions; the
|
||||
scheme's viability at 9.5 tok/s suggests unpacking is amortized inside the
|
||||
SIMD kernels.
|
||||
- RuView precedent: ADR-175 measured INT8 quantization of a RuView pose model
|
||||
and is our internal quality bar for how quantization claims must be
|
||||
validated (accuracy delta on a held-out split, not vibes).
|
||||
|
||||
### 3.3 Memory-hierarchy mapping (claimed layout of `esp32-ai`)
|
||||
|
||||
| Tier | Size | Speed class | Contents |
|
||||
|---|---|---|---|
|
||||
| Internal SRAM | 512 KB | fastest | activations, normalization weights, hot scratch |
|
||||
| Octal/Quad PSRAM | 8 MB | ~80 MHz SPI | dense transformer core (~3.9 M params) + output head, KV cache |
|
||||
| SPI flash (memory-mapped) | 16 MB | slowest | 25 M-param PLE table (14.9 MB artifact total) |
|
||||
|
||||
### 3.4 Compute optimizations (from `esp32-llm`, inherited by successors)
|
||||
|
||||
- ESP-DSP dot-product kernels using the ESP32-S3's PIE 128-bit SIMD
|
||||
extensions (the S3's differentiator vs. plain ESP32/C-series).
|
||||
- Both Xtensa LX7 cores active during matmuls (second core normally runs the
|
||||
WiFi stack — these demos do not run WiFi during inference).
|
||||
- 240 MHz CPU, 80 MHz PSRAM clock, enlarged instruction cache.
|
||||
- DaveBben's measured jump to 19.13 tok/s on a 260 K model (`CLAIMED`) came
|
||||
primarily from SIMD + dual-core, an indication of how memory- and
|
||||
compute-bound the naive loop is.
|
||||
|
||||
### 3.5 What the model actually does
|
||||
|
||||
TinyStories-class models emit grammatical, mostly-coherent short fiction.
|
||||
They have no instruction following, no factual recall, no structured-output
|
||||
reliability. Every secondary source and the author agree on this. Any plan
|
||||
that assumes "small LLM = small ChatGPT" is wrong at this parameter scale.
|
||||
|
||||
## 4. Fit analysis against RuView hardware and workload
|
||||
|
||||
### 4.1 Flash: the demo does not fit our fleet
|
||||
|
||||
RuView production partition table (ADR-045, `partitions_display.csv`):
|
||||
8 MB flash = 2× 2 MB OTA app slots + 1.875 MB SPIFFS + NVS/PHY/otadata.
|
||||
The 4 MB variant (issue #265) is tighter still. A 14.9 MB model artifact
|
||||
**cannot be stored on either** — not in SPIFFS, not memory-mapped, not at
|
||||
all (`SYNTHETIC`, from partition arithmetic). Direct replication requires a
|
||||
16 MB-flash S3 variant (e.g. N16R8 modules), which is a *hardware SKU change*,
|
||||
not a firmware change.
|
||||
|
||||
Fitting inside the existing 1.875 MB SPIFFS would cap a 4-bit model near
|
||||
~3.5 M total parameters (`SYNTHETIC`) — a regime where even TinyStories
|
||||
coherence degrades sharply.
|
||||
|
||||
### 4.2 RAM: feasible in isolation, contended in practice
|
||||
|
||||
Current firmware budget (firmware README): ~35 KB SRAM used with ~280 KB SRAM
|
||||
free; 640 KB PSRAM for WASM arenas with ~7.36 MB PSRAM free. A ~2 MB dense
|
||||
core + KV cache would fit free PSRAM (`SYNTHETIC`). But:
|
||||
|
||||
### 4.3 Compute and scheduling: direct conflict with the day job
|
||||
|
||||
The published demos own **both cores** and disable radio work during
|
||||
inference. RuView nodes run WiFi + 20 Hz CSI capture on core 0 and the
|
||||
Tier 1–2 DSP pipeline (plus WASM3 modules) on core 1 (ADR-039). CSI capture
|
||||
is the product; a micro-LLM competing for core 1 and PSRAM bandwidth would
|
||||
degrade the primary sensing path. On-node LLM inference is therefore only
|
||||
plausible as (a) a mutually exclusive duty mode, or (b) a dedicated companion
|
||||
node that does no CSI capture.
|
||||
|
||||
### 4.4 ESP32-C6 is out of scope
|
||||
|
||||
The C6 (ADR-110 research target) is a single-core RISC-V without the S3's
|
||||
PIE SIMD and typically without PSRAM — no published micro-LLM result exists
|
||||
for it and the arithmetic is unfavorable. S3-only.
|
||||
|
||||
### 4.5 Capability fit: what would a node even say?
|
||||
|
||||
RuView's edge outputs are compact typed packets (32-byte vitals, ADR-039) and
|
||||
server-side semantics. Candidate LLM uses on-node and their honest status:
|
||||
|
||||
| Candidate use | Verdict | Why |
|
||||
|---|---|---|
|
||||
| Natural-language event narration ("someone fell in the kitchen") | Weak | A template engine does this deterministically in <1 KB; a TinyStories-class model adds hallucination risk to a safety-relevant path with zero benefit |
|
||||
| On-device Q&A / configuration assistant | No | Requires instruction following; not available at this scale |
|
||||
| Semantic compression of CSI events | Research-only | Would require a custom domain model trained on RuView event→text pairs; no dataset exists today |
|
||||
| Demonstrating the platform's headroom / marketing | Real but bounded | Legitimate as a clearly labeled demo, never as a shipped claim |
|
||||
|
||||
The durable value is the **memory architecture**, not the language model:
|
||||
flash-resident memory-mapped quantized weights + PLE-style tables would let
|
||||
RuView's *own* future neural presence/pose edge models grow well beyond
|
||||
current RAM budgets on 8 MB parts (ADR-325).
|
||||
|
||||
## 5. Risks and honest-labeling obligations
|
||||
|
||||
- All throughput/size numbers above are `CLAIMED` until reproduced on RuView
|
||||
silicon with a captured boot/runtime log (witness-log discipline, cf.
|
||||
ADR-028/WITNESS-LOG-028). A successful build or QEMU run is not hardware
|
||||
evidence.
|
||||
- Third-party code and model weights (`esp32-ai` MIT; `llama2.c` MIT;
|
||||
`esp32-llm` license to be verified at intake) require license review before
|
||||
entering any RuView tree; model weights are unreviewed generated artifacts
|
||||
and must not be committed (repository non-negotiables).
|
||||
- Never present micro-LLM output as perception evidence. It must not enter
|
||||
the evidence engine (ADR-304) or any published sensing claim.
|
||||
- Marketing risk: "LLM on our sensor" invites camera-grade-style
|
||||
overclaiming. Any public statement must carry the demo framing and
|
||||
evidence tags.
|
||||
|
||||
## 6. Recommended validation plan (if the ADR-324 spike is funded)
|
||||
|
||||
1. Acquire an ESP32-S3 N16R8 (16 MB flash / 8 MB PSRAM) devkit — companion
|
||||
hardware, not a fleet SKU.
|
||||
2. Reproduce `esp32-ai` unmodified; capture serial witness log (boot, model
|
||||
SHA-256, tok/s over ≥500 tokens). This produces the first `MEASURED` row.
|
||||
3. Measure tok/s with WiFi stack active vs. disabled to quantify the
|
||||
coexistence penalty.
|
||||
4. Attempt a ≤3.5 M-parameter retrain fitting the 8 MB partition map, solely
|
||||
to characterize the quality cliff (`SYNTHETIC` sizing above).
|
||||
5. Separately benchmark the transferable kernels (ESP-DSP SIMD matmul,
|
||||
memory-mapped flash weight streaming) against RuView's own edge-model
|
||||
workloads — this feeds ADR-325 regardless of the LLM outcome.
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- Geeky-Gadgets coverage (task prompt): https://www.geeky-gadgets.com/run-llm-esp32-microcontroller/
|
||||
- Project repo: https://github.com/slvDev/esp32-ai (MIT)
|
||||
- Prior art: https://github.com/DaveBben/esp32-llm ; https://github.com/karpathy/llama2.c
|
||||
- Distributed follow-on: https://github.com/wladimiravila/esp32s3-distributed-ai
|
||||
- Tom's Hardware: https://www.tomshardware.com/tech-industry/artificial-intelligence/ai-developer-runs-28-9-million-parameter-model-on-usd10-esp32-s3-microcontroller-uses-googles-per-layer-embeddings-technique-stores-table-on-16mb-flash-memory
|
||||
- The Register: https://www.theregister.com/edge-and-iot/2026/08/04/dev-proves-llms-will-run-on-anything-even-a-10-microcontroller/5283088
|
||||
- CNX Software: https://www.cnx-software.com/2026/08/03/28-9m-parameter-llm-runs-locally-on-esp32-s3-at-9-tokens-s/
|
||||
- Hackster: https://www.hackster.io/news/running-a-28-9m-parameter-llm-on-an-8-microcontroller-173f1f370708
|
||||
- XDA: https://www.xda-developers.com/someone-squeezed-a-289m-llm-onto-an-esp32-s3-and-so-can-you/
|
||||
- TinyStories: Eldan & Li 2023, arXiv:2305.07759
|
||||
- Gemma 3n / Per-Layer Embeddings: Google AI developer announcements, 2025
|
||||
Reference in New Issue
Block a user