[REF] core: replace streaming pipeline with staged batch pipeline

The deduplication core was a three-stage streaming producer/consumer
(scan -> group-by-size -> group-by-hash) orchestrated by a Server struct
that ran the stages concurrently on a threadpool. Coordination relied on
AtomicBool flags polled in busy-wait loops, an Arc<Mutex<Vec>> hand-off
queue, DashMap stores, and a per-file Arc<Mutex<FileState>>. Every stage
had to receive its own hand-cloned Arcs with ad-hoc names, and the
busy-wait branches burned a core spinning while waiting for the producer.

Replace it with a staged batch pipeline over owned collections:
pipeline::run(&Params) drives scan -> group_by_size -> group_by_hash in
sequence, with rayon supplying parallelism. With no shared mutable state
between stages, all the Arc plumbing, the AtomicBool coordination, the
Mutex queue, the per-file lock, server.rs, and the threadpool/dashmap
dependencies are gone. FileInfo becomes plain data and each stage is a
pure, independently testable function.

Behavior is preserved except for three authorized deviations: progress
spinners render sequentially rather than concurrently under -p; the
interactive empty-result message prints once instead of twice; and the
interactive "Duplicate Set X of Y" total now counts the confirmed
duplicate groups shown instead of the internal candidate-hash store size.

This lands as the foundation for the cache, CLI, and TUI work that follows.
This commit is contained in:
Sreedev Kodichath
2026-07-26 14:27:02 +02:00
parent d102d92bfa
commit 3ca931fa8e
11 changed files with 316 additions and 732 deletions

View File

@@ -21,7 +21,6 @@ anyhow = "1.0.68"
bytesize = "2.0.1"
chrono = "0.4.23"
clap = { version = "4.0.32", features = ["derive"] }
dashmap = { version = "6.1.0", features = ["rayon"] }
globwalk = "0.9.1"
gxhash = { version = "3.4.1", default-features = false }
indicatif = { version = "0.18.0", features = ["rayon"] }
@@ -30,7 +29,6 @@ pathdiff = "0.2.1"
prettytable-rs = "0.10.0"
rand = "0.9.1"
rayon = "1.6.1"
threadpool = "1.8.1"
unicode-segmentation = "1.12.0"
[profile.release]