mirror of
https://github.com/sreedevk/deduplicator.git
synced 2026-09-07 10:34:22 +00:00
[REF] core: replace streaming pipeline with staged batch pipeline
The deduplication core was a three-stage streaming producer/consumer (scan -> group-by-size -> group-by-hash) orchestrated by a Server struct that ran the stages concurrently on a threadpool. Coordination relied on AtomicBool flags polled in busy-wait loops, an Arc<Mutex<Vec>> hand-off queue, DashMap stores, and a per-file Arc<Mutex<FileState>>. Every stage had to receive its own hand-cloned Arcs with ad-hoc names, and the busy-wait branches burned a core spinning while waiting for the producer. Replace it with a staged batch pipeline over owned collections: pipeline::run(&Params) drives scan -> group_by_size -> group_by_hash in sequence, with rayon supplying parallelism. With no shared mutable state between stages, all the Arc plumbing, the AtomicBool coordination, the Mutex queue, the per-file lock, server.rs, and the threadpool/dashmap dependencies are gone. FileInfo becomes plain data and each stage is a pure, independently testable function. Behavior is preserved except for three authorized deviations: progress spinners render sequentially rather than concurrently under -p; the interactive empty-result message prints once instead of twice; and the interactive "Duplicate Set X of Y" total now counts the confirmed duplicate groups shown instead of the internal candidate-hash store size. This lands as the foundation for the cache, CLI, and TUI work that follows.
This commit is contained in:
@@ -21,7 +21,6 @@ anyhow = "1.0.68"
|
||||
bytesize = "2.0.1"
|
||||
chrono = "0.4.23"
|
||||
clap = { version = "4.0.32", features = ["derive"] }
|
||||
dashmap = { version = "6.1.0", features = ["rayon"] }
|
||||
globwalk = "0.9.1"
|
||||
gxhash = { version = "3.4.1", default-features = false }
|
||||
indicatif = { version = "0.18.0", features = ["rayon"] }
|
||||
@@ -30,7 +29,6 @@ pathdiff = "0.2.1"
|
||||
prettytable-rs = "0.10.0"
|
||||
rand = "0.9.1"
|
||||
rayon = "1.6.1"
|
||||
threadpool = "1.8.1"
|
||||
unicode-segmentation = "1.12.0"
|
||||
|
||||
[profile.release]
|
||||
|
||||
Reference in New Issue
Block a user