Sreedev Kodichath
3ca931fa8e
[REF] core: replace streaming pipeline with staged batch pipeline
...
The deduplication core was a three-stage streaming producer/consumer
(scan -> group-by-size -> group-by-hash) orchestrated by a Server struct
that ran the stages concurrently on a threadpool. Coordination relied on
AtomicBool flags polled in busy-wait loops, an Arc<Mutex<Vec>> hand-off
queue, DashMap stores, and a per-file Arc<Mutex<FileState>>. Every stage
had to receive its own hand-cloned Arcs with ad-hoc names, and the
busy-wait branches burned a core spinning while waiting for the producer.
Replace it with a staged batch pipeline over owned collections:
pipeline::run(&Params) drives scan -> group_by_size -> group_by_hash in
sequence, with rayon supplying parallelism. With no shared mutable state
between stages, all the Arc plumbing, the AtomicBool coordination, the
Mutex queue, the per-file lock, server.rs, and the threadpool/dashmap
dependencies are gone. FileInfo becomes plain data and each stage is a
pure, independently testable function.
Behavior is preserved except for three authorized deviations: progress
spinners render sequentially rather than concurrently under -p; the
interactive empty-result message prints once instead of twice; and the
interactive "Duplicate Set X of Y" total now counts the confirmed
duplicate groups shown instead of the internal candidate-hash store size.
This lands as the foundation for the cache, CLI, and TUI work that follows.
2026-07-26 14:27:02 +02:00
sreedevk
d102d92bfa
cleanup clone littering in server.rs
2025-08-07 23:18:02 +00:00
sreedevk
da24efbb36
count graphemes instead of chars in path length check
2025-07-30 10:47:58 +00:00
sreedevk
b84510f18e
tree rendering ui improvements
2025-07-19 16:38:17 +00:00
sreedevk
21f0eb3cb0
all file type related filtering issues fixed
2025-07-19 16:26:12 +00:00
sreedevk
f5d2d4e22c
fix: single file hash groups printed to screen
2025-07-19 14:22:43 +00:00
sreedevk
4894d1b4db
minor ui improvements
2025-07-19 11:43:26 +00:00
sreedevk
c0286b17cc
fix: hash collision edge case
2025-07-19 11:33:52 +00:00
sreedevk
711eb36fb9
output formatting and printing performance improvements
2025-07-19 01:26:46 +00:00
sreedevk
130a8f99ba
partial hash page count increased
2025-07-18 11:25:37 +00:00
sreedevk
d97af6b8f6
improved parallelization + updated benchmarks
2025-07-17 00:31:40 +00:00
sreedevk
90198502a3
added exclude file types.
...
updated documentation to include exclude file types.
2025-07-14 12:56:07 +00:00
sreedevk
e1b81b9981
page hash aggregation method improvement
2025-07-14 01:02:14 +00:00
Sreedev Kodichath
58e33cc8da
v0.3 ( #61 )
...
- [x] parallelization
- [x] (scanning) + (processing sw & processing hw & formatting & printing)
- [x] reduce cloning values on the heap
- [x] add a partial hashing mode (--strict)
- [x] add unit tests
- [x] add silent mode
- [x] update documentation
- [x] remove color output
- [x] progress bar improvements
- [x] use progress bar groups
- [x] remove broken json rendering
- [x] add benchmarks
2025-07-13 19:03:14 -04:00
sreedevk
f9e86f8522
performance improvements
2025-07-12 00:24:25 +00:00
Sreedev Kodichath
028b868ea9
version 0.2.2 ( #59 )
...
* replaced fxhash with gxhash
2024-07-04 23:23:01 -04:00
sreedev
080cd791dc
removed early return
2023-11-14 18:37:48 -05:00
sreedev
3e407f69c8
added json output for further processing using other tools
2023-11-14 18:33:40 -05:00
sreedev
ffa5295598
clippy
2023-07-17 14:40:10 -04:00
sreedev
6be8596992
restored interactive mode
2023-07-17 14:34:42 -04:00
sreedev
a40f251e30
subtract overflow issue fixed
2023-07-17 14:19:57 -04:00
sreedev
02d05172da
minsize issues fixed
2023-07-17 14:13:18 -04:00
sreedev
dcc709a666
added filetype filter
2023-07-17 14:06:57 -04:00
sreedev
e3d48ec505
v0.2.0
2023-07-17 13:59:58 -04:00
sreedev
a65e22269c
stabilize ui progress bars
2023-02-06 09:35:24 -05:00
sreedev
f0dbf05705
removed tokio & other unused dependencies
2023-01-26 14:07:50 -05:00
sreedev
ef1e9a1fce
replace path String in File type with PathBuf
2023-01-26 00:04:17 -05:00
sreedev
d06ca7897d
fix critical error
2023-01-25 23:12:52 -05:00
sreedev
f9b6d57968
added short params
2023-01-23 20:05:14 -05:00
sreedev
5c86a080c4
Merge branch 'development' into feature/add_scan_control_args
2023-01-23 20:03:30 -05:00
sreedev
9f4d9139b6
added --min-depth --max-depth --follow-links and renamed --minsize to --min-size
2023-01-23 20:03:08 -05:00
sreedev
fa12f85b6a
make --dir a positional argument
2023-01-23 19:12:55 -05:00
sreedev
10329015f7
added more progress bars to avoid blank states
2023-01-23 18:29:41 -05:00
beeb
850c274adc
refactor: use globwalk builder pattern
2023-01-23 22:28:48 +01:00
beeb
6f8b1d55df
feat: switch to globwalk
2023-01-23 21:53:44 +01:00
sreedev
7b275d40d0
min opts 2
2023-01-19 23:27:40 -05:00
sreedev
30398c3ca9
min opts
2023-01-19 23:25:16 -05:00
Sreedev Kodichath
f0ff1ec325
Merge pull request #27 from beeb/sorted_by_size
...
Sort table results by ascending file size
2023-01-18 10:36:11 -05:00
Valentin Bersier
84b194efca
feat(cli): add value hint for --dir argument
2023-01-18 15:40:10 +01:00
Valentin Bersier
d162ca98ef
docs: comment
2023-01-18 15:34:58 +01:00
Valentin Bersier
dfb73ceb5c
feat: sort interactive results by descending size
2023-01-18 15:33:23 +01:00
Valentin Bersier
72ca7c7a44
Merge branch 'development' into sorted_by_size
2023-01-18 15:24:47 +01:00
sreedev
e24603c645
output enhancements
2023-01-18 02:34:08 -05:00
sreedev
0e624b042d
added minsize filter
2023-01-18 02:30:48 -05:00
Valentin Bersier
e61d1f1475
perf: unwrap the size once
2023-01-18 08:07:51 +01:00
Valentin Bersier
8c76c6a3c8
refactor: simplify iterator to get size
2023-01-18 08:00:53 +01:00
Valentin Bersier
df3bea3d4f
feat: display results in ascending size order
2023-01-18 07:59:51 +01:00
sreedev
7ace0bde63
scanner optimizations
2023-01-18 01:33:51 -05:00
sreedev
1ea5705474
fix confusing delete confirmation
2023-01-18 00:30:39 -05:00
sreedev
ea748c60d8
performance tweaks
2023-01-18 00:24:50 -05:00
