Sreedev Kodichath
3ca931fa8e
[REF] core: replace streaming pipeline with staged batch pipeline
...
The deduplication core was a three-stage streaming producer/consumer
(scan -> group-by-size -> group-by-hash) orchestrated by a Server struct
that ran the stages concurrently on a threadpool. Coordination relied on
AtomicBool flags polled in busy-wait loops, an Arc<Mutex<Vec>> hand-off
queue, DashMap stores, and a per-file Arc<Mutex<FileState>>. Every stage
had to receive its own hand-cloned Arcs with ad-hoc names, and the
busy-wait branches burned a core spinning while waiting for the producer.
Replace it with a staged batch pipeline over owned collections:
pipeline::run(&Params) drives scan -> group_by_size -> group_by_hash in
sequence, with rayon supplying parallelism. With no shared mutable state
between stages, all the Arc plumbing, the AtomicBool coordination, the
Mutex queue, the per-file lock, server.rs, and the threadpool/dashmap
dependencies are gone. FileInfo becomes plain data and each stage is a
pure, independently testable function.
Behavior is preserved except for three authorized deviations: progress
spinners render sequentially rather than concurrently under -p; the
interactive empty-result message prints once instead of twice; and the
interactive "Duplicate Set X of Y" total now counts the confirmed
duplicate groups shown instead of the internal candidate-hash store size.
This lands as the foundation for the cache, CLI, and TUI work that follows.
2026-07-26 14:27:02 +02:00
sreedevk
d102d92bfa
cleanup clone littering in server.rs
2025-08-07 23:18:02 +00:00
sreedevk
68c2491fd0
added file store partitioning idea to readme.
2025-07-30 13:08:54 +00:00
sreedevk
da24efbb36
count graphemes instead of chars in path length check
2025-07-30 10:47:58 +00:00
sreedevk
2191763b4d
added potential optimizations to readme
2025-07-28 14:22:13 +00:00
sreedevk
b84510f18e
tree rendering ui improvements
2025-07-19 16:38:17 +00:00
Sreedev Kodichath
bfb89edd16
Update README.md
2025-07-19 12:32:09 -04:00
sreedevk
21f0eb3cb0
all file type related filtering issues fixed
2025-07-19 16:26:12 +00:00
sreedevk
f5d2d4e22c
fix: single file hash groups printed to screen
2025-07-19 14:22:43 +00:00
sreedevk
360af25318
updated readme
2025-07-19 13:57:26 +00:00
sreedevk
d06fe93171
fix pipeline breaks
2025-07-19 13:41:52 +00:00
sreedevk
6a24ca5ecf
updated github release pipeline
2025-07-19 12:15:42 +00:00
sreedevk
6f23a51a53
created a automatic release ci pipeline
2025-07-19 12:00:25 +00:00
sreedevk
4894d1b4db
minor ui improvements
2025-07-19 11:43:26 +00:00
sreedevk
c0286b17cc
fix: hash collision edge case
2025-07-19 11:33:52 +00:00
sreedevk
e78dfc4211
updated dependencies
2025-07-19 02:01:15 +00:00
sreedevk
66f49a9aee
benchmarks updated
2025-07-19 01:35:24 +00:00
sreedevk
711eb36fb9
output formatting and printing performance improvements
2025-07-19 01:26:46 +00:00
sreedevk
130a8f99ba
partial hash page count increased
2025-07-18 11:25:37 +00:00
sreedevk
d97af6b8f6
improved parallelization + updated benchmarks
2025-07-17 00:31:40 +00:00
sreedevk
90198502a3
added exclude file types.
...
updated documentation to include exclude file types.
2025-07-14 12:56:07 +00:00
sreedevk
e1b81b9981
page hash aggregation method improvement
2025-07-14 01:02:14 +00:00
sreedevk
48e7c62036
added planned features
2025-07-14 00:23:16 +00:00
Sreedev Kodichath
58e33cc8da
v0.3 ( #61 )
...
- [x] parallelization
- [x] (scanning) + (processing sw & processing hw & formatting & printing)
- [x] reduce cloning values on the heap
- [x] add a partial hashing mode (--strict)
- [x] add unit tests
- [x] add silent mode
- [x] update documentation
- [x] remove color output
- [x] progress bar improvements
- [x] use progress bar groups
- [x] remove broken json rendering
- [x] add benchmarks
2025-07-13 19:03:14 -04:00
sreedevk
f9e86f8522
performance improvements
2025-07-12 00:24:25 +00:00
sreedev
b4bf5d4fb3
docs updated
2024-07-05 04:34:12 +00:00
Sreedev Kodichath
028b868ea9
version 0.2.2 ( #59 )
...
* replaced fxhash with gxhash
2024-07-04 23:23:01 -04:00
Sreedev Kodichath
a56d194ee3
Merge pull request #56 from sreedevk/feature/add-json-output
...
Version 0.2.1
2023-11-14 18:38:50 -05:00
sreedev
080cd791dc
removed early return
2023-11-14 18:37:48 -05:00
sreedev
b16236763c
updated readme + version number
2023-11-14 18:36:27 -05:00
sreedev
3e407f69c8
added json output for further processing using other tools
2023-11-14 18:33:40 -05:00
Sreedev Kodichath
7d386c9420
Merge pull request #54 from sreedevk/distribution-improvements
...
Improve Distribution Methods & Create Compiled Binaries for More Platforms
2023-08-10 19:54:29 -04:00
sreedev
0dc681d4e7
added release yml
2023-08-10 19:51:57 -04:00
sreedev
442cb4b519
added cargo dist options
2023-08-10 13:05:29 -04:00
sreedev
8463e72f2d
removed debug information from distrbution & release profiles to reduce binary size
2023-08-10 12:59:07 -04:00
sreedev
05c95bb67a
added roadmap to readme
2023-08-02 09:41:15 -04:00
Sreedev Kodichath
062d44acd9
Merge pull request #53 from sreedevk/v0.2.0
...
v0.2.0 Architecture Improvements
2023-07-17 18:17:57 -04:00
sreedev
ee6655de9b
removed examples
2023-07-17 18:14:13 -04:00
sreedev
ffa5295598
clippy
2023-07-17 14:40:10 -04:00
sreedev
6be8596992
restored interactive mode
2023-07-17 14:34:42 -04:00
sreedev
a40f251e30
subtract overflow issue fixed
2023-07-17 14:19:57 -04:00
sreedev
02d05172da
minsize issues fixed
2023-07-17 14:13:18 -04:00
sreedev
dcc709a666
added filetype filter
2023-07-17 14:06:57 -04:00
sreedev
e3d48ec505
v0.2.0
2023-07-17 13:59:58 -04:00
Sreedev Kodichath
7bd88f1642
Merge pull request #49 from sreedevk/development
...
Version 0.1.6
2023-02-06 09:37:01 -05:00
sreedev
a65e22269c
stabilize ui progress bars
2023-02-06 09:35:24 -05:00
sreedev
6e3fe4a37d
cargo dist
2023-02-03 10:18:26 -05:00
sreedev
eadffb5fea
doc fixes
2023-01-27 09:56:10 -05:00
sreedev
5532ded331
updated benchmarks
2023-01-27 09:54:39 -05:00
sreedev
7119f019d3
benchmark updates
2023-01-27 09:52:14 -05:00
Sreedev Kodichath