From 70de402eab76b40437abc5afeb2899b8ffa16f41 Mon Sep 17 00:00:00 2001
From: sreedev
- Deduplicator uses fxhash (a non-cryptographic hashing algorithm) which is extremely fast. As a result, deduplicator is able to process huge amounts of data in a couple of seconds.Performance
couple of seconds. few milliseconds.
- While testing, Deduplicator was able to go through 8.6GB of pdf files and detect duplicates in 2.9 seconds
+ While testing, Deduplicator was able to go through 8.6GB of pdf files and detect duplicates in 2.9 seconds
+ As of version 0.1.1, on testing locally, deduplicator was able to process and find duplicates in 120GB of files (Videos, PDFs, Images) in ~300ms