| Operation | Standard Command | xtool -dd | |-----------|----------------|-----------| | Remove dupes (unsorted) | awk '!seen[$0]++' | xtool -dd | | Remove dupes (sorted) | sort -u | sort -u | | Remove dupes & execute | xargs -n1 + awk | xtool -dd xargs |
xTool -dd -c 10M -p 4 /data/source /data/deduplicated xtool -dd deduplication option