-.TH "FINDDUP" 1 "Mar 2010" "Francois Fleuret" "User Commands"
+.TH "FINDDUP" "1.2" "Apr 2011" "Francois Fleuret" "User Commands"
\" This man page was written by Francois Fleuret <francois@fleuret.org>
\" and is distributed under a Creative Commons Attribution-Share Alike
.SH "SYNOPSIS"
-\fBfinddup\fP [OPTION]... DIR1 [[and:|not:]DIR2]
+\fBfinddup\fP [OPTION]... [DIR-OR-FILE1 [[and:|not:]DIR-OR-FILE2]]
.SH "DESCRIPTION"
-With a single directory argument, \fBfinddup\fP prints the duplicated
-files found in it.
+With one directory as argument, \fBfinddup\fP prints the duplicated
+files found in it. If no directory is provided, it uses the current
+one as default.
With two directories, it prints either the files common to both DIR1
and DIR2 or, with the `not:' prefix, the ones present in DIR1 and not
-in DIR2. The and: prefix is assumed by default and necessary only if
-you have a directory name starting with `not:'.
+in DIR2. The `and:' prefix is assumed by default and necessary only if
+you have a directory name starting with `not:'. Files are handled like
+directories containing a single file.
This command compares files by first comparing their sizes, hence goes
reasonably fast.
.SH "OPTIONS"
.TP
+\fB-v\fR, \fB--version\fR
+print the version number and exit
+.TP
\fB-h\fR, \fB--help\fR
-display help and exit
+print the help and exit
.TP
\fB-d\fR, \fB--ignore-dots\fR
ignore files and directories starting with a dot
ignore empty files
.TP
\fB-c\fR, \fB--hide-matchings\fR
-do not show which files from DIR2 corresponds to files from DIR1
+do not show which files from DIR2 correspond to files from DIR1
(hence, show only the files from DIR1 which have an identical twin in
DIR2)
.TP
\fB-g\fR, \fB--no-group-ids\fR
do not show the file group IDs
.TP
+\fB-t\fR, \fB--time-sort\fR
+sort files in each group according to the modification times
+.TP
+\fB-q\fR, \fB--trim-first\fR
+do not print the first file in each group
+.TP
\fB-p\fR, \fB--show-progress\fR
show progress information in stderr
.TP
\fB-i\fR, \fB--same-inodes-are-different\fR
files with same inode are considered as different
.TP
-\fB-m\fR, \fB--md5\fR
-use MD5 hashing
+\fB-e \fI<command>\fR, \fB--exec \fI<command>\fR
+execute the provided command for each group of identical files, with
+their names as arguments
+.TP
+\fB-f \fI<string>\fR, \fB--result-prefix \fI<string>\fR
+for each group of identical files, write one result file whose name is
+the given prefix string followed by the group number, and containing
+one file name per line
.SH "BUGS"
None known, probably many. Valgrind does not complain though.
-The MD5 hashing is not satisfactory. It is computed for a file only if
-the said file has to be read fully for a comparison (i.e. two files
-match and we have to read them completely).
+Since files with same inodes are considered as different when looking
+for duplicates in a single directory, there are weird behaviors -- not
+bugs -- with hard links.
-Hence, in practice lot of partial MD5s are computed, which costs a lot
-of cpu and is useless. This often hurts more than it helps, hence it
-is off by default. The only case when it should really be useful is
-when you have plenty of different files of same size, and lot of
-similar ones, which does not happen often.
+The current algorithm is dumb, as it does not use any hashing of the
+file content.
-Forcing the files to be read fully so that the MD5s are properly
-computed is not okay neither, since it would fully read certain files,
-even if we will never need their MD5s.
+Here are the things I tried, which did not help at all: (1) Computing
+md5s on the whole files, which is not satisfactory because files are
+often not read entirely, hence the md5s cannot be properly computed,
+(2) computing XORs of the first 4, 16 and 256 bytes with rejection as
+soon as one does not match, (3) reading files in parts of increasing
+sizes so that rejection could be done with only a small fraction read
+when possible, (4) using mmap instead of open/read.
.SH "WISH LIST"
The format of the output should definitely be improved. Not clear how.
-Their could be some fancy option to link two instances of the command
-running on different machines to reduce network disk accesses. Again,
-this may not help much, for the reason given above.
+There could be some fancy option to link two instances of the command
+running on different machines to reduce network disk accesses. This
+may not help much though.
.SH "EXAMPLES"
ignore empty files, and ignore files and directories starting with a
dot.
+.B finddup -qtg
+
+.fi
+List all files which are duplicated in the current directory, do not
+show the oldest in each each group of identical ones, and do not show
+group numbers. This is what you could use to list what files to
+remove.
+
.P
.B finddup sources not:/mnt/backup
.fi
List groups of files with same content which exist both in
\fB./tralala/\fR and \fB./cuicui/\fR. Do not show group IDs, instead
-write an empty lines between groups of files of same content.
+write empty lines between groups of files of same content.
.SH "AUTHOR"
projects / finddup.git / blobdiff