+- Support Forums (https://www.supportforums.net)
+-- Forum: Categories (https://www.supportforums.net/forumdisplay.php?fid=87)
+--- Forum: Coding Support Forums (https://www.supportforums.net/forumdisplay.php?fid=18)
+---- Forum: Perl Programming Support (https://www.supportforums.net/forumdisplay.php?fid=31)
+---- Thread: Duplicate File Finder (/showthread.php?tid=15675)
Duplicate File Finder - AceInfinity - 01-17-2011
Here's a duplicate file finder perl script that i've modified a few times. It will look through files of a similar size and compare them by looking through their MD5 hashes to find duplicate files. This is VERY useful if you have copies of music or something within a same directory but with a different name, because this will detect those copies, and you can delete them manually to free up some space on your hard drive.
I didn't make it to automatically remove files just so that you have the option yourself to decide whether or not to delete them.
Code:
#!/usr/bin/perl -w
use strict;
use File::Find;
use Digest::MD5;
my %files;
my $wasted = 0;
find(\&check_file, $ARGV[0] || ".");
local $" = "\n";
foreach my $size (sort {$b <=> $a} keys %files) {
next unless @{$files{$size}} > 1;
my %md5;
foreach my $file (@{$files{$size}}) {
open(FILE, $file) or next;
binmode(FILE);
push @{$md5{Digest::MD5->new->addfile(*FILE)->hexdigest}},$file;
}
foreach my $hash (keys %md5) {
next unless @{$md5{$hash}} > 1;
print "\n";
print "\n";
print "($size bytes) Duplicate Files:\n";
print "@{$md5{$hash}}\n";
print "\n";
$wasted += $size * (@{$md5{$hash}} - 1);
}
}
1 while $wasted =~ s/^([-+]?\d+)(\d{3})/$1,$2/;
print "\n";
print "######################################################\n";
print " \n";
print " You have $wasted bytes total in duplicate files \n";
print " \n";
print "######################################################\n";
print "\n";
sub check_file {
-f && push @{$files{(stat(_))[7]}}, $File::Find::name;
}Put this in the directory that you want to look though and run it from it's filename within cmd prompt. It will compare files from different folders as well.
Enjoy
RE: Duplicate File Finder - AceInfinity - 01-22-2011
No one active in perl programming i'm assuming?
Even if you don't understand it, I would recommend using this script to find file duplicates. It will find file duplicates by using it's MD5 hash, and compares files by file size to check. So unless the MD5 changes, it will detect any file duplicates. If the MD5 changes, that means the file is not the same or has been modified. Enjoy
RE: Duplicate File Finder - Caaz - 02-14-2011
This is very interesting. I might actually use this later since I have a ton of images I need to sort through. It's pretty simple for what it does as well. I like.
RE: Duplicate File Finder - AceInfinity - 02-14-2011
You'll find it very useful. I'm suprised there aren't more people who have experience in perl scripting. They are missing out on lots of good things you can do with the programming language.
RE: Duplicate File Finder - eax - 04-30-2011
(02-14-2011, 09:06 PM)Infinity Wrote: You'll find it very useful. I'm suprised there aren't more people who have experience in perl scripting. They are missing out on lots of good things you can do with the programming language.
I personally like Ruby, it has the best bits of Perl and Python. Nice useful script though. Bad music in the YT vid.
RE: Duplicate File Finder - AceInfinity - 04-30-2011
(04-30-2011, 03:06 AM)eax Wrote: I personally like Ruby, it has the best bits of Perl and Python. Nice useful script though. Bad music in the YT vid.
Found it through Audioswap or whatever that youtube function is called. I already went through the rest of my youtube playlist from iTunes so I didn't know what to add. The music is just there so you don't get bored of listening to the silence though, it really has no effect on the actual video. You can mute it if you want
RE: Duplicate File Finder - Bengan - 05-08-2011
What to do with that code?
RE: Duplicate File Finder - AceInfinity - 05-08-2011
(05-08-2011, 09:28 AM)Bengan Wrote: What to do with that code?
You put it into a .pl file and you open it with your perl command line interpreter. You have to have a version of ActivePerl downloaded.
RE: Duplicate File Finder - andrewgail - 05-14-2011
A script to find and remove duplicate files in one or more directory. The program gets a speed-up by abbreviation file reads to a minimum. In a lot of cases, it alone reads small chunks from unique files and only files with duplicates are read completely.
RE: Duplicate File Finder - AceInfinity - 05-14-2011
(05-14-2011, 11:49 AM)andrewgail Wrote: A script to find and remove duplicate files in one or more directory. The program gets a speed-up by abbreviation file reads to a minimum. In a lot of cases, it alone reads small chunks from unique files and only files with duplicates are read completely.
You don't know what you're talking about sorry to say.
1) it doesn't remove the duplicate file
2) it's not a program, nothing is compiled, it's only a script that gets interpreted
3) it doesn't read small chunks of files and read the full file of duplicates because it needs to determine what the duplicate files are first