General :: App To Find And Remove Duplicate Images ?
Dec 13, 2010I am looking for a Linux app that can find and remove duplicate images (with different filenames if that's at all possible).
View 5 Replies
ADVERTISEMENT
General :: Find And Remove Duplicate Phrases In A Document ?
Mar 29, 2011I would like to find a command which automatically finds and removes phrases which appear more than once in a text file. I still want to keep one of these phrases, but I only want to see one of them.
View 9 Replies View RelatedSoftware :: Imagemagick To Find Duplicate Images
Nov 18, 2010I have a folder that over 70,000 images (within it is a complex hierarchy of subfolders). Of these 70,000 images I assume that I only have ~10,000 unique images; the rest are copies that have been resized. I would like to somehow delete all of the resized copies of the larger originals and remove them, keeping only the original image.
Is there a way to use imagemagick (or any other application) to scan this folder recursively and determine which files may be (resized) copies?
Ubuntu :: Software That Can Compare And Find Similar & Duplicate Images For Ubuntu?
Apr 1, 2011Here's the problem I hope to solve: I had to recover a hard drive with pictures and other data on it after the file system was erased. Now, in some cases there are 5 or more instances of the same image with different sizes and different file names. Sometimes the images are also in different formats such as jpeg or pnm...
Some of these images may be duplicates of images in a backup hard drive. I am just asking to see if there is a software that can visually compare images, find visual similarities, and ask me which to keep.
General :: Using Sed To Remove Lines With Duplicate ID's?
May 8, 2010I have a file that contains lines representing the nodes of a polyline but I only need the first point in each segment. With the following text:
0,"013A",0.57,260739.891,4379258.87
0,"013A",0.57,260737.674,4379258.94
0,"013A",0.57,260684.628,4379258.35
1,"013A",0.545,260769.915,4379257.84
1,"013A",0.545,260739.891,4379258.87
[Code]....
The problem with uniq is that the last two colums will differ. I don't care about the x/y for any points following the first one.
General :: Remove Duplicate Program - Older One?
Feb 23, 2010I did apt-get install qtcreator and it installed qt 4.5.3(qt4.5.2real) I had qt 4.5.2. If I go in Applications->programming I see 2 shortcuts for qtcreator, one of them being newer. How do I remove the older one? On another note, if I want to update Qt to 4.6 what would be the steps if I already have qt 4.5
View 1 Replies View RelatedGeneral :: Remove Duplicate Files From Two Folders?
Aug 19, 2010I have two folders - Folder abc and Folder xyz which contains 1000's of files with few of them having the same file names. How can I remove the duplicates from Folder abc?
View 14 Replies View RelatedGeneral :: Remove Duplicate Words Within A Particular Text In A File?
Jul 22, 2011I am basically trying to remove duplicate words in my <title></title> tag after I got hit by Google Panda. I have around 750 .html files and it will be difficult for to me remove one by one. I am looking for a way to remove only from within <title> </title>
Example of a duplicate title I have:
Code:
<title>Pasta, Pasta Recipe and Pasta Guide</title>
I dont want to replace those words anywhere else in the file except for within the <title>
General :: Find Duplicate Files By Name ?
Jul 24, 2010We have a huge amount of duplicate files in a folder and I would like some pointers on to writing a bash script to create a list of the duplicate files. I've seen examples that check for the md5 sum of files... but I dont need that, the file name is enough.
View 4 Replies View RelatedGeneral :: Remove Lines From A Syslog Text File That Have Duplicate Strings
Mar 17, 2011Trying to remove lines from a syslog text file that have duplicate strings
Mar 10 06:51:11[http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360]
then a few lines down
Mar 10 06:52:03 [http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360
got the same thing in terms of a u: number but the issue is I need to remove duplicates and just leave one and the file has multiple duplicates of different u: numbers and it's 14,000 lines long. can anyone tell me if I can use awk? sed? or sort for something like this to? removing lines that have a certain string in there that's a duplicate.
General :: End Of Support For Mint6 - Safely Remove The Partitions And Duplicate Operating Systems
Apr 14, 2010The bad news comes that active support for Mint6 is set to end Apr. 30. The worse news is I don't know what to do about it. Complicating this is that I have about 5 drive partitions and duplicate Mint6 operating systems because of password problems and just partitioning the drive and rebooting the OS instead of trying to fix the issue. I hear good things about Mint8, but my 80 Gig drive is getting pretty thin on partitions. I know there must be a way to safely remove the partitions and duplicate operating systems. I just don't know how to do it.
View 6 Replies View RelatedGeneral :: Find Duplicate Files Using Bash Script ?
Jun 8, 2010What kind of method to find the duplicates files on linux,
1.how to find just using the file name, sometimes i figure out people often to copy their files to another directory and i want to find out if there any same file name in the linux box.
2. what about if i want to find the duplicate files based on contents of the file, example is in picture file if users store picture files from digital camera first they just save the file name in default but when they want to give that picture to others they will rename it, i've been used method md5 for this situation in python script but it takes long time
I'm asking this question just to know to use bash script a lot in work and i want to test out fdupes at home, is fdupes use similar md5 scan to find duplicate files?
General :: BASH: Find Duplicate Files (MAC/LINUX Compatible)?
Mar 18, 2011I am looking for a bash script which is compatible with Mac, to find duplicate files in a directory.
View 2 Replies View RelatedGeneral :: Filtering Out Duplicate Lines From A Find/grep Output
Mar 22, 2010I have some big files of logs that contain errors printed by an app. They are most of the time relevant, however most of them are similar. So i figured i could check what happened between a time interval with a find.
Im using this one
Code:
And I get an output similar to this one.
Code:
Is there a way to condensate the output lines to get only one or two, indicating the start and last occurrence of a block? Or I need to create a program to do so?
Because right now I get thousands of similar lines, but when I'm scrolling through them i sometimes miss relevant information that i would've otherwise noted if it wasn't all that spammy.
General :: Find Duplicate Files And Delete All Except Most Recent Version
Aug 17, 2010I have a directory containing a ton of photos, some of which are duplicates but just with different names. Is there any way in linux to find all the duplicates and remove all of them except the most recent version? I know on Windows there are utilities that will do this through a GUI, but I'm using Linux through the CLI only.
View 6 Replies View RelatedGeneral :: Find All Debian Release Iso Images With Kernel 2.4.x?
Jun 9, 2010where can I find all Debian release iso images with Linux kernel 2.4.x?
View 3 Replies View RelatedProgramming :: Remove Duplicate Entries On A Row?
Nov 1, 2010we have a variable LIBS:
LIBS='-lpq -lmysqlclient -lssl -lpthread -lresolv -lssl -lpthread -lresolv'
I'd want to get:
LIBS='-lpq -lmysqlclient -lssl -lpthread -lresolv'
Bear in mind that LIBS can be variable, I mean I need to drop any duplicate and only retain the last one of each different entry. And we must keep the order as is, I must not sort out them.
So, if LIBS is:
LIBS='-lpq -lmysqlclient -lpthread -lresolv -lpthread -lresolv'
I need: LIBS='-lpq -lmysqlclient -lpthread -lresolv'
Fedora :: Remove Duplicate Menu Items?
Oct 21, 2009I have Fedora 11 installed-32bit-with xfce installed as the desktop. When I click on the fedora icon for the menu and select Preferences, there are 2 input methods listed even though I did not have any installed.Since there is no menu editor any more, does anybody know how to edit the menu so that I can get rid of these entries?
View 1 Replies View RelatedOpenSUSE :: Remove Duplicate Notification Popup?
Dec 19, 2010there is this unknown notification popup appear on top left of the screen other than the one on the panel. Anyone have experience on remove the top notification popup? this is my root account that i mainly use everyday, but if i created new account the top navigation not exist.
View 2 Replies View RelatedProgramming :: Remove Duplicate Entries From First Column?
May 12, 2010I have a huge (over 10 gb) file with a list of IP's each followed by a corresponding number like this:
Code:
12.32.34.23 10
143.32.34.543 11
232.32.45.65 12
54.23.5.232 13
143.32.34.43 14
and so on..
I'm trying to sort this file numerically and weed out any duplicate IP addresses. How do I do this on bash? I have come up with this but obviously it does'nt work.
Code:
$sort -n myfile.txt | cut -f1 | uniq -u
Programming :: Script To Remove Duplicate Jpg Files
May 25, 2010Thanks y'all for the great script and explanation. This helped a lot in my own project. I thought I'd share the efforts.The project is this: I've got lots of duplicate JPGs from all the family members who've named the same photo with different names. Since md5sum generates a "fingerprint" based on the file contents, not the name, I want to use the md5sum of each jpg to uniquely name each photo and also remove exact duplicates.
It has the following flaws:
0) it doesn't handle certain non-alphanumerics
1) it keeps both photo-shopped and unaltered photos (different md5s)
2) it (currently) doesn't preserve descriptive filenames.
(For me, removal of duplicates is more important than keeping the filenames. I may change that to concatenate the md5 and the filename.)Please note that the commented "rename" command should be used to strip non-aphanumerics from the file names, and the script should be launched with the commented "find" command.
Ubuntu Multimedia :: Program To Remove Duplicate Songs?
Feb 12, 2010I'm looking for a program to remove songs I downloaded more than once. They're all tagged differently, and some are of varying sizes/lengths/types.
View 1 Replies View RelatedUbuntu :: Remove Duplicate Lines From A Text File?
Dec 16, 2010Contained within each of these 67 text files is about 1 million urls. Yes. I have 67 text files that contain 1 million lines of urls each. I am sure I am swimming in duplicates. I tried opening one text file and clicking sort ----->remove duplicates. Now Gedit is not responding my processor is maxed out to 100% and I think I am finally ready to delve into some command line code. Can anyone give me idiot proof instructions on how to sort the duplicates out of each one of these 67 text files? How about no duplicates across all 67?
View 7 Replies View RelatedProgramming :: Remove Duplicate Lines From Shell Script
Apr 8, 2010I have a file with semi duplicate lines, like:
abc 12 32
agsi 82
sha 26
abc 1
iaij
agsi 3
Now I want to edit my file and make it:
abc 12 32
agsi 82
sha 26
iaij
i.e. remove second occurrence of line when 1st column is abc or agsi.
Ubuntu :: Remove Duplicate Lines In Plain Text File?
Apr 14, 2010i have a big file of random numbers i generated at some point in time, after working with it with different things(how fun that was)... i want to remove duplicate lines and i'm not sure i'm doing this right
heres the command
Code:
sort random.txt | uniq -u > rand-shorter.txt
the file is pretty big, everything on a new line. i found the command on a web site so i'm sure its correct(bit of a command line in linux newbie)
can anyone confirm if this will remove lines duplicate lines (keeping one copy) and dump what is left in a file named rand-shorter.txt?
EDIT: i think its actually working, just taking a reallllly long time (on an old pen 4 from 2000)
Debian :: Duplicate Files In User Library - Safe To Remove Outdated
Feb 8, 2011I have just upped from lenny to squeeze. I didn't mean to, really, but the package manager was well into its stride by the time I realised what was happening. Mostly all went well, BUT /usr is now 100% full. I notice that there are duplicate files in /usr/lib, eg Oct 11 22:35 libgcj.so.10.0.0 and Sep 14 2008 libgcj.so.90.0.0 (I assume the latter has been replaced by the former?). Is it safe to remove the "outdated" lib files? Is there an elegant way of doing it?
View 4 Replies View RelatedApplications :: Find Duplicate Files In My Folders
Jul 2, 2010I have found some duplicate files in my folders. Is there a way to clean them out?
View 2 Replies View RelatedSoftware :: Best Tool To Find Duplicate Files ?
May 16, 2010I have a 1TB drive that has MANY duplicate files all over it. a good linux tool that can find duplicate files on such a large drive (almost full) drive?
View 1 Replies View RelatedSoftware :: Find Duplicate Files - Program With Gui ?
Jan 29, 2011I would like to check two folders for duplicate files (two pretty old backup instances).
MY folders are quite alike so I would like to stop the NON-duplicated files for that I want to be able to do some checks not only for the filename but alsofor the filesize (might be the case that two files have same name but not size).
The ideal would be to suggest me such a program with gui but if not I will try run any script code that is available outhere.
OpenSUSE :: Find Duplicate Files In Multiple Disks?
Jan 11, 2011I am looking for an application (better kde one) that can search two external hard disks I have and find any duplicate files. I did some backups before to one disk which i copied few years ago to the other disk. Right now I would like some program to check files and tell me if there are the same.
View 9 Replies View Related