Ubuntu :: Write A Script To Go Through And Delete Bulk Duplicate Files?
Jul 7, 2010
I am using my Ubuntu machine to serve as a media server and network storage. The problem I have is iTunes on my desktop managed to make 2 copies of every song on the machine so instead of the 30GB I have its up to almost 100gb. I was wondering if there was a way to write a script to go through and delete the duplicates. The duplicates are the same filename as the original except a 1 or 2 following. Wasn't looking forward to deleting 12,000 files by hand.
Is there a way to remove duplicate files from a specific folder through SSH? I've uploaded a lot of flash games on my server and I can see in the Webmin's file manager that I have many duplicates. Their names are different, of course.
I'm using a mac, and just transferred a bunch o photos from another computer, and as it turns out, there is a bunch of duplicates.I'm not too familiar with the mac terminal, but if there is a solution for linux, it will probably work for the mac.Just need to be able to recursively scan all folders in my Pictures folder and then Delete them.
I have a directory containing a ton of photos, some of which are duplicates but just with different names. Is there any way in linux to find all the duplicates and remove all of them except the most recent version? I know on Windows there are utilities that will do this through a GUI, but I'm using Linux through the CLI only.
Is there a way to remove users in bulk? Maybe by a range of user id numbers.
I help run a school server for our Linux class. I create temp accounts for tests and now I just want to delete them. I really should delete the old users too but the Professor likes to let them have access even after the class.
i have to write a shell script that will delete all the .dat files in /var/oracle/etl/incoming which the created date of the file is 7 days before the currrent date.
I couldn't find my previous posting about udev at booting, so I created a new posting with the same material. Subsequently my earlier post showed up again----or I managed to find it. I can't seem to find any information about how I go about deleting the earlier post. Supposedly we are supposed to be able to notify the managers of abusive posts. I thought I might use that to ask them to delete the earlier post, so I will try that..
I have a Netgear ReadyNAS NV in the basement, that I want to use to serve up video files over my network to a TV in the living room.
Now, I have a lot of files that HandBrake encoded and it gave the files an m4v suffix. Even when the files are in a codec that the TV can handle, it refuses to load them because of this suffix... so I want to rename them all.
This is fairly simple for files on a local filesystem. I can simply cd into the directory containing the files, and do something like the commands below.
Code: $ for a in `ls`; > do > stem=`echo ${a} | cut -f1 -d"."` ; > mv ${a} ${stem}.mpg ;
[Code]....
Although there are a few smb commands available (smbstatus, smbget, etc.), I've not found any commands like smbls or smbmv.
Are there any special commands or utilities around that can do the kind of thing I'm trying to do?
I used awk "'!x[$0]++' test.txt > file.new" ,but it deleted #1 also.I tried using uniq command but i didn't work. Can anyone Please let me know is there any way to do this using shell script.
I need to copy a large number of files, it comes to 1 lakh from one server to another. When I tried various commands using scp , ftp etc. It is saying "Arg list too long". In which way can we copy all the files. The Two servers are under Linux.
I've been spinning my wheels for a bit on this one not getting any traction. I have a list of pictures that have a bad extension that I would like to rename with the good extension. Here's a snippet of the list that I'm looking at:
listold Code: /Volumes/MyFiles/Pictures/2009/02 - February/Qeirstyn Playing Inside/._IMG_3347.JPG /Volumes/MyFiles/Pictures/2009/02 - February/Qeirstyn Playing Inside/._IMG_3349.JPG /Volumes/MyFiles/Pictures/2009/02 - February/Qeirstyn Playing Inside/._IMG_3350.JPG /Volumes/MyFiles/Pictures/2009/02 - February/Qeirstyn Playing Inside/._IMG_3354.JPG
I am trying to use an old box as backup server. I have tried a couple of possibilities along the lines of:
Quote:
rsync -a --delete --progress --log-file=/home/$USER/info.txt -e ssh /home /etc root@192.168.0.106:/mnt/back
The problem is it does not delete files that has been removed from my local system? I run the command as root on the local system.
(I realize I should properly not ssh into the server as the server's root but I'm having trouble with the permissions and I want to make sure everything else works before messing around with it)
Was wondering if any perl guru's could help me with a quick log file adjustment. I have a text file that looks like so (tabs and newlines are revealed so you can see what separates the data):
There are maybe 100 lines of text in this file at any given time. I need to delete all duplicate lines only looking at the first bit of text prior to the first tab. It doesn't matter which one gets deleted as long as there are no two lines that begin with that same text at the beginning before the first tab. So in this example, either the fist line "1234" or the last line "1234" would need to be deleted. I already have code in my script that opens the files - I just need the code to read the text into an array and the part that would find matches based on the above criteria, and make the deletions.
If it would be easier, I can even do a system call and use SED (v4.1.5) and/or AWK (3.1.5) instead.
# ls -1 *.mp3 Megadeth - Killing Is My Business...And Business Is Good! - 01 - Last Rites-Loved To Death.mp3 Megadeth - Killing Is My Business...And Business Is Good! - 02 - Killing Is My Business...And Business Is Good.mp3 Megadeth - Killing Is My Business...And Business Is Good! - 03 - Skull Beneath The Skin.mp3 [Code]....
i want to rename it to:
01 - Last Rites-Loved To Death.mp3 02 - Killing Is My Business...And Business Is Good.mp3 03 - Skull Beneath The Skin.mp3 04 - These Boots.mp3 05 - Rattlehead.mp3 06 - Chosen Ones.mp3 07 - Looking Down The Cross.mp3 08 - Mechanix.mp3
here is my RnD so far:
[Code]....
so it does show that i can run "mv" and reame the files the way i want. BUT...there are lots of spaces in each file, and hence when I append "|sh" to above command to have him actually rename the files, my "mv" command is failing. BTW, I tried exploring perl "rename" command, but I could not understand it's regex.
I just can't stand knowing that there's a slight problem with my PC.I have roughly 12.5 Gigs of files, mostly movies that are multiple clones of a particular movie (which was an entirely different problem altogether) and I CANNOT DELETE THESE THINGS! There has to be a simple way to do it from terminal, problem is, I can't seem to find the trash directory in terminal.
I want to write a shell script which will simultaneously collect OS user information and write in an individual text files.Can anyone tell me the syntax of the script.N.B. The user name will be mentioned in an array within the shell script.
share a mount"/RAID" on my server: 192.168.0.2 with everyone on my network.192.168.0.* From what I have listed below I am able to mount the share but I can not write or delete anything. It is almost like it is ro only permissions.
From the Server:
Code:
sudo cat /etc/exports [sudo] password for jesse: # /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5).
[code]....
Eventually I need to share 192.168.0.2:/RAID on two OSX computers as well. I read some places where you need to add insecure to your /etc/exports on your server in order for the OSX client to access the the share.
I am working through a collection of many thousand photographs in many collections, spanning decades. One of the folders is a "Selected_images" folder which should only contain copies of images from the collections.
How can I check that the files in "Selected_images" are all indeed copies and have an identical counterpart somewhere in the collections hierarchy, not necessarily with the same name?
At present I am scanning the output of fdupes -r collection, which is tedious (although, in fact, no photographs should ever be duplicated except in "Selected_images").
Code: mount /dev/sdd1 on /media/E0FD-1813 type vfat (rw,nosuid,nodev,uhelper=udisks,uid=1000,gid=1000,shortname=mixed,dmask=0077,utf8=1,showexec,flush) cp 'Aankhon Aankhon Mein hum tum ho gaye deewane.mp3' /media/E0FD-1813/Music/sumeet/a cp: cannot create regular file '/media/E0FD-1813/Music/sumeet/a/Aankhon Aankhon Mein hum tum ho gaye deewane.mp3': Read-only file system The micro sd card mounts & works as fat32 in windows xp just fine.
We have a huge amount of duplicate files in a folder and I would like some pointers on to writing a bash script to create a list of the duplicate files. I've seen examples that check for the md5 sum of files... but I dont need that, the file name is enough.
I have two directories, I want to know which files in the second directory also appear in the first and delete the duplicate in the second directory. Filenames might be different (so that rules out diff).
My problem is that various programs (such as fdupes and freedup) are very capable of finding duplicate files but randomly delete (or link) files from the first or the second directory.
Here an example with fdupes:
Code:
As you can see, the file in the third pair is removed from dir1 instead of from dir2. My aim is to have only files deleted from the dir1. I know that fdupes can't do this, as I emailed with the author.
I just got a new PC with 2TB of HDD space and it's running Ubuntu 10.04.1 32bit. I have created a user for networking, called share and have created a folder within the home folder called public. I then shared the folder and gave 777 access to it as I want this to be public - but for my local network and my later for ssh access. I have installed samba, and under the sharing options I allowed for guest logins and people to write and delete from folder. SO how would I be able to "see" this from my windows machine? It's running XP pro and it has a domain other than workgroup - but that shouldn't affect it should it?
Also, the laptop with XP is on the wireless with and internal IP of 10.0.0.135 and my PC is on the LAN connection with and internal IP of 10.0.0.1 I'm sort of familiar with networks, but I have NO clue as to how to do this.
Long time reader, first time poster. I've got, what has become to me, a brain bender. It seems ACL's are the best way to go, but I am not 100% sure. Each user should be able to create files and modify each others'files, but should not be able to delete any one elses files in a directory.chmod -1777?setfacl?
On a server in an office, I want a situation where people on the client PC are able to write to a particular file, but not have privileges to delete it -
