Bear in mind that LIBS can be variable, I mean I need to drop any duplicate and only retain the last one of each different entry. And we must keep the order as is, I must not sort out them.
I have a huge (over 10 gb) file with a list of IP's each followed by a corresponding number like this:
Code:
12.32.34.23 10 143.32.34.543 11 232.32.45.65 12 54.23.5.232 13 143.32.34.43 14 and so on..
I'm trying to sort this file numerically and weed out any duplicate IP addresses. How do I do this on bash? I have come up with this but obviously it does'nt work.
I'm getting duplicate entries in mysql database when submitting a new item from a form. I initially loaded the database via IMPORT function in phpMyAdmin. This is my input form:
Theres two main things im trying to remove. First, Wine which I need to keep, has about 15 different entries which I want to remove. Second I installed OpenMPT, a music tracking program through Wine. This added about 10 duplicate open with entries.. I uninstalled OpenMPT, because I decided I didnt want it, but all the entries remain.
When I highlight one of the entries, the remove button is still disabled. Its disabled for Wine, OpenMPT, and every other program in the list.
Thanks y'all for the great script and explanation. This helped a lot in my own project. I thought I'd share the efforts.The project is this: I've got lots of duplicate JPGs from all the family members who've named the same photo with different names. Since md5sum generates a "fingerprint" based on the file contents, not the name, I want to use the md5sum of each jpg to uniquely name each photo and also remove exact duplicates.
It has the following flaws: 0) it doesn't handle certain non-alphanumerics 1) it keeps both photo-shopped and unaltered photos (different md5s) 2) it (currently) doesn't preserve descriptive filenames.
(For me, removal of duplicates is more important than keeping the filenames. I may change that to concatenate the md5 and the filename.)Please note that the commented "rename" command should be used to strip non-aphanumerics from the file names, and the script should be launched with the commented "find" command.
I have made a custom grub2 menu however, both the default and the custom show together. So my grub looks like the list below, the bolded entries are my custom ones. How do I get rid of the duplicates? I have tried apt-get remove and deleting old kernels.
ubuntu,linux ... ubuntu,linux recovery memtest memtest windows7 windows7 ubuntu linux ubuntu linux recover
I used to be able to resolve this problem by simply opening my sources.list file and doing a search and delete, but now Ubuntu appears to have two completely different methods of handling sources. What is is the simple solution? I would really like to have everything all in one place.
Code: W: Duplicate sources.list entry http://ppa.launchpad.net karmic/main Packages (/var/lib/apt/lists/ppa.launchpad.net_sunab_ppa_ubuntu_dists_karmic_main_binary-i386_Packages) W: Duplicate sources.list entry http://ppa.launchpad.net karmic/main Packages (/var/lib/apt/lists/ppa.launchpad.net_mozillateam_firefox-stable_ubuntu_dists_karmic_main_binary-i386_Packages) W: You may want to run apt-get update to correct these problems
To start by saying that I am brand new to anything linux. I just installed Ubuntu 10.10 32-bit to dual boot on my Windows 7 32-bit PC. This last time that I booted my pc the grub looked like this:
Ubuntu, Linux 2.6.32-25-generic Ubuntu, Linux 2.6.32-25-generic (recovery mode) Ubuntu, Linux 2.6.32-24-generic Ubuntu, Linux 2.6.32-24-generic (recovery mode) Windows 7 (loader) (on/dev/sda1) Windows 7 (loader) (on/dev/sda2)
From what I understand the duplicate for Ubuntu is just an updated Kernal, hence the -24 to -25. What I didn't understand is why there is a duplicate for Windows 7 and what the sda1 and sda2 is? Someone please explain. When I first installed Ubuntu, there was only one listing for Ubuntu and one for Windows 7.
Also, I was wondering what you do to keep up with new kernals and remove old one's. How do you keep the Kernals from piling up? I assume that when another Kernal becomes available, there will then be 3 Ubuntu listings in the grub. One last thing is I would like some suggestions for the best way of going about learning linux. (i.e. terminal commands)
For the past few weeks I have been using the lastest Ubuntu OS. The following menu pops-up during the boot process.
GNU GRUB version 1.98+20100804-5ubuntu3 Ubuntu, with Linux 2.6.35-22-generic-pae Ubuntu, with Linux 2.6.35-22-generic-pae (recovery mode) Memory test (mentest86+) Memory test (mentest86+, serial console 115200) Windows 7 (loader) (on /dev/sdb1)
Just recently the menu displays duplicate entries. The first two entries are repeated (see below). What might have caused these duplicate entries to appear and how to correct them? Before this problem occurred I did not issue any sudo commands.
GNU GRUB version 1.98+20100804-5ubuntu3 Ubuntu, with Linux 2.6.35-22-generic-pae Ubuntu, with Linux 2.6.35-22-generic-pae (recovery mode) Ubuntu, with Linux 2.6.35-22-generic-pae Ubuntu, with Linux 2.6.35-22-generic-pae (recovery mode) Memory test (mentest86+) Memory test (mentest86+, serial console 115200) Windows 7 (loader) (on /dev/sdb1)
I see this posted in several places in the past but no answers. If I use UUID in fstab to mount additional partitions I get duplicate entries for the partitions in the Places Menu. I can solve the problem by mounting the partitions by Label ( /dev/sdb2 /mountpoint ) but half of the time grub2 will not mount the drive correctly using this method and will change the volume ID (sdb2 becomes sdf2) .
I just upgraded from 10.04 to 10.10. I originally had installed the nvidia drivers using the bash script from nvidia's website. This time around I used the ones from synaptic. However now when I go and look in System -> Administration there are 2 entries for "NVIDIA X Server Settings". The first one has the NVIDIA logo, the other one has a grey question mark. Does anyone know why there are 2 entries and how I can remove one of them?
I have Fedora 11 installed-32bit-with xfce installed as the desktop. When I click on the fedora icon for the menu and select Preferences, there are 2 input methods listed even though I did not have any installed.Since there is no menu editor any more, does anybody know how to edit the menu so that I can get rid of these entries?
there is this unknown notification popup appear on top left of the screen other than the one on the panel. Anyone have experience on remove the top notification popup? this is my root account that i mainly use everyday, but if i created new account the top navigation not exist.
I did apt-get install qtcreator and it installed qt 4.5.3(qt4.5.2real) I had qt 4.5.2. If I go in Applications->programming I see 2 shortcuts for qtcreator, one of them being newer. How do I remove the older one? On another note, if I want to update Qt to 4.6 what would be the steps if I already have qt 4.5
I have two folders - Folder abc and Folder xyz which contains 1000's of files with few of them having the same file names. How can I remove the duplicates from Folder abc?
Contained within each of these 67 text files is about 1 million urls. Yes. I have 67 text files that contain 1 million lines of urls each. I am sure I am swimming in duplicates. I tried opening one text file and clicking sort ----->remove duplicates. Now Gedit is not responding my processor is maxed out to 100% and I think I am finally ready to delve into some command line code. Can anyone give me idiot proof instructions on how to sort the duplicates out of each one of these 67 text files? How about no duplicates across all 67?
I would like to find a command which automatically finds and removes phrases which appear more than once in a text file. I still want to keep one of these phrases, but I only want to see one of them.
I am basically trying to remove duplicate words in my <title></title> tag after I got hit by Google Panda. I have around 750 .html files and it will be difficult for to me remove one by one. I am looking for a way to remove only from within <title> </title>
Example of a duplicate title I have:
Code:
<title>Pasta, Pasta Recipe and Pasta Guide</title>
I dont want to replace those words anywhere else in the file except for within the <title>
In Xubuntu, the xfrun4 dialog (alt + F2) doesn't have auto-completion for every command, only for those commands successfully called at some previous point in time. My question is: where are these entries located, and how can I remove some of them?
I'm just curious because I'd like to keep my entries uncluttered. I was experimenting earlier to see whether xfrun4 would retain an invalid command, so I tried the inexistent (so I thought) command "terminal", but lo and behold, it opened a new tab in Firefox and brought me to www.terminal.com. Now I have the useless command "terminal" saved in xfrun4, and I'd like to get rid of it.
i have a big file of random numbers i generated at some point in time, after working with it with different things(how fun that was)... i want to remove duplicate lines and i'm not sure i'm doing this right
I have just upped from lenny to squeeze. I didn't mean to, really, but the package manager was well into its stride by the time I realised what was happening. Mostly all went well, BUT /usr is now 100% full. I notice that there are duplicate files in /usr/lib, eg Oct 11 22:35 libgcj.so.10.0.0 and Sep 14 2008 libgcj.so.90.0.0 (I assume the latter has been replaced by the former?). Is it safe to remove the "outdated" lib files? Is there an elegant way of doing it?
Trying to remove lines from a syslog text file that have duplicate strings
Mar 10 06:51:11[http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360]
then a few lines down
Mar 10 06:52:03 [http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360
got the same thing in terms of a u: number but the issue is I need to remove duplicates and just leave one and the file has multiple duplicates of different u: numbers and it's 14,000 lines long. can anyone tell me if I can use awk? sed? or sort for something like this to? removing lines that have a certain string in there that's a duplicate.
I'm trying to automatically remove some of the old GRUB entries that pile up after a few updates and just want to leave live 2 or 3 there along with my Win7 entry. I was looking and saw that there is a utility for Ubuntu but I couldn't find anything for Fedora.
I have to hide Unix partitions to succeed in installing Linux and the winner is Ubuntu 10.10! I had to remove whole partitions entries. DOS based primary/logical partitions are now called (hd0,msdos1), (hd0,msdos2), etc. And BSD slices would have been (hd0, bsd1) (hd0,bsd2). I guess I saw those names during installation. update-grub failed in three attempts in a row on 2 machines (32 and 64 bit). Everything went fine after I removed the Unix partitions (which are going to be restored by openSUSE legacy Grub)
