Programming :: Remove Duplicate Entries From First Column?
May 12, 2010
I have a huge (over 10 gb) file with a list of IP's each followed by a corresponding number like this:
Code:
12.32.34.23 10
143.32.34.543 11
232.32.45.65 12
54.23.5.232 13
143.32.34.43 14
and so on..
I'm trying to sort this file numerically and weed out any duplicate IP addresses. How do I do this on bash? I have come up with this but obviously it does'nt work.
Code:
$sort -n myfile.txt | cut -f1 | uniq -u
View 2 Replies
ADVERTISEMENT
Nov 1, 2010
we have a variable LIBS:
LIBS='-lpq -lmysqlclient -lssl -lpthread -lresolv -lssl -lpthread -lresolv'
I'd want to get:
LIBS='-lpq -lmysqlclient -lssl -lpthread -lresolv'
Bear in mind that LIBS can be variable, I mean I need to drop any duplicate and only retain the last one of each different entry. And we must keep the order as is, I must not sort out them.
So, if LIBS is:
LIBS='-lpq -lmysqlclient -lpthread -lresolv -lpthread -lresolv'
I need: LIBS='-lpq -lmysqlclient -lpthread -lresolv'
View 3 Replies
View Related
Apr 13, 2011
I'm getting duplicate entries in mysql database when submitting a new item from a form. I initially loaded the database via IMPORT function in phpMyAdmin. This is my input form:
<center>
<br />
<FONT SIZE="5" ><b>Add Item Template</b></FONT>
<br />
[code].....
However, I do need to return the input form in order to keep entering new items into the database.
View 8 Replies
View Related
Dec 14, 2010
Theres two main things im trying to remove. First, Wine which I need to keep, has about 15 different entries which I want to remove. Second I installed OpenMPT, a music tracking program through Wine. This added about 10 duplicate open with entries.. I uninstalled OpenMPT, because I decided I didnt want it, but all the entries remain.
When I highlight one of the entries, the remove button is still disabled. Its disabled for Wine, OpenMPT, and every other program in the list.
How can I remove these?
View 2 Replies
View Related
May 25, 2010
Thanks y'all for the great script and explanation. This helped a lot in my own project. I thought I'd share the efforts.The project is this: I've got lots of duplicate JPGs from all the family members who've named the same photo with different names. Since md5sum generates a "fingerprint" based on the file contents, not the name, I want to use the md5sum of each jpg to uniquely name each photo and also remove exact duplicates.
It has the following flaws:
0) it doesn't handle certain non-alphanumerics
1) it keeps both photo-shopped and unaltered photos (different md5s)
2) it (currently) doesn't preserve descriptive filenames.
(For me, removal of duplicates is more important than keeping the filenames. I may change that to concatenate the md5 and the filename.)Please note that the commented "rename" command should be used to strip non-aphanumerics from the file names, and the script should be launched with the commented "find" command.
View 1 Replies
View Related
Apr 8, 2010
I have a file with semi duplicate lines, like:
abc 12 32
agsi 82
sha 26
abc 1
iaij
agsi 3
Now I want to edit my file and make it:
abc 12 32
agsi 82
sha 26
iaij
i.e. remove second occurrence of line when 1st column is abc or agsi.
View 13 Replies
View Related
Mar 6, 2010
I have made a custom grub2 menu however, both the default and the custom show together. So my grub looks like the list below, the bolded entries are my custom ones. How do I get rid of the duplicates? I have tried apt-get remove and deleting old kernels.
ubuntu,linux ...
ubuntu,linux recovery
memtest
memtest
windows7
windows7
ubuntu linux
ubuntu linux recover
View 7 Replies
View Related
Mar 23, 2010
I used to be able to resolve this problem by simply opening my sources.list file and doing a search and delete, but now Ubuntu appears to have two completely different methods of handling sources. What is is the simple solution? I would really like to have everything all in one place.
Code:
W: Duplicate sources.list entry http://ppa.launchpad.net karmic/main Packages (/var/lib/apt/lists/ppa.launchpad.net_sunab_ppa_ubuntu_dists_karmic_main_binary-i386_Packages)
W: Duplicate sources.list entry http://ppa.launchpad.net karmic/main Packages (/var/lib/apt/lists/ppa.launchpad.net_mozillateam_firefox-stable_ubuntu_dists_karmic_main_binary-i386_Packages)
W: You may want to run apt-get update to correct these problems
View 2 Replies
View Related
Oct 12, 2010
To start by saying that I am brand new to anything linux. I just installed Ubuntu 10.10 32-bit to dual boot on my Windows 7 32-bit PC. This last time that I booted my pc the grub looked like this:
Ubuntu, Linux 2.6.32-25-generic
Ubuntu, Linux 2.6.32-25-generic (recovery mode)
Ubuntu, Linux 2.6.32-24-generic
Ubuntu, Linux 2.6.32-24-generic (recovery mode)
Windows 7 (loader) (on/dev/sda1)
Windows 7 (loader) (on/dev/sda2)
From what I understand the duplicate for Ubuntu is just an updated Kernal, hence the -24 to -25. What I didn't understand is why there is a duplicate for Windows 7 and what the sda1 and sda2 is? Someone please explain. When I first installed Ubuntu, there was only one listing for Ubuntu and one for Windows 7.
Also, I was wondering what you do to keep up with new kernals and remove old one's. How do you keep the Kernals from piling up? I assume that when another Kernal becomes available, there will then be 3 Ubuntu listings in the grub. One last thing is I would like some suggestions for the best way of going about learning linux. (i.e. terminal commands)
View 7 Replies
View Related
Dec 5, 2010
For the past few weeks I have been using the lastest Ubuntu OS. The following menu pops-up during the boot process.
GNU GRUB version 1.98+20100804-5ubuntu3
Ubuntu, with Linux 2.6.35-22-generic-pae
Ubuntu, with Linux 2.6.35-22-generic-pae (recovery mode)
Memory test (mentest86+)
Memory test (mentest86+, serial console 115200)
Windows 7 (loader) (on /dev/sdb1)
Just recently the menu displays duplicate entries. The first two entries are repeated (see below). What might have caused these duplicate entries to appear and how to correct them? Before this problem occurred I did not issue any sudo commands.
GNU GRUB version 1.98+20100804-5ubuntu3
Ubuntu, with Linux 2.6.35-22-generic-pae
Ubuntu, with Linux 2.6.35-22-generic-pae (recovery mode)
Ubuntu, with Linux 2.6.35-22-generic-pae
Ubuntu, with Linux 2.6.35-22-generic-pae (recovery mode)
Memory test (mentest86+)
Memory test (mentest86+, serial console 115200)
Windows 7 (loader) (on /dev/sdb1)
View 5 Replies
View Related
Apr 2, 2011
I see this posted in several places in the past but no answers. If I use UUID in fstab to mount additional partitions I get duplicate entries for the partitions in the Places Menu. I can solve the problem by mounting the partitions by Label ( /dev/sdb2 /mountpoint ) but half of the time grub2 will not mount the drive correctly using this method and will change the volume ID (sdb2 becomes sdf2) .
View 2 Replies
View Related
Nov 22, 2010
I just upgraded from 10.04 to 10.10. I originally had installed the nvidia drivers using the bash script from nvidia's website. This time around I used the ones from synaptic. However now when I go and look in System -> Administration there are 2 entries for "NVIDIA X Server Settings". The first one has the NVIDIA logo, the other one has a grey question mark. Does anyone know why there are 2 entries and how I can remove one of them?
View 1 Replies
View Related
Jul 15, 2010
I have a multicolumn datas, like
a1 b1 ... f1
a2 b2 ... f2
. . ... .
[code]...
I would like to make a file with all these data in one column, like
a1
a2
.
.
[code]....
Can it be done with awk or some other command? Also, is it possible then do add another column in front of this one with numbers of the lines (for every previous column), like
1 a1
2 a2
. .
. .
[code].....
View 14 Replies
View Related
Mar 23, 2010
I am unable to find a suitable forum to post unless anyone knows of one. V1.9c Basically I need to remove a 'ctcl' column which was configured a few years ago and is now redundant. I have trawled through the documentation and google and am basically stuck.
To cut a long story short, I have removed the $BBVAR/hist,hostlog and log but unfortunately they all reappear after 5 mins (same as bb.html and bb2.html). Ive followed nearly every suggestion again to no avail and as im not a BB expert ive hit a brick wall. I was thinking there must be a command to remove the column (as bbrm).
View 4 Replies
View Related
May 8, 2010
I have a file that contains lines representing the nodes of a polyline but I only need the first point in each segment. With the following text:
0,"013A",0.57,260739.891,4379258.87
0,"013A",0.57,260737.674,4379258.94
0,"013A",0.57,260684.628,4379258.35
1,"013A",0.545,260769.915,4379257.84
1,"013A",0.545,260739.891,4379258.87
[Code]....
The problem with uniq is that the last two colums will differ. I don't care about the x/y for any points following the first one.
View 4 Replies
View Related
Oct 21, 2009
I have Fedora 11 installed-32bit-with xfce installed as the desktop. When I click on the fedora icon for the menu and select Preferences, there are 2 input methods listed even though I did not have any installed.Since there is no menu editor any more, does anybody know how to edit the menu so that I can get rid of these entries?
View 1 Replies
View Related
Dec 19, 2010
there is this unknown notification popup appear on top left of the screen other than the one on the panel. Anyone have experience on remove the top notification popup? this is my root account that i mainly use everyday, but if i created new account the top navigation not exist.
View 2 Replies
View Related
Feb 23, 2010
I did apt-get install qtcreator and it installed qt 4.5.3(qt4.5.2real) I had qt 4.5.2. If I go in Applications->programming I see 2 shortcuts for qtcreator, one of them being newer. How do I remove the older one? On another note, if I want to update Qt to 4.6 what would be the steps if I already have qt 4.5
View 1 Replies
View Related
Dec 13, 2010
I am looking for a Linux app that can find and remove duplicate images (with different filenames if that's at all possible).
View 5 Replies
View Related
Aug 19, 2010
I have two folders - Folder abc and Folder xyz which contains 1000's of files with few of them having the same file names. How can I remove the duplicates from Folder abc?
View 14 Replies
View Related
Feb 12, 2010
I'm looking for a program to remove songs I downloaded more than once. They're all tagged differently, and some are of varying sizes/lengths/types.
View 1 Replies
View Related
Dec 16, 2010
Contained within each of these 67 text files is about 1 million urls. Yes. I have 67 text files that contain 1 million lines of urls each. I am sure I am swimming in duplicates. I tried opening one text file and clicking sort ----->remove duplicates. Now Gedit is not responding my processor is maxed out to 100% and I think I am finally ready to delve into some command line code. Can anyone give me idiot proof instructions on how to sort the duplicates out of each one of these 67 text files? How about no duplicates across all 67?
View 7 Replies
View Related
Mar 29, 2011
I would like to find a command which automatically finds and removes phrases which appear more than once in a text file. I still want to keep one of these phrases, but I only want to see one of them.
View 9 Replies
View Related
Jul 22, 2011
I am basically trying to remove duplicate words in my <title></title> tag after I got hit by Google Panda. I have around 750 .html files and it will be difficult for to me remove one by one. I am looking for a way to remove only from within <title> </title>
Example of a duplicate title I have:
Code:
<title>Pasta, Pasta Recipe and Pasta Guide</title>
I dont want to replace those words anywhere else in the file except for within the <title>
View 14 Replies
View Related
Sep 23, 2010
For some reason, Skype starts up in two instances. How can remove one of them in KDE (4.4.4)?
View 5 Replies
View Related
Feb 13, 2010
I have Googled this one and I have tried several fixes so far nothing has worked. All I want to keep is these entries in my grub menu:
"Ubuntu, Linux 2.6.31-19-generic-pae"
"Ubuntu, Linux 2.6.31-17-generic-pae"
"Microsoft Windows XP Professional"
1st - Through Synaptic I removed:
linux-headers-2.6.31-14
linux-headers-2.6.31-14-generic-pae
linux-headers-2.6.31-15
linux-headers-2.6.31-15-generic-pae
[Code]...
View 9 Replies
View Related
May 26, 2011
In Xubuntu, the xfrun4 dialog (alt + F2) doesn't have auto-completion for every command, only for those commands successfully called at some previous point in time. My question is: where are these entries located, and how can I remove some of them?
I'm just curious because I'd like to keep my entries uncluttered. I was experimenting earlier to see whether xfrun4 would retain an invalid command, so I tried the inexistent (so I thought) command "terminal", but lo and behold, it opened a new tab in Firefox and brought me to www.terminal.com. Now I have the useless command "terminal" saved in xfrun4, and I'd like to get rid of it.
View 2 Replies
View Related
Apr 14, 2010
i have a big file of random numbers i generated at some point in time, after working with it with different things(how fun that was)... i want to remove duplicate lines and i'm not sure i'm doing this right
heres the command
Code:
sort random.txt | uniq -u > rand-shorter.txt
the file is pretty big, everything on a new line. i found the command on a web site so i'm sure its correct(bit of a command line in linux newbie)
can anyone confirm if this will remove lines duplicate lines (keeping one copy) and dump what is left in a file named rand-shorter.txt?
EDIT: i think its actually working, just taking a reallllly long time (on an old pen 4 from 2000)
View 8 Replies
View Related
Feb 8, 2011
I have just upped from lenny to squeeze. I didn't mean to, really, but the package manager was well into its stride by the time I realised what was happening. Mostly all went well, BUT /usr is now 100% full. I notice that there are duplicate files in /usr/lib, eg Oct 11 22:35 libgcj.so.10.0.0 and Sep 14 2008 libgcj.so.90.0.0 (I assume the latter has been replaced by the former?). Is it safe to remove the "outdated" lib files? Is there an elegant way of doing it?
View 4 Replies
View Related
Mar 17, 2011
Trying to remove lines from a syslog text file that have duplicate strings
Mar 10 06:51:11[http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360]
then a few lines down
Mar 10 06:52:03 [http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360
got the same thing in terms of a u: number but the issue is I need to remove duplicates and just leave one and the file has multiple duplicates of different u: numbers and it's 14,000 lines long. can anyone tell me if I can use awk? sed? or sort for something like this to? removing lines that have a certain string in there that's a duplicate.
View 4 Replies
View Related
