General :: Filtering Out Duplicate Lines From A Find/grep Output
Mar 22, 2010
I have some big files of logs that contain errors printed by an app. They are most of the time relevant, however most of them are similar. So i figured i could check what happened between a time interval with a find.
Im using this one
Code:
And I get an output similar to this one.
Code:
Is there a way to condensate the output lines to get only one or two, indicating the start and last occurrence of a block? Or I need to create a program to do so?
Because right now I get thousands of similar lines, but when I'm scrolling through them i sometimes miss relevant information that i would've otherwise noted if it wasn't all that spammy.
View 10 Replies
ADVERTISEMENT
Aug 13, 2010
Is there some way to filter output of command by OR condition in Linux? There is filtering by AND condition with grep in way like:
ls -l | grep "^a" | grep "z$"
That says: list all files that beggins with "a" AND ends with "z" (so there is shorter way to write this: grep "^a.*z$", but it is not matter). Is there some way to perform test by OR condition? For example: files that starts exactly with "xen" OR files that ends exactly with ".rpm". But exactly, not something like:
grep "[xen]{0,3}.*[.rpm]{0,4}"
View 2 Replies
View Related
Aug 24, 2010
how I cat to filter out information about Unix Domain Sockets from netstat output without grep? Is there some option for command (I not found it in man of netstat).
View 2 Replies
View Related
May 8, 2010
I have a file that contains lines representing the nodes of a polyline but I only need the first point in each segment. With the following text:
0,"013A",0.57,260739.891,4379258.87
0,"013A",0.57,260737.674,4379258.94
0,"013A",0.57,260684.628,4379258.35
1,"013A",0.545,260769.915,4379257.84
1,"013A",0.545,260739.891,4379258.87
[Code]....
The problem with uniq is that the last two colums will differ. I don't care about the x/y for any points following the first one.
View 4 Replies
View Related
Feb 7, 2011
I have a file "test.txt" with following data
#1
aaa
#2
bbb
#3
aaa
#4
ddd
I wanted it to be displayed as
#1
aaa
#2
bbb
#4
ddd
I used awk "'!x[$0]++' test.txt > file.new"
,but it deleted #1 also.I tried using uniq command but i didn't work.
Can anyone Please let me know is there any way to do this using shell script.
View 2 Replies
View Related
Mar 17, 2011
Trying to remove lines from a syslog text file that have duplicate strings
Mar 10 06:51:11[http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360]
then a few lines down
Mar 10 06:52:03 [http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360
got the same thing in terms of a u: number but the issue is I need to remove duplicates and just leave one and the file has multiple duplicates of different u: numbers and it's 14,000 lines long. can anyone tell me if I can use awk? sed? or sort for something like this to? removing lines that have a certain string in there that's a duplicate.
View 4 Replies
View Related
Aug 7, 2010
I know how to use grep to output a line that matches a string. But what if I also want to output one line above every line containing a matching string, how do I do that?
View 1 Replies
View Related
Feb 21, 2011
Does anyone know why something like:
echo lowercase | grep [A-Z]
Would still return the lowercase line even tho there is no match?
Using --color will show there is no match but the line is still returned.
I am still new but from what i've learned this should not work like that. ubuntu 10.10
View 13 Replies
View Related
Feb 16, 2011
I have following contents
I want to grep "#2" and want the output as
How to using shell script?
View 6 Replies
View Related
Mar 6, 2011
I have a large file in which each line has three or more blank-delimited words. I'd like to code a grep to keep only those lines which have the letter M in the last word. the M (if present) will be the first character in the last word.
View 14 Replies
View Related
Apr 11, 2011
I have a file that goes like this:
I need to grep the lines between pattern 1 and pattern 2 and not the lines following pattern 2. Cannot use grep -A(num), as there are varying number of lines following pattern 1. Also, used awk one-liners, but results are erroneous.
View 14 Replies
View Related
Feb 3, 2011
I have done this before but i cant really recall now
I have a text file and i want to grep the word "interface" and 10 lines following it. I think there was some switch like grep -A or something ?
View 2 Replies
View Related
Jun 17, 2009
I have a list of words that I want to grep in many files to see which ones have it and which ones dont. in the text file I have all the words listed line by line, ex: list.txt:
check
try this
word1
word2
open space
list ..
I want to grep each line one by one. like I want it to
grep "check" *.log
grep "try this" *.log
grep "word1" *.log .. etc how can I do this?
and maybe write the output to a file.
View 5 Replies
View Related
May 17, 2010
I forgot a lot of my command line. I am doing cat file | grep "error" and i would like it to show everything to the right of G:/ including G:/ if possible. I figure its an awk command but i dont know what. I tried awk '{print $8+}' but + does not work like i hoped and guessed.
View 2 Replies
View Related
Mar 6, 2011
Following is my ifconfig output code...
I want to do some thing grep that I see the IP corresponding to each LAN card?
Is that possible?
View 8 Replies
View Related
Sep 11, 2009
I am trying to do a find/grep/wc command to find matching files, print the filename and then the word count of a specific pattern per file. Here is my best (non-working) attempt so far:
wc `find . ( -name "*.as" -o -name "*.mxml" ) -exec grep -H HeightResizableList {}` ;
View 10 Replies
View Related
Feb 28, 2011
I have used diff command in past.I faced a situation to which I did not had a cluehere are some text strings (which can be stored in a file)Quote:
CONFIG_XEN=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128
[code]...
View 3 Replies
View Related
Feb 15, 2011
I have a requirement to find the files having its name as ack_reply. However, there are many other files in the same directory as these resides. Now I have to remove these files from the folder and retain others after 7 days. So I tried to write the below script with grep command.
find $directory -type f -mtime +7 | grep ack_reply
how can I pass this output to -exec command.
If I am not using grep command my script would be as
find $directory -type f -mtime +7 -exec remove.sh {}\;;
How can I use -exec with grep and find.
View 4 Replies
View Related
Feb 5, 2010
I'm trying to grep the output of ngrep. Unfortunately when I add another grep to the pipeline, I get no output at all. It can be some other command too - cat / grep / tee - everything breaks the chain. Example:
[Code]....
If I use cat somefile instead of ngrep at the start, everything works as expected.
View 2 Replies
View Related
Feb 21, 2010
In linux, I can grep a string from a file using grep mySearchString myFile.txt.
How can I only get the result which are unique?
View 1 Replies
View Related
Feb 3, 2010
I would like to grep two numbers out of a text file, and divide them.
Here is the script code...
It feels like grep saves a new line too? or what is happening? i simply can't divide them, as it handles the variables as they are empty (and prints the two numbers although they were not printed
View 6 Replies
View Related
Jul 24, 2010
We have a huge amount of duplicate files in a folder and I would like some pointers on to writing a bash script to create a list of the duplicate files. I've seen examples that check for the md5 sum of files... but I dont need that, the file name is enough.
View 4 Replies
View Related
Oct 11, 2010
I would like to know how to remove X lines from output. i have a test file and i want the output without the first 2 lines
[root@node1 ~]# cat test
1
2
3
[code]....
View 5 Replies
View Related
Dec 16, 2010
Contained within each of these 67 text files is about 1 million urls. Yes. I have 67 text files that contain 1 million lines of urls each. I am sure I am swimming in duplicates. I tried opening one text file and clicking sort ----->remove duplicates. Now Gedit is not responding my processor is maxed out to 100% and I think I am finally ready to delve into some command line code. Can anyone give me idiot proof instructions on how to sort the duplicates out of each one of these 67 text files? How about no duplicates across all 67?
View 7 Replies
View Related
Apr 8, 2010
I have a file with semi duplicate lines, like:
abc 12 32
agsi 82
sha 26
abc 1
iaij
agsi 3
Now I want to edit my file and make it:
abc 12 32
agsi 82
sha 26
iaij
i.e. remove second occurrence of line when 1st column is abc or agsi.
View 13 Replies
View Related
Dec 13, 2010
I am looking for a Linux app that can find and remove duplicate images (with different filenames if that's at all possible).
View 5 Replies
View Related
Apr 14, 2010
i have a big file of random numbers i generated at some point in time, after working with it with different things(how fun that was)... i want to remove duplicate lines and i'm not sure i'm doing this right
heres the command
Code:
sort random.txt | uniq -u > rand-shorter.txt
the file is pretty big, everything on a new line. i found the command on a web site so i'm sure its correct(bit of a command line in linux newbie)
can anyone confirm if this will remove lines duplicate lines (keeping one copy) and dump what is left in a file named rand-shorter.txt?
EDIT: i think its actually working, just taking a reallllly long time (on an old pen 4 from 2000)
View 8 Replies
View Related
Mar 29, 2011
I would like to find a command which automatically finds and removes phrases which appear more than once in a text file. I still want to keep one of these phrases, but I only want to see one of them.
View 9 Replies
View Related
Jun 8, 2010
What kind of method to find the duplicates files on linux,
1.how to find just using the file name, sometimes i figure out people often to copy their files to another directory and i want to find out if there any same file name in the linux box.
2. what about if i want to find the duplicate files based on contents of the file, example is in picture file if users store picture files from digital camera first they just save the file name in default but when they want to give that picture to others they will rename it, i've been used method md5 for this situation in python script but it takes long time
I'm asking this question just to know to use bash script a lot in work and i want to test out fdupes at home, is fdupes use similar md5 scan to find duplicate files?
View 4 Replies
View Related
Mar 18, 2011
I am looking for a bash script which is compatible with Mac, to find duplicate files in a directory.
View 2 Replies
View Related
