General :: Filtering Out Duplicate Lines From A Find/grep Output

Mar 22, 2010

I have some big files of logs that contain errors printed by an app. They are most of the time relevant, however most of them are similar. So i figured i could check what happened between a time interval with a find.

Im using this one

Code:

And I get an output similar to this one.

Code:

Is there a way to condensate the output lines to get only one or two, indicating the start and last occurrence of a block? Or I need to create a program to do so?

Because right now I get thousands of similar lines, but when I'm scrolling through them i sometimes miss relevant information that i would've otherwise noted if it wasn't all that spammy.

View 10 Replies


ADVERTISEMENT

General :: Grep - Filtering Output By OR Condition

Aug 13, 2010

Is there some way to filter output of command by OR condition in Linux? There is filtering by AND condition with grep in way like:
ls -l | grep "^a" | grep "z$"

That says: list all files that beggins with "a" AND ends with "z" (so there is shorter way to write this: grep "^a.*z$", but it is not matter). Is there some way to perform test by OR condition? For example: files that starts exactly with "xen" OR files that ends exactly with ".rpm". But exactly, not something like:
grep "[xen]{0,3}.*[.rpm]{0,4}"

View 2 Replies View Related

General :: Filtering UDS From Output Of Netstat Without Grep?

Aug 24, 2010

how I cat to filter out information about Unix Domain Sockets from netstat output without grep? Is there some option for command (I not found it in man of netstat).

View 2 Replies View Related

General :: Using Sed To Remove Lines With Duplicate ID's?

May 8, 2010

I have a file that contains lines representing the nodes of a polyline but I only need the first point in each segment. With the following text:

0,"013A",0.57,260739.891,4379258.87
0,"013A",0.57,260737.674,4379258.94
0,"013A",0.57,260684.628,4379258.35
1,"013A",0.545,260769.915,4379257.84
1,"013A",0.545,260739.891,4379258.87

[Code]....

The problem with uniq is that the last two colums will differ. I don't care about the x/y for any points following the first one.

View 4 Replies View Related

General :: Delete Duplicate Lines Without Removing First Instance Of That Line?

Feb 7, 2011

I have a file "test.txt" with following data

#1
aaa
#2
bbb
#3
aaa
#4
ddd

I wanted it to be displayed as

#1
aaa
#2
bbb
#4
ddd

I used awk "'!x[$0]++' test.txt > file.new"
,but it deleted #1 also.I tried using uniq command but i didn't work.
Can anyone Please let me know is there any way to do this using shell script.

View 2 Replies View Related

General :: Remove Lines From A Syslog Text File That Have Duplicate Strings

Mar 17, 2011

Trying to remove lines from a syslog text file that have duplicate strings

Mar 10 06:51:11[http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360]

then a few lines down

Mar 10 06:52:03 [http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360

got the same thing in terms of a u: number but the issue is I need to remove duplicates and just leave one and the file has multiple duplicates of different u: numbers and it's 14,000 lines long. can anyone tell me if I can use awk? sed? or sort for something like this to? removing lines that have a certain string in there that's a duplicate.

View 4 Replies View Related

General :: Finding Surrounding Lines With Grep?

Aug 7, 2010

I know how to use grep to output a line that matches a string. But what if I also want to output one line above every line containing a matching string, how do I do that?

View 1 Replies View Related

General :: Grep - A-Z - Returns Unmatched Lines ?

Feb 21, 2011

Does anyone know why something like:

echo lowercase | grep [A-Z]

Would still return the lowercase line even tho there is no match?

Using --color will show there is no match but the line is still returned.

I am still new but from what i've learned this should not work like that. ubuntu 10.10

View 13 Replies View Related

General :: Grep And Store Two Lines In Other File

Feb 16, 2011

I have following contents

I want to grep "#2" and want the output as

How to using shell script?

View 6 Replies View Related

General :: Grep To Select Lines With M In Last Word

Mar 6, 2011

I have a large file in which each line has three or more blank-delimited words. I'd like to code a grep to keep only those lines which have the letter M in the last word. the M (if present) will be the first character in the last word.

View 14 Replies View Related

General :: Grep Varying No. Of Lines Between Two Patterns

Apr 11, 2011

I have a file that goes like this:

I need to grep the lines between pattern 1 and pattern 2 and not the lines following pattern 2. Cannot use grep -A(num), as there are varying number of lines following pattern 1. Also, used awk one-liners, but results are erroneous.

View 14 Replies View Related

General :: Use Grep To Match Multiple Lines?

Feb 3, 2011

I have done this before but i cant really recall now

I have a text file and i want to grep the word "interface" and 10 lines following it. I think there was some switch like grep -A or something ?

View 2 Replies View Related

General :: Grep Multiple Lines From A Text File

Jun 17, 2009

I have a list of words that I want to grep in many files to see which ones have it and which ones dont. in the text file I have all the words listed line by line, ex: list.txt:

check
try this
word1
word2
open space
list ..

I want to grep each line one by one. like I want it to

grep "check" *.log
grep "try this" *.log
grep "word1" *.log .. etc how can I do this?

and maybe write the output to a file.

View 5 Replies View Related

General :: Grep - Output Show Only Right Of?

May 17, 2010

I forgot a lot of my command line. I am doing cat file | grep "error" and i would like it to show everything to the right of G:/ including G:/ if possible. I figure its an awk command but i dont know what. I tried awk '{print $8+}' but + does not work like i hoped and guessed.

View 2 Replies View Related

General :: Grep IPs From Ifconfig Output?

Mar 6, 2011

Following is my ifconfig output code...

I want to do some thing grep that I see the IP corresponding to each LAN card?
Is that possible?

View 8 Replies View Related

General :: Find/grep/wc Command To Find Matching Files - Print Filename And Word Count?

Sep 11, 2009

I am trying to do a find/grep/wc command to find matching files, print the filename and then the word count of a specific pattern per file. Here is my best (non-working) attempt so far:

wc `find . ( -name "*.as" -o -name "*.mxml" ) -exec grep -H HeightResizableList {}` ;

View 10 Replies View Related

General :: Using Find Or Grep To Find A Group Of Text Strings?

Feb 28, 2011

I have used diff command in past.I faced a situation to which I did not had a cluehere are some text strings (which can be stored in a file)Quote:

CONFIG_XEN=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128

[code]...

View 3 Replies View Related

General :: Exec Command With Grep Output?

Feb 15, 2011

I have a requirement to find the files having its name as ack_reply. However, there are many other files in the same directory as these resides. Now I have to remove these files from the folder and retain others after 7 days. So I tried to write the below script with grep command.

find $directory -type f -mtime +7 | grep ack_reply

how can I pass this output to -exec command.

If I am not using grep command my script would be as

find $directory -type f -mtime +7 -exec remove.sh {}\;;

How can I use -exec with grep and find.

View 4 Replies View Related

General :: Grepping Grep Output Fails

Feb 5, 2010

I'm trying to grep the output of ngrep. Unfortunately when I add another grep to the pipeline, I get no output at all. It can be some other command too - cat / grep / tee - everything breaks the chain. Example:

[Code]....

If I use cat somefile instead of ngrep at the start, everything works as expected.

View 2 Replies View Related

General :: Filter Out Unique Results From Grep Output?

Feb 21, 2010

In linux, I can grep a string from a file using grep mySearchString myFile.txt.
How can I only get the result which are unique?

View 1 Replies View Related

General :: Can't Save Grep Output Into Variable / Solution For This?

Feb 3, 2010

I would like to grep two numbers out of a text file, and divide them.

Here is the script code...

It feels like grep saves a new line too? or what is happening? i simply can't divide them, as it handles the variables as they are empty (and prints the two numbers although they were not printed

View 6 Replies View Related

General :: Find Duplicate Files By Name ?

Jul 24, 2010

We have a huge amount of duplicate files in a folder and I would like some pointers on to writing a bash script to create a list of the duplicate files. I've seen examples that check for the md5 sum of files... but I dont need that, the file name is enough.

View 4 Replies View Related

General :: Remove X Lines From Output?

Oct 11, 2010

I would like to know how to remove X lines from output. i have a test file and i want the output without the first 2 lines

[root@node1 ~]# cat test
1
2
3

[code]....

View 5 Replies View Related

Ubuntu :: Remove Duplicate Lines From A Text File?

Dec 16, 2010

Contained within each of these 67 text files is about 1 million urls. Yes. I have 67 text files that contain 1 million lines of urls each. I am sure I am swimming in duplicates. I tried opening one text file and clicking sort ----->remove duplicates. Now Gedit is not responding my processor is maxed out to 100% and I think I am finally ready to delve into some command line code. Can anyone give me idiot proof instructions on how to sort the duplicates out of each one of these 67 text files? How about no duplicates across all 67?

View 7 Replies View Related

Programming :: Remove Duplicate Lines From Shell Script

Apr 8, 2010

I have a file with semi duplicate lines, like:
abc 12 32
agsi 82
sha 26
abc 1
iaij
agsi 3

Now I want to edit my file and make it:
abc 12 32
agsi 82
sha 26
iaij
i.e. remove second occurrence of line when 1st column is abc or agsi.

View 13 Replies View Related

General :: App To Find And Remove Duplicate Images ?

Dec 13, 2010

I am looking for a Linux app that can find and remove duplicate images (with different filenames if that's at all possible).

View 5 Replies View Related

Ubuntu :: Remove Duplicate Lines In Plain Text File?

Apr 14, 2010

i have a big file of random numbers i generated at some point in time, after working with it with different things(how fun that was)... i want to remove duplicate lines and i'm not sure i'm doing this right

heres the command

Code:
sort random.txt | uniq -u > rand-shorter.txt

the file is pretty big, everything on a new line. i found the command on a web site so i'm sure its correct(bit of a command line in linux newbie)

can anyone confirm if this will remove lines duplicate lines (keeping one copy) and dump what is left in a file named rand-shorter.txt?

EDIT: i think its actually working, just taking a reallllly long time (on an old pen 4 from 2000)

View 8 Replies View Related

General :: Find And Remove Duplicate Phrases In A Document ?

Mar 29, 2011

I would like to find a command which automatically finds and removes phrases which appear more than once in a text file. I still want to keep one of these phrases, but I only want to see one of them.

View 9 Replies View Related

General :: Find Duplicate Files Using Bash Script ?

Jun 8, 2010

What kind of method to find the duplicates files on linux,

1.how to find just using the file name, sometimes i figure out people often to copy their files to another directory and i want to find out if there any same file name in the linux box.

2. what about if i want to find the duplicate files based on contents of the file, example is in picture file if users store picture files from digital camera first they just save the file name in default but when they want to give that picture to others they will rename it, i've been used method md5 for this situation in python script but it takes long time

I'm asking this question just to know to use bash script a lot in work and i want to test out fdupes at home, is fdupes use similar md5 scan to find duplicate files?

View 4 Replies View Related

General :: BASH: Find Duplicate Files (MAC/LINUX Compatible)?

Mar 18, 2011

I am looking for a bash script which is compatible with Mac, to find duplicate files in a directory.

View 2 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved