General :: Grep - Manipulating Large Text File Full Of Records

Nov 26, 2010

I'm trying to manipulate a large text file full of records (metadata - one complete record per line). I need to delete every line on which certain words appear - there are five different words, all pretty simple all-caps strings with occasional whitespace. I tried using grep -v, which worked a treat, but only string-by-string. Ideally I'd like to run this as grep -v -f, where the file targeted by the -f contains the strings I need to match in order to delete the lines they're in.

i.e. grep -v -f filecontainingSTRINGS.txt targetfile.txt > outputfile.txt

When I try this, however, I don't get any matches - or more specifically, no changes are made in the output file. It works fine if there's only one string in filecontainingSTRINGS, but it doesn't work if there's more than one (I'm using newline as the delimiter). (Also my machine doesn't recognise /usr/xpg4/bin/grep - no idea what that's all about!)

View 5 Replies


ADVERTISEMENT

Software :: Manipulating Fixed-column Records In Bash?

Jul 20, 2011

I have a performance report that provides all the information I need to report the following: total transactions per day, average transactions per second, and peak transactions per second. It just doesn't provide any of it in a very accessible manner, so I want to parse on it and just capture the bits I care about. Ideally, I'd like the output to look something like this:

Code:
Date Total Avg Peak
07/11/11 12,328,033 24.05 64
07/12/11 9,328,429 21.98 56

The problem is the format of the input file, which is somewhat complicated. The report gives a summary of all transactions within any given second, and then totals at the end of each day, with page breaks in the middle, like so:

[Code]...

So first, the easy part that takes me to the daily summary, which gives me the date, the total transactions, and I can divide the total by 86400 to get the average per second, too. No problem. It's the last part that's got me stumped... the daily peak. I can't just do a while loop on the date, because it's missing from most of the records. And it also means I can't use positional parameters, because depending on the page break, the total will move between $2 or $3. And I need the date as a conditional to find the daily peak, because this output will have many days' worth of data.

Any ideas? Some kind of awk or sed command to insert the date wherever it's missing (I'm not particularly good at either utility)? Is there a method to parse these things based on column location that I'm not aware of?

View 12 Replies View Related

General :: Grep Multiple Lines From A Text File

Jun 17, 2009

I have a list of words that I want to grep in many files to see which ones have it and which ones dont. in the text file I have all the words listed line by line, ex: list.txt:

check
try this
word1
word2
open space
list ..

I want to grep each line one by one. like I want it to

grep "check" *.log
grep "try this" *.log
grep "word1" *.log .. etc how can I do this?

and maybe write the output to a file.

View 5 Replies View Related

General :: Convert A Large Number Of File Types From None Standard To Text?

Sep 28, 2009

I have on my windows machine several hundred files that are a format of .nc .ncs for a CNC machine. I need to convert them to txt which is something as easy as opening in notepad and then saving as .txt but there are so many that this kind of action would take way too long.

The reason I am writing the linuxquestions is because I would feel more comfortable in loading a live CD and using some sort of terminal command to do this than I would to download one of the many "freeware" type programs I have found for windows (even more so since I have had a root kit before and had to start all the way over to get rid of it).

I need to know:

1. Is this possible to do with the terminal without super advanced knowledge.

2. Can one please point me in the right direction; something to read or an example

View 2 Replies View Related

General :: Sed/awk/grep Search For Number String Of Variable Length In Text File?

Jan 19, 2010

I need to search a text file for a string of numbers which are different lengths, and always are between number=" and " like:

number="1234567890"
number="22390"

I need to grab those numbers and pipe each one to a line in a file. I've already tried something with awk and that didn't seem to work.

View 10 Replies View Related

Ubuntu :: Copying Large File Blocks Externally Causes Disk Full

Oct 15, 2010

This has happened several times now, with 9.10 and 10.04. I back up my photos periodically to external drives, using Nautilus. At the next attempted login Gnome won't start and sometimes gives power manager incorrect installation error.

First time this happened I was stumped and eventually did a clean install. Second time, I found advice elsewhere in this forum to solve this by emptying root trash, which did the trick. This time, however, root trash has nothing in it and 2 users trash were insignificant (I emptied them all anyway with rm -r). Tried looking for enormous directories but couldn't find a smoking gun. I would rather not end up doing another clean install - a painful and extreme solution. I'm continuing to look for solutions to the immediate problem, but my question really is, what causes this and how do I prevent it in the future? I've run Computer Janitor regularly and ran apt-get clean but no help. Should I do all my large scale copying from terminal? I'm not a total noob, but close.

View 9 Replies View Related

Programming :: Split / Break Up Large Text File

Feb 18, 2011

I have a large text file with three columns. I'm trying to write a PERL script that splits the file up based on the value of the 3rd column. So every time the third column reads 0, a new file is created and all the data up until the next 0 is found is written to that new file. This should happen over and over until the initial file has been entirely split up.

ex data:
000
2024
2243
2143
96962871
97972878
000
2034
3034
3352

So with the data above, the file would be split into two files:
data_1.txt would contain
000
2024
2243
2143
96962871
97972878

and data_2.txt would contain
230
2034
3034
3352

View 10 Replies View Related

Programming :: Get Data From Multi Lined Text File Using Awk / Sed Or Perl - Grep And Cut Not Upto Par?

Jul 1, 2010

I need a loop that pulls out the user name into a variable and then pulls out the LastUpdate field into another variable so I can then perform a comparison against the last update field. Requirements are AIX tools including AWK, SED and Perl I am writing a script to check AIX users password expiration dates and if they are within the alerting period (ie. 7 days etc) it will email the user. I will release the full script into the public domain once completed. The text file I want to parse is formatted like:

Code:

colettel:
password = XSON0m4SdIQDw
lastupdate = 1260829398
andrewwa:

[code].....

View 4 Replies View Related

Programming :: Storing Grep Output To Feed Awk To Retrieve Entire Records Matching Variable

Jul 28, 2010

I have two files :

FileA
prot1
prot5
prot9
prot15

[Code]....

What I need to do is to extract from fileB the fields containing only the strings in fileA.

I thought awk could do the job easily with :

Code:
awk 'BEGIN { RS = "###" } /'$variable'/' fileB > output

where variable would maybe be the output of grep from fileA. So can I store the output of grep in a variable to use it afterwards with awk ?

something like that:

Code:
result=`grep prot. fileA` ; awk 'BEGIN { RS = "###" } /'$result'/' fileB > output
but that doesn't work. I'm always getting the entire fileB.

The output of grep get stored in the variable, I verified that with echo. So there is something that I just don't get... It seems to me that the above line should work.

View 11 Replies View Related

Ubuntu :: Searching WITHIN Doc - Xls - Ppt Files - Grep Is Unable To Locate Text INSIDE File Types

Sep 1, 2011

I am on Ubuntu 11.04 and using Libre Office 3.3.2 to compose new documents and am saving them using .doc, .ppt and .xls files. (due to having to share them with others who are on Windows systems)

I have a lot of doc files and I need to search for text INSIDE these files. I am perplexed with the fact that no search tool is able to search for text INSIDE these file types. "cat" can display them of course, but grep is not able to locate text INSIDE these file types. I even tried to save a .doc file as an .odt file, but no luck. The Applications>Accessories>Search for Files does not search INSIDE doc, xls or ppt with the option "Contains the text".

View 6 Replies View Related

General :: Using Sed And Grep With A Text String?

Apr 28, 2011

Q: Is there any way to use grep and sed with a string variable rather than with a file?

The problem: Im running through a LARGE (about 10,000 lines) xhtml file and need to replace every instance of lines beginning <p>~

The following code works but takes a long time mainlly because an in/out operation needs to be carried out on each line. If I could read from a string rather than a file it would take a much shorter time!

Code:
#!/bin/bash
count=$((0))
>tpf
echo "waiting....."

[Code].....

View 4 Replies View Related

General :: Using Grep / Awk To Search For Colored Text

Sep 17, 2010

I have done a bunch of searches on this but the terms seem to get tangled in the more popular search of "colouring the output of grep / awk". I am trying to find a way to grep/awk through the output of a command to find text of a specific colour. The command's output has a range of colours signifying too many different things to specify using text, with colour being the only form of grouping.

View 5 Replies View Related

Ubuntu :: Enable Full Text Copy Of A Locked Pdf File?

Sep 17, 2010

I have an ebook that has copy protection on; only 25 pages can be copied every 14 days. This is very annoying, as I need to copy more pages for my study group.

how I can enable full text copy of a locked pdf file?

View 5 Replies View Related

General :: Searching Text Files For Large Numbers?

Dec 31, 2010

I am looking for a way to search for large numbers in text files and print the nearby lines.

For example if I had a text file like:

Event: 11
blah: 3
blah: 41 bleh: 19
Event: 2
blah: 31

[Code].....

View 1 Replies View Related

Programming :: Accessing File And Manipulating With With Folders?

Mar 12, 2010

I have a .csv file with a list of Drugs Name that i need to remove from the folder. The folder consist of files that stored as drugname.mol format.

What i need to do is to sieve out those mol files from the folder?

How do i

(1) access the drugs names from the .csv file line by line (variable x)

(2) how do i access the files in the folder one by one (variable y)

(3) how can i do a comparison whether $x.mol == $y

(4) and shift them to another folder if $x.mol == $y

View 11 Replies View Related

General :: Increase Text Size In 10.04 Due To Having Large Resolution/monitors?

Apr 3, 2010

I have 24" dual monitors with 1920x1080 resolution on both of them. Consequently the text appears so small. I use the following text-intensive applications frequently:

Web browser (Google Chrome)
IDE (Komodo)
Terminal (Gnome Terminal)
Email (Thunderbird)

I can configure text size on IDE, Terminal and Email. But for Chrome, it is not a good idea to set proportional font size because often one wants to see the entire (not just proportional fonts) site to be zoomed. So I am asking: Is it possible to increase DPI in Ubuntu (much like on Windows) so as to increase the text size across all apps? OR Is it possible to set permanent 'zoom' in Google Chrome, using a third-party extension maybe?

View 1 Replies View Related

General :: Cp Adds Exclamation Points When Copying Very Large Text Files?

Jul 13, 2009

For my research I have some very large files that are basically millions of lines of ten columns of numbers. These files can be up to 5 GB in size. Recently I noticed that when I made a copy of one of my files, some exclamation points appeared in it where there should not be any: in front of random numbers throughout the file. Making another copy of the file would result in exclamation points in front of different numbers in different parts of the file. Doing this many times has given me up to four exclamation points in different parts of the file. Sometimes the file copies just fine without producing any extraneous exclamation points.Additionally, I have occasionally seen a "^K" where there should be a newline (the data that should have been on the next line was instead on the previous line with a ^K in front of it) in copies that I have made of my files. I don't know if this is related or not.

View 7 Replies View Related

Programming :: Input File - Grep Within Grep

Nov 22, 2010

I need to kind of grep within grep. My input file would be something like:

[Code]....

and I need to find the first occurrence of hello before MY PATTERN (hello 9008 in this case), so the output should be:

[Code]....

View 4 Replies View Related

General :: View A Particular Ten Lines In A Large File Where Can't Open The File In Vi

May 12, 2010

I am using RHEL 5.I have a very large test file which cannot be opened in vi.The content of the file has some 8000 lines.I need to view ten lines between 5680 to 5690.How can i view these particular lines in a large file.what is command and option i need to use.

View 1 Replies View Related

General :: Search By Content On Ubuntu - Grep -rl "text Here" It Freezes?

Feb 19, 2011

I'd like to search the entire server by content. (text file) When I try grep -rl "text here", it freezes. How would you do it? And how long does it usually take?

View 4 Replies View Related

General :: Using Find Or Grep To Find A Group Of Text Strings?

Feb 28, 2011

I have used diff command in past.I faced a situation to which I did not had a cluehere are some text strings (which can be stored in a file)Quote:

CONFIG_XEN=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128

[code]...

View 3 Replies View Related

General :: Manipulating Fonts From The Command Line?

Jul 30, 2010

I would like to manipulate font size of a text file, preferably from command line, before printing it.. Any Linux tool you can suggest?

View 1 Replies View Related

General :: Use Grep, Cut, Or Awk To Get An IP From A File?

Feb 3, 2010

I have a file which contains a line like this:SERVER=10.205.110.19 How can I cut out the IP and assign it to a variable?

View 4 Replies View Related

General :: Grep The Contents Of A File With Another File?

Sep 25, 2010

someone once told me that use can pass a file to grep and use that to search the contents of another file. if that is the case I'm not entirely sure why the following isn't working for me.

Code:

[root@LCENT01:~]#grep -i id_rsa.pub .ssh/authorized_keys
[root@LCENT01:~]#cat id_rsa.pub >> .ssh/authorized_keys
[root@LCENT01:~]#grep -i id_rsa.pub .ssh/authorized_keys

View 3 Replies View Related

Debian Configuration :: Jessie LVM - Full Disk / Large Logs And GParted

Sep 23, 2015

So, my issues since upgrading to Jessie seem to compound. When I fix one issue, two more arise. Right now, I have a full system disk. How it got so full. So I started poking around. I ran

Code: Select all find / -type f -size +50M -exec ls -lh {} ; | awk '{ print $NF ": " $5 }'

Found a few files I could delete, and did, but I also found Code: Select all/var/log/syslog.1: 33G
/var/log/messages: 33G
/var/log/user.log: 33G

What I find strange is that they're all exactly 33G each. So that accounts for the missing 99GB I deleted them, however only recovered 27Gb. Whats weird is when I type df -h I get

Code: Select allFilesystem      Size  Used Avail Use% Mounted on
/dev/dm-0       106G   74G   27G  74% /
udev             10M     0   10M   0% /dev
tmpfs           3.2G  9.7M  3.2G   1% /run
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda1       228M   27M  189M  13% /boot
/dev/sdb1       1.9T   62G  1.8T   4% /media/ntfs
tmpfs           1.6G     0  1.6G   0% /run/user/0

What are the tmpfs's and how can I reclaim that space, and what is /dev/dm-0 and why is that taking up so much space?

I have 2 LVGs vgdisplay -v

Code: Select allroot@SETV-007-WOWZA:~# vgdisplay -v
    DEGRADED MODE. Incomplete RAID LVs will be processed.
    Finding all volume groups
    Finding volume group "WOWZASERVER"

[Code] ....

After deleting the log files, I was able to regain access to my GDM session. But I still cant find out what /dev/dm-0 is, and where all the 75 GB is being taken up.

I just noticed, however, even though I can access the drive A-OK via browser, terminal, and web services (Our wowza) when I enter gParted I get this error for sda, my primary OS drive!

Code: Select all  Libparted Bug Found!

Error informing the kernel about modifications to partition /dev/sda2 -- Invalid argument. This means Linux won't know about any changes you made to /dev/sda2 until you reboot -- so you shouldn't mount it or use it in any way before rebooting

Now that I'm in gParted I see 3 partitions: [URL] ....

It reports now, that I have used ALL of my disk space.

Post Log delete, and fresh reboot, this is what Code: Select alldf -h outputs

Code:
Select all Filesystem      Size  Used Avail Use% Mounted on
/dev/dm-0       106G  8.7G   92G   9% /
udev             10M     0   10M   0% /dev
tmpfs           3.2G  9.8M  3.2G   1% /run
tmpfs           7.9G   80K  7.9G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock

[Code] ....

What the heck is going on?

View 0 Replies View Related

General :: Grep Or Sed Or Awk For File Manipulation?

Dec 7, 2010

Let me *try* and explain what I'm trying to do, and keep in mind aside from a little command line stuff I'm a beginner to any of what I'm asking about.

I have a directory structure something like this:

Code:

/usr/local/chatlogs/
webserver1/
1.chatlog
2.chatlog

[Code]....

So that whatever was captured in the () in the first part of the statement would be used in the 1 in the back part of the statement for every n.chatlog that might be in any of the /webserver directories at that time.

View 2 Replies View Related

General :: Grep 2 Items In A File?

Dec 31, 2010

I have collection my routers configuration through rancid. Now i want to do the following,

Display all the files that contain the following words

C1841 AND HWIC-4ESW

I want to display all the files that contains both these words.

View 14 Replies View Related

General :: Replace Text In One File With Text From Another File Using Sed?

Apr 28, 2010

How could I replace text in one file with text from another file using sed?

The text in each of the files would be surrounded by a starting and ending delimiter or a starting tag, say like /* */ so to easily find them.

View 8 Replies View Related

General :: Gnome - Configure The Definition Of "full Screen" To Make Large VNC Windows Behave Nicely On A Dual Screen Setup?

Dec 15, 2010

In my corporate environment, I'm required to run a Windows machine that acquires a VNC session on a machine in the server farm. My windows machine is dual head with different resolution monitors ( 1600x1080 on left and 1920x1200 on right). If I create a VNC session that spans the monitors, then maximizing a window in the VNC session causes it to stretch across both my monitors.

Instead, I want a "maximize" event to behave like it does on my windows machine -- I only want to maximize to the display that the window is on.

How can I define what, what I'll call, "maximize regions"? Regions in the VNC graphical plane where when I click "maximize", the window only expands to the region it currently ( and mostly) resides in.

Can I do this in gnome, X, xrandr, or some other magical interface?

View 2 Replies View Related

General :: Read File And Pipe To Grep

May 26, 2011

I'm storing a list of strings in a file and would like to read the file and pipe each line returned to grep which in turn searches a directory for files containing the string.However this is not returning any output.

View 2 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved