Programming :: Remove Many Lines Based On Content Of Other File

Jun 21, 2011

I a csv-file (A.csv) with a total of 4.600.000 lines. Thats to many and only a few is necessary. I have a txt-file with 150 lines (X.txt) (all lines is dataset from a mainframe and looks like abc.def.123.456. How do I remove lines from A.csv where none of the dataset from x.txt is present?

View 13 Replies


ADVERTISEMENT

Programming :: Remove Lines In A Text File Based On Another Text File?

Jan 28, 2009

I have a text file called file1.txt containing many lines eg.

line1
line2
line3
line4
line5
line6

Then i have another text file called file2.txt contains

3
5
6

Is there a command to remove the lines in file1.txt based on the keywords in file2.txt? note: It should remove line3,line5,line6 based on 3,5,6

View 10 Replies View Related

Programming :: Remove Specific Lines From File

Jan 21, 2011

I'm trying to come up with ideas for a simple way to strip a specific "entry" from a text file.I know tools like sed and perl can remove specific lines from a file but I haven't been able to come up with an elegant way to do my group of lines.In my file, the first "Location" line and the "SVNPath" line should be unique every time... but are they enough to strip out the whole set of the group plus the trailing one line of white space separating each group? Add to this, my file will grow as new entries are added (always appended to the end) but new entries will have the same formatting.

View 9 Replies View Related

Programming :: Remove Lines And Parts Of Lines From Python Strings?

Nov 24, 2009

How do you remove parts of strings using python? Such as, if I have something like:

Code:
erme1 sdifskenklsd
erme2 sdfjksliel

[code]....

View 3 Replies View Related

Programming :: Script To Remove Lines In A File With More Than "x" Instances Of Any Character ?

Oct 4, 2010

I'm looking for a script (bash, python, perl etc) or even a one liner (sed, awk etc) that can take a set of files and remove any line that has more than "x" instances of any character (case sensitive). I have been doing a lot of searching and can only come up with examples of how to remove blank lines, lines that start with a certain character or lines that contain a certain string. This will be used on a system running a Kubuntu derivative.

As a very poor and basic example, I would like to take files that contain lines like:

Code:

And end up with the files only containing the lines:

Code:

If I tell the script that 2 is the maximun number of times any character can appear in any line.

I know this must be possible, but for the life of me I cannot find even an example that will lead me in the right direction or better yet a piece of code I can use.

View 15 Replies View Related

Ubuntu :: [Scripting] Remove Content Of One File From Another?

Jun 22, 2011

I've been puzzling over this for a while and have not been able to reach a solution, so turning to your good selves for advice! I currently have two files, let's say they look like this:*File A*

Code:
item1
item3

[code]....

View 6 Replies View Related

Programming :: Remove All Lines Containing 'A,,,,,,?

Nov 3, 2010

How can I remove all lines which contain A,,,,,, I tried the following sed statements but no luck.

Code:

sed "/A,,,,,,/d file"
sed "/A,,,,,,/d file"

View 6 Replies View Related

Programming :: Remove All Lines Containing Extended Characters

Nov 7, 2010

I am using 'sed -e /foo/d' to match lines which I want to delete from a file. I discovered I have some lines which contain random (extended?) characters like 'ủ' which I would also like to delete. The lines in the file should only contain alpha numeric characters.

View 8 Replies View Related

Programming :: Remove Extra New Lines (BASH)

Jul 15, 2010

I'm trying to search through some pdf files and I'm doing so by converting them to text files using pdftotext which is fine but I'm trying to get the number of occurrences in a paragraph of different words and it's adding a new line character at what it thinks is the right hand margin. I'm trying to remove all these singe new line characters but keep the doubles and I can't seem to work it out. i.e.

This is some text that has been broken.
Another paragraph.
becomes
This is some text that has been broken.
Another paragraph

View 9 Replies View Related

Programming :: Remove Duplicate Lines From Shell Script

Apr 8, 2010

I have a file with semi duplicate lines, like:
abc 12 32
agsi 82
sha 26
abc 1
iaij
agsi 3

Now I want to edit my file and make it:
abc 12 32
agsi 82
sha 26
iaij
i.e. remove second occurrence of line when 1st column is abc or agsi.

View 13 Replies View Related

General :: Remove All Lines In A File Containing Sameword?

Aug 6, 2010

When i want to remove particular lines containing a specific word in from entire document at a time,i am using the following command.

awk '$columnno !~/specificword/' inputfile > outputfile

But here, coulmn no is my problem, because iam having this in different columns. So i need a solution for it.

How to write such removal command without mentioning column no. , ie irrespective of column no, it has to remove all lines having that specific word.

View 10 Replies View Related

Ubuntu :: Python : Remove Some Lines From A Text File?

Sep 6, 2010

I am creating my own address book Python program and I want to create a nction that removes some specified entries. The code looks like this now.

Code:
def remove():
delentry= raw_input('Enter the entry name to delete: ')

[code]...

View 1 Replies View Related

Ubuntu :: Remove Duplicate Lines From A Text File?

Dec 16, 2010

Contained within each of these 67 text files is about 1 million urls. Yes. I have 67 text files that contain 1 million lines of urls each. I am sure I am swimming in duplicates. I tried opening one text file and clicking sort ----->remove duplicates. Now Gedit is not responding my processor is maxed out to 100% and I think I am finally ready to delve into some command line code. Can anyone give me idiot proof instructions on how to sort the duplicates out of each one of these 67 text files? How about no duplicates across all 67?

View 7 Replies View Related

General :: Commands To Remove Particular Few Lines In History File?

Jun 16, 2010

Is there any commands or scripts to remove only selected line in the history file.

View 1 Replies View Related

General :: Remove Lone Lines From A Text File?

Jul 6, 2011

anyone has ideas how to remove lone lines from a text file?

If I have a file that is like this:
-----------------------------------
line 1

[code]...

View 14 Replies View Related

Programming :: String Editing: Remove Lines Consisting Of A Single Character?

Jul 16, 2010

I have a large text file that's formatted sort of like this:

Code:
foo bar
blah

[code]...

View 2 Replies View Related

Programming :: Sed - Save Output To File With Filename From Content Of Another File?

Feb 28, 2011

My employer issues pdf files with everyones work schedules. I copy the content and save it as plain text in a file called unformatted (hope to be able to automate this step someday). Im working on a SED script that reduces unformatted to only display what I want to see and saves the result in a file Ive named formatted. After that I have to manually copy formatted and save it with that days date as a filename e.g. 2011-02-25 or whatever day is scheduled in the pdf, for use on a mobile device (Nokia N900). I noticed that the date occurs on certain lines in the file so I added a line like:

sed -n 's/^Date: (201[1-9])/([0-1][0-9])/([0-3][0-9]).*/1-2-3/p' < unformatted >theDate
That creates a file theDate with the date in it that I wish to use as the filename for this particular instance. So I would like to skip the file formatted all together and have the sed- script write to a new file using the content of the Date as a filename, but how do I make that happen? And of course it would be more elegant if I could skip the intermediate theDate file as well.

View 4 Replies View Related

Ubuntu :: Remove Duplicate Lines In Plain Text File?

Apr 14, 2010

i have a big file of random numbers i generated at some point in time, after working with it with different things(how fun that was)... i want to remove duplicate lines and i'm not sure i'm doing this right

heres the command

Code:
sort random.txt | uniq -u > rand-shorter.txt

the file is pretty big, everything on a new line. i found the command on a web site so i'm sure its correct(bit of a command line in linux newbie)

can anyone confirm if this will remove lines duplicate lines (keeping one copy) and dump what is left in a file named rand-shorter.txt?

EDIT: i think its actually working, just taking a reallllly long time (on an old pen 4 from 2000)

View 8 Replies View Related

Programming :: Bash Script To Find And Remove Similar Lines From Multiple Files?

Jun 5, 2009

I want to remove duplicate or multiple similar lines from multiple files. I.e. if I have four files file1.txt file2.txt file3.txt and file4.txt and would like to find and remove similar lines from all these files keeping only one line from these similar lines. I only that uniq can be used to remove similar lines from a sorted file.

View 9 Replies View Related

Programming :: Copy One File Content To Multiple File

May 28, 2010

I wanted to copy one file to multiple new files. I have an idea to write a script and do the operation. But here i m looking for any particular command to do this operation.

View 1 Replies View Related

General :: Remove Lines From A Syslog Text File That Have Duplicate Strings

Mar 17, 2011

Trying to remove lines from a syslog text file that have duplicate strings

Mar 10 06:51:11[http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360]

then a few lines down

Mar 10 06:52:03 [http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360

got the same thing in terms of a u: number but the issue is I need to remove duplicates and just leave one and the file has multiple duplicates of different u: numbers and it's 14,000 lines long. can anyone tell me if I can use awk? sed? or sort for something like this to? removing lines that have a certain string in there that's a duplicate.

View 4 Replies View Related

Programming :: C - Can't Print In Screen The Content Of File

Mar 20, 2010

why this code can't print in the while loop the content of the file ?

Code:

#include <stdio.h>
int main()
{
FILE *fp;

[code]...

View 7 Replies View Related

Programming :: Display Content Of Txt File In Xbmc?

Apr 25, 2011

go about developing this add-on. i am testing this on my xbmc-live set-up; i am fairly affluent in bash/ c but unfortunately i dont have experience with python.i trimmed the data using this bash 1-liner so the output looks like:

Code:
xbmc@XBMCLive:~$ wget -q --output-document=- www.google.com/movies | sed -n 3p | tr '<' "
" | egrep "(title_bar|id=link_1_theater|class=info|mid=|Rated|class=times|fandango)" | grep

[code]...

View 2 Replies View Related

Programming :: Sed Delete Lines From File One If Regexp Are Listed In File Two?

Sep 17, 2009

I am trying to delete lines of a file if they contain text that is present on another file. For example

> cat one.txt:

a
b
c
d

[code]....

I get the following output:

> ./test.sh one.txt two.txt
a
b
d
e

[code]....

View 6 Replies View Related

Programming :: Iterate The Content Of A File In The Shell Script?

May 29, 2010

i need to check group of URLs and there https requests from browser. Recently i got some command line web browsers to know the HTTPS status of the URL like curl, wget etc... Now all of i need to do is write a shell script. I will put all my URLs in a text file and my shell script should read each URL one by one and log the status along with the corresponding URL.

Sample output:

./myscript.sh url.txt
1. site - 200 OK

View 6 Replies View Related

Programming :: Insert Each Line Of One File To The Another File After Certain Lines?

Jul 2, 2010

i have two files with thousands of line, I am trying to combine these two files but i want to insert each line of one file to the another file after certain lines. I am using awk with the following command but it does not work.cat file1 | awk ' { print $0; if (NR%3004==0) {print "file2"}}' > outputfile

View 14 Replies View Related

Programming :: How To Write Using Fwrite Without Affecting Existing Content Of File?

Oct 30, 2010

How can I write to a file multiple times using fwrite without affecting the previous writes?The method shown below accepts a file name, buffer and offset. The method opens the file in reading/writing mode and writes the content of the buffer at offset.

View 1 Replies View Related

Programming :: Lilypond Parser For Python - Typeset The Content Of File

Nov 12, 2009

I'm woring on a personal research project and would like to know if there are lilypond parsers for python available or I'll have to create my own. Just in case you are wondering: I don't need to typeset the content of the lilypond file, just understand what's written in the file (what notes, what duration, when in time to play each one, etc). [url]

View 1 Replies View Related

Programming :: Splitting A File Based On The Values Read From An Input File?

Jul 27, 2010

I am splitting a file based on the values read from an input file. The below one is the script.

1)How do I add the header which is present in the original file to the new split files created?(For eg. pharmacyf conatins header as table column names. The new files created (ODS.POS.$pharmacyid.$tablename.$CURRENT_DATE.dat) are without the header).

2) Also the script is creating 0 byte files for the pharmacyids which are not available in the intial file? Can this be avoided?

for pharmacyf in *
do
tablename=`echo $pharmacyf |cut -f4 -d'.' `
while read pharmacyid
do
grep -w $pharmacyid $pharmacyf >> $OUT/ODS.POS.$pharmacyid.$tablename.$CURRENT_DATE.dat
done< inputfile
done

View 2 Replies View Related

Programming :: How To Add Characters And Lines To File

Jan 29, 2011

For example, I have a file called "file" like this one:
type=strongsubj len=1 word=absolve pos=verb stemmed=y priorpolarity=positive
type=strongsubj len=1 word=unique pos=adj stemmed=n priorpolarity=neutral
type=strongsubj len=1 word=absolutely pos=adj stemmed=n priorpolarity=neutral
type=weaksubj len=1 word=taking pos=verb stemmed=y priorpolarity=positive
type=weaksubj len=1 word=friend pos=noun stemmed=n priorpolarity=positive
type=weaksubj len=1 word=usually pos=adverb stemmed=n priorpolarity=positive
type=strongsubj len=1 word=purecolor pos=anypos stemmed=n priorpolarity=negative
type=strongsubj len=1 word=accusingly pos=anypos stemmed=n priorpolarity=negative

I want to add the plural for the noun, for example if find this line:
type=weaksubj len=1 word=friend pos=noun stemmed=n priorpolarity=positive
will add one more line :
type=weaksubj len=1 word=friends pos=noun stemmed=n priorpolarity=positive
where we add "s" for the word friend
I did try to do like that:
<code>
cat file | while read LINE ; do

set -- ${line}
if [[ "${4#pos1=}" == "noun" ]];then
#I tried this line but it doesn't work properly:
v3==$(echo $line |sed 's/$3/$s') #I want to find the third word "word=friend" in that line and add "s" after that word
# I don't know what command to add this new line "$v3" to the file ???
done
</code>

View 12 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved