Programming :: Bash Script To Find And Remove Similar Lines From Multiple Files?
Jun 5, 2009
I want to remove duplicate or multiple similar lines from multiple files. I.e. if I have four files file1.txt file2.txt file3.txt and file4.txt and would like to find and remove similar lines from all these files keeping only one line from these similar lines. I only that uniq can be used to remove similar lines from a sorted file.
View 9 Replies
ADVERTISEMENT
Jul 15, 2010
I'm trying to search through some pdf files and I'm doing so by converting them to text files using pdftotext which is fine but I'm trying to get the number of occurrences in a paragraph of different words and it's adding a new line character at what it thinks is the right hand margin. I'm trying to remove all these singe new line characters but keep the doubles and I can't seem to work it out. i.e.
This is some text that has been broken.
Another paragraph.
becomes
This is some text that has been broken.
Another paragraph
View 9 Replies
View Related
Mar 11, 2011
For example, I have a text file with data which lists numerical values from two separate individuals
Code:
Person A
100
200
300
400
500
600
700
800
900
1000
1100
1200
Person B
1200
1100
1000
900
800
700
600
500
400
300
200
100
How would I go about reading the values for each Person, then being able to perform mathematical equations for each Person (finding the sum for example)?
View 13 Replies
View Related
May 13, 2011
My server was hit with an injection script which has placed code across many of my clients files. I need a script that can remove a block of php code that spans multiple lines, multiple directories/files and is dynamic, meaning that part of the code changes. I think using find/sed is what I need but cannot seem to figure out how to get it to work.The following is the script that is being injected everywhere. The catch is that they have generated dynamic code at the start/end of the script. (I have commented the parts that are dynamically changing on EVERY instance).PLEASE NOTE: Directly following this script is the start of a valid php script that I do not want to delete.
<?php
//{{65281980 - DYNAMIC!!
GLOBAL $alreadyxxx;
[code]...
View 2 Replies
View Related
Aug 25, 2010
I've been trying to sort this out for several hours and I?m totally lost? I?ve been searching around, but haven?t found the solution to my problem. I have a directory with 100 files. I need to copy 10 lines of each files (let?s say from line 45 to 55) into one unique file. So I guess I could use sed ?w, but I didn?t manage to write the right script. I also tried using a loop to create 100 different files, each one with the 10 lines) to concatenate them later on. But I only got 1 file, not 100.
View 12 Replies
View Related
Nov 24, 2009
How do you remove parts of strings using python? Such as, if I have something like:
Code:
erme1 sdifskenklsd
erme2 sdfjksliel
[code]....
View 3 Replies
View Related
Jan 26, 2011
I have wrote a 1 line command that parses a file, locates the IP Address in the file and then trims the output the way I want it, and then sorts numerically and by uniqueness and then >> appends to output.txt
I can get all the IP's into 1 file "output.txt", but what I am really looking for is some type of way to create a text file, for each IP it finds labeled xxx.xxx.xxx.xxx.txt and also put that ip address into that file..
xxx.xxx.xxx.xxx = the ip address it finds
View 14 Replies
View Related
May 9, 2011
bash script to give sensible names to a large number of photos. I hope to be able to run a script with an argument which will become the filename followed by a number beginning at 1.
Code:
./file_rename.sh Summer2009_
Summer2009_0001
Summer2009_0002
Summer2009_0003
[Code]....
View 1 Replies
View Related
Nov 9, 2010
However, the ffmpeg command generates a temporary file blahblah.mpg.tmp of about 1GB per hour of transcoded video.My issue is that I can't seem to delete these files automatically from any bash script.Now from the command line, I can cd to the directory and just rm -f *.tmp and they get deleted. However, from my script, that same command doesn't remove those files. I thought maybe the file was in use, so I put a sleep command in for like an hour before the delete happens, but it still fails. I also put rm -f /mnt/mythtv/*.tmp in a root cronjob and it still doesn't delete the files.
If I just rm *.tmp I do get a prompt about "Are you sure you want to delete this write protected file?". But the -f switch seems to work fine as a normal user from the command line and just delete them.Does anyone have an idea how to troubleshoot this problem? The particular filesystem that the tmp files get generated on is on it's own xfs partition mounted as /mnt/mythtv.
View 8 Replies
View Related
Oct 3, 2009
i have lots of files with dots insde file name for example:
document.1.is.bigger.doc
resume.version12.doc
10.photo.www.cnn.com.jpg
i want to rename all files in that directory to be the same (with same extention) but convert dots to underscore "_"
how can i do that on bash script / php.
View 6 Replies
View Related
Mar 6, 2011
I have this cool bash script that I worked hard on. But it broke down when it can across files that had non-English characters. Another small problem was getting it to descend into a directory. If it renamed a directory it would not descend into that dir to rename the other files. I would have to run the script twice on the same directory.
Here is the script:
Code:
find -type d -o -regextype egrep -iregex '(.*.ogg|.*.mp3|.*.wav)' | while read s
do
rename -v 'y/A-Z/a-z/' "$s"
done
find -type d -o -regextype egrep -iregex '(.*.ogg|.*.mp3|.*.wav)' | while read n
do
rename -v 's/ /_/g' "$n"
done
A French name like this:
Code:
Chateau De Sable (imagine accents above the letter a)
became this:
Code:
ch303242tea_de_sable
This is not what I wanted.
Why would the script not descend into a directory after it was renamed?
View 7 Replies
View Related
Oct 19, 2010
I want to delete all files within a specific folder without actually deleting the folder, what is a good bash command for this?. I found this one but encountered some errors even though I am executing it within the specific folder:
useratdebian:/home/user/folder# find . -type f -exec rm -rf {} ;
[1] 5052
useratdebian:/home/user/folder# find: missing argument to `-exec'
[1]+ Exit 1 find . -type f -exec rm -rf
The command as it appears is:
find . -type f -exec rm -rf {} ;
how to delete only the files contained within the folder called "folder" for example?
View 4 Replies
View Related
Mar 11, 2011
I am trying to find a nightly backup if it was successfully copied over, rename it and curl, but it's always passing the check even if the file is older than specified. From the command line it does as it should. Example is here;
Code:
find /backup -type f -mmin +4440 -exec echo "found" {} ;
- nothing returned (good). Then I change the time
[code].....
View 4 Replies
View Related
May 12, 2010
I have a txt file with couple of comment lines:
Number of title = !num!
#line1
#line2
#line3
I wrote a script with "sed" to replace !num! in this file, which is very straightforward. However, based on the !num!, I want to remove the number of "#" based on the !num! value. Is there an easy way to do that with "sed"; otherwise, i will have to write a script to loop through the file.
View 8 Replies
View Related
Aug 20, 2011
I often use the rpl command to make changes to multiple html files at once. For example:
rpl -R '<br />' '<br /><br />' mydirectory However, I haven't been able to figure out how to change multiple lines. For example, let's say I want to change all occurrences of :
<br />
<br />
to:
<br />
I've tried
rpl -R '<br />
<br />' '<br />' mydirectory
but that didn't work. how to do this with rpl or some other way?
View 9 Replies
View Related
Feb 3, 2011
i want to find all files with .h or .c extension and print them on the screen. How can i do it with bash script programming?
View 5 Replies
View Related
Apr 30, 2010
I have this script:
Code:
a=awk'{print $8}' f1.txt
How can I make it store lines in $a ? In the way above it creates one big line.
View 2 Replies
View Related
Feb 25, 2010
I'm working on a bash script that will go through a directory, find the sub-directories that have been created since the last time the script ran, count the results, and output that integer (will most likely be '1' or less per each instance run) to a file. Give the circumstances, my previous (and very limited) experience with bash is not sufficient for me to pull this off. since it probably has bearing, is that my mail server stores files that it flags as viruses in a folder. It creates a sub-directory for each virus that it quarantines .I want to count those subdirectories and graph them with MRTG. Hence the script. I'm going to post what I've got so far and the purpose of it, because I'm told I have a very odd and efficient way of doing scripting.
[Code]...
But then it dawned on me that it wouldn't work because I would have to not count the directories that have already been counted and count the ones that have not been counted. Given that the purpose of this is to generate a graph about every 5 minutes, using find won't work because, to my knowledge, that will only find things based on whole day values, I need it almost down to the minute.
View 1 Replies
View Related
Dec 20, 2010
I would like to compare them, in terms of amplitude, length, tempo, ... Which program could basically analyse them giving an sufficient output data
View 2 Replies
View Related
Jan 23, 2010
I want to be able to remove the first character of a line when I highlight multiple lines in gedit. Example:
%Example is
%Commented Code
%Uncomment using this shortcut
I would then highlight/select these lines, and remove the first character to make it look like this:
Example is
Commented Code
Uncomment using this shortcut
I'm pretty sure there is an actual shortcut for this. If there is another text editor on Linux that it would work in, it would be nice to know how to do it in that editor as well.
View 2 Replies
View Related
Mar 22, 2010
i have a lot off windows - thumbs.db files om my Debian Apache server.How to remove all (without define all directory's)Is there any way with RM?- or maybe some other smart solution to do it?
View 8 Replies
View Related
Mar 13, 2011
I'm trying to write a bash script to find all lines containing two different strings in many files. I don't have access to egrep so I want to use sed for this purpose.
The files will look like this:
FileX
------
Info:18
Data:76
Contact:me@home.com
Start:1500
I want to generate a new file from these files with only the rows containing Data and Start. Something like this:
for y in `ls /file*.db`;
do sed '/Data|Start/p' $y > newfile
done
View 7 Replies
View Related
Sep 12, 2010
I currently have 3 files with floating point data that I wish to have in a single file with the format:
Code:
F1 F1 F3 Output
a1 b1 c1 a1 b1 c1
a2 b2 c2 a2 b2 c2
a3 b3 c3 a3 b3 c3
a4 b4 c4 a4 b4 c4
View 3 Replies
View Related
Mar 8, 2011
I have a set of files containing data that I need to re-arrange into one single property list.
The files that I have look like this:
Code:
# cat uk
<string>10</string>
<string>11</string>
...
<string>29</string>
[Code]....
So the lines in the files match up but I haven't found a way of reading several files line by line.
View 3 Replies
View Related
Nov 3, 2010
How can I remove all lines which contain A,,,,,, I tried the following sed statements but no luck.
Code:
sed "/A,,,,,,/d file"
sed "/A,,,,,,/d file"
View 6 Replies
View Related
Mar 30, 2010
I have a file witch I need to list 10 line by 10 lines with something like press enter to go on in between.
Well, the problem is that i have absolutely no idea on how to implement this.
Can you give me some points?
View 3 Replies
View Related
Mar 8, 2009
I need to replace part of a line in a file with multiple lines, however need to separate each line with a newline is this possible?
Example, need to remove line 2
And insert line similar to below:
If the multiple lines were part of a text file, would inserting a text file automatically include newline characters?
View 1 Replies
View Related
Jan 28, 2010
i have a sorted file with many repeated lines like this:
1ujw
1ujw
1ul1
1us7
[code].....
i need to calculated haw many times each line is repeated, for ex
1ujw 2
1ul1 1
1us7 5
etc.
View 5 Replies
View Related
Mar 26, 2011
Using awk I pull the first field of a random line from my datafile.myvar1=`awk -F" " 'NR=='$randline' {printf "%s", $1}' myfileThis works fine. The problem is there will be empty lines at the end of the file. Rather than using awkto filter out blank lines I would like to figure this out first.So I test $myvar1 for a blank string after setting $randline to one that I know is blank:test -z "$myvar1" && echo "true" || echo "false"But, this returns "false"? So the string is not zero length. Why? It's a tab-separated file. Is awk storing the tab with the $1 field or something.This is where I get headache. I try to echo my variable to see what it looks like.
echo "$myvar1"
outputs: nothing
echo "My variable is [$myvar1]"
outputs: [y variable is [
Why is the closing bracket at the beginning? What character could be stored in $myvar1 that would do such a thing and how did it get there?
View 7 Replies
View Related
Jun 9, 2009
I would like to know how I can get the ouput from the following dmidecode command in example 1 to look like example 2 without having to grep -v all the unwanted lines.Is there a way in awk or sed?Example 1
Code:
Processor Information
Socket Designation: Socket 1 CPU 1
[code]...
View 7 Replies
View Related