General :: Delete Rows Based On Values In A Column Using Sed?
Jan 18, 2011
I have a lot of files with thousands of rows and usually two columns of data. Column 1 is a coordinate, column 2 is a score. I want to delete rows that have a value in column 2 over, in this case, 50.
For example, here's a clip from a file:
2999898 39
2999899 40.9
2999900 42.7
2999901 44.7
2999902 46.7
2999903 48.9
2999904 51.1
2999905 53.3
2999906 55.7
I want the output to be this:
2999904 51.1
2999905 53.3
2999906 55.7
I know how to use sed for very basic things like deleting rows or ranges of rows but I can't figure out this ostensibly easy command.
View 6 Replies
ADVERTISEMENT
Apr 7, 2011
I need to transpose a file with over a 1000 rows of 5 columns of numbers into a file with a single column of numbers. The numbers are separated by a single space and range from one digit to 5 digits each. I tried using awk, but can only get it to grab one column of numbers.
Input:
1 2 3 4 50
600 7 8 9000 10
11 12000 13 14 15
Desired output:
1
2
3
4
[code]....
Tried using: awk '{split($0,a,""); print $NF}' <filename> and got:
50
10
15
It only grabbed the last number in each row.
View 13 Replies
View Related
Nov 5, 2009
I have a file that stores employee login IDs, names, types, and permissions. Our software reads the information based on byte-columns, so it reads a column as any ASCII character (spaces, letters, numbers, punctuation, etc.). I want to create a web-interface for adding and removing users, and storing the data in a MySQL database. However, if I am creating the files from the MySQL output, I need a way to write to specific column locations in the file ...
User ID: Columns 1-4
User Name: Columns 6-30
Type: 32-40
Permissions: 42-45
I want to use a scripting language, preferably C-Shell, to call MySQL for the data and write the data to the correct columns of the file. I wrote a script that takes the data from the file, and dumps it into the MySQL table, so maybe I can pad the remaining space in the table column to fill with spaces ...
View 12 Replies
View Related
Dec 1, 2008
I am writing a script to mine data from my game server logs which are going to mysql. My first question is syslog-ng is sticking $msg into one colum it self, $msg tends to be a long string here are a few examples:
cn name team frag death flags role host.
0 Maik CLA 2 0 0 normal *.70.233.118.
1 lizardncd RVSF -1 3 0 normal
[*.22.247.40] lizardncd fragged Maik.
And they are not always in the same format, How could I parse this data to do something useful, like creating a user stats page with frag counts and flag counts. Should I create a script that runs in a cronjob that parses this data to a new table maybe once a day. Even then the $msg string value is random and I am thinking in terms of awk.
Second. I would like to keep track of what ip's are using what nicknames. Kinda in the sense of and ip associated to a array of names that have been used on that ip address so I can keep track of users. How would this work logically with a mysql db, could a column allow multiple values, then loadable into an array, which then can add, remove, and store to mysql.
View 1 Replies
View Related
Jan 18, 2010
I have found many information about how to delete/grab a line in a text file including delete line with match a pattern but I did not find info about how to delete a line which match a pattern of a particular column only.
for example mydata.txt:
id type x y z
1 6 0.474611 0.227223 0.583947
[code]....
View 6 Replies
View Related
Apr 6, 2010
How can I delete selected rows of a table in perl/CGI??
View 5 Replies
View Related
Jul 13, 2011
I have tried to find the solution for my problem on this site and other sites but haven't found a good enough answer yet. Maybe some of you can help me out here?What i need is a script (bash preferrably) that can delete directories based on a date in its dirname.For example.I have a bunch of directories that is named
data-20110623/
data-20110624/
data-20110625/
[code]...
View 5 Replies
View Related
May 12, 2010
I've got a set of objects (all of the same type). I'm trying to think of a good way to divide it into equivalence classes, with equivalence of two objects defined as meaning a specified set of attributes are equal for both objects. More concretely, I've got:
- a Java class with around 50 fields
- a bunch of instances of the class
I want:
- to divide the instances into a few sets
- in each set, each instance has field 1 - field 5 equal to fields 1-5 of the other instances in the set.
The method I've come up with is to generate a hashcode for each instance based on the hashcodes of fields 1-5*, and map the hashcode to one of my sets. Ignoring problems with potential hashcode collisions (which I'm expecting to be too rare to worry about for now), does that sound reasonable? It seems simple enough, but I'm wondering if there's a simpler method I haven't thought of.
* I'll generate the hashcode using a method based on Eclipse's generic hashcode method, which looks like this:
Code:
public int hashCode() {
final int prime = 31;
int result = 1;
[code]....
View 5 Replies
View Related
Jul 19, 2010
How would i calculate following values.Initial file
10 3
20 4
How would i calculate 3rd column which should be addition of value in 1 and 2nd column.File after calculation
10 3 13
20 4 24
View 14 Replies
View Related
Mar 7, 2010
i use this script to get the time and date of back and fourth transactions for a particular execution id. I use a substr command on the 5th column to to cut the milli seconds off the time value. - otherwise the times would look like 08:30:04.235
grep <executionID> <auditfile> | awk '{ print $1, $2, $3, $4, substr($5,1,8}
FIX -> Mon 3/1/2010 08:30:04
FIX <- Mon 3/1/2010 08:32:36
FIX <- Mon 3/1/2010 08:35:08
[code].....
anyhow - i append two sed commands to further clarify the direction of the message.
awk '{ print $1, $2, $3, $4, substr($5,1,8} | sed -e 's/->/ ->IN/g' | sed -e 's/<-/<-OUT/g'
FIX -> IN Mon 3/1/2010 08:30:04
FIX <- OUT Mon 3/1/2010 08:32:36
[code]....
I tried using an awk gsub () command within the string instead of the two seds, but it did not work:
awk '{ print gsub(/<regex>/, <replace with>,$1), $2, $3, $4, substr($5,1,8}
the sed works ok, but it would be cooler to make the replacement within the awk command:
gsub(/->/,-> IN, $1)
Is there a way where i could replace the value of the $1 column in the awk print string?
View 1 Replies
View Related
Jul 27, 2010
I am splitting a file based on the values read from an input file. The below one is the script.
1)How do I add the header which is present in the original file to the new split files created?(For eg. pharmacyf conatins header as table column names. The new files created (ODS.POS.$pharmacyid.$tablename.$CURRENT_DATE.dat) are without the header).
2) Also the script is creating 0 byte files for the pharmacyids which are not available in the intial file? Can this be avoided?
for pharmacyf in *
do
tablename=`echo $pharmacyf |cut -f4 -d'.' `
while read pharmacyid
do
grep -w $pharmacyid $pharmacyf >> $OUT/ODS.POS.$pharmacyid.$tablename.$CURRENT_DATE.dat
done< inputfile
done
View 2 Replies
View Related
Aug 18, 2011
I want find a bunch of log files and delete ones that are older than say 5 days. Ideally I would then like to add this my crontab to run once a day.
The log files are in /var/log and are owned by root. They have a standard naming convention which is [date]RootCronRsync-backupHOME.log An example file is 20100621RootCronRsync-backupHOME.log Trying to put together a bash script to do this I think I need something like
Code: find /var/log/ -name *RootCronRsync-backupHOME.log -mtime +5 -exec rm {} ; However if I try this without the -exec rm (ie to see if I can find the right files first) I get the following error find: paths must precede expression: 20090405RootCronRsync-backupHOME.log Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]
View 8 Replies
View Related
Jul 15, 2010
I have a multicolumn datas, like
a1 b1 ... f1
a2 b2 ... f2
. . ... .
[code]...
I would like to make a file with all these data in one column, like
a1
a2
.
.
[code]....
Can it be done with awk or some other command? Also, is it possible then do add another column in front of this one with numbers of the lines (for every previous column), like
1 a1
2 a2
. .
. .
[code].....
View 14 Replies
View Related
Jun 7, 2010
I notice that on DistroWatch, the descriptions include "debian based", "arch based", etc. For the newbie, is there any practical difference between the different bases, or is it a Coke vs Pepsi type question?
Is a distro based on Debian say, easier for a newbie to learn or work with than say one that is FreeBSD based?
View 9 Replies
View Related
Apr 3, 2011
How can i concatenate rows of 2 files.
File1:
File2:
How want file3 to be like?
View 14 Replies
View Related
Apr 18, 2011
Does anyone have a solution for merging files if the number of rows in the two (or more) files is non-equivalent.To exemplify, how about merging the following 3 files:
file1:
1
2
[code]...
View 8 Replies
View Related
Jul 29, 2011
I would like to transform rows into columns from a file in bash.
View 4 Replies
View Related
Apr 7, 2011
When Debian 6 (amd 64, testing) is loaded, before desktop appears, for a while, the screen is black and three strange green rows appear. Then, all is ok and I can see a normale desktop and start to use Debian. So, an error in video config could be possible. Linked to this fact (I think) , there is the impossibility to modify the resolution screen (that is fixed to 1024x768) and the refresh frequency. I've tried to find "xorg.conf" in order to modify it like various guides suggest me, but this file is not present.
View 1 Replies
View Related
Mar 11, 2010
I have a csv file that has around 3 million rows. I want to process this file so that it creates a new file that contains only the rows that have 2 characters or less in the first column column.At the moment i am using SQL Loader but its taking too long so im wondering whether this would be easier if done in Unix?
View 8 Replies
View Related
Apr 18, 2010
Is the maximum number of letters a database table column can contain 255?and is it a bad idea to build a table with about 15 columns?
View 1 Replies
View Related
Apr 5, 2010
When you use an IDE and it asks you if you want frame based or dialog based, what is the difference?
View 1 Replies
View Related
Feb 2, 2011
what distro should i use, rpm based or debian based?
View 2 Replies
View Related
Jul 20, 2010
I have a big file full of records like this
1, 2, 4, 5, 6
1, 3, 5, 6, 3
1, 4, 5, 6, 6
2, 4, 5, 5, 5
2, 3, 4, 5, 2
anyway i need to take the average of all rows with the same 1st number(key). i.e.
1, 3, 4.66, 5.66, 5
2, 3.5, 4.5, 5, 3.5
I know this is something awk/sed would be great for, i just dont have enough experience with them to accomplish it, Also, what about averaging those columns together? so, after I output this to a file, id like to get another like:
1, 4.58
1, 4.125
The number of colums to add might not always be 4 either. EDIT: this might be easier to do in gnuplot, so I mainly just need an answer to the first part.
View 3 Replies
View Related
Oct 26, 2010
i am trying to get a script that i'm calling to have information from a sql populate into rows... but i'm not getting the data to output correctly into the rows. can someone please help?
<table>
<thead>
<tr>
<th>Name</th>
<th>Help</th>
<th>Folder</th>
code....
View 14 Replies
View Related
Jul 22, 2010
I'm not that familiar with sed and awk in order to be able to solve this problem on my own, so I'm calling on you for a bit of assistance. I'm writing a Nagios plugin to check our Oracle tablespaces and the output is given in one line like this: 1.04007771 TEMP 0 UNDOTBS1 .005340579 USERS 0 7 rows selected. I've been playing around with sed like below to delete the obsolete info and change every second space into a newline:
[code]...
how many tablespaces there are so I'd have to check all databases and 'hardcode' the tablespaces in my script. Is there any way to 'automate' this knowing that 'rows selected' preceded by a number is always the last line and using a sort of counter to auto-adjust the number to put in the -e 's/ / /2' part?
View 7 Replies
View Related
Mar 4, 2010
if there's a tab-delimited file under /usr/desktop, how can I determine the number of rows and columns of the file in shell?And, if told the the 3rd column of the file contains only numerical values and all values in the 5th column are unique, how can I verify these in shell?
View 13 Replies
View Related
Oct 8, 2010
I need to write shell script which can take number of files and count total rows from all CSVs and display total number of rows counted in all files. Is there any possibility of doing that using shell script and if yes then how.
View 4 Replies
View Related
Jan 18, 2011
I have a two column file and I need to create a new column in between the first and second but the new column adds a value to the first. E.g.code...
I thought I had figured out how to do it with the following but it just hangs:
awk -F '{print $0,$0+25,$1}' file_in > file_out
Also tried the following to no avail:
awk -F,-v OFS,'{print $0,$0+25,$1}' file_in > file_out
I can add the new column with the added value to the last column easy ( awk '{print $0,$0+25}' file_in > file_out).
View 8 Replies
View Related
Feb 23, 2011
I have not defined a user vimrc, OS is redhat 4.6 and after a search and replace the first column in the editor is highlighted yellow and it stays that way as I close and open vim. This is the /etc/vimrc that came on the system. anyone see a bug or a reason it would do that?
if v:lang =~ "utf8$" || v:lang =~ "UTF-8$"
set fileencodings=utf-8,latin1
endif
set nocompatible " Use Vim defaults (much better!)
[Code]....
View 2 Replies
View Related
Oct 20, 2010
1. when we run od command it displays octal values. But the first column will be always 000000. after that the actual file contents are displayed. Can anyone tell the meaning of that.
2. When we run ls -l command, in the first line of the output, we can see some integer value. What is the significance of that value.
View 4 Replies
View Related