General :: Bash Script To Read Csv File With Multiple Length Columns
Jul 27, 2011
I've searched everywhere and I can't come up with a good solution. For each line I need to find the average, min, and max. I've seen plenty of solutions where the number of columns is fixed, unfortunately for me these lines can get pretty large. My thought was to read each line individually into an array, loop through the array and find the avg, min, and max that way but i haven't had much luck. I can read each line using a while loop but I'm having trouble with the array part, or perhaps that's not the best solution?
View 14 Replies
ADVERTISEMENT
Jan 20, 2011
I have a text file with 2 columns. Column A has 69,000 rows. Column B has 49,000 rows. Column A has our complete product list Column B has product list from Manufacturer 1 There are only certain/some rows which are common between 2 columns. and also, column B is not a subset of column A. Column A has extra entries and so does column B. I need to know, which rows from Column B, are common with Column A which rows from Column B are not common with Column A. Essentially I want to know from this list, how many of our products are from my manufacturer 1, how many does my manufacturer has which we dont carry.
How would I acheive this? My natural approach to solving this kind of obstacle is to reach for MS excel and use its lookup function, but its not working... Its taking forever and hanging up. since the file is so huge and probably my excel skills are really bad.
How can I do this from command line? I am looking for awk command if possible instead of sed since I am trying to pick up its syntax and usage etc. My thought process is, sort column A,B, for every row in A, lookup and output based on condition. Dont know if I am on the right track.
View 7 Replies
View Related
Mar 11, 2011
For example, I have a text file with data which lists numerical values from two separate individuals
Code:
Person A
100
200
300
400
500
600
700
800
900
1000
1100
1200
Person B
1200
1100
1000
900
800
700
600
500
400
300
200
100
How would I go about reading the values for each Person, then being able to perform mathematical equations for each Person (finding the sum for example)?
View 13 Replies
View Related
May 26, 2010
First post from a very new Linux user....I am trying to create a BASH script that will allow user to provide multiple directory names, Checks if the directory exists and if not create the directory.
I am using the following code:
Which works fine as long as the user enters a single directory name. How can I modify this so it will process all directory names user enters on the read response?
View 5 Replies
View Related
Nov 10, 2009
Each line of the file I am sorting is in the following format:
<url> <month> <day>
For example:
[URL]
I wrote the following to sort:
Code:
#!/usr/bin/perl
$in = shift;
chomp($in);
[code]....
The script worked fine for my small testing files, but failed in my input file. The input file is 18MB and containing more than 300,000 lines. The output will contains some lines like that:
url_one 10 1
url_two 10 1
url_three 10 3
url_four 10 1
Is that because my file is too big for perl to handle ?
View 10 Replies
View Related
Aug 18, 2010
I am having a souce file (say SOURCEFILE) as below with 5 columns and many rows.
pol name status loan address
123 aaa 21 100 RJY
234 bbb 31 200 RMY
345 ccc 21 300 SJY
456 ddd 41 400 pqr
i want to compare the file (say COMPAREFILE ) which is below :
123
345
I need the following output file (sayOUTPUTFILE) as below :
( ie for file COMPAREFILE i need column 2, column 4 ie name and loan from source file )
ie output file required is to be like below :
123 aaa 100
345 ccc 400
Till now iam using the following method.
1. cutting column 2nd column
2. again cutting 4th column
3. in third, step merging the 2nd and 4 th column.
method of writting single line command or any awk programme to do this in one step.
I need output file (say OUTPUTFILE) as below
View 1 Replies
View Related
Sep 22, 2010
I'd like to extract a single column from 5 different files and put them gether in an output file. I saw a similar question for 2 input files, and the line of code workd very well, the code is:awk 'NR==FNR{a[NR]=$2; next} {print a[FNR], $2}' file1 file2I added the file3, file4 and file5 at the end, but it doesn't work. Does anyone know what do I have to do?
View 5 Replies
View Related
Dec 20, 2010
I want to merge columns (selectively) from several files and create a new file with the merge output. I saw some suggestions to use pr/paste to join the columns and then awk to pick-up the columns.
Code:
pr -m -t -s file1 file2 | gawk '{print $4,$5,$6,$1}'
But I have hundreds of files and I cannot manually pick up columns using awk as given in
[code]....
View 14 Replies
View Related
Jul 29, 2011
I would like to transform rows into columns from a file in bash.
View 4 Replies
View Related
May 7, 2010
As I'm starting to learn bash scripting I'm trying to automatize some tasks I usually perform. I have a notification mail I need to send several times a day. It has this structure:
Quote:
Dear user,
blah blah blah blah
You need to contact the following people:
[code]...
To replace "user", I found this:
Code:
read -p "Please enter username: " username
echo "Dear $username,"
Which probe to be very useful with other simple notifications like this. But I don't know how to manage the email addresses as they are usually more than one and could vary from 1 to 10. They should appear one above the other. I found this: "Here is a little work around. The only thing the user needs to do is hit enter without anything else on a line and it will close out"
Code:
#!/usr/bin/ksh
word=a
until [[ $word = "" ]];do
[code]....
I tried to use it and modify for my needs but I failed, I don't realize yet how can I use it. If possible, I would like to use the until loop like the above example just for learning purposes but any other form will be accepted as well.
View 10 Replies
View Related
Jun 20, 2011
I have many files with name in order of number. e.g)
u0101.asc
u0102.asc
u0103.asc
[code]...
I am trying to read file using for loop.
for ((date=01; date<=31; date++))
do
echo ${date}
done
but '01' is read(print) as '1' How can I make it read from '1' to '01'?
View 6 Replies
View Related
Sep 28, 2009
as a result of a find command, i have
852065 72: /bin/gunzip
852065 72: /bin/gzip
852065 72: /bin/zcat
(the first column is inode number and the second is size if you're curious)
I want to be able to format it in a way such that:
852065 72:
/bin/gunzip
/bin/gzip
/bin/zcat
I know I can get the bottom half using awk- but I can't figure out how to extract the first set of numbers and get them to appear only once
View 9 Replies
View Related
Nov 22, 2010
I wanted to be as clear as possible. I have to add hundreds of new servers to hundreds of configuration files in Nagios. Here is a sample structure:
servers.txt has:
SERVER3
SERVER4
[code]....
View 10 Replies
View Related
Jul 30, 2010
At last i finish my script to find the status of the URL , in this code i used all the status codes, because of this it take too much of length. with the help of wiki only i find all the status codes.code...
View 7 Replies
View Related
Mar 19, 2011
At last i finish my script to find the status of the URL , in this code i used all the status codes, because of this it take too much of length. with the help of wiki only i find all the status codes.Quote:and my question is do we need all the status codes necessary in this script..and is there any other way to reduce the code using any conditonal statements..code...
View 6 Replies
View Related
Nov 24, 2010
First, I made a simply script which let me download a file from a filehosting site on my server, but I can only put one link there, so I need help how to put multiple links in.Here is my script:
Code:
#!/bin/bash
echo "Enter URL:"
[code]...
View 3 Replies
View Related
Feb 13, 2011
Q=What will this do?
A=This will read from /root/Desktop/iplist (or anywhere you want to put the list) after it reads the ip it will ping the ip two times. If the remote host is up then it will write the ip to a log. If the ping fails it will display it on the screen, write the ip to syslog so that you can do remote logging, and last it will send the ip by email to a user or a group of users.
Code:
#!/bin/bash
#wrote by Zach Underwood feb-12-2011
#test and wrote on centos 5.5 with sendmail
rm -f /root/Desktop/iplog #This will remove the good iplog
exec 9</root/Desktop/iplist #put full path the file where ip list is. Make sure that the ips are listed one per line.
while read -u9 ip
do
ping -c 2 $ip > /dev/null 2> /dev/null # ping and discard output
if [ $? -eq 0 ]; then # check the exit code
echo "$ip is up" >> /root/Desktop/iplog #write the ip of the hosts that are up
else
echo "$ip is down" # will show on the screen the ips that are not up
logger -p auth.notice "$ip is down" # will write the down host to syslog
echo "$ip is down" | mail -v -s alert user@anydomain.com > /dev/null 2> /dev/null #will send mail users about host down
fi
done
View 3 Replies
View Related
Oct 9, 2009
I have a script almost working except for 1 thing. What I'm trying to do is read a file that has the files that need to be FTP'd using a bash script. I have everything working except the reading of the file. It works outside of the ftp script I've wrote but once I put it in the FTP script it doesn't.
Here's the Script:
#Here's where the problem is that I know of
I've been playing w/ the exclamation points to see if that could be the problem, but so far no luck.
View 6 Replies
View Related
Jun 10, 2011
I have a text file that contains a single word and I want to write a bash script that will read the word from the text file... The following is my incorrect attempt, as it assigns the name of the textfile to the variable as opposed to the word stored within the textfile:(assume I have a text file value.txt that has its contents a single word, say wordone)
Code:
#!/bin/sh
for f in value.txt
do
echo $f
done
so the output of the above script is value.txt, however I want it to be wordone.to summarise: how do I assign the value of the word contained within a textfile to a variable?
View 1 Replies
View Related
May 5, 2011
Despite the history file is unique (~/.bash_history) I see that the multiple bash processes run on different windows are not all updating that file. I presume that the bash is not taking into account this possibility (multiple bashes on multiple windows) and writes thus the history file in a simple straightforward way. This would mean that a number of history entries are lost. I've tried to find information but had no luck so far.
View 1 Replies
View Related
Mar 29, 2010
I am trying to write a script which compares a log file with reference file. The log file has a table, the LHS of the table are constants strings and RHS of table values changes if there are any changes in configuration.code...
Here I am looking for a script which compares test.log file ( whose RHS data-types are known prior whether it is digit or string) with test.Ref which is reference file for test.log file. It will be really helpful for me if any of you give some idea about writing this script.
View 2 Replies
View Related
Dec 19, 2010
I was trying to do a school assignment in LaTeX. The assignment involves having the page split into three columns: the first column is for the quote, the second is for a reaction, and the third is for questions. These columns must be able to break in the middle of a row. I was trying to accomplish this using LaTeX.
First, I tried the longtable environment, but that would not allow me to pagebreak in the middle of a row. Then, I tried parcolumns, but for some reason, the second "row" had a huge space between the first two words. Does anyone know of an environment suitable for this kind of work?
View 4 Replies
View Related
Feb 23, 2010
I have a folder with only 24 files named <number>.dat (i.e. 4.dat, 6.dat and so on) where <number> is between 0 and 256. Each file has just two columns of data and nothing else.
I'm trying to combine all the second columns ($2) together. I've been fiddling around with getline and so far have
awk '{ getline ln < "6.dat" ; print ln" "$2 }' 4.dat
which takes file 4.dat and adds $2 from 6.dat, but I want a single command to take each $2 from every file and add them to (for example) 4.dat (having $1 from 4.dat is no problem). A command that takes every file in the folder and grabs $2 and places them in a common file would be ideal. Frankly I can work around if you combine both columns from every file.
View 5 Replies
View Related
Feb 4, 2010
What are the possible problem when Windows access the file from Ubuntu got Read Only even though have a full permission to read, write and execute the file? Ubuntu to Ubuntu accessing the file there is no problem only Windows got a problem.
View 1 Replies
View Related
Aug 8, 2010
I have a script that reads part of a line, delimited between the first and second intended part by a colon. Then it "chops" the part after the colon, which are words offset by commas (counting them beforehand so as to catch every word in the string's second part), like this:
Code:
"COLORS.JPG:red,orange,yellow,green,"
(Returning)
red
[code]....
single script that parses/breaks both parts of a line like this "COLORS.JPG:red,orange,yellow,green;blue,indigo,violet," so that the two parts, separated into single words (or two and three words, sometimes with spaces) can be used as single-line annotations and written to JPEG files using Exiv2. So far, I haven't been able to come up with a script that does this without one part of the total string(usually that part after the colon) becoming the first word in the second array. In other words, I look for this:
KEYWORDS:
[ ]red
[ ]orange
[ ]yellow
[code]....
Or vice-versa (ie, the second array winds up as a single-line "member" of the first). I think it's because I'm using a single while read loop to read the text file in which the filenames and substrings happen to be. If there's some way of reading a file once and going back to the beginning to read it again in another while loop, I haven't found it.
View 14 Replies
View Related
Mar 14, 2011
I am struggling with Bash scripting at the moment (I can't seem how anyone can write scripts with this language!!!) I have a need at home to have a cron job execute daily to lookup my downloads.txt file, read each url (per line) and download content from that url. Then that entry needs to be removed (well I keep all urls in memory and clear the file afterwards). If an error occurred during the download process, then the url is written to a downloads.err file. I got all the above working except for properly reading the url from the text file without including newline characters. I am using the following to read:
while read url; do
--Do whatever here--
done < downloads.txt
How can I get it not to let the url variable have newline characters?
View 11 Replies
View Related
Nov 16, 2010
Multiple dirs full of mp3s All strictly encoded with exactly the same parameters (CBR 128kbps, Joint-Stereo, etc) Is it possible to determine the total playing time (to within ~98% accuracy) by some formula based on the total file size? I say ~98% accurate since ID3 tags do consume a small amount of space.
View 3 Replies
View Related
Apr 27, 2010
i've got a file with sorted words - one on each line.How could it be possible to delete thouse lines that have words of length 1 or 2 (1-2 letters). I guess a good way it will be with AWK, n its fuction length(), but getting it, i dont know how to delete those very lines.
View 14 Replies
View Related
Apr 19, 2010
in gcc how to read a blank line ie a string of length 0.my code:
Code:
#include<stdio.h>
#include<string.h>
[code]...
View 3 Replies
View Related
Aug 12, 2010
I've written a usb device driver and a program that sends and receives data over the bulk pipe. The read function sometimes returnsI'm reading an unknown amount of data. However, using a usb tracker I can see that the correct data is being sent.The error only occurs sometimes.I expect that the read function is told to read more data that it receives it would fail and return -1, however if this was the case then every read call would fail.
View 4 Replies
View Related
