Programming :: Using AWK Within Bash Command Via Xargs
Mar 24, 2011
I often want to extract some info using awk from a variable/filename while running other things using xargs and sh. Below is an example:
Code: ls -1 *.txt | xargs -i sh -c 'NEW=`echo $0 | awk -F'_' '{print $1}'`; echo $NEW' {}
In the above case I would like to grab just the first field from a filename (delimited by '_') from within an sh command. This is a simplified example, where normally I would be doing some further data processing with the sh command(s).
The error message that I get is:
Code: }`; echo $NEW: -c: line 0: unexpected EOF while looking for matching ``'
}`; echo $NEW: -c: line 1: syntax error: unexpected end of file.
I haven't been able to figure out how to escape the awk command properly.
I get this behavior on Slackware 13.37, which includes BASH 4.1.010. Yes, BASH is my shell. I have a file called a.flac and I'm in the directory that contains it.
The output of the ls command is expected: Code: ls *.flac gives: Code: a.flac
Removing the extension with basename works as expected: Code: basename a.flac .flac gives: Code: a
Putting the above command in a variable substitution works as expected: Code: echo `basename a.flac .flac` gives: Code: a
Using xargs with ls and a variable substitution works as expected: Code: ls *.flac | xargs -i echo `echo {}` gives: Code: a.flac
However, when I try to add the basename command to the above command, it stops working. Code: ls *.flac | xargs -i echo `basename {} .flac` gives: Code: a.flac
Whereas the result I expect is: Code: a Why is it not working, and how do I make it work?
I would like to ask the following: 1) ls -l |grep test -> this will grep every "ls -l" output line 2) ls -1 |xargs grep test -> this will grep every single file with test 3) ls -1 |xargs echo -> this will echo directory list 4) ls -1 |echo -> this does nothing!!!
My question is: how some command can receive input from "both sides" (grep can grep whole output or every single file - xargs, the same is for i.e. wc command). 4) echo does nothing (it's a single echo command).
I'm testing some multi-plat java code and I'm getting a bit frustrated with the Linux tests. I need to run the command: Code: $ java -jar /home/developer/TCO/TabletComicOptimizer.jar <file> <args[]> against all the files that match a specific criteria. I've tried various find syntax and I can't seem to get it right.
Normally I would just create a bash script and populate the results of find into an array and then just enumerate the collection but in this specific case I want to demonstrate this operation at the bash terminal.
I've tried things like: Code: ~/TCO $ find . -type f -iname "*.cb[rz]" | xargs java -jar TabletComicOptimizer.jar {} 1200x1800 ; Thinking that the {} is the substitution for each file returned by find but it's not working. How do I execute my java program against each result in the find operation?
I have an awk program that finds all files of a specific filename and deletes them from selected subdirectories. There is logic in the awk to avoid certain subdirectories, and this is initialized via a parameter in the beginning statement of the awk. The parameter should have all of the subdirectory names at the top level. This varies from time to time, so I cannot hard-code the value.I'm having a problem initializing the awk parameter using sed. I'm setting a variable (named subdir) using an "ls" command piped to "xargs". I'm then trying to substitute that value into the awk using the sed command.
I am trying to grep multiple numbers from file, grep does have the -f option for that.
Code: grep -f <`seq 500 520` /etc/passwd I know this could be done with
Code: for i in `seq 500 520`; do grep "$i" /etc/passwd; done But my question is fare more behind this example. It is possible to redirect one command output which will be treat as a content of file for another command ?
I understand that $! is the PID of a command. For example:
Code: #!/bin/bash myprogram & echo "PID of myprogram is $!"
I'd like to send the output of "myprogram" to both console and to a log file using the "tee" command but I also want to store the PID of "myprogam". Something like this:
Code: #!/bin/bash myprogram | tee ./logfile & echo "PID of myprogram is $!"
The problem is that $! is now the PID of "tee" rather than the PID of "myprogram".
I have the following command that works Code: ssh root{at}IPADDRESS 'vim-cmd vmsvc/power.getstate 64 | grep Powered | awk "{ print $2}"' Which outputs the following text:- Powered on I would like to Append some text so the output is:- Ubuntu Server: Powered on Every different variation that I have tried ends up in an unexpected token.
I am trying to process a column separated data file, with a few bash command. For example, I have
Code:
file1 aaaa yes file2 aaaa no file3 bbbb yes
Let say I want to create new file with the output of first column and do something else with the output of 3rd column. Of course there are many ways to process this data file, but I wish to know by using awk, how could I do it. I'm trying:
Code:
awk '{system("touch $1")}' datafile
but the shell command will not able to get the awk '$1' output. How do I get this done ? And for another question, if the data file contains the variable name of a shell variable, how could I make use of it during a awk output ? For example I have a datafile1:
Code:
server1 yes server2 no
And in another server declaration data file, I got this datafile2:
Code:
server1=xxx1 server2=yyy1
And in my awk script, I want to achieve something like (the syntax is definitely wrong, just to demonstrate what I assume it will like):
As part of my script I need to compress a 50Gb file. but I need to check that the compressed is not corrupt if it is ok it will then send it over if not it will report an error.
cd /home/ops/Desktop/temp tar czvf backup-"$(date +%d-%b-%y)".tgz /home/ops/Desktop/temp
I need some here to check the compress file then somelike if the file is
if send the file else send an e-mail reporting a failure
I am having all sorts of trouble trying to assign a variable within an awk script with the system command. I know there is a lot of ways around this problem, but for efficiency reasons, I would like to, within my awk script, do something like
system(x=3)
or
system(x=NR)
and, latter on the shell script which calls the awk script, use the variable $x. But nothing is passed to x. I have already tried things like
command = "x=3" system(command)
and also used a pipeline within the system to pipe it to /bin/sh In fact tried a lot of stuff like that, using $(( )) etc etc etc I can create directories e write to files (yes, i could write to a file and read from there, but I dont think it is efficient, plus I am puzzled).
I'm trying to do something here:: I'm writing a bash script, I want to [open a new terminal and run a bash command in it] inside the script. I tried to use this, but apparently I get syntax errors.
This pretend to be a script for rename a lot of files automatically. So I put the list of files in an array named @lista. But, as you can see, at the end of the command I use a sed filter to print out a backslash for those files that have spaces in their names, so the path for those files could be rightly interpreted.
But there's no way I could print a backslash. It works well when I use the Perl's sed substitution s///, but I need every path in the array to be fixed.
I'd like to add that the bash command works perfectly well alone. I mean outside the Perl script.
I am new to scripting and been working on this bash script for awhile now. I been researching this problem, but I can't seem to find a solution. I was wondering if someone could please help me out. Here is my script:
I cannot get this script to run the "ps -ef" command on the client. It get its value from the host machine that I am running this script from. I need this command to execute on the client. When I run the command (ps -ef | grep NO | grep -v grep) on the client, I get something back. Here is what I get when I try to debug the script.
I want to delete all files within a specific folder without actually deleting the folder, what is a good bash command for this?. I found this one but encountered some errors even though I am executing it within the specific folder:
useratdebian:/home/user/folder# find . -type f -exec rm -rf {} ; [1] 5052 useratdebian:/home/user/folder# find: missing argument to `-exec' [1]+ Exit 1 find . -type f -exec rm -rf
The command as it appears is:
find . -type f -exec rm -rf {} ;
how to delete only the files contained within the folder called "folder" for example?
I'm writing a Bash script to take IPTC keywords from a text file and write them, via Exiv2, to several (first batch is 100) JPEG files in a single directory. The script has one while loop inside another while loop, both terminated, but I'm pretty sure that's not my problem. I think it's how I'm incrementing the "counter" variable, although it could also be the method of parsing the text lines from the file (using cut with delimiters that have worked fine in simpler scripts).
Here's the code as I've worked it up to this point.
Code:
And yes, "keywords" checks out in Crimson Editor, Emacs GUI and nano as an ASCII file with UNIX line endings. No issues on that score.
Feeding each line consecutively into a terminal (excepting the exiv2 command) works fine: each variable echoes with the part of the text line used as a variable value as it should, even when the b variable is incremented the quick&dirty way (up arrow three commands and hit enter).
Running the above script in eval mode (sh -x) stalls after setting the b variable to one and reading in the first line of text. I'd like to know why. I'd also like some advice on another reliable method of parsing the read-in lines.
Writing script to create backup of file by adding datetime to file name. Basically test for file presence if there, cp with datetime then rm original cp works fine from command line but get cannot stat `full path to file': No such file or directory
Code:
Here are the errors: cp: cannot stat `~/html/CVP_dadamail/.dada_files/.logs/errors.txt': No such file or directory rm: cannot remove `...': No such file or directory
The for statement is a placeholder as I have same file to backup out of several directories. using "bash -x scriptname" -OR- inserting echos, I can see I've constructed the strings properly. Believing it might be related to the hidden directories, I tried setting the shopt "glob" options to no avail.
Ultimately I'll add the other directories to the for loop and then run this from a cron job, so if you see potential pitfalls knowing I'm headed in that direction...believe construct would be
I'm trying to write a bash script program in the Linux command terminal that will write to a fellow user and then continue reading down the program. this is what i have (kind of explains the idea too):
#!/bin/sh
clear echo "this is before the write command" write jcummins this message should go to jerry echo "the message didn't send and this string will not appear" echo "it appears it has stopped at the write command"
I wanted to find and replace a string from a perl file. I have written a script in bash which runs the following command.
perl -pi -e "s/$findstring/$replacestring/" testfile where as $findstring = print F_WC_TMP"$line "; and $replaceString = $line = join ' ', split ' ', $line; print F_WC_TMP"$line ";
But when I am running the above command, i think it is replacing the $findstring with the above mentioned string and hence it contains a $line, it is looking for the variable $line and not finding the exact string. I am confused about how to search for a string that contains $ in it and replace it with another $string.
I wonder if there is anyway to make a user-defined bash shell function global, meaning the function can be use in any bash shell scripts, interactively or not. This is what I attempted:
I am trying to fix a perl script, and I really suck at perl. But I think this problem will be easy for people who know it.
The problem is, I have an old setup script someone wrote many years ago. It fails if the standard shell is dash and not bash. The only way I've gotten it to work is to point /bin/sh to bash. I looked thru the script and it uses "system" many places, and I think that's the problem.
I searched for it and found this link:url
My plan is to include this function:
Code: sub system_bash { my @args = ( "bash", "-c", shift ); system(@args); } Then I could simply change all calls to system into system_bash and it should work?
The parameter to the system calls is usually some variable. What if the parameter is a list already? Do I need to test for it somehow, and if it's a list, prepend "bash" and "-c" to the list? How do I do that?
In the script there are lots of places like this:
my $error = system($cmd); if ($error) { die/warn "some error message"; }
Shouldn't there be a return in the system_bash function?
Trying to create a small script that will read user's input, test if user entered some input and if not display some message or display a text using user's input.
The script is the following but i get an error saying "[: 6: =: argument expected"
I use command "find" in my bash script: if the filename exist command find work quiet, and if the filename not exist I see the message "find: /tmp/filename: No such file or directory". My problem is following, i want to have in my script something like this:
find "/tmp/filename" -type f -delete | "if no_any_errors execute command1" , if file_not_found execute command2"
Is there a way to use exec, but if exec fails to go on with the script?
Example:
Code: #!/usr/bin/env bash exec startx echo "Starting of X failed"
If startx fails, the echo will be seen on the screen. I tried all kind of stuff, but guess it ain't of much use to post it here. I searched the web, but searching for "exec and bash" in one sentence does give results which are not what i am looking for.
