I need to process billions of small files using bash shell commands with limited memory size (256MB). If any of those files contain certain "keywords", the file will be removed. I tried with command:
find . -type f -exec grep -i -l -H "keyword" '{}' + | xargs rm -rf
where all files are located within the current directory. But the command above failed in "out of memory".
I have an Ubuntu server in which a file is dumped every hour and a new file for the next hour and the process continues. If there is any problem due to which the creation of file stops then empty files are created every minute till the process is killed & started again. I need help to make a shell script to check if the empty files are being created and then kill the process and start it again.It would be a great help if anyone can help me regarding this.
View 9 Replies View RelatedIn below program I want to add (as part of the valid_cmds string) the pwd (print working directory), lo (logout), and cd (change directory) commands. However when I add those into original program ;
char *valid_cmds = " ls ps df pwd lo cd";
they are not working I have the cout message huh?
Original source code is below code...
This is a really odd bug I can't seem to figure it out. Basically, commands like ls can see all the files in the current directory, however when I go to execute the file it will give errors like "file not found", even when it most obviously is. If you look at my command history in the screenshot, you can see I can ls into a directory and see it's contents. When I try to run the file, I get the "no such file or directory" error.
However, if I type simply 'vm', I can't use tab completion to complete the directory name, and my third command is me typing 'vm' and hitting tabtab, it lists a bunch of vmware specific tools instead of the subdirectory name. I can then ls and see my current directory contents, and it will list only the single subdirectory. However, then I tried to use the full filepath from root to run the file, still to no avail. If anyone has any insight,
I was trying to run small shell script, but could not run. I got the error like in subject.
This is exact way i was trying to do.
I have a command which on the command line needs to look like this
rlam -if3 '!pvalue -H image1.jpg' > image2.jpg
Nevermind what rlam or pvalue do ... they are part of a program package I am using. The above command works on the command line, and also when written verbatim in a bash shell script.
My problem is: in the script I wish to replace image1.jpg with the content of a variable, e.g.
IM1=image1.jpg
How to I get the script to insert the value of $IM into the command when the pvalue part of it needs to be quoted?
I upgraded from Slackware 64 13.1 to Slackware 64 13.37 a week or so ago. I am now having a perceptible delay of a few seconds when launching commands from the command line, say for example: screen -R.
My laptop is an X200 with 3GB of RAM.
I named a number of files with spaces in them, and I want to replace the space with "_". However, every time I write a command in the shell with the file name (eg "Spring 2011"), the shell doesn't recognize the file or directory. What can I do about this? Is there any way to use the unicode character for a space?
View 5 Replies View Relatedthere is a folder. Its empty. When every I drag a new file and put into it it echo out "there is file in there" and keep monitoring the folder. How can I do it?
View 7 Replies View RelatedI have a computer with 16GB of ram. At the moment, top shows all the RAM is taken, (NOT by cache), but the RAM used by the various processes is very far from 16GB.I have seen this problem several times, but I don't understand what is happening.My only remedy so far has been to reboot the machine.
View 1 Replies View Relatedclarify me with ulimit output and memory limit?
ulimit -a output:
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 1572864
[code]...
I need show the number of process per user, and after the date of the oldest process per user also.
With "ps -eo user | sort -u" i get all users that are running any process. And with "ps U username | wc -l" i get the number of process that the user "username" is running.
But how can i merge both commands for do what i need? Like a FOR or something like that. There is any method of make a FOR using the list that i get with the first command?
And then for show the date of the oldest process.. with "ps U username | sort -k 4" (4 is TIME field) i can show the process of the user "username" sorting they by time. But how can i get the date of the process takes longer running?? I can get only the time, but no the date.
below are the details of my system. I have bash as my current shell, some really common commands aren't working.
Do I need to do a re-installation of bash? Or how do I install a selection of bash commands which I need? (for example a subset of [URL])
Code:
root@sdptfw:~ # uname -a
Linux sdptfw.sdpt.co.za 2.4.36 #1 Tue Jul 22 13:13:24 GMT 2008 i686 i686 i386 GNU/Linux
root@sdptfw:~ # echo $SHELL$
/bin/bash$
[Code]....
What bash command can I use to rename or change the extension or name of a batch of files (for example, from .php to .html)?
Furthermore, is there a simple bash or python script/command that can be used to open a batch of plain text files one-by-one, search for all instances of a specific word, and replace all of those instances with another word?
I am supposed to take some small files, and print them to a specific printer, such that the small files are concatenated into one file. The file name has to be included in the file that gets printed.
Should I be looking to concatenate the files into one file with the file names included, and then print them?
something like: -printfunction -printername < file*
I'm looking for a way to insert an SD memory card into my computer and have it copy the files from it (a specific directory) in the background while I view the images from the desktop.
View 2 Replies View RelatedI've been trying to find a bug in this test script, but haven't been able to so far. I'm not lazy, I promise...just new to Bash so am having a hard time catching syntax errors. I call the script with the option -disableVenusBld, and it still prints "Starting build", which it shouldn't be doing right?
[Code]...
I'm trying to make a shell script that will list the 50 newest files in a directory with several subdirectories in. I've been trying with the find-command with no luck and now I've figured I should probably use ls. The problem is when I do "ls -lRt | head -50" it will do 1 directory at the time. It will not first make the full list and then sort it. This will display all items in first directory, sorted, then the newest directory will be sorted and displayed. So I figured I have to sort the whole process of ls before I limit the head. So this is where I am at now: ls -lRt | sort <something clever here> | head -50
Only doing a "|sort|" will sort it by name if I understand it right and I don't know how to solve it. Here's also my first attempt if that is of any interrest or help, this was limited by the change status time of files (so some lists got very large). These lists dit not get sorted by time and I could not find any way to do so.find $ftpDir -ctime $time -type f -print > $ftpFileLsAny help on this would be appreciated since I'm sort of stuck now. After reading manuals for all the options I can think of and still there's just a big blur in my head..
I have had a fresh install of Ubuntu 9.10 and installed some software after that.Since third some, some process is eating half of my memory.I have checked processes running in system manager but everything is normal.Maximum is consumed by compiz which is about 26 mb, seems very normal.I did restarted my computer several times, and in the start for 5 mins, its fine after that again my cpu fans runs at very fast speed and my one cpu is used up 95 % (I have dual core).Please help me out, this invisible thing is driving me crazy.I am attaching my htop screen shot (sorted by cpu %), now the cpu is not used by completely but fan is still struggling hard and fast.
View 9 Replies View RelatedI need a command that can get the memory consumption of a process called "job_runner_o".
Running ubuntu 10.10
Programs like matlab/octave and I'm sure many other ones allow you to start typing a command, and then hit Up to recall the last command that starts with the typed characters. Common linux shell bash doesnot do this. Is there a different shell that does? I'm not asking how to find out the last command, I'm asking if there's a shell that's a little friendlier.
View 3 Replies View RelatedWhat happens when the script executes is that the ssh connection works and parks me at the remote hosts's shell login. Therefore, the "firefox" command refuses to execute. I need to know how to make the "ssh" connection occur, stay open, and go into the background so that the rest of the script can execute.If I could also do this with the "firefox" line so that the entire term window could be closed would also be helpful.
View 3 Replies View RelatedHow to stop a thread by using commands? Other threads in that process shouldn't disturbed. Is it possible to do the same?
View 2 Replies View RelatedIn linux, how can I display memory usage of each process if i do a 'ps -ef'?I would like to the 'virtual memory', 'res memory', 'shared memory' of each progress. I can get that via 'top', but I want the same info in 'ps -ef ' so that I can pipe the output to 'grep {my process name}'.
View 3 Replies View RelatedI am running a series of tests for an implementation of a remote pager that sends page faults to other computers in a network. Long story short, I was wondering if there is an easy way to force a process to use virtual memory as oppose to physical RAM so that I can better measure the performance of my implementation against how the system would perform while swapping to the hard drive.
View 4 Replies View RelatedIm using SUSE, i have 31GB of memory Mem: 31908592k total, 31429632k used, 478960k free, 12176k buffers. How do I find out what process are eating up all my memory.
View 3 Replies View RelatedDoes anyone know of a linux utility which will prevent all memory in a forked process from being swapped out to disk? I've seen the 'mlockall' call, but hacking the app sounds like overkill.My reason for needing this is that I'm running Windows XP under VirtualBox on my linux netbook, and I'm concerned there are basically two levels of swapping going on, which on a single dinky netbook hard disk isn't
View 3 Replies View RelatedWhat originally seemed like an easy thing to calculate has given me a big headache. Perhaps someone can help me with my issue. I am trying to find, in particular, how much memory certain application processes are taking. The process always is the same name, main_server, but with an argument to tell it what to do when running as a daemon.
When running the following command against all "main_server" processes, it produces a result in megabytes based on the output of the rss field in 'ps'.
Code:
CALC=0
for ea in
`ps -e -orss=,args= |
sort -b -k1,1n |
pr -TW$COLUMNS |
grep main_server |
grep -v "grep main_server" |
[Code]...
Currently I am left scratching my head. For capacity planning purposes, it would be nice to know how many more 'main_server' processes could run on the system without causing it to swap. Knowing the buffer and cache usage will go down as running processes demand more memory, I prefer to look at the free memory excluding cache and buffers. However, since 'ps' is reporting the processes are using more memory than free reports is in use without those things, I have no way to know how many more processes the system can support. I played around with different fields in 'ps', such as vsize, size, etc, but with no luck in matching up any numbers.
Is there any link where i can get information about below?
Dirty memory
RSS
PSS
One more?
if a set of process are getting executed in a use case say 50 times. How do one know the memory leak for a particular process?
I would be running SQL commands (UPDATE/SELECT) from within my bash script. I am completely new to this subject. Is MYSQL used for this purpose? Alternatively, what is sqlplus?
View 6 Replies View Related