Programming :: Bash Time Comparison Within A Fast Moving Log?
Jun 1, 2010
I have a bash script that is tracking ERROR connections (running in a cron every 15 minutes), sometimes the errors are true and are not connecting. But most of the time, the disconnects/reconnects are between 30 seconds are sending out false positives. I am looking for a time comparison code to pick out the 30 second disconnects and know that this is and "ok" error. Here is a section of the log with timestamps, errors, etc.
Tue 2010-06-01 22:01:30 ERROR [DataBufferSendService] 172.31.0.1:8016 caught exception: [NetworkAPI] Exception: [SocketStream] socket error
Tue 2010-06-01 22:01:32 ERROR [DataBufferSendService] 172.31.0.2:8016 caught exception: [NetworkAPI] Exception: [SocketStream] socket error
Tue 2010-06-01 22:02:00 INFO [Client] 172.31.0.1:8016 connecting to 172.31.0.1 on port 8016
Tue 2010-06-01 22:02:02 INFO [Client] 172.31.0.2:8016 connecting to 172.31.0.2 on port 8016
I have no Bash skills but Im badly in need of a script to move the first 20,000 or whatever number of files from a directory containing over 200,000 files to a new directory. The problem is that I cant access the directory because its so large so I want to break it into chunks, but keep the files in order if possible. If you know of a script to meet my needs, please post it. Otherwise Ill post my fumblings with Bash until I find the right way to fix my problem.
I'm starting bash shell script and I'm looping without any solution.
I'm trying to find some files under a folder hierarchy and in case of errors moving these files to a destination folder under the same hierarchy recreating this hierarchy if not exists.
Finding all ._* files under /src and moving them to /dest recreating folder1 or the others which contains ._* files but without moving files which does not correspond to the pattern.
Code:
I tried find command and I'am getting all needed files
Code:
But I don't know how to use the output to get the parent folder of files which are found to
1- create folder with mkdir -p /dest/folder1 or /dest/folder1/folder4
2- move found files from /src/... to /dest/... with rm command
I'm working on a find command as this trying to do all in the same line but ... little lost
My install of 10.10 went fine with the exception of Audio. If I attempt to use audio with the Rythmbox application or through Adobe Flash (in firefox), it sounds like an old fashioned 33 1/3 rpm phonograph record playing at 45 rpms. This problem has been around for a while and I had hoped that it would be cleaned up by now.
I'm running an AMD 64 processor (in 32 bit mode) and my audio card is a "Sound Blaster Live Value" (EMU10k1X). My on-board sound card is turned off in the bios. It works beautifully under Ubuntu 10.04 and FC13 with Pulse Audio.
Originally I thought that it might be a problem with Rythmbox which is now at v 0.13.1, a later version then I have been using in 10.04. However, with the Rhythmbox application off, I still get the same effect when trying to use audio in firefox. I also noticed that my cpu utilization climbs to about 80% when Rhythmbox is on.
I am trying to find a nightly backup if it was successfully copied over, rename it and curl, but it's always passing the check even if the file is older than specified. From the command line it does as it should. Example is here;
Code: find /backup -type f -mmin +4440 -exec echo "found" {} ; - nothing returned (good). Then I change the time
I have scowered the internet for the answer to this one. I need a script to rename multiple files to the same exact name, run a program on the file then do the same for the next file. We have a unix backend system that is expecting to load the file with the filename of cards.in So I will have files named card.2009xxx, like i said i will have around 4 or five of those. I want the script to rename card.2009xxx to cards.in, run our unix program on cards.in which inturn changes the file name and once complete i want the script to rename the next cards.2009 to "cards.in"Until there are no more left in the directory and the unix program has processed all the files. All of this is occurring in the same directory. I have written some scripts but they fail by moving for example cards.200901 to cards.in then immediately moving cards.200902 to cards.in and that is not good because it is overwriting valuable data
I write a bash script to time some downloads from a server using ftp connection. ( something like: "time ftp 'options' ) I need to count the time for a list of files in a loop and output the result in a file.
I'm trying to write a simple program that lists a menu and then asks you for your decision, and you can answer with a number or the name. However, I don't know how to add the second options (name).
I'm very new at Bash scripting and have a bone head issue that I'm trying (and failing) to resolve. I cannot get this one IF statement to work, it seems the comparison operator does not think the resulting number from the $b*$c+$b operation is an integer even though it is a number. Below is a small proof of concept script with the bit I'm having trouble with.
I know it is wrong to use the "==" operator to compare the equality of two floating point numbers. Logically it would seem that if the "==" operator is not usable for floating point comparison, then the "<=" operator would also not be usable. Is this true? The lack of google search results on this topic made me think that it must be true. If that is true, then is it true that the only way to compare floats with <= or >= is with something like the code below? Code: bool smaller_than_or_equal(float a, float b) {
if ( fabs(a-b) < EPSILON){return true;} else if (a < b) {return true;} else {return false;}
} I think this is a general question, but if specifics are necessary, I am using the C++ language to code at the moment.
my issue is that the when i put in the correct input its going into the loop like i put in incorrect input and wont break out of the loop even with the correct input entered
I am fairly new to linux still and I'm currently trying to write a shell script (which I have never done) that is supposed to run the xrestop(like top) program. What I want to do is execute the program and check the identifier column to see if a certain program is running. If it is running I want to strip the data from that row of the terminal say every 5 seconds or so and put it in a text file. Is this going to be a complicated thing?
Heres a list of the columns when you run xrestop program res-base Wins GCs Fnts Pxms Misc PxmMem Other Total PID Identifier.
how do I get this regular expression to work in an if/else statement? This is just a little script for learning BASH. don't be too harsh.
This script will test if a certain number of files with 1-4 in their filename exist and print their filename. An error message will be printed if not.
# for i in `ls file[1-9]` do if [[ "$i" == *1-4 ]] ; then echo "This file, $i, ends in a number between 1-4" else echo "Error, this file, $i, does not end with a number between 1-4" fi done
I get this error. ./file_test.sh: 13: [[: not found
I am trying to use the Linux Ramdisk y machine and followed the instructions oninux/Ramdisk/ramdisk.html to create a 4GB ramdisk (total memory is 8GB, linux 2.6.9)The issue is that it is taking me almost exactly the same time to read a 1GB file from ramdisk as it is from disk. I was expecting it to be atleast twice as fast.Has anyone encountered this before?
I use basic ssh and scp on a regular basis and sometimes for file transfer... certainly over public networks. However, often I want to transfer large (several GB) files from one place on my LAN to another.
I've read that FTP is the fastest, and when I'm transfering many GB, fast matters. Of course I know it's not secure, but I don't need security on my home wired LAN. What I do need many times is preservation of date and time of each file when uploading. Filezilla does that and I use it, but sometimes I need a CLI solution. I have use ncFTP and been happy with it until I realized it doesn't preserve date and time on uploads (probably not on downloads either, haven't looked).
Is there a CLI FTP solution out there that will preserve file date and time information on file transfers (including uploads)?
I want to read values from a text file about 2 times a second and update a textLabel with those values but it only inputs the values with the last value from the text file? I tried the same thing with an LCDNumber but it only updated every 10 value changes? After each change do I need to clear the textLabel or refresh the window? For testing the print z and print x in the code prints the values correctly.Heres my code:
So I've been writing a code and I have been adding functions to make life easy, but all of a sudden when i added the last function I got this error *** glibc detected *** /home/ahzeeper/Desktop/C-code/check6: free(): invalid next size (fast): 0x0804c048 ***followed by a huge backtraceNow here is my code
I have an academic simulator software and I want to visualize its output at the same time the simulation is happening. However I want to separate visualization and simulation modules. The simulation data will be held in an array of a size around 0.5M and will be read only to visualization software (but updated regularly by simulator).
- In past I have used shared memory to share small variables among two applications.
- TCP/IP adds the option of having the simulator and visualization applications on separate machines but the implementation will be more difficult.
- I have also thought about an abstraction layer which allows to replace the communication/interconnection layer with other methods later (file/network/shared memory/pipe).
I would like to know how do I print the line # in a script. My requirement is, I have a script which is about ~5000 lines long. If there are any errors happen I just exit. And I would like to add the line # of the script where the error happened.
Code: #!/bin/bash trap "echo 'you got me'" SIGINT SIGTERM # to trap ctrl+c echo "Press ctrl+c during 5 sec loop" for ((i=0;i<5;i++)); do
[Code]...
How come code behaves normally and stops when ctrl+c signal is caught and resumes, but after I use at least one timeout read in the code it looks like, if signal is caught again it doesn't pause the execution but skips the loop. If you remove -t (timeout) option from the read, both loops look the same!
I am using Debian linux. I have 100 timers running. If a timer expired which will generate a signal and it was mapped to a same function handler. All the timers are mapped to one function handler. The problem is if the timer expires one at a time, the function handler called at a time. But if the 2 timers expires at a time, the function handler is called one time only instead 2 times. Is it possible to invoke the function handler as many times based on timer expirary happens simultaneoulsy?
Now in my bash script, I want to get the output /home/user instead of $HOME once read. So far, I have managed to get the $HOME variable but I can't get it to echo the variable. All I get is the output $HOME.
I have written quite a few separate bash & scripts and php scripts that up to now I have run from cron jobs. However I have to estimate how long each takes to run, before running the next and so it probably takes much longer than necessary to run them all. They have to run in order.
Now there are so many I am thinking it would be better to have a master bash script that would run one after the other, but I am not sure how to get the master script to wait before starting to run the next script. Is this possible and is there a command that will make the script wait between bash and php scripts , for them to finish, before running the next?
I am developing a program in a system where the Linux does not take care of the sync command automatically. So I have to run it from my application always I save some data in the disk, which in my case is a 2GB sdcard. It is true that I can make the operation system takes care of the syncronization, using a proper mount option, but in this case the programm's performance drops drastically. In particular I use the shelve module from Python to save data that comes from a socket/TCP connection and I have to deal with the potencial risk of the system being turned off suddenly Initially I wrote something like that to save data using shelve:
But that takes too much time to save the data. Note that I use the sync from the OS every time I close a file to prevent data corruption in the case of the "computer" being turned off with data even in the buffer. To improve the performance I made something like that:
Code:
def saveListData( list ) fd = shelve.open('file_name', 'c') for itemVo in list: fd[itemVo.key] = itemVo fd.close() os.system("sync")
Thus, first I saved an amount of objects in a list then I open the file and save the objects. In this way I have to open the file just one time to save a lot of objects.However I would like to know if adding a lot of objects before closing the file would increase the risk of data corruption.I known that turning off the system after fd.close() and before os.sync may cause problems. But what about turning off the system after
I am trying to set time using settimeofday in linux. But it sets local time. i.e works like SetLocalTime in windows. But I want to set system time(like SetSystemTime in windows). I could'nt find no other api in linux. What should i do? I had tried with mktime/gmtime apis
