I'm writing a script to automate rsync backups which so far is working great. I prefer to send output to a text file instead of watching it fly by but the blank output is slightly boring? I would love to use a progress bar. Not for each file but for overall progress. Something like
syncing /home directory 50% [++++++++]
I already know it's way over my head and maybe bash isn't even the right tool for the job.
Example from zenity:
Code:
#!/bin/sh
(
[code].....
After searching I cant find script that can deal with directories. All the found scripts work file to file and not directory to directory. Someone know script that can deal with all this situation?
View 8 Replies View RelatedI would really like to capture the output of scp and my file's progress. Scp updates the transfer rate every 1 second, and I will like to save the transfer rate at every update. So for example, if the file transfer takes 30 seconds, I would like 30 reports of the transfer rate.
The output looks like:
Code:
file.dat 1% 3664KB 938.5KB/s 05:48
Whenever I try a simple redirect like:
Code:
scp file.dat 192.168.1.100:~/ &> output
... it does not save the rate at every update, it only shows the final rate.
If I try using typescript by starting "script" ... it's the same deal.
I've got a loop which is able to echo out how complete a process is after each loop (e.g says 25% if the loop has 4 elements and has run once through). Is there any way I can print characters, perhaps a progress bar? I was thinking of using the percentage as a count or something and using.
Code:
I got told off for including code from work here.
I am busy writing this script to get a pixel count of a large batch of images in a directory for a research project. To do this I'm using a loop to get all the images in the directory. To make the whole procedure more user friendly for some colleagues I decided to use zenity, something I haven't yet really worked with. As not to make sure that the users don't have to watch to this blank screen during the actual processing of the images I'm trying to get the zenity progress bar working. So far the progress bar comes up but instead of changing over time it stays "empty" till all the images are processed and then becomes 100% when all images are done. What servers my needs for now, but since I am trying to learn more about zenity I was wondering how I could change the script to get the progress bar properly working? Below is the piece of script I am talking about.
Code:
cd $input #goto path of image directory
(for a in *jpg *JPG *jpeg *JPEG; do
if [ -f "$a" ]; then #was it a image file
b="$a" #set b to $a
convert -resize 1000x1000! ${b} -depth 1 -format %c histogram:info:$b.txt #get pixelcount and write to .txt file
fi
done) | zenity --progress
I am attempting to "export" the progress bar from wget display using sed. Basically, we have an app that starts wget to download a large file and we want to show a progress bar. Our application has a dbus interface to receive the download progress.
So we were think of a command like:
wget [] | sed [] | dbus-send[]
The problem at the moment is, how do you get the matched string out of sed and into dbus-send? I can get the progress string by:
sed -u 's/[0-9]*%/&/'
This populated '&' with the correct percentage, but I cannot seem to get this out of sed.
I've got a bash script that uses rsync via SSH to back up data from a remote webserver.
This script works fine when I run it myself but when it runs as a cron job I get the following error:
Code:
2010/02/20 08:15:01 [6786] Unexpected remote arg: user@www.server.com:~/www
2010/02/20 08:15:01 [6786] rsync error: syntax or usage error (code 1) at main.c(1219)
[code]....
i have written a bash script to convert flv files (e.g. videos) to xvid to be played on my dvd player. the conversion is not really the issue, as it works. what i am dealing with now are 2 problems i have not found a good workarouond:
1. surpressing the shell output of mencoder completely ( i do not want to see all the mencoder messages). this i have only solved to a certain extend.
2. i want to create a progress bar. this one i have not yet cracked.
the script:
code:
#!/bin/bash
## settings
v_bitrate=768
video_opts="-ovc xvid -xvidencopts bitrate=$v_bitrate:autoaspect"
audio_opts="-oac mp3lame -lameopts fast:preset=standard"
store_dir=./converted_files .....
#start the script
main_menu
1. mencoder output: using /dev/null works more or less. for example: if mencoder detects a double frame while converting, a message is still printed on the bash screen
2. progressbar: i have no clue yet on how to do this. any link or tip is welcome.
my goal is something like this:
[56%] [time in progress] [timeleft]
having my data backed up properly, but it's also in different places (e.g. photos and music on separate flash drives), so I have a few rsync's that I can execute manually, drives are mounted. Unfortunately, it gets confusing very quickly, I'd like a simple bash script that goes something like this (this pseudo-code would be most like REXX):
pair.1 = (/dev/sdb1, /dev/mmcblkp01)
pair.2 = (/some/other/device, /yet/another/device)
for i = 1 to #pairs
[code]....
Seems simple enough, but I've never written a bash script and wouldn't know where to start.
I would like to know how do I print the line # in a script. My requirement is, I have a script which is about ~5000 lines long. If there are any errors happen I just exit. And I would like to add the line # of the script where the error happened.
View 3 Replies View Relatedsimple bash code:
Code:
#!/bin/bash
trap "echo 'you got me'" SIGINT SIGTERM # to trap ctrl+c
echo "Press ctrl+c during 5 sec loop"
for ((i=0;i<5;i++)); do
[Code]...
How come code behaves normally and stops when ctrl+c signal is caught and resumes, but after I use at least one timeout read in the code it looks like, if signal is caught again it doesn't pause the execution but skips the loop. If you remove -t (timeout) option from the read, both loops look the same!
I have a config file that contains:
my.config:
Code:
Now in my bash script, I want to get the output /home/user instead of $HOME once read. So far, I have managed to get the $HOME variable but I can't get it to echo the variable. All I get is the output $HOME.
Here is my parse_cmd script:
Code:
I have written quite a few separate bash & scripts and php scripts that up to now I have run from cron jobs. However I have to estimate how long each takes to run, before running the next and so it probably takes much longer than necessary to run them all. They have to run in order.
Now there are so many I am thinking it would be better to have a master bash script that would run one after the other, but I am not sure how to get the master script to wait before starting to run the next script. Is this possible and is there a command that will make the script wait between bash and php scripts , for them to finish, before running the next?
I would like to get the filename (without extension) and the extension separately. The best solution I found so far is:
Let FILE="thefilenameofsomefilesfor_instance.txt"
Code:
NAME=`echo "$FILE" | cut -d'.' -f1`
EXTENSION=`echo "$FILE" | cut -d'.' -f2`
I think it would be better to count the len and remove 3 chars to right to get the extension, but it can be macintosh filenames with have 4 chars for extensions.
I've been given an old rsync script to update some data files on a few different servers... I am trying to understand what directory on the server it gets it's files as the way it seems to refer to the data on the server is:
data_source='random-server.fqdominan.net::Firstlogic'
Not sure what the "::Firstlogic" part with the two :: means.
Here is the entire script:
Code:
#!/bin/sh
# Begin user-configurable options #
# Who can run this script
authorized_user='appdtools'
# From where we rsync the update data
data_source='random-server.fqdominan.net::Firstlogic'
# From where we rsync the rollback data
rollback_data_source='random-server.fqdominan.net::Firstlogic_rollback'
# Where updates will go
data_target='/Firstlogic/postware/dirs'
# How to do the rsync .....
# Here is where we do the "work".
splash# Check user and print warning
if [ "${rsync_test}" == "-n" ]; then
update_address_data# Test the monthly update
else
stop_address# Stop the address standardizer
update_address_data# Load the monthly update
clean_logs# Empty the old log files
start_address# Start the address standardizer
fi
I have not tested this script yet but evidentialy it used to work. I just need to find out where to stick the files on random-server.fqdomain.net to test it out. The server it points to no longer has the files on them...
I have a cron job that runs overnight for a rsync script that backups my home directory to a external hard disk that is connected to the computer via usb 2.0. I also output the results to a log and follow it via tail -f command from a terminal, to monitor it.
Here is the script:
Code:
#!/bin/bash
# backup data on a daily basis via rsync and a cron job
echo
echo backup started `date` >>/home/user/scripts/backup/backuplog
echo
rsync -avh /home/user/ /media/Linux_ext3/
echo
echo backup complete `date` >>/home/user/scripts/backup/backuplog
echo
echo disk used: `du -csh /media/Linux_ext3` >>/home/user/scripts/backup/backuplog
echo
echo disk free: `df -h /media/Linux_ext3` >>/home/user/scripts/backup/backuplog
echo
Two things
- The rsync is doing a complete mirror of my home directory to the usb drive. For example, say I have a .txt file at the root of /home/user. It gets copied over. However I delete the .txt file the next day and if I go to /media/Linux_ext3/, the .txt file is still there.
I was always under the impression that rsync would mirror the two directories, correct?
-For my log, I think it is creating an entry, I think, for each file. Again, was under the impression that rsync would only copy over files that have be updated, correct?
That is the whole script:
Code:
#!/usr/bin/env bash
# backup device for me, m.tornow, probably:
# /dev/sde1
[Code].....
and BU_MOUNTPOINT being the mount-point where a device is being mounted (duh).
I am trying to create a simple bash script to rsync some folders within a directory stucture. I am using wild cards, in the rsync source directory structure, but my command always fails. I believe it is the way I am using wild cards within my for loop. Here is my command ;
Code:
for seq in `cat test.txt` ; do rsync -nvP /folder/folder/folder/folder/folder/**/$seq /folder/folder/folder/ ; done This always fails, where if I do a ls to the destination, to test the path, it always works.
Thought I'd post it here because it's more server related than desktop... I have a script that does:
[Code]....
This is used to sync my local development snapshot with the live web server. There has to be a more compact way of doing this? Can I combine some of the rsyncs? Can I make the rsync set or keep the user and group affiliations? Can I exclude .* yet include .htaccess?
When I run rsync --recursive --times --perms --links --delete --exclude-from='Documents/exclude.txt' ./ /media/myusb/
where Documents/exclude.txt is
- /Downloads/
- /Desktop/books/
the files in those directories are still copied onto my USB.
And...
I used fetchmail to download all my gmail emails. When I run rsync -ar --exclude-from='/home/xtheunknown0/Documents/exclude.txt' ./ /media/myusb/ I get the first image at url.
I need to create a script that will compare the differences between two folders and then to copy only the updated and new files only to another directory. I know I need to use rsync here, I can write scripts so really it not how to create a script it is how do I accomplish the transfer of only new or changes files between two folders to a new file. Do I need to link these two folders first and then use the "--compare-dest" switch.
View 8 Replies View RelatedI have a tiny shell script to rsync files between two servers and remove the source files.
This script works fine, when it has been initiated manually or even when the rsync command is executed on the command line.
But the same script doesn't work, when I try to automate it through crontab.
I am using 'abc' user to execute this rsync, instead of root, as root login to servers are restricted in all of our servers, by us.
As I mentioned earlier, manual execution works like charm!
When this rsync.sh is initiated through crontab, it runs the first command(chown abc.abc ...) perfectly without any issues. But the second line is not at all executed, and there is no log entry i can find at /mnt/xyz/folder/rsync.log.
I just tried to sync files from one server to another. After the sync process, I found the files are bigger than original ones.
I looked up the web and found someone mentions the rsync daemon. So I have to run the daemon on one server before I run the rsync?
The command I used is rsync --partial --progress -r source destination
I've got quite a decent rsync script setup, however I'd like to invoke it whenever there's change to a file. My initial idea was to use find, however this has two major flaws - the first being my particular unix veriant cant understand -print0 which means this doesn't work, the second is that I'm not 100% sure how to put variables into quotation marks so ls can understand the target:
Code:
for i in `find /shares/ -mtime -1 -print`; do ls -ltr $i;done
use rsync to cp such files and dirs under /var/www/html/mydir directory but these two files(/dir4/1.html /dir4/2.html) cant rsync to dest mechine.
rsync configure file,below...
I need to rename the resulted searched files from a loopI have the following code:
find . -name DOC* | while read i
do
find $i -type f -name '*.txt'
done
basically, I am searching for all txt files inside any folder starting with DOC name.this code is working fine with me.I need to rename those .txt files to .txtOLDOS: Ubuntu 10.4Bash shell
I'm using Ubuntu 10.04 LTS server and Postgresql 8.4. I have a .sh script that is run by cron every other hour. That works fine. The .sh script includes an rsync command that copies a postgresql dump .tar file to a remote archive location via ssh. That fails when run by cron; I think because it is (quietly) asking for the remote user's password (and not getting it). I set up the public/private ssh key arrangement. The script succeeds when run manually as the same user that the cron job uses, and does not ask for the password. I am able to ssh to the remote server from the source server (using the same username) and not get the password prompt (both directions), so why doesn't rsync work? I even put a .pgpass file in the root of that user's directory with that user's password, and the user/password are identical on both servers.
I think the problem is rsync is not able to use the ssh key correctly. I tried adding this to my script but it didn't help.
Code:
Here is the rsync command embedding in the .sh script.
Code:
Here is the cron entry:
Code:
I am trying to write a bash script that sources another bash script. Essentially, I need a few lines to check to see if a certain variable is set. If not, I set it manually, and then source a scripts with that variable in the path. I wrote a test script to try it, but for some reason the last line does not work. Here is what I wrote:
#!/bin/sh
source ~setupdir/setup.shrc #just a test, this line works
echo ${#SETUP} # prints 0 if setup is not set, which it isn't
if [ ${#SETUP} -eq 0 ]
then
SETUP="~setupdir"
fi
echo $SETUP # prints ~setupdir
[Code]...
i'm in the process of learing C++. currently i'm creating shell scripts to get things done. i'm just curious how, as a programmer using C++ you would get a similar job done.as an example i have a script that takes the contents of files, pipes it to some sed and awk commands, which is piped to create a new file. that file is then imported into a mysql database.if you were going to do this in C++, would you call the sed/awk programs to modify the file, or can it be done within the program itself? i'm probably jumping the gun here because i've just started learing about pointers so this is above my ability
View 12 Replies View Related