I have a tiny shell script to rsync files between two servers and remove the source files.
This script works fine, when it has been initiated manually or even when the rsync command is executed on the command line.
But the same script doesn't work, when I try to automate it through crontab.
I am using 'abc' user to execute this rsync, instead of root, as root login to servers are restricted in all of our servers, by us.
As I mentioned earlier, manual execution works like charm!
When this rsync.sh is initiated through crontab, it runs the first command(chown abc.abc ...) perfectly without any issues. But the second line is not at all executed, and there is no log entry i can find at /mnt/xyz/folder/rsync.log.
I'm using Ubuntu 10.04 LTS server and Postgresql 8.4. I have a .sh script that is run by cron every other hour. That works fine. The .sh script includes an rsync command that copies a postgresql dump .tar file to a remote archive location via ssh. That fails when run by cron; I think because it is (quietly) asking for the remote user's password (and not getting it). I set up the public/private ssh key arrangement. The script succeeds when run manually as the same user that the cron job uses, and does not ask for the password. I am able to ssh to the remote server from the source server (using the same username) and not get the password prompt (both directions), so why doesn't rsync work? I even put a .pgpass file in the root of that user's directory with that user's password, and the user/password are identical on both servers.
I think the problem is rsync is not able to use the ssh key correctly. I tried adding this to my script but it didn't help.
Here is the rsync command embedding in the .sh script.
I've 15 web servers (in private network) running RHEL, Apache. Needs to sync web files between them. each server is accessible to each other via public key (with passphares).
1) Main server is web1 (where dev upload files initially). So I can make all other servers accessible by web1 without password/passphares and run rsync periodically to update files between them. But security is an issue here as all servers will become easily accessible.
2) Run rsync daemon in all other servers (except web1) on designated port and run rsync command from web1 to sync files. This will do the work but running daemon in all servers might increase overhead and making sure that daemon is running all the time etc. are my concern for this implementation.
Thought I'd post it here because it's more server related than desktop... I have a script that does:
This is used to sync my local development snapshot with the live web server. There has to be a more compact way of doing this? Can I combine some of the rsyncs? Can I make the rsync set or keep the user and group affiliations? Can I exclude .* yet include .htaccess?
I have a Linux box, running rsync, only problem is, I have to turn it off at night. What I want to do is turn it on and when it turns on, rsync will already be running in daemon mode rather then having to run rsync daemon. Does anyone know the solution? I'm sure its simple, I just cannot figure it out
I'm trying to rsync files and directories from a RedHat linux host(v 4.5 & 4.7) to a Windows server 2003R2 Standard Edition with cygwin running. I'm executing the rsync command from the cygwin shell. The transfer involves rsync'ing approximately 1 TB of data from the linux server to the windows server. After about 280+GB of data transfer, the transfer just dies.
There seems to be no particular file or directory that the transfer stops at. I'm able to rsync GB's of data from other linux hosts to this cygwin server with no problem. Files and directories rsync fine.The network infrastructure is essentially the same regardless of the server being rsync'ed in that it is GB Ethernet running through Cisco GB switches. There appear to be no glitches or hiccups across the network path.
I've asked the folks at rsync.samba.org if they know of any problems or issues. Their response has been neutral in that if the version of rsync that cygwin has ported is within standards then there is no rsync reason this problem should happen.I've asked the cygwin support site if they know of any issues and they have yet to reply. So, my question is whether the version of rsync that is ported to cygwin is standard. If so, is there any reason cygwin & rsync keep failing like this?
I've asked the local rsync on linux guru's and they can't see any reason this should fail from a linux perspective. Apparently I am our company cygwin knowledge base by default.
I use Fedora 14 32 bit at home and I d not have internet in home , so I download all packages are depend on Fedora 14 in my work place and move all of them to home by USB flashI use rsync for download all packages , in home I make local repo and install all packages I need by use local repo ,right now I want update my kernel , and I want use rsync foe download kernel update for fedora 14 .How I can do this , I want rsync only download update are depend on kernel and does not download other thing How I can do this ?
in home I do not have internet connection, but in work I have internet connection, in home I install Fedora 14 for my 6 years old daughter and she use it for play games like supertux and openarena and .So I want install opensuse for my daughter and I want test it. for fedora , I download all packages with rsync in work and move them to home by USN flash and then I make localrepo in home and install all packages , I need , I want do this for OpenSUSE , all of us know DVD , does not has all packages , I need , so I have to download all packages and make localrepo in home and install all packages , I need , Can I do this for OpenSuse or not ,I want download all packages need by OpenSuse by rsysc and make loacl repo, How I can do this for Opensuse ?
According to the man page, rsync should copy "special" and "device" files if the corresponding options are set. Also the man page says that the -a option (--archive) includes the -D option, which is "same as --devices and --specials". BUT We use the -a option, and still get the following in our report:
My Source folder contains 424.8 GB in 502,474 files. My Destination folder was created fresh, and after the copy contains 394.0 GB in 486.514 files. I am running it as grsync with root authority. The only options are to preserve time, permissions, owner and group., and to produce a verose output and transfer progress. There are no exceptions specified to skip any files.
I have two directories, dirA whicht contains N gb of data and dirB which is supposed to contain only the newest M gb of data from dirA. When files are added to dirA, they sould also be added to dirB, while the oldest files in dirB should be deleted.Is that possible with rsync? or any other software?
So I just used rsync to backup about 400gb of data to my NAS. Look just over a day to complete, which is what I figured. I decided I should run rsync again to see how its going to handle comparing and only adding new files to the remote location. So I added a few new files and then ran the backup again. Well rsync is trying to do a complete copy of all of my original data, even though they have not changed.
Is there a way that I can tell rsync to compare the two directories and only add the new files and delete the ones that are no longer in the original location?
I am running Ubuntu 10.04. I am transferring roughly 62 GB of data libraries to my 84 GB /home partition. I'm using rsync because scp kept stalling, and I had to restart it over and over. Things were going great until recently when it began to show an error: "failed: No space left on device (28)" These are the things I've done so far: Used the GUI to find out how much I have copied so far: 5,149,552 which take up 30.2 GB. df -h, it tells me that my /home partition is 56% used, and that I have 33 GB available. (42 GB used out of 78 GB with 33 GB available) Also, none of my other partitions are anywhere near 100%. the /home partition is the most-used and it's only a little over half-full. du -s in the directory where I'm copying all of this: it also returned 42206500. Additionally, when I try to save screen captures, it sometimes fails with a "device full" error. What's going on? Am I really out of space? Why doesn't it show me that I'm out of space?
Is there a hidden temp file that rsync uses that just got too full? I did a little research on wikipedia and it said that ext4 has a 64,000 directory limit. Could it be that I somehow broke that limit with all of these files? Solution: not enough inodes for the vast amount of subdirectories on hard drive. This wasn't an RSYNC problem, rather, a partition configuration issue. To check inode usage: df -i If you want to add any inodes, you will need to backup your partition and format it using mke2fs (man mke2fs). Be sure to change the respective inode setting.
rsync -r -v -e ssh email@example.com:/usr/local/websites/* /usr/local/websites and each time I run it it copies everything - all files. I thought rsync was only supposed to copy files that had been added or modified.
During my backups I'm finding that rsync is copying all files, instead of just what's changed.
I'm rsyncing between 2 USB external hard drives. One hard drive is FAT32 and one is NTFS. I've examined some of the files and believe that the difference is that there's a 1-second modtime difference developing in some of the files somehow.
Here's an example. These duplicity files were synced from /media/BACKUPHD (the NTFS drive) to /media/VIDEOHD (the FAT32 drive) only a few hours ago this morning. They have not been touched or changed since then, but that 1-second difference in their time stamps has appeared:
Code: tim@localhost:~> stat /media/BACKUPHD/backups/duplicity/duplicity-full.20110107T145955Z.vol10.difftar.gpg File: `/media/BACKUPHD/backups/duplicity/duplicity-full.20110107T145955Z.vol10.difftar.gpg'
I have Ubuntu on both my laptop and desktop machines, both are connected to the same network. I back up the laptop to the desktop by running the following on the laptop:
rsync -avv --stats /home/alisdt firstname.lastname@example.org:/home/alisdt/laptop_backup (with the IP address of the desktop instead of the many x, obviously). Whenever rsync hits a large file (greater than a few MB), the network use rapidly drops to ~60KB/s (that's kilobytes not bits). When I copy the same file to the same place using scp, I get > 500KB/s throughout the transfer. Things I've tried:
* mounting the desktop home dir on the laptop using SSHFS -- a simple file copy is fast, rsync is still slow * ditto with NFS * rsync --whole-file option, in case the delta-transfer algorithm was choking on large files * rsync --inplace option * HPN-SSH (http://www.psc.edu/networking/projects/hpn-ssh/) to enable dynamic window and unencrypted bulk transfer, just in case it was some ssh bottleneck I think it's either an rsync application problem, or a network problem that is only affecting rsync. Any ideas, or other ideas of what I can try to debug? In case it's relevant, I'm using 9.04 on both machines. (A standing bug prevents me from upgrading the laptop, and I haven't bothered to upgrade the desktop).
well, i know ther are issues when using rsync to copy files to ntfs partition like file permission blah blah. the thing is, i need to backup my music files periodically onto a ntfs partition from ext4. i really dont care about file permissions or any other stuff. when i use rsync, it should update the mp3 files on my ntfs (external) disc with the new ones.can i give a go with this operation? i have lot more important files on the external disc and i dont want this rsync corrupt or delete those files coz they are highly important files.
I built a script that downloads my podcasts using Gpodder into the directory /HOME/SHARED/PODCASTS/ (with a subdirectory for each podcast)The script then selects the latest episode and copies it over to a target directory (it empies the target directory first and copies over everything) I want to use RSYNC to make sure the 'not so fresh' episodes get deleted and the "fresh" episodes get copied over. Then dropbox can sync the "new" files over to the cloud where i can access them via my ipad/iphone (whole other story).The thing is : i've replaced the cp command with the RSYNC command and now the script is acting strangely.
It selects and sync's over the "newest" podcasts to the destination directory. Then it suddenly DELETES all the episodes in the destination directory and copies over the three last files.
I'm using the command below to sync two directories. Problem is insted of deleting the files on the target directory it simply appends a ~ character at the end of the file name. Not sure why this is happening?I'd like to have all deletes on the source replicated on target.
I was going to do a rsync -r -a -z -v -p -e sshto move some files frome server to another, but then realized all I really need are files which have dates starting June 1, 2008 to current. Is there a way to have rsync only sync those files?he directory structure that's my source goes all the way back to 2004.
I synchronise my data disk to a backup disk daily. Yesterday I noticed that rsync listed in its output my password-managers directory, which contains an old Keepass database and other Windows password programs I used long ago. I haven't accessed this directory in ages, so I was wondering why rsync felt the need to list itthe directories it updated.Modification dates and last accessed dates are what I would expect; nothing recen