Fedora :: Making An Rsync Backup More Robust?
Aug 24, 2009
I'm syncing a server over the internet with rsync, but it only works for a few hours before the backup fails with a "No route to host". I can restart the job and it'll will pick up where it left off, but is there an automated way to do this, or protect against a connection failure? I have about 170GB to copy over initially, but I can only get through about 4-5GB before the connection drops--manually restarting the sync everytime it drops will make the initial backup take days...
View 2 Replies
ADVERTISEMENT
Mar 6, 2010
I'm currently learning to use rsync to backup my music collection. I have a Firefox tab open to the rsync manual page(s) and have been reading man rsync and running experimental rsync operations.I've been doing this for the last 3-4 hours. I've used rsync for this purpose in the past with disastrous results. What was and is once again (due to a month and a half of file pruning) a 9000 file music collection had mysteriously grown to over 25,000 music files and 80GB of data! This was likely due to the fact that I didn't really know what I was doing with rsync and had never spent too much time learning about all the parameters, what their functions are and how they may relate to my goal.Here are the particulars:
* Source drive is a 500GB disk, /media/sata500/music/.
* Destination drive is a 250GB USB disk, /media/FreeAgent/music, connected to the same computer that houses the 500GB disk.
* I want to copy or backup files from /media/sata500/music to /media/FreeAgent/music.
* I do not want to create ANY duplicates of files that exist.
* I only want to add files to the destination drive if they are new on the source drive, like if I rip a CD and add the contents to the source. I want them copied over next time I run rsync.
Here's the rsync command in it's most recently used form, and probably very immature at this point.
Code:
rsync -t -r -vv --stats -i --log-file=/home/glenn/rsync.log /media/sata500/music/* /media/FreeAgent/music/
This appears to have copied all files and folders and I'm satisfied that my goal has been met with some success. To convince myself of this I ran the command and then once it was complete I added 2 new songs putting them in their respective folders on the source drive and ran the same command again. The resulting output was
[code]....
Two files transferred. Exactly what I want.Both folders now house 20,931 files and use 40.6GB. Identical as far as I can tell.What I'm concerned about are time stamps and play count data, etc. Anything that changes the original file. I don't want this data to cause a file to be transferred as I'm afraid that the new file will be created along side the old file of the same name thereby starting this whole music collection expansion thing all over again. I've invested a lot of time and effort to get it pruned down to where there are virtually no duplicates and albums are correct in that they contain the proper songs in the proper order.
View 14 Replies
View Related
Oct 29, 2009
I'm making my own yum repository - firstly so that all the machines I administer can be updatedvia the internal network, secondly so that I can test any updates on a spare machine before passing them on, and thirdly so that I can add my own repo for internal software.
I've created the necessary folders under my webserver, and used rsync to update them from my local CentOS mirror, following the instructions at [URL]
I notice it says to run "createrepo" on the base repository, created by copying the rpms from release DVD.
When I rsynced the updates repo, but I notice that the files in repodata are very small. In fact, having a look inside them, filelists.xml contains no file details. But if I run createrepo in the updates directory, filelists.xml gets lots of file details inside it.
I wondered if maybe the local mirror hadn't been updated properly, but checking against mirror.centos.org shows that has the same files.
how does the (real, live, CentOS) updates repo work when there is nothing in the filelists?
View 11 Replies
View Related
Nov 18, 2009
I'm running Fedora 11, and just did an update of the system. It looks like there was an update to the package that takes care of knetwork manager. In the past, I have played around with it, but it hasn't met my needs, so I went back to nm-applet, which works fine.
I'm curious about the update, and would like to see if it works. However, knetwork manager won't start as there is another network manager already running (there is a message about this everytime I log in). I can easily kill nm-applet, but logging out and back in causes it to reload, and I still get the message from knetwork manager. So here are my questions:
1) With the update, is knetwork manager stable and robust enough for regular use (the docs for F11 said it wasn't upon F11 release, so included gnome's network manager as the default network manager)?
2) How do I get knetwork manager to work (I presume this means I turn off Gnome's network management and switch to knetwork management)?
View 2 Replies
View Related
Nov 15, 2010
I was wondering if there is a way to tell rsync to only apply changes (delete, overwrite,create) only if all files in the file list transferred successfully.Just to clarify, this would essentially be putting a transaction around the transfer.
View 3 Replies
View Related
Mar 21, 2011
As an example, I have two servers, sm-i222 and fileserv. sm-i222 is a Win2k3 system running cygwin. fileserv is a linux box running RHEL 4.7. sm-i222 maps /cygdrive/c to the c: drive and /cygdrive/d maps to the d: drive(actually a single 4TB RAID). from the sm-i222 server /cygdrive/c I call a small script from the crontab. The internal IP for fileserv is 10.0.0.7. See below.
cd /cygdrive/d/fileserv/home
rm -rf /cygdrive/d/fileserv/home/*
rsync -avz -e 'ssh -c blowfish' root@10.0.0.7:/home/* .
These three lines perform well in that they make a full transfer of the fileserv:/home/ directory on fileserv to the appropriate place on sm-i222 using rsync. I use rsync instead of scp because I have to traverse subdirectories and symbolic links in the /home/... filesystem on fileserv. What I'm looking to do is use rsync to do an incremental transfer/backup of only the files that have changed since the last full backup. I'll manage the times I do this manually or in crontab. A colleague says this is do-able, but not how. Rsync.org says this is do-able but not how. Cygwin says this is do-able... see rsync.org. I believe what I'm looking for is a single rsync line like I have above that only transfers the changed files on fileserv to sm-i222.
View 14 Replies
View Related
Mar 2, 2010
I saw in a magazine reference to using rsync to have identical copies of folders. This looks like something I could find useful as I have a large number of items in need of safe backup.
I have the folders on an old system on a home network and would like to copy these over to a USB Hard Drive.
Currently the folders reside on SFTP xxx.xxx.xxx.xxx and I wish to sync them to a USB port on my laptop.
View 9 Replies
View Related
Aug 2, 2010
I have a samba share to a windows 7 computer I do not know if I will be able to use backintime or not so I want to know how to have rsync do my backup.I read the man but I'm not sure if I understand the it.on same computer different hard drive to run every hour in a script. Leanne is windows 7 share and backup is the other hard drive in the computer rsync -arvRzEP /media/leanne /media/backup.
View 1 Replies
View Related
Jan 18, 2016
I switched last summer from Windows (used it since Windows 95) to Debian. I'm using Debian Jessie for a couple of months now and I'm getting used a little.
There are problems here and there, but I can solved them with some reading on the web. Not really a big problem...till now
I run Debian 8.2 om my PC (PC1). Bought an older PC (PC2) that I want to use as a backup server.
I'm using PC2 only for making backups, after the backup I switch it off again.
So I installed Debian 8.2 (net-install without DE and with SSH) on PC2 and tried to configure it to let it work as my backup location. Made a public SSH key and exported it to the root account (no problem) and to the user account (sensdeb), but there was an error "Access Denied"
Gave the user (sensdeb) sudo-rights via visudo file
# User privilege specification
root ALL=(ALL:ALL) ALL
sensdeb ALL=(ALL:ALL) ALL
I installed rsync.
The problem is that Rsync only works when I use the root account.
Code: Select allrsync -r -n -t -v --progress --delete -u -l -H -s /media/Data/Mp3/Anastacia root@192.168.1.102:/test/Mp3
When I try to sync with he normal user account sensdeb
Code: Select allrsync -r -n -t -v --progress --delete -u -l -H -s /media/Data/Mp3/Anastacia sensdeb@192.168.1.102:/test/Mp3
I get error's. Access Denied
I don know how to give the user sensdeb the rights so that I can use that account for my backup tasks. Now it's possible to sync with the root account, but that should not be the way to do it, I read many times.
View 7 Replies
View Related
Mar 14, 2011
had some 'files disappearing' probs for rsync after upgrade to openSuSE 11.4 - solution seems to be to add :
--exclude=/var/lib
View 1 Replies
View Related
Feb 27, 2011
I want to save a backup of my data on a remote server, but never want the backup server to see the data unencrypted. Editing a single file and backing up should not result in everything being encrypted and sent again. The remote server should preferably not even know the directory structure (and especially not the directory names).
View 2 Replies
View Related
Jun 18, 2010
I'm trying to learn how rsync works to backup my system. I tried:
Code:
rsync -azvv /home /media/Elements
I get a folder called home on my external hard drive but when I use ls -l to see the permissions they are all wrong.
On my /home folder the permissions for /nathan are
drwxr-xr-x 48 nathan nathan
The permissions on the backup /nathan folder are
drwx------ 1 nathan nathan
I also tried using the long version of -a which is -rlptgoD and that didn't work either. What do the 48 and 1 mean when I used ls -l? When I look in the /nathan folder the permissions are all screwed up too. A lot of the files are backed up as executable and the permissions are all screwed up. I also ran it with sudo, and that didn't work either. The permissions were still screwed up and ownership is messed up too.
View 3 Replies
View Related
Jul 20, 2010
This should be a quick one. I'm trying to backup a single directory and it's subdirectories on my Lucid Server to a freenas box across my network. This is what I'm using to do that rsync -r -a -v -z * --delete freenas: DSIBackups..It almost works perfectly except for one problem. When a file is deleted at the source, this command doesn't seem to delete it on the receiving end. I assumed that the --delete would do that but aparently not.
View 1 Replies
View Related
Oct 20, 2010
when rsync is finished the update, or in the meantime - i need to move the updated files to a different location - like date +%Y%m%d something or what ..the reason is, because of the development, i need the modified files, but all of them, not just the last one - so i have to store them daily, but i dont want to store the whole dir - just that few files which are updated does it make sense?
View 5 Replies
View Related
Jul 13, 2010
I've been trying to make a three stage backup with stage 0 being a full monthly back up, stage 1 being a weekly backup, and stage 2 being a daily backup. I've been trying very hard to use rsync for this but sorting files by date is proving to be problematic. Sometimes it seems to work from the command line directly, but the same command causes errors and warnings from a script while entirely failing to sort the correct files.
The common example I see for this involves commands like this:
Code:
rsync -Rav `find /home/ -ctime -7 -print` /path/to/home_backup The problem seems to be that since the user directories in /home contain files that have been altered within the time frame specified the whole directory is matched first which means that the whole directory is recursively archived as opposed to just the changed files.
I've also seen examples using the --files-from tag using the same find parameters and this one seems to ALMOST work but gives me strange warnings and fails to run at all when launched from a script.
Many of the things I've googled about using rsync to backup stuff by the date modified involves a rather snarky "You're missing the point of rsync!" to which I respond by yelling at my computer monitor followed by "JUST TELL ME WHAT I NEED TO KNOW!" I understand that rsync is meant to take care of incremental backups on it's own, that's why I want to use it specifically for a traditional 3 stage backup scheme.
View 5 Replies
View Related
Jul 11, 2010
To copy from production to standby over the internet I use a cron job doing rsync -avze 'ssh -p 8022' --exclude-from= ....
My question is: should the cron job run on the production or the standby system. Root access to the remote system is given by a pass phrase-less ssh key. Currently I run rsync on the production system. I guess that it is more secure because the standby needs no ssh login to production. Running rsync on the standby would use less resources on production. I am concerned that in this case there would be pass phrase-less access from standby to production.
View 3 Replies
View Related
Jul 10, 2011
I'm doing an rsync backup to an external drive in order to take a shot at setting up partition encryption. My rsync command is, as root: Code: rsync -av / /external1/backup.Once I've finished my cryptsetup and done a fresh Linux install, what command should I use to properly restore my backup (without messing up the encryption setup)?
View 1 Replies
View Related
Nov 28, 2010
I'd like to backup my whole system to a 2nd disk using rsync (other tools not possible).Which paths should I exclude from the packup?I was thinking about /proc, /dev, the lost+found directories...What other paths am I forgetting?
View 2 Replies
View Related
Apr 14, 2010
I'm hoping somebody can find something here that I haven't. I'm trying to use rsync to backup home directories to a nas. First, I NFS mounted the nas and ran an rsync and everything worked out fine. the transfer completed after a few hours and everyting was transferred (lots of stuff!). I then decided that I don't want to leave the nas mounted all the time and I didn't want to automate mounting and unmounting of the nas as I didn't think I could produce a script that would work reliably enough. So I decided to start an rsync daemon on the nas and upgrade via that. I run the following command (results are included. the ^C is me killing it after it hangs).
Code:
ryan@server:/etc/backup$ sudo rsync -ax --stats --progress --delete /data root@192.168.0.98:backups1
root@192.168.0.98's password:
sending incremental file list
data/home/user/Documents/
data/home/user/Documents/The File.wmv
[Code]...
View 3 Replies
View Related
May 31, 2010
well, i know ther are issues when using rsync to copy files to ntfs partition like file permission blah blah. the thing is, i need to backup my music files periodically onto a ntfs partition from ext4. i really dont care about file permissions or any other stuff. when i use rsync, it should update the mp3 files on my ntfs (external) disc with the new ones.can i give a go with this operation? i have lot more important files on the external disc and i dont want this rsync corrupt or delete those files coz they are highly important files.
View 2 Replies
View Related
Sep 15, 2010
I have a Linux host acting as an ISCSI server for a Windows box. I want to keep an off site backup, so I figure rsync will keep the ISCSI server synced with an offsite Linux host. I understand that Rsync does block level incremental transfer to conserve bandwidth ok, awesome.The trick is, that I also want an archival copy kept. Say I want to go back to a revision of a file from 10 days ago, I need to be able to do that.
I was planning on using Backup Exec, since we currently have a licensed copy. Throw the archives from Backup Exec onto the ISCSI server as well, and have it keep a rotating 30 day backup, or something like that. The issue I see here is that this will be creating a deleting files as it does its daily backup rotation. I'm guessing RSYNC will see these as new files, and likely retransmit everything on a daily basis. The question then becomes, is this assumption correct, or will it still know to do a block level incremental transfer even when file names and such are changing?
View 5 Replies
View Related
Jan 29, 2011
Our backup script was working fine (ssh to the server, back up /home to a second hard drive on my computer). Then right after an ubuntu update, it quit working. I investigated and found that "something" had changed the label on the backup hdd to what looked like gibberish to me. But the script identified the backup hdd by its uuid, which didn't change. Yet, here is the error I get when the backup fails: receiving file list ... done [took about 5 seconds] rsync: mkdir "/media/14D9-3B1F/server-backup" failed: No such file or directory (2) rsync error: error in file IO (code 11) at main.c(594) [receiver=3.0.6]
Note that the backup hdd IS mounted, uuid is correct, and the folder 'server-backup' DOES exist. Does anyone have a clue for me? I'm moderately experienced in Linux and ubuntu. Our server runs centos 5. And as stated, the backup ran fine for several weeks. I think there was a new linux kernel on that update, but at this point a while later I don't know which one. Current kernel is .2.6.31-22-generic.
View 1 Replies
View Related
Feb 4, 2011
I support a small business which has an Ubuntu server running as a file server. The server is running Ubuntu 10.4. There is one hard drive which is mounted as /media/hdd. Each night this is backed up to an external USB hard drive mounted as /media/backup. The backup is carried out using the command:
Code:
rsync -av /media/hdd/ /media/backup/
Is there a way to encrypt this back-up so that if the USB hard drive is plugged into another machine it cannot be read?
View 5 Replies
View Related
Jul 30, 2011
i am trying to keep a backup of my root on a second partition using rsync.
sda1 system
sda2 system-BAK
sdb1 /home
[code]....
View 3 Replies
View Related
Mar 23, 2011
I'm going to be using this command to back up my files:
Should I change anything or is it ok?
View 4 Replies
View Related
May 22, 2010
I want to use rsync to backup a directory on my local hard drive to a USB hard drive.
I created the initial backup with:
rsync -av /share/chris/docs/* /usb/docs
View 7 Replies
View Related
Mar 6, 2011
/etc is where your gnu/linux stores it's configuration. Its function is similar to the Windoze registry. As you install, uninstall, and work with various programs, /etc changes. Sometimes, things change that make your system act unexpectedly.
1) open a console
2) su enter root password
3) # mkdir /var/backup
4) # vim /usr/sbin/prep4change replace "vim" with text editor of your choice
5) cut and paste the following short script
[code]....
Next time you are about to install, remove, or change something, just issue prep4change as root. If the unthinkable ( but expected ) should happen, and things stop working correctly, you have your /etc and /boot already backed up. Your backup will be located in /var/backup You can also $ tail /var/backup/notes to see why you made what backup Again, this isn't the most elegant, but it's simple, and it works!
View 3 Replies
View Related
Mar 4, 2010
I'm creating a backup scheme with rsync.
It is pretty clear not to include these:
/var/lib/named/proc
/var/lib/ntp/proc
[code]...
View 3 Replies
View Related
Aug 10, 2010
I'm going to make a nightly backup copy from one server to another, using rsync. If I have a sufficiently large file, say 4+ GB or so, I'm not interested in copying the whole file if only a small change has been made. Can rsync detect small changes on block level and backup only those if needed?
View 2 Replies
View Related
Oct 14, 2010
I went ahead and created this directory
mkdir /tmp/rsync-backup
and, ran this
rsync av /home /tmp/rsync-backup
this is the result
rsync: link_stat "/av" failed: No such file or directory (2)
skipping directory home
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1060) [sender=3.0.7]
View 5 Replies
View Related
