Server :: Using Rsync For Incremental Backups?
Jan 29, 2010Can I use rsync for incremental backups of the running linux server?
View 5 Replies
ADVERTISEMENT
General :: Rsync Incremental Backups Rather Than Full Backups?
Nov 12, 2009How do you get Rsync to do incremental backups rather than full backups? At the moment I have a script that will create a backup folder (if it doesnt already exist) then copy the source files into the backup directory with the command
rsync $VERBOSE --exclude=$TARGET/ $EXCLUDE --exclude '/Ls-wtgl1c8/**' -rt --delete $source/ $TARGET/$source/ >> $LOG_FILE
Target is where the files will be backed up to Sources is the dir(s) to be backed up Exclude files is the list of files not to backup
log file is where the output will be saved to. At the moment it only does full backups, but I would only like to do incremental, how would this be achieved? Am I missing out an option in the Rsync that is required.
General :: Rsync Not Doing Incremental Backups?
Jan 21, 2010I am using rsync to backup dirs on my ubuntu server onto a NAS (which is mounted onto the filesystem), but the problem is that it is constantly doing full backups rather than doing incrementals and I am not really sure why. After doing a bit of expermienting with the script I noticed that if I just backed up a home dir (/home/user) the incremental backups work fine. If however I was to back up a dir like (/home/domain/user) it always does full backups.I have tried various different scripts but still the same end result. The latest script is a variation on the a script found on the samba rsync examples webpage, see below...
#!/bin/bash
# rsyncbu.sh -- backup to nas using rsync
# This script backups files listed in BDIR to the BSERVER. The verbose output along with the date is listed in the LOG_FILE specified
# verbose output
[code]....
Software :: Rsync Incremental Backups To Be Restored?
Dec 2, 2010With the --backup and --backup-dir= options on rsync, I can tell it another tree where to put files that are deleted or replaced. I'm hoping it fills out the tree with a replica of the original directory paths (at least for the files put there) or else it's a show stopper. What I'm wanting to find out applies when I'm restoring files. Assuming each time I run rsync (once a day) I make a new directory tree (named by the date) for the backup directory. For each file name/path in the tree, I would start with whatever is in the main tree (the rsync target) and work through the incremental trees going backwards until I reach the date of interest to restore to. If along the way I encounter a file in an incremental, I would replace the previous file at that path with this next one. So by the time I get back to a given date, I should have the version of the file which was present at that date. Do this for each file in the tree and it should be a full restore.
But ... and this is the hard part, it seems. What about files that did not exist at the intended restore date, but do exist (were created) on a date after the intended restore date. What I'd want for a correct restore would be for such files to be absent in the restored tree (just as they were absent in the source tree on that date). How can such a restore be done to correctly exclude these files? Wouldn't rsync have to store some kind of sentinel that indicates that on dates prior, the file did not exist. I suspect someone might suggest I just make a complete hard linked replica tree for each date, and this way absent files will clearly be absent. I can assure you this is completely impractical because I have actually done this before. I ended up with backup filesystems that have so many directories and nodes that it could take over a day, maybe even days, to just do something like "du -s" on it. I'm intending to keep daily changes for at least a couple years, if not more. So that means the 40 million plus files would be multiplied by over 700, making programs like "du -s" have to check over 28 BILLION file names (and that's assuming the number of files does not grow over the next two years).
Server :: Create Daily Incremental Backups Easily?
Nov 29, 2010I've had several HDD crashes on my personal server over the years and it's just gotten to be a real pain in the rear. Crashed again this morning. Currently, I make monthly tarball backups of the entire filesystem using my script:
Code:
#!/bin/sh
# Removes the tarball from the previous execution.
rm -rf /backup/data/*.tar.gz
[code].....
Server :: Making Incremental Copies/transfers With Rsync In Cygwin?
Mar 21, 2011As an example, I have two servers, sm-i222 and fileserv. sm-i222 is a Win2k3 system running cygwin. fileserv is a linux box running RHEL 4.7. sm-i222 maps /cygdrive/c to the c: drive and /cygdrive/d maps to the d: drive(actually a single 4TB RAID). from the sm-i222 server /cygdrive/c I call a small script from the crontab. The internal IP for fileserv is 10.0.0.7. See below.
cd /cygdrive/d/fileserv/home
rm -rf /cygdrive/d/fileserv/home/*
rsync -avz -e 'ssh -c blowfish' root@10.0.0.7:/home/* .
These three lines perform well in that they make a full transfer of the fileserv:/home/ directory on fileserv to the appropriate place on sm-i222 using rsync. I use rsync instead of scp because I have to traverse subdirectories and symbolic links in the /home/... filesystem on fileserv. What I'm looking to do is use rsync to do an incremental transfer/backup of only the files that have changed since the last full backup. I'll manage the times I do this manually or in crontab. A colleague says this is do-able, but not how. Rsync.org says this is do-able but not how. Cygwin says this is do-able... see rsync.org. I believe what I'm looking for is a single rsync line like I have above that only transfers the changed files on fileserv to sm-i222.
General :: Make Incremental Backups On A Separate Partition Of The Same Hard Drive?
Jan 22, 2010one would have to exclude certain folders / directories but would the backup be possible if the system is up and running in its native "live" state ? Which directories could be excluded ? Does swap need to be turned off ? I would like to make incremental backups on a separate partition of the same hard drive. I will endeavour to backup the MBR/ Partition table using dd.
View 6 Replies View RelatedUbuntu :: Rsync Like Incremental Backup
Oct 20, 2010when rsync is finished the update, or in the meantime - i need to move the updated files to a different location - like date +%Y%m%d something or what ..the reason is, because of the development, i need the modified files, but all of them, not just the last one - so i have to store them daily, but i dont want to store the whole dir - just that few files which are updated does it make sense?
View 5 Replies View RelatedGeneral :: Rsync Command - Sending Incremental File List?
Feb 14, 2010I received the following output from an rsync (3.0.0) command that was executed:
sending incremental file list
sent 77214 bytes received 484 bytes 155396.00 bytes/sec
total size is 254531170 speedup is 3275.90
What does "sending incremental file list" mean?
Ubuntu :: What Can Use For Backups That Is Faster Then Rsync?
May 9, 2010I am currently backing up my data but find that it takes way to long to do a rsync, it takes forever to just find the differences and transfer them.Out of 3 separate rsyncs the main one that is slow is my www.skins.be mirror directory which is 41GB and has 392,200 files, sorted into multiple directories. Which grows by around 100 every couple days.I think that something that would be able to track changes by inotify time on directories will speed it up since Picasa sure finds the changes fast when I open it and it is tracking over 26,200 pictures. I just don't know of a backup solution that does that.
View 4 Replies View RelatedSecurity :: Securing Backups Via Rsync And SSL?
Apr 27, 2011There are multiple servers to be backed up. Different access rights exist in each server. There are two backup servers with plenty of disk space, one local, and one offsite. The local one feeds to the offsite one. The rsync command is being used to make a replica of backed up data. Deleted data is also being archived. There are two methods that have been considered: One is to have the individual servers run rsync which logs in to the backup server to push data. Two is to have the backup server run rsync which logs in to each individual server to pull data. Because system data is involved and meta information (like owning user) must be stored, root is required to access the data as well as to store it. That means everything runs as root both ends. So method one was quickly dismissed because each server would effectively have rights to access ALL the data on the backup server since it logs into the backup server as root. The security containment here involves different groups using different servers, and they need to be isolated from each other.
But even method two involves some risks that are a concern. This means one machine has access rights to every server. If the backup server were compromised, every machine could be compromised.What I'd like to find is some way to allow backups to be run without either machine granting root access to the other, while still running as root, or something equivalent, that allows accessing all data and storing all metadata. So I was looking at setting up an rsync daemon on each individual server (running as root so it can access what it is specified to access), and running an rsync client on the backup server (as root so it can store metadata). This opens network access issues. Any user on the network can connect to the rsync daemon. So password protection is needed. But this communication is also not encrypted, which exposes the password and the data should the network be sniffed.
So now I'm thinking about a non-root ssh login between machines. The backup server would login to a non-privileged user on each individual server and set up a secure forwarding channel to the rsync daemon. Is this the best that can be done? Is there a way to run rsync via SSL with key verification so it can all be done together? I'd like to have the rsync daemons configured to always talk SSL, and always verify the client's key against a list of authorized keys, and likewise the client verify the server's key against the known public key for that server.
Debian :: Rsync Error In Rsnapshot Backups
Apr 5, 2011I am backing up my debian server with rsnapshot which actually uses rsync to perform the backup. The backups are located in an external storage of size 1.4T .
[code]....
I tried to understand what this error message means and i founde that error code 12 : 12 Error in rsync protocol data stream I understand that when rsync find that a file on the target was changed , it will send only the block/blocks that contain the changes and in the destination rsync will create new file and not update the old one (new inod...) . I want to know if this error i get is due to full disk or perhaps it is some other factor
Security :: Logging In As Root Over Rsync/ssh For Backups?
Sep 30, 2010I need to login as root, or at least get root privileges, in a cron triggered backup run. The straight way to do this would be the backup server making an ssh connection to the server to be backed up (this way because I want to avoid many servers being backed up in parallel and the backup server itself would be managing this diversity), via the rsync command which would be performing the backup's synchronization step.
I'm looking for alternatives to this in some form. I'd like to disallow direct root login to my ssh port (not 22One idea I have is to have the backup server initiate an ssh login as a non-root user, to either the actual source server, or to a server that can reach the source server ... and set up port forwarding. Over the forwarded port, then initiate the rsync that logs in as root via another port that allows direct root, but cannot be reached from the internet at all (because the border firewall doesn't include this port as allowed in).FYI, these logins will be using ssh keys, not passwords. I do need to keep ownership metadata for files being backed up, so this is why I am using root. Also, rsync is needed to get the incremental updates to keep bandwidth usage lower (otherwise I could just transfer a tarball each day).Anyone have any other ideas or comments, for security issues, based on experience doing things like this (backups, routine data replication, etc)?
Software :: Rsync: Backups And Hard Links?
Jul 13, 2011I am using rsync for incremental backups. I am backing up to a second hard drive on my computer. When I check the individual backup directories (backup.0 through backup.4) with du -hs they each show 12G; when I check the parent directory squeeze it shows 15G. Over 4 backups I have added 3G. I haven't made very much for changes to directories I'm backing up and am using hard links. I have included some info below.
Quote:
Backup script:
#!/bin/bash
mount /mnt/backup
cd /mnt/backup/squeeze/
rm -rf backup.7
[code]....
Ubuntu :: Rsync For Initial Backup Then Maintenance Backups?
May 9, 2011So I am using rsync (3.0.7 on MAC OSX) to backup one hard drive to a folder on another one. The is USB drive to USB drive and I have done the initial backup from one drive to a new formatted other drive with the following command:
Code:
rsync -avX --progress /Volumes/Source /Volumes/Destination
This all appears to be going smoothly as I type. I am going to write a script to do subsequent backups in the
[code]....
Networking :: Malformed Rsync Command (backups) On ReadyNAS
Oct 6, 2010I'm trying to set up rsync backups on my ReadyNAS and I'm getting the following error: ERROR: The remote path must start with a module name not a / This error is accompanied by the following information:
[Code]...
Ubuntu :: Rsync And Full-disk Backups - Failure Takes Too Long
Mar 20, 2010I've been using dump/restore for backups, for quite some time. It's worked fine, but the process of recovering from a HD failure takes too long. What with eSATA and external drive docks, what I'd really like is to use rsync to maintain a current clone of my entire system drive. That is, start with a full disk clone, and then use rsync to keep it current.
I've seen plenty of instructions on how to do this with a directory tree, but I've seen none for doing it with a copy of the entire disk. If, for example, I copy /etc/fdisk, then the copied disk would have entries with the same UUIDs as the original disk. Which would mean that if the clone disk were to be bootable, its partitions would need the same UUIDs as the original disk. Which they would be, if the cloned disk started as a full-disk clone, I think. Am I wrong? But that means that when the clone disk was active, I'd have partitions with duplicated UUIDs. Is this going to cause problems? When I boot, will I get the correct partitions loaded?
Ubuntu Security :: Rsync Automated Backups Of Entire File Structure Over Ssh?
May 16, 2011I am in the process of writing an rsync script to run unattended backups of my entire file system to another system located on my local network using ssh and password-less rsa keys.
I will absolutely will not use password-less keys with the root account and this is the limitation preventing me from accomplishing my goal because root is required by rsync to access the / tree and copy it to another location. I decided that if I compiled the script into a binary that I didn't have a problem with the password being contained within the binary itself but from what I've read there is no way to elevate to root and then back down to user level from within the script/binary.
I can create the script as the user and use chroot to make it owned by root but retain execution permission for the user but it will still cause the ssh login to be under root and therefore require either that I am there to enter my password or the use of password-less keys under the root account which I reiterate I will NOT do. Currently the script is executed by the user on the machine containing the files to be backed up.
Server :: Rsync Execution - With Crontab - Have Given Full Path To Rsync Too
Apr 12, 2011I have a tiny shell script to rsync files between two servers and remove the source files.
This script works fine, when it has been initiated manually or even when the rsync command is executed on the command line.
But the same script doesn't work, when I try to automate it through crontab.
I am using 'abc' user to execute this rsync, instead of root, as root login to servers are restricted in all of our servers, by us.
As I mentioned earlier, manual execution works like charm!
When this rsync.sh is initiated through crontab, it runs the first command(chown abc.abc ...) perfectly without any issues. But the second line is not at all executed, and there is no log entry i can find at /mnt/xyz/folder/rsync.log.
Software :: Run Rsync To Download Files From A Server Without Rsync Daemon?
Sep 18, 2009I just tried to sync files from one server to another. After the sync process, I found the files are bigger than original ones.
I looked up the web and found someone mentions the rsync daemon. So I have to run the daemon on one server before I run the rsync?
The command I used is rsync --partial --progress -r source destination
Server :: Rsync Can Not Rsync Files With Include Filter
Jul 21, 2010use rsync to cp such files and dirs under /var/www/html/mydir directory but these two files(/dir4/1.html /dir4/2.html) cant rsync to dest mechine.
rsync configure file,below...
Server :: Incremental Backup In Linux
Sep 18, 2010A complete back up using tar takes consumes more time. so is there any way to take incremental backups using tar.And i also want to take incremental backup dump of my databases too.Any suggestions and links will be very helpful.i keep on googling for this,but could find any exact for this.
View 14 Replies View RelatedServer :: Rsync Fails In Cron - Ssh Key - For Rsync?
Dec 8, 2010I'm using Ubuntu 10.04 LTS server and Postgresql 8.4. I have a .sh script that is run by cron every other hour. That works fine. The .sh script includes an rsync command that copies a postgresql dump .tar file to a remote archive location via ssh. That fails when run by cron; I think because it is (quietly) asking for the remote user's password (and not getting it). I set up the public/private ssh key arrangement. The script succeeds when run manually as the same user that the cron job uses, and does not ask for the password. I am able to ssh to the remote server from the source server (using the same username) and not get the password prompt (both directions), so why doesn't rsync work? I even put a .pgpass file in the root of that user's directory with that user's password, and the user/password are identical on both servers.
I think the problem is rsync is not able to use the ssh key correctly. I tried adding this to my script but it didn't help.
Code:
Here is the rsync command embedding in the .sh script.
Code:
Here is the cron entry:
Code:
CentOS 5 :: Scheduled Unattended Backups With Alerts If The Backups Fail
Feb 3, 2011I've been a DOS/Windows guy for 20 years, and recently became a SW test lab helper. My company uses CentOS for a lot, so I've become familiar with it, but obviously not as comfortable as I am with Windows.
Here's what I have planned:
machine: Core 2 Duo E8400, 8GB DDR2, 60GB SSD OS drive, ATI 4650 video card, other storage is flexible (I have 3 1TB drives and 4 750GB drives around that can be used in this machine.)
uses: HTPC, Network Storage, VMWare server host: SMTP, FTP server, and Web server virtual machines
I've figured out how to do much of this, but I haven't figured out how to do backups in Linux. I've been spoiled with Windows, with the built in backup system so simple to use. I find myself overwhelmed with the array of backup software, and unable to determine which to use. none of them seem to do everything I need them to do, but some come close, I think. I'm hoping someone here can help me out in figuring out which program to use and how to use it.
Here is what I need the backup software to do:
1. scheduled unattended backups, with alerts if the backups fail
2. a weekly full backup with incremental every 12 hours
3. removing the old backups when the new full backup runs, I would prefer to keep 2 weeks of backups, but that's not necessary
4. a GUI would be preferable, since my arthritic fingers don't always do as I want them to do. I typo things a lot, and the label worn off my backspace can attest to that.
Server :: Cron Job That Does The Backups?
Mar 21, 2010I have a crontab related question which I am hoping someone can answer. I recently took over a Redhat Enterprise 5 Server, and I was told by the previous Server Admin that there is a cron job that does the backups. I ran the following command to get a list of all users:
Code:
cat /etc/passwd | grep "/home" |cut -d: -f1
I then ran the following command for each of those users to see if they have any crontabs associated with them:
Code:
crontab -u USER -l
It doesn't show any crontab entries for any users (including root). But I am positive that there is a scheduled job somewhere because the backups are still running every night.
Server :: Use FileSystem For Backups?
Jul 8, 2010With so many filesystems available which one should I use to make backups? All I care about is reliability and stability. I don't care at all about portability.
View 8 Replies View RelatedServer :: Sync File Server Data Into Backup Server Machine By Command- Rsync -avu?
Jun 21, 2011iam trying to sync file server data into backup server machine by command- rsync -avu path/of/data ipaddress-of-backup-server:/path/where/to/save after running it ask for root password and manually it is successful.but i want to make it automatic.for that i also tried cronjob and also generated authentication key but iam not successful in login automatically..anybody know how to authenticate root to login for storing data in backup server.
View 14 Replies View RelatedGeneral :: Use Bacula To Make Backups Of The Files Stored On The Server?
Apr 14, 2010I have an ubuntu (8.04.3) server where I use bacula to make backups of the files stored on the server. Ive been trying to find a solution (with no luck) trying to succesfully implement the following:-
A Backup tape for each day of the week besides Thurs which is resused on a weekly basis. For the thursday tapes we have a backup tape corresponding to the week number that the thursday falls so for the first thursday of the month it would be ThursOne For example. These tapes are resued on a monthly basis. We then have a monthly tape that is used on the last thursday of the month. These tapes will be resused on a yearly basis.
Another requirement is just in case a tape is accidently not changed a backup should still occur regardless of what tape is in the drive (so if its tuesday and mondays tape is still in the tape drive it should rewrite that tape).
I did have this successfully set up where the tape was appended after each use rather than being recycled after the nightly backup. But then after a few weeks I would have to manually purge tapes when they became full (which isnt ideal - as Im not always in the office so in my absence it may be that a backup may not take place), so have been playing around and have now got the tapes to be marked as used after a max of 2 jobs (so the backup of the files and the catalog of the night). I also added this line 'Recycle Current Volume = yes' so that it would hopefully recycle the volume in the drive.
However what I am finding is that the tape that should be recycled is not, but in yesterday case the Mondays tape was recycled rather than the Tuesday although Mondays was the last written so Im not even sure why it choose to recycle this tape.
Server :: Setup House Mostly (for The First Time) For Backups And File Sharing?
Jan 28, 2011**Edit: path for mount was incorrect Distro Server: CentOS 5.5
Clients:
Fedora(latest)
OSX(latest)
Backround I am attempting to setup a server in my house mostly(for the first time) for backups and file sharing. It is important to me that file permissions are preserved. So its my understanding that I must use idmapd in order for this to work. As of now I'm only working with the linux distros while osx will be dealt with once these two work together. portmapper is up and running, along with lockd on both machines. Firewalls are also down on both machines for now. The server side was all setup using the GUI interface with no extra options selected. Problem When attempting to "mount -t nfs4 10.0.0.2/$sharedfolder /mnt" I get an error operation not permitted with no error printing in /var/log/message. If I use "mount -t -o nolock nfs4 10.0.0.2/$sharedfolder /mnt" it mounts just fine. Ive checked both machines multiple times to make sure that lockd is up and running. In the idmapd.conf file I the domain as "localdomain" for both machines but I doubt that is right; like I stated above this is my first attempt at a server. I'm assuming the problem is a whole missing step that involves some kind of id mapping server I need to setup.
Security :: Secure And Automated Backups - Add Public Key To Authorized_hosts File On Prod Server?
Mar 13, 2010I'm trying to find a secure way to backup files on my Prod Server to Backup Server. It must be automated, so I will need to run a command with cron which will login to Prod Server from Backup Server and backup data. 1. Do you think it would be secure enough to do this by creating an passwordless RSA private key on Backup Server and adding it's public key to authorized_hosts file on Prod Server? I can't think of a way to Automate this without having to enter any passwords without passwordless RSA key. Is there another. more secure way? 2. Should I create a special user for backup, which will only have read access to all files in the directory that I am backing up? If so, How can I run a check that this new backup user indeed has read access to ALL files in the folder that I intent to back up? How can I ensure the backup process will not skip files due to some permission problem? 3. I'm thinking of using rsnapshot tool, which uses rsync.
View 10 Replies View Related