Debian :: Rsync Error In Rsnapshot Backups
Apr 5, 2011
I am backing up my debian server with rsnapshot which actually uses rsync to perform the backup. The backups are located in an external storage of size 1.4T .
[code]....
I tried to understand what this error message means and i founde that error code 12 : 12 Error in rsync protocol data stream I understand that when rsync find that a file on the target was changed , it will send only the block/blocks that contain the changes and in the destination rsync will create new file and not update the old one (new inod...) . I want to know if this error i get is due to full disk or perhaps it is some other factor
View 2 Replies
ADVERTISEMENT
Nov 12, 2009
How do you get Rsync to do incremental backups rather than full backups? At the moment I have a script that will create a backup folder (if it doesnt already exist) then copy the source files into the backup directory with the command
rsync $VERBOSE --exclude=$TARGET/ $EXCLUDE --exclude '/Ls-wtgl1c8/**' -rt --delete $source/ $TARGET/$source/ >> $LOG_FILE
Target is where the files will be backed up to Sources is the dir(s) to be backed up Exclude files is the list of files not to backup
log file is where the output will be saved to. At the moment it only does full backups, but I would only like to do incremental, how would this be achieved? Am I missing out an option in the Rsync that is required.
View 9 Replies
View Related
May 9, 2010
I am currently backing up my data but find that it takes way to long to do a rsync, it takes forever to just find the differences and transfer them.Out of 3 separate rsyncs the main one that is slow is my www.skins.be mirror directory which is 41GB and has 392,200 files, sorted into multiple directories. Which grows by around 100 every couple days.I think that something that would be able to track changes by inotify time on directories will speed it up since Picasa sure finds the changes fast when I open it and it is tracking over 26,200 pictures. I just don't know of a backup solution that does that.
View 4 Replies
View Related
Jan 21, 2010
I am using rsync to backup dirs on my ubuntu server onto a NAS (which is mounted onto the filesystem), but the problem is that it is constantly doing full backups rather than doing incrementals and I am not really sure why. After doing a bit of expermienting with the script I noticed that if I just backed up a home dir (/home/user) the incremental backups work fine. If however I was to back up a dir like (/home/domain/user) it always does full backups.I have tried various different scripts but still the same end result. The latest script is a variation on the a script found on the samba rsync examples webpage, see below...
#!/bin/bash
# rsyncbu.sh -- backup to nas using rsync
# This script backups files listed in BDIR to the BSERVER. The verbose output along with the date is listed in the LOG_FILE specified
# verbose output
[code]....
View 4 Replies
View Related
Apr 27, 2011
There are multiple servers to be backed up. Different access rights exist in each server. There are two backup servers with plenty of disk space, one local, and one offsite. The local one feeds to the offsite one. The rsync command is being used to make a replica of backed up data. Deleted data is also being archived. There are two methods that have been considered: One is to have the individual servers run rsync which logs in to the backup server to push data. Two is to have the backup server run rsync which logs in to each individual server to pull data. Because system data is involved and meta information (like owning user) must be stored, root is required to access the data as well as to store it. That means everything runs as root both ends. So method one was quickly dismissed because each server would effectively have rights to access ALL the data on the backup server since it logs into the backup server as root. The security containment here involves different groups using different servers, and they need to be isolated from each other.
But even method two involves some risks that are a concern. This means one machine has access rights to every server. If the backup server were compromised, every machine could be compromised.What I'd like to find is some way to allow backups to be run without either machine granting root access to the other, while still running as root, or something equivalent, that allows accessing all data and storing all metadata. So I was looking at setting up an rsync daemon on each individual server (running as root so it can access what it is specified to access), and running an rsync client on the backup server (as root so it can store metadata). This opens network access issues. Any user on the network can connect to the rsync daemon. So password protection is needed. But this communication is also not encrypted, which exposes the password and the data should the network be sniffed.
So now I'm thinking about a non-root ssh login between machines. The backup server would login to a non-privileged user on each individual server and set up a secure forwarding channel to the rsync daemon. Is this the best that can be done? Is there a way to run rsync via SSL with key verification so it can all be done together? I'd like to have the rsync daemons configured to always talk SSL, and always verify the client's key against a list of authorized keys, and likewise the client verify the server's key against the known public key for that server.
View 14 Replies
View Related
Jan 29, 2010
Can I use rsync for incremental backups of the running linux server?
View 5 Replies
View Related
Sep 30, 2010
I need to login as root, or at least get root privileges, in a cron triggered backup run. The straight way to do this would be the backup server making an ssh connection to the server to be backed up (this way because I want to avoid many servers being backed up in parallel and the backup server itself would be managing this diversity), via the rsync command which would be performing the backup's synchronization step.
I'm looking for alternatives to this in some form. I'd like to disallow direct root login to my ssh port (not 22One idea I have is to have the backup server initiate an ssh login as a non-root user, to either the actual source server, or to a server that can reach the source server ... and set up port forwarding. Over the forwarded port, then initiate the rsync that logs in as root via another port that allows direct root, but cannot be reached from the internet at all (because the border firewall doesn't include this port as allowed in).FYI, these logins will be using ssh keys, not passwords. I do need to keep ownership metadata for files being backed up, so this is why I am using root. Also, rsync is needed to get the incremental updates to keep bandwidth usage lower (otherwise I could just transfer a tarball each day).Anyone have any other ideas or comments, for security issues, based on experience doing things like this (backups, routine data replication, etc)?
View 5 Replies
View Related
Jul 13, 2011
I am using rsync for incremental backups. I am backing up to a second hard drive on my computer. When I check the individual backup directories (backup.0 through backup.4) with du -hs they each show 12G; when I check the parent directory squeeze it shows 15G. Over 4 backups I have added 3G. I haven't made very much for changes to directories I'm backing up and am using hard links. I have included some info below.
Quote:
Backup script:
#!/bin/bash
mount /mnt/backup
cd /mnt/backup/squeeze/
rm -rf backup.7
[code]....
View 2 Replies
View Related
Dec 2, 2010
With the --backup and --backup-dir= options on rsync, I can tell it another tree where to put files that are deleted or replaced. I'm hoping it fills out the tree with a replica of the original directory paths (at least for the files put there) or else it's a show stopper. What I'm wanting to find out applies when I'm restoring files. Assuming each time I run rsync (once a day) I make a new directory tree (named by the date) for the backup directory. For each file name/path in the tree, I would start with whatever is in the main tree (the rsync target) and work through the incremental trees going backwards until I reach the date of interest to restore to. If along the way I encounter a file in an incremental, I would replace the previous file at that path with this next one. So by the time I get back to a given date, I should have the version of the file which was present at that date. Do this for each file in the tree and it should be a full restore.
But ... and this is the hard part, it seems. What about files that did not exist at the intended restore date, but do exist (were created) on a date after the intended restore date. What I'd want for a correct restore would be for such files to be absent in the restored tree (just as they were absent in the source tree on that date). How can such a restore be done to correctly exclude these files? Wouldn't rsync have to store some kind of sentinel that indicates that on dates prior, the file did not exist. I suspect someone might suggest I just make a complete hard linked replica tree for each date, and this way absent files will clearly be absent. I can assure you this is completely impractical because I have actually done this before. I ended up with backup filesystems that have so many directories and nodes that it could take over a day, maybe even days, to just do something like "du -s" on it. I'm intending to keep daily changes for at least a couple years, if not more. So that means the 40 million plus files would be multiplied by over 700, making programs like "du -s" have to check over 28 BILLION file names (and that's assuming the number of files does not grow over the next two years).
View 2 Replies
View Related
May 9, 2011
So I am using rsync (3.0.7 on MAC OSX) to backup one hard drive to a folder on another one. The is USB drive to USB drive and I have done the initial backup from one drive to a new formatted other drive with the following command:
Code:
rsync -avX --progress /Volumes/Source /Volumes/Destination
This all appears to be going smoothly as I type. I am going to write a script to do subsequent backups in the
[code]....
View 2 Replies
View Related
Oct 6, 2010
I'm trying to set up rsync backups on my ReadyNAS and I'm getting the following error: ERROR: The remote path must start with a module name not a / This error is accompanied by the following information:
[Code]...
View 1 Replies
View Related
Mar 20, 2010
I've been using dump/restore for backups, for quite some time. It's worked fine, but the process of recovering from a HD failure takes too long. What with eSATA and external drive docks, what I'd really like is to use rsync to maintain a current clone of my entire system drive. That is, start with a full disk clone, and then use rsync to keep it current.
I've seen plenty of instructions on how to do this with a directory tree, but I've seen none for doing it with a copy of the entire disk. If, for example, I copy /etc/fdisk, then the copied disk would have entries with the same UUIDs as the original disk. Which would mean that if the clone disk were to be bootable, its partitions would need the same UUIDs as the original disk. Which they would be, if the cloned disk started as a full-disk clone, I think. Am I wrong? But that means that when the clone disk was active, I'd have partitions with duplicated UUIDs. Is this going to cause problems? When I boot, will I get the correct partitions loaded?
View 4 Replies
View Related
May 16, 2011
I am in the process of writing an rsync script to run unattended backups of my entire file system to another system located on my local network using ssh and password-less rsa keys.
I will absolutely will not use password-less keys with the root account and this is the limitation preventing me from accomplishing my goal because root is required by rsync to access the / tree and copy it to another location. I decided that if I compiled the script into a binary that I didn't have a problem with the password being contained within the binary itself but from what I've read there is no way to elevate to root and then back down to user level from within the script/binary.
I can create the script as the user and use chroot to make it owned by root but retain execution permission for the user but it will still cause the ssh login to be under root and therefore require either that I am there to enter my password or the use of password-less keys under the root account which I reiterate I will NOT do. Currently the script is executed by the user on the machine containing the files to be backed up.
View 9 Replies
View Related
Jan 31, 2010
How to backup incremental running debian server using rsnapshot?
View 2 Replies
View Related
May 26, 2011
I've been using grsync to backup my home directory to an external drive (connected via USB), but lately I've been getting errors very early in the process and I was hoping someone might be able to help me fix them (or at least understand what I'm doing wrong).
Here's the output of a window titled "rsync: stopped" and in red "completed with errors!"
rsync: write failed on "/media/LaCie/myhomefolder/.cache/champlain/osm-mapnik/6/12/23.png": Input/output error (5)
kyconway/.cache/champlain/osm-mapnik/6/12/26.png
rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.8]
rsync: connection unexpectedly closed (486 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.8]
Rsync process exit status: 12
View 4 Replies
View Related
Feb 3, 2011
I've been a DOS/Windows guy for 20 years, and recently became a SW test lab helper. My company uses CentOS for a lot, so I've become familiar with it, but obviously not as comfortable as I am with Windows.
Here's what I have planned:
machine: Core 2 Duo E8400, 8GB DDR2, 60GB SSD OS drive, ATI 4650 video card, other storage is flexible (I have 3 1TB drives and 4 750GB drives around that can be used in this machine.)
uses: HTPC, Network Storage, VMWare server host: SMTP, FTP server, and Web server virtual machines
I've figured out how to do much of this, but I haven't figured out how to do backups in Linux. I've been spoiled with Windows, with the built in backup system so simple to use. I find myself overwhelmed with the array of backup software, and unable to determine which to use. none of them seem to do everything I need them to do, but some come close, I think. I'm hoping someone here can help me out in figuring out which program to use and how to use it.
Here is what I need the backup software to do:
1. scheduled unattended backups, with alerts if the backups fail
2. a weekly full backup with incremental every 12 hours
3. removing the old backups when the new full backup runs, I would prefer to keep 2 weeks of backups, but that's not necessary
4. a GUI would be preferable, since my arthritic fingers don't always do as I want them to do. I typo things a lot, and the label worn off my backspace can attest to that.
View 7 Replies
View Related
Aug 21, 2009
I am trying to use rsync to keep 2 machines in sync. I get the following error sometimes. If I rerun the rsync it usually works.
IO error encountered - skipping file deletion
View 1 Replies
View Related
Sep 19, 2010
I'm trying to sync a folder between my Windows install and my PCLinux install. Same machine, Windows 7 on the C drive and Linux on an external USB hard drive.
Using the Grsync GUI, I searched for the source and destination folders, and then executed the command. Here is the output:
** Launching RSYNC command (simulation mode): pkexec rsync -r -n -t -v --progress -s /media/Win7/Documents and Settings/Paul/Application Data/The Word file:///root/.wine/drive_c/Program Files/The Word/myData
ssh: connect to host file port 22: No route to host
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
Rsync process exit status: 255
It seems to be doing an FTP operation, whereas all I need is a file transfer. Can anyone point out my error here?
View 5 Replies
View Related
Apr 6, 2010
For a project I have had to migrate from FreeBSD to Linux and I have decided to choose Debian because I have had good experiences with it.However, since my main development machines are completely offline I have run into a little bit of trouble.Is there a way to specify a .deb package and get a tool to recursively list (and fetch) all dependency packages that are not included in the base install and put them in a folder?
I do not like to be tied to the internet (Never a good idea) and so this would save me a heap of trouble (and journeys to an online PC)Once I have these folders containing packages, then I can simple cd into one of them, dpkg -i the relevent .deb and not have to rely on the DVDs or worry about connecting to an online repositoryLife will be good!Suffice to say FreeBSD can do this well so I strongly believe that Debian will be able to aswell but since it is not normally done, I havn't found much on google about it
View 8 Replies
View Related
Nov 17, 2010
I'm currently backupping our home data (pictures, videos, our CDs ripped to FLAC which I spent a lot of time to tag accurately ), totalling almost 300 Gb, on 2 external USB drives, one of which is meant to stay at a friend's. I left the factory msdos filesystem as it was, thinking it could be useful to be able to connect the drives to a windows machine with no problems. It's certainly useful to have «normal» data that I can take with me e.g. when visiting my family.
I'm simply using rsync manually, checking for suspicious changed or deleted file before commiting the change. I do that every 2 weeks or so.
Now I want to add a file integrity management to my backupping scheme: I want to be able to check that new data I'll be committing has not been tampered with (integrity check before updating tags on my main drive), and I want to be able to check that backupped data is still sane on my USB drives, especially if I need to recover from data corruption on my main drive.
Since I'm essentially mirroring the data, I thought run of the mill integrity software would let me just rsync the integrity database, and I'm done.
But after browsing through the docs of tripware, afick and the like, I fear they work only with absolute paths, so the database for my main drive wouldn't work for my USB drive, that's mounted elsewhere when I plug it in, obviously.
So, I feel I'm missing something. It looks to me I'm trying to solve a very common problem, how do people do it?
Did I miss a file integrity software that works with backups?
Is there a trick like using a symbolic link pointing to whatever file hierarchy I want to check, and have tripware/afick/... monitor that link?
Should I run a more elaborate backupping system than plain rsync? Which one? (Storebackup for instance looks promising since it involves md5 sums, but it's targetting a completely different problem, and I'm not sure I can use it at all for what I need.)
View 3 Replies
View Related
Apr 29, 2010
Right, just a quick question about rsnapshot over sshfs and encfs. I've set up an encfs filesystem, and when mounted on the remote machine remotely:
Code:
touch foo.bar
Code:
cp -al foo.bar foo.car
Works as one would expect it to.
The same is true on the local machine (The EncFS has External IV chaining disabled). However, when the remote dir is sshfs mounted on my computer here, and then encfs'd to a decrypt mount on my computer, I can move files to it, and they go over the network and get encrypted, however:
Code:
cp -al <file> <file>
No longer works, I get 'not implemented' errors...
I thought since I don't have External IV chaining this shouldn't be an issue - I've tried without any of the file chaining options, again to no effect. All work remotely, or with both locally, but not over sshfs. Is this a quirk of sshfs?
View 1 Replies
View Related
Jun 20, 2010
I just did my first rsnapshot backup of my /home/ to an external harddisk. When I am not at my computer for a couple of hours, I always shut it down. Therefore, there are no predictable hours of the day where I know that my computer is running. So, how should I schedule/crontab my rotating rsnapshot backups?
Is anyone using rsnapshot in combination with a schedule which is not based on exact times but rather on the time the computer is running?
View 2 Replies
View Related
Dec 9, 2010
I have a server with a postgres database, apache and a custom java application.
I am trying to run rsnapshot to backup /home /etc and /var folders.
But I am running into issues with rsnapshot and permissions. More specifically these kind of errors,
Code:
I look at the permissions on these files with ls -la, I get
Code:
The owner of the files is root and postgres users. I am using passwordless login to connect to server as user XYZ. XYZ has root access to the server and to the database.
These files are all over the place. Some in /etc and some in /var/lib for instance. How can I best copy these remaining files.
View 1 Replies
View Related
Apr 24, 2010
I have installed rsnapshot from slackbuilds.How to run it to backup home directory?I'm also reading the official docs.
View 3 Replies
View Related
Nov 21, 2010
I am trying to run rsnapshot from cron via root's crontab file (crontab -e). If I run rsnapshot from the command line with sudo it works perfectly, however, if I run it from cron:
Code:
* * * * * /usr/bin/rsnapshot hourly >/tmp/crontab.out 2>/tmp/crontab.err
This does not work. The crontab.err file shows:
[Code]....
View 2 Replies
View Related
May 25, 2010
I have a machine on my network and that machine is a mass storage server that I will eventually use as a media server (to stream movies, videoclips and music on my home theater system). I use slackware 13 on ALL of my machines.
I am trying to automate the backup of the "/home" folder of my laptop onto the mass storage server. I currently use rsnapshot and it works great, but I would like to automate the whole process, even if I am not home or in front of my machines...
Here's what I imagined (in pseudo code):
1) Poll if server is active (up);
1.1) If not:
1.1.1) Wake up the server (WOL);
1.1.2) Wait for the server to boot;
1.1.3) Confirm the server has made it to the login prompt (normal boot);
1.1.3.1) If not, send an alarm via email;
[Code].....
View 14 Replies
View Related
Mar 31, 2010
I attempted today to Rsync data from 'serverold' to 'servernew' and also delete data on 'servernew' that has been deleted / removed from 'serverold'. When I run the Rsync command I get the following error:
[Code]...
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1042) [sender=3.0.7] why this is not working? Is my command syntax wrong or missing switches? I read the 'Man' page and gathered this would be the correct command but I could be wrong.
View 1 Replies
View Related
Aug 28, 2010
Rsnapshot is a software written in Perl to make backup of local and remote file system. The well proven rsync is behind this utility. rsnapshot does not need root user intervention to restore the data of a normal user. It does not take much space in your Backup server. It can be easily automated (scheduled) to make life easier. Just setup once and forget it configuration. Basically it takes snapshot of file system (or a part of) in regular interval such as hourly, daily, weekly and monthly.
This can be configured easily through a simple text based configuration file. The above task can be setup in a few easy steps in a few minutes. Two major tasks are configuring rsnapshot and openssh automatic login. To make the backup automatically, we need to automate the remote login in a secured way. This can be done through openssh tools. This scenario depicts backup of desktop (assuming that IP address is 192.168.0.100) data to a backup server. My desktop runs on Ubuntu 10.04 and backup server runs on Debian Squeeze. [URL]
View 2 Replies
View Related
Jan 29, 2011
Our backup script was working fine (ssh to the server, back up /home to a second hard drive on my computer). Then right after an ubuntu update, it quit working. I investigated and found that "something" had changed the label on the backup hdd to what looked like gibberish to me. But the script identified the backup hdd by its uuid, which didn't change. Yet, here is the error I get when the backup fails: receiving file list ... done [took about 5 seconds] rsync: mkdir "/media/14D9-3B1F/server-backup" failed: No such file or directory (2) rsync error: error in file IO (code 11) at main.c(594) [receiver=3.0.6]
Note that the backup hdd IS mounted, uuid is correct, and the folder 'server-backup' DOES exist. Does anyone have a clue for me? I'm moderately experienced in Linux and ubuntu. Our server runs centos 5. And as stated, the backup ran fine for several weeks. I think there was a new linux kernel on that update, but at this point a while later I don't know which one. Current kernel is .2.6.31-22-generic.
View 1 Replies
View Related
Sep 18, 2010
I back up with rsync like this:
rsync -av --delete /mnt/hda1/ /mnt/hdb1/
hda1 had an error (bad superblock, i think), but rsync didn't exit! It just went on on deleting my files until i hit Ctrl+C. Now I can't access the original drive and the back-up drive has lost a lot of files. Why doesn't rsync exit when it encounters and error? How can I stop this behavior?
View 2 Replies
View Related
