Ubuntu :: Rsync And Full-disk Backups - Failure Takes Too Long
Mar 20, 2010
I've been using dump/restore for backups, for quite some time. It's worked fine, but the process of recovering from a HD failure takes too long. What with eSATA and external drive docks, what I'd really like is to use rsync to maintain a current clone of my entire system drive. That is, start with a full disk clone, and then use rsync to keep it current.
I've seen plenty of instructions on how to do this with a directory tree, but I've seen none for doing it with a copy of the entire disk. If, for example, I copy /etc/fdisk, then the copied disk would have entries with the same UUIDs as the original disk. Which would mean that if the clone disk were to be bootable, its partitions would need the same UUIDs as the original disk. Which they would be, if the cloned disk started as a full-disk clone, I think. Am I wrong? But that means that when the clone disk was active, I'd have partitions with duplicated UUIDs. Is this going to cause problems? When I boot, will I get the correct partitions loaded?
View 4 Replies
ADVERTISEMENT
Nov 12, 2009
How do you get Rsync to do incremental backups rather than full backups? At the moment I have a script that will create a backup folder (if it doesnt already exist) then copy the source files into the backup directory with the command
rsync $VERBOSE --exclude=$TARGET/ $EXCLUDE --exclude '/Ls-wtgl1c8/**' -rt --delete $source/ $TARGET/$source/ >> $LOG_FILE
Target is where the files will be backed up to Sources is the dir(s) to be backed up Exclude files is the list of files not to backup
log file is where the output will be saved to. At the moment it only does full backups, but I would only like to do incremental, how would this be achieved? Am I missing out an option in the Rsync that is required.
View 9 Replies
View Related
Oct 13, 2009
I am running Centos 5.3. I ran no updates, performed no installs, nor changed any configuration immediately prior to this issue. My problem is this: when I run the command startx (default runlevel 3), it is a long time (5-10 minutes) before Gnome startx, and once it does start applications will not run. Also, when I try to use sudo (from any environment, even ssh), it is a long time (5-10) before the command is executed.
I cannot say for sure, but it seems like this is an intermittent problem. Sometimes X takes a long time to start, but once it starts it will launch programs. Sometimes X takes a long time to launch, but once it starts it will only launch certain programs. Though presently X always takes a long time to start, and I cannot successfully launch any programs.
A while back a had a similar problem to this (x taking long time to start, sudo taking long time to execute) and it ended up being a DNS problem. Unfortunately, I cannot remember exactly what it was and I stupidly did not document it. Maybe this is also DNS related, I don't know.
I don't know what log files to look at for problems with X, Gnome, and sudo taking a long time to start.
View 19 Replies
View Related
Apr 12, 2011
I have a tiny shell script to rsync files between two servers and remove the source files.
This script works fine, when it has been initiated manually or even when the rsync command is executed on the command line.
But the same script doesn't work, when I try to automate it through crontab.
I am using 'abc' user to execute this rsync, instead of root, as root login to servers are restricted in all of our servers, by us.
As I mentioned earlier, manual execution works like charm!
When this rsync.sh is initiated through crontab, it runs the first command(chown abc.abc ...) perfectly without any issues. But the second line is not at all executed, and there is no log entry i can find at /mnt/xyz/folder/rsync.log.
View 6 Replies
View Related
May 9, 2010
I am currently backing up my data but find that it takes way to long to do a rsync, it takes forever to just find the differences and transfer them.Out of 3 separate rsyncs the main one that is slow is my www.skins.be mirror directory which is 41GB and has 392,200 files, sorted into multiple directories. Which grows by around 100 every couple days.I think that something that would be able to track changes by inotify time on directories will speed it up since Picasa sure finds the changes fast when I open it and it is tracking over 26,200 pictures. I just don't know of a backup solution that does that.
View 4 Replies
View Related
Jan 21, 2010
I am using rsync to backup dirs on my ubuntu server onto a NAS (which is mounted onto the filesystem), but the problem is that it is constantly doing full backups rather than doing incrementals and I am not really sure why. After doing a bit of expermienting with the script I noticed that if I just backed up a home dir (/home/user) the incremental backups work fine. If however I was to back up a dir like (/home/domain/user) it always does full backups.I have tried various different scripts but still the same end result. The latest script is a variation on the a script found on the samba rsync examples webpage, see below...
#!/bin/bash
# rsyncbu.sh -- backup to nas using rsync
# This script backups files listed in BDIR to the BSERVER. The verbose output along with the date is listed in the LOG_FILE specified
# verbose output
[code]....
View 4 Replies
View Related
Apr 27, 2011
There are multiple servers to be backed up. Different access rights exist in each server. There are two backup servers with plenty of disk space, one local, and one offsite. The local one feeds to the offsite one. The rsync command is being used to make a replica of backed up data. Deleted data is also being archived. There are two methods that have been considered: One is to have the individual servers run rsync which logs in to the backup server to push data. Two is to have the backup server run rsync which logs in to each individual server to pull data. Because system data is involved and meta information (like owning user) must be stored, root is required to access the data as well as to store it. That means everything runs as root both ends. So method one was quickly dismissed because each server would effectively have rights to access ALL the data on the backup server since it logs into the backup server as root. The security containment here involves different groups using different servers, and they need to be isolated from each other.
But even method two involves some risks that are a concern. This means one machine has access rights to every server. If the backup server were compromised, every machine could be compromised.What I'd like to find is some way to allow backups to be run without either machine granting root access to the other, while still running as root, or something equivalent, that allows accessing all data and storing all metadata. So I was looking at setting up an rsync daemon on each individual server (running as root so it can access what it is specified to access), and running an rsync client on the backup server (as root so it can store metadata). This opens network access issues. Any user on the network can connect to the rsync daemon. So password protection is needed. But this communication is also not encrypted, which exposes the password and the data should the network be sniffed.
So now I'm thinking about a non-root ssh login between machines. The backup server would login to a non-privileged user on each individual server and set up a secure forwarding channel to the rsync daemon. Is this the best that can be done? Is there a way to run rsync via SSL with key verification so it can all be done together? I'd like to have the rsync daemons configured to always talk SSL, and always verify the client's key against a list of authorized keys, and likewise the client verify the server's key against the known public key for that server.
View 14 Replies
View Related
Jan 29, 2010
Can I use rsync for incremental backups of the running linux server?
View 5 Replies
View Related
May 9, 2011
So I am using rsync (3.0.7 on MAC OSX) to backup one hard drive to a folder on another one. The is USB drive to USB drive and I have done the initial backup from one drive to a new formatted other drive with the following command:
Code:
rsync -avX --progress /Volumes/Source /Volumes/Destination
This all appears to be going smoothly as I type. I am going to write a script to do subsequent backups in the
[code]....
View 2 Replies
View Related
Apr 5, 2011
I am backing up my debian server with rsnapshot which actually uses rsync to perform the backup. The backups are located in an external storage of size 1.4T .
[code]....
I tried to understand what this error message means and i founde that error code 12 : 12 Error in rsync protocol data stream I understand that when rsync find that a file on the target was changed , it will send only the block/blocks that contain the changes and in the destination rsync will create new file and not update the old one (new inod...) . I want to know if this error i get is due to full disk or perhaps it is some other factor
View 2 Replies
View Related
Sep 30, 2010
I need to login as root, or at least get root privileges, in a cron triggered backup run. The straight way to do this would be the backup server making an ssh connection to the server to be backed up (this way because I want to avoid many servers being backed up in parallel and the backup server itself would be managing this diversity), via the rsync command which would be performing the backup's synchronization step.
I'm looking for alternatives to this in some form. I'd like to disallow direct root login to my ssh port (not 22One idea I have is to have the backup server initiate an ssh login as a non-root user, to either the actual source server, or to a server that can reach the source server ... and set up port forwarding. Over the forwarded port, then initiate the rsync that logs in as root via another port that allows direct root, but cannot be reached from the internet at all (because the border firewall doesn't include this port as allowed in).FYI, these logins will be using ssh keys, not passwords. I do need to keep ownership metadata for files being backed up, so this is why I am using root. Also, rsync is needed to get the incremental updates to keep bandwidth usage lower (otherwise I could just transfer a tarball each day).Anyone have any other ideas or comments, for security issues, based on experience doing things like this (backups, routine data replication, etc)?
View 5 Replies
View Related
Jul 13, 2011
I am using rsync for incremental backups. I am backing up to a second hard drive on my computer. When I check the individual backup directories (backup.0 through backup.4) with du -hs they each show 12G; when I check the parent directory squeeze it shows 15G. Over 4 backups I have added 3G. I haven't made very much for changes to directories I'm backing up and am using hard links. I have included some info below.
Quote:
Backup script:
#!/bin/bash
mount /mnt/backup
cd /mnt/backup/squeeze/
rm -rf backup.7
[code]....
View 2 Replies
View Related
Dec 2, 2010
With the --backup and --backup-dir= options on rsync, I can tell it another tree where to put files that are deleted or replaced. I'm hoping it fills out the tree with a replica of the original directory paths (at least for the files put there) or else it's a show stopper. What I'm wanting to find out applies when I'm restoring files. Assuming each time I run rsync (once a day) I make a new directory tree (named by the date) for the backup directory. For each file name/path in the tree, I would start with whatever is in the main tree (the rsync target) and work through the incremental trees going backwards until I reach the date of interest to restore to. If along the way I encounter a file in an incremental, I would replace the previous file at that path with this next one. So by the time I get back to a given date, I should have the version of the file which was present at that date. Do this for each file in the tree and it should be a full restore.
But ... and this is the hard part, it seems. What about files that did not exist at the intended restore date, but do exist (were created) on a date after the intended restore date. What I'd want for a correct restore would be for such files to be absent in the restored tree (just as they were absent in the source tree on that date). How can such a restore be done to correctly exclude these files? Wouldn't rsync have to store some kind of sentinel that indicates that on dates prior, the file did not exist. I suspect someone might suggest I just make a complete hard linked replica tree for each date, and this way absent files will clearly be absent. I can assure you this is completely impractical because I have actually done this before. I ended up with backup filesystems that have so many directories and nodes that it could take over a day, maybe even days, to just do something like "du -s" on it. I'm intending to keep daily changes for at least a couple years, if not more. So that means the 40 million plus files would be multiplied by over 700, making programs like "du -s" have to check over 28 BILLION file names (and that's assuming the number of files does not grow over the next two years).
View 2 Replies
View Related
Jan 16, 2010
I am having a problem with a slow wake up from the hibernate in karmic with gnome.My desktop has 2GB of ram and 4GB of swap.m doing standard hibernate from the menu.Hibernation takes 44s, which I can live with for now. But wake up measured from grub boot menu to unlock dialog takes 2m 47s. I think it's reasonable to expect it to take less than a minute.
View 1 Replies
View Related
Jun 12, 2010
I am running Ubuntu 10.04 Lucid, and it works perfectly.However, since last week (probably after some update), the gnome menu bar takes much longer to appear when I login. After it appears, it works just fine.To clarify, I turn on the system, login, the desktop background image and icons appear, and only after about 30 seconds the upper and lower gnome menu bars appear (a week ago all of it loaded almost instantly).
View 1 Replies
View Related
Jul 26, 2010
After I installed a new hard drive, when I booted up into Ubuntu, it would give me this error: "failed command: WRITE DMA". So I tried the workarounds and I guess it just covered the log with the Boot Splash, now it's taking a long time just to boot up.
View 9 Replies
View Related
Nov 15, 2010
I recently changed my computer so it would auto boot into Ubuntu, but if I hold down the shift key, I can bring up the GRUB menu so I can choose Windows in the rare times I use it[URL]Anyway, I noticed that when I did that, it actually took longer to boot. I timed it today and it took 45 seconds from the time I pressed the power button to get to the login screen! My friend uses Linux a lot and says even with Ubuntu (which he thinks is "too slow") it shouldn't take that long.
View 1 Replies
View Related
May 16, 2011
So as the title says shutdown time takes unusually long. Upwards of 4-6 minutes. I'm used to my linux systems taking about 10 seconds to shutdown, 20 tops. In fact this problem seemed to stem from the natty release because I didn't have this issue with the betas or 10.10....
It seems the longer I use my laptop the longer it takes to shutdown. When I do a quick task like a file backup the shutdown takes 10seconds or so, but if I open up a browser and start surfing for a few hours it takes more like 5 minutes.
View 6 Replies
View Related
Oct 6, 2010
I'm trying to set up rsync backups on my ReadyNAS and I'm getting the following error: ERROR: The remote path must start with a module name not a / This error is accompanied by the following information:
[Code]...
View 1 Replies
View Related
May 16, 2011
I am in the process of writing an rsync script to run unattended backups of my entire file system to another system located on my local network using ssh and password-less rsa keys.
I will absolutely will not use password-less keys with the root account and this is the limitation preventing me from accomplishing my goal because root is required by rsync to access the / tree and copy it to another location. I decided that if I compiled the script into a binary that I didn't have a problem with the password being contained within the binary itself but from what I've read there is no way to elevate to root and then back down to user level from within the script/binary.
I can create the script as the user and use chroot to make it owned by root but retain execution permission for the user but it will still cause the ssh login to be under root and therefore require either that I am there to enter my password or the use of password-less keys under the root account which I reiterate I will NOT do. Currently the script is executed by the user on the machine containing the files to be backed up.
View 9 Replies
View Related
Apr 19, 2011
Got some speed problems with my backup script, need to save 250GB data (28 network-shares (20 user homefolders included) - each one gets an own zip; approx. 100'000 files)Backupdata are stored on our nas that is embedded with mount -t cifs -o user,pw //networkadress /yet/another/backup/folderPacking process takes about 60! hours (on an intel xeon 3.0 GHz RHEL4 system) - connected via 100mbit networkzip -r /yet/another/backup/folder/asdf.zip /home/asdf/Is there a way to speed up this whole process? Saving via tar.gz instead of ziping?
View 2 Replies
View Related
May 22, 2010
I installed bootchart and uploaded an image of my latest boot on imageshack: [URL] but I do not know really how to interpret it. Bare in mind that I did not use to have this problem while running Karmic...back then the OS started twice as faster as it does now after the update (now it takes at least 75 seconds to boot) .
View 9 Replies
View Related
Jul 25, 2010
I have an Acer Aspire 1810TZ laptop with Windows 7 on it. I decided to dual boot w/ Ubuntu 10.04 32-bit Lucid Lynx. I was initially getting this issue
Code:
gave up waiting for root device. Common problems:
-Boot args (cat /proc/cmdline)
[code]....
View 1 Replies
View Related
Aug 23, 2010
I have just got some new sheet music; and I was wondering why it takes so long to print from Document viewer. I have a lexmark optra e312 which is connected to a windows network, and other programs seem to work just fine, but this one prints slowly whenever I:connect it to my computer directlyreinstall the printerreinstall ubuntu, even to other variantsMaybe someone could suggest another document viewer? I need to view .pdf, .ps, and .djvu as well as a few others if possible
View 4 Replies
View Related
Jan 25, 2011
It's been a while since this problem started. I have an Acer Aspire 4720z laptop with Ubuntu 10.10 installed. My laptop takes a whole damn 1.5 minutes to boot up and login (measured according to bootchart; I have auto-login enabled) (The majority of this 1.5 minutes is taken up after boot up, so it might indicate a problem with Xorg.)I don't know whether this is relevant, but when I boot up, a message gets displayed: "ata4.01: failed to resume link (SControl 0)". Also, this problem started right around the time I upgraded from Lucid to Maverick, so it could be some problem with my upgrade.find the source of this issue.ATTACHED: bootchart image from last login.boot.log:
Code:
fsck from util-linux-ng 2.17.2
udevd[370]: can not read '/etc/udev/rules.d/z80_user.rules'
[code]....
View 5 Replies
View Related
Apr 3, 2011
I did used the Alien-Arena yesterday as usual and did work fine from start to finish! When I launch it today, it is taking about 4 minutes to load the main screen. I did a reinstall and with no success! Completely removed it with synaptic and reinstall... but the problem still remains! Something went wrong and I can't figure out what
View 1 Replies
View Related
May 11, 2011
After upgrading to a 64-bit Ubuntu 11.04, I have found that it takes a lot longer to open any browsers such as Firefox, Google Chrome or Opera. Does anyone know why? I thought 64 bit Ubuntu should be faster than the 32 bit version. Nevertheless, I see no difference besides the browser problem mentioned above.
View 9 Replies
View Related
Nov 12, 2010
I booted fedora from an live cd not i have a problem when i want to install fedora formatting is taking vey long its already busy for 1 hour and this is te second time im trying to install fedora does it need to take this long ? I've got an unexpected fault for the second time and pressed debug but nothing is happening keeps formatting.
View 3 Replies
View Related
Feb 23, 2010
I'm having trouble with Vim in any terminal emulator I use. I have a link (vi) to vim. Occasionally it will take very long to load, whether I use 'vi' or 'vi file'. Before, if I could I would restart X, and then it would load instantly again, but I waited this time and it did load, after a minute or so. Is this a problem with X or vim?
View 9 Replies
View Related
May 6, 2010
I have a newly installed Kubuntu 10.04 running here, works fine except for one thing.
I have a kind of "fileserver" and it has a samba share that I have mounted in the home folder of my desktop computer ("/home/xxx/fileserver", the server is running an older version of Ubuntu, can't exactly remember what it is but the filesystem is ext2, if that's of any importance).
I have large files on the server, mostly video. When I use Dolphin (or Konqueror, doesn't make any difference) and right click one of these large files and choose Properties, it takes a LONG time to load the properties window. As if it copies the file to local hd before opening properties, or something.
The reason why I posted here and not in the networking section is, that I had the exact same setup with my previous installation which was Kubuntu 8.04, and also at least three different Ubuntu's before that. Never had this problem before, so I think my server and networking thingies are okay.
View 1 Replies
View Related
