I'm trying to set up rsync backups on my ReadyNAS and I'm getting the following error: ERROR: The remote path must start with a module name not a / This error is accompanied by the following information:
[Code]...
How do you get Rsync to do incremental backups rather than full backups? At the moment I have a script that will create a backup folder (if it doesnt already exist) then copy the source files into the backup directory with the command
rsync $VERBOSE --exclude=$TARGET/ $EXCLUDE --exclude '/Ls-wtgl1c8/**' -rt --delete $source/ $TARGET/$source/ >> $LOG_FILE
Target is where the files will be backed up to Sources is the dir(s) to be backed up Exclude files is the list of files not to backup
log file is where the output will be saved to. At the moment it only does full backups, but I would only like to do incremental, how would this be achieved? Am I missing out an option in the Rsync that is required.
I am currently backing up my data but find that it takes way to long to do a rsync, it takes forever to just find the differences and transfer them.Out of 3 separate rsyncs the main one that is slow is my www.skins.be mirror directory which is 41GB and has 392,200 files, sorted into multiple directories. Which grows by around 100 every couple days.I think that something that would be able to track changes by inotify time on directories will speed it up since Picasa sure finds the changes fast when I open it and it is tracking over 26,200 pictures. I just don't know of a backup solution that does that.
View 4 Replies View RelatedI am using rsync to backup dirs on my ubuntu server onto a NAS (which is mounted onto the filesystem), but the problem is that it is constantly doing full backups rather than doing incrementals and I am not really sure why. After doing a bit of expermienting with the script I noticed that if I just backed up a home dir (/home/user) the incremental backups work fine. If however I was to back up a dir like (/home/domain/user) it always does full backups.I have tried various different scripts but still the same end result. The latest script is a variation on the a script found on the samba rsync examples webpage, see below...
#!/bin/bash
# rsyncbu.sh -- backup to nas using rsync
# This script backups files listed in BDIR to the BSERVER. The verbose output along with the date is listed in the LOG_FILE specified
# verbose output
[code]....
There are multiple servers to be backed up. Different access rights exist in each server. There are two backup servers with plenty of disk space, one local, and one offsite. The local one feeds to the offsite one. The rsync command is being used to make a replica of backed up data. Deleted data is also being archived. There are two methods that have been considered: One is to have the individual servers run rsync which logs in to the backup server to push data. Two is to have the backup server run rsync which logs in to each individual server to pull data. Because system data is involved and meta information (like owning user) must be stored, root is required to access the data as well as to store it. That means everything runs as root both ends. So method one was quickly dismissed because each server would effectively have rights to access ALL the data on the backup server since it logs into the backup server as root. The security containment here involves different groups using different servers, and they need to be isolated from each other.
But even method two involves some risks that are a concern. This means one machine has access rights to every server. If the backup server were compromised, every machine could be compromised.What I'd like to find is some way to allow backups to be run without either machine granting root access to the other, while still running as root, or something equivalent, that allows accessing all data and storing all metadata. So I was looking at setting up an rsync daemon on each individual server (running as root so it can access what it is specified to access), and running an rsync client on the backup server (as root so it can store metadata). This opens network access issues. Any user on the network can connect to the rsync daemon. So password protection is needed. But this communication is also not encrypted, which exposes the password and the data should the network be sniffed.
So now I'm thinking about a non-root ssh login between machines. The backup server would login to a non-privileged user on each individual server and set up a secure forwarding channel to the rsync daemon. Is this the best that can be done? Is there a way to run rsync via SSL with key verification so it can all be done together? I'd like to have the rsync daemons configured to always talk SSL, and always verify the client's key against a list of authorized keys, and likewise the client verify the server's key against the known public key for that server.
Can I use rsync for incremental backups of the running linux server?
View 5 Replies View RelatedI am backing up my debian server with rsnapshot which actually uses rsync to perform the backup. The backups are located in an external storage of size 1.4T .
[code]....
I tried to understand what this error message means and i founde that error code 12 : 12 Error in rsync protocol data stream I understand that when rsync find that a file on the target was changed , it will send only the block/blocks that contain the changes and in the destination rsync will create new file and not update the old one (new inod...) . I want to know if this error i get is due to full disk or perhaps it is some other factor
I need to login as root, or at least get root privileges, in a cron triggered backup run. The straight way to do this would be the backup server making an ssh connection to the server to be backed up (this way because I want to avoid many servers being backed up in parallel and the backup server itself would be managing this diversity), via the rsync command which would be performing the backup's synchronization step.
I'm looking for alternatives to this in some form. I'd like to disallow direct root login to my ssh port (not 22One idea I have is to have the backup server initiate an ssh login as a non-root user, to either the actual source server, or to a server that can reach the source server ... and set up port forwarding. Over the forwarded port, then initiate the rsync that logs in as root via another port that allows direct root, but cannot be reached from the internet at all (because the border firewall doesn't include this port as allowed in).FYI, these logins will be using ssh keys, not passwords. I do need to keep ownership metadata for files being backed up, so this is why I am using root. Also, rsync is needed to get the incremental updates to keep bandwidth usage lower (otherwise I could just transfer a tarball each day).Anyone have any other ideas or comments, for security issues, based on experience doing things like this (backups, routine data replication, etc)?
I am using rsync for incremental backups. I am backing up to a second hard drive on my computer. When I check the individual backup directories (backup.0 through backup.4) with du -hs they each show 12G; when I check the parent directory squeeze it shows 15G. Over 4 backups I have added 3G. I haven't made very much for changes to directories I'm backing up and am using hard links. I have included some info below.
Quote:
Backup script:
#!/bin/bash
mount /mnt/backup
cd /mnt/backup/squeeze/
rm -rf backup.7
[code]....
With the --backup and --backup-dir= options on rsync, I can tell it another tree where to put files that are deleted or replaced. I'm hoping it fills out the tree with a replica of the original directory paths (at least for the files put there) or else it's a show stopper. What I'm wanting to find out applies when I'm restoring files. Assuming each time I run rsync (once a day) I make a new directory tree (named by the date) for the backup directory. For each file name/path in the tree, I would start with whatever is in the main tree (the rsync target) and work through the incremental trees going backwards until I reach the date of interest to restore to. If along the way I encounter a file in an incremental, I would replace the previous file at that path with this next one. So by the time I get back to a given date, I should have the version of the file which was present at that date. Do this for each file in the tree and it should be a full restore.
But ... and this is the hard part, it seems. What about files that did not exist at the intended restore date, but do exist (were created) on a date after the intended restore date. What I'd want for a correct restore would be for such files to be absent in the restored tree (just as they were absent in the source tree on that date). How can such a restore be done to correctly exclude these files? Wouldn't rsync have to store some kind of sentinel that indicates that on dates prior, the file did not exist. I suspect someone might suggest I just make a complete hard linked replica tree for each date, and this way absent files will clearly be absent. I can assure you this is completely impractical because I have actually done this before. I ended up with backup filesystems that have so many directories and nodes that it could take over a day, maybe even days, to just do something like "du -s" on it. I'm intending to keep daily changes for at least a couple years, if not more. So that means the 40 million plus files would be multiplied by over 700, making programs like "du -s" have to check over 28 BILLION file names (and that's assuming the number of files does not grow over the next two years).
So I am using rsync (3.0.7 on MAC OSX) to backup one hard drive to a folder on another one. The is USB drive to USB drive and I have done the initial backup from one drive to a new formatted other drive with the following command:
Code:
rsync -avX --progress /Volumes/Source /Volumes/Destination
This all appears to be going smoothly as I type. I am going to write a script to do subsequent backups in the
[code]....
I've been using dump/restore for backups, for quite some time. It's worked fine, but the process of recovering from a HD failure takes too long. What with eSATA and external drive docks, what I'd really like is to use rsync to maintain a current clone of my entire system drive. That is, start with a full disk clone, and then use rsync to keep it current.
I've seen plenty of instructions on how to do this with a directory tree, but I've seen none for doing it with a copy of the entire disk. If, for example, I copy /etc/fdisk, then the copied disk would have entries with the same UUIDs as the original disk. Which would mean that if the clone disk were to be bootable, its partitions would need the same UUIDs as the original disk. Which they would be, if the cloned disk started as a full-disk clone, I think. Am I wrong? But that means that when the clone disk was active, I'd have partitions with duplicated UUIDs. Is this going to cause problems? When I boot, will I get the correct partitions loaded?
I am in the process of writing an rsync script to run unattended backups of my entire file system to another system located on my local network using ssh and password-less rsa keys.
I will absolutely will not use password-less keys with the root account and this is the limitation preventing me from accomplishing my goal because root is required by rsync to access the / tree and copy it to another location. I decided that if I compiled the script into a binary that I didn't have a problem with the password being contained within the binary itself but from what I've read there is no way to elevate to root and then back down to user level from within the script/binary.
I can create the script as the user and use chroot to make it owned by root but retain execution permission for the user but it will still cause the ssh login to be under root and therefore require either that I am there to enter my password or the use of password-less keys under the root account which I reiterate I will NOT do. Currently the script is executed by the user on the machine containing the files to be backed up.
I have managed to connect to the above NAS using the supplied RAIDar software from my base unit. The unit is duel boot with Ubuntu 10.10 and Windows XP. Both operating systems work with the NAS both by wireless and wired using RAIDar and any other relevant programme
However my laptop, running Ubuntu, will not find it using Raidar, the IP address in a browser nor with a file manager or any other programme. I have tried both wired and wireless connections
However if I go to "places" from the top bar, which opens Nautilus, I can access it and all the files on it.
I've been a DOS/Windows guy for 20 years, and recently became a SW test lab helper. My company uses CentOS for a lot, so I've become familiar with it, but obviously not as comfortable as I am with Windows.
Here's what I have planned:
machine: Core 2 Duo E8400, 8GB DDR2, 60GB SSD OS drive, ATI 4650 video card, other storage is flexible (I have 3 1TB drives and 4 750GB drives around that can be used in this machine.)
uses: HTPC, Network Storage, VMWare server host: SMTP, FTP server, and Web server virtual machines
I've figured out how to do much of this, but I haven't figured out how to do backups in Linux. I've been spoiled with Windows, with the built in backup system so simple to use. I find myself overwhelmed with the array of backup software, and unable to determine which to use. none of them seem to do everything I need them to do, but some come close, I think. I'm hoping someone here can help me out in figuring out which program to use and how to use it.
Here is what I need the backup software to do:
1. scheduled unattended backups, with alerts if the backups fail
2. a weekly full backup with incremental every 12 hours
3. removing the old backups when the new full backup runs, I would prefer to keep 2 weeks of backups, but that's not necessary
4. a GUI would be preferable, since my arthritic fingers don't always do as I want them to do. I typo things a lot, and the label worn off my backspace can attest to that.
I'm am not remotely familiar with the linux operating sytem and just need a simple 'cut and paste solution' that will allow me to send .gz files as mail attachments, from the command line.Recently I moved my website to new hosting company, they are using cpanel and I have set up a couple of commands; the cpanel scheduler runs them for me as required.
1.creates an mysql dump and saves it to folder ( in root of my space )Works fine.2. delete the file created above, after 15 mins. Also works fineWhat I really want to do is email the .gz to myself before it is deleted, this is were I am stuck.This is possible ( I believe ) but it is beyond my understanding and ability to write the scipt/command that will make this happen.Please make any instructions really simple and clear as I am really new to this.on the server and this information is provided below and may be helpful to you (but I certainly to not know anything about this type of stuff)
On the ReadyNAS Duo, a RAID resync synchronizes the entire contents of one disk to another. It's a slow operation. So it's not something to be done lightly. By default, it is done whenever the unit is shut down uncleanly (generally the result of a hang or power interruption). It's something that can be disabled, though, such that a resync would be done only when manually requested. The authors of the software RAID driver in Linux (called "md") came up with a nice solution to the problem of slow RAID resyncs: the md driver allows you to assign a write intent bitmap to an md device, and where it's stored is configurable (the default is to store it on the md device in question, in the superblock). So when the array goes down unexpectedly (e.g., in a power outage), only the blocks that the md device was going to write will actually be resynced.
It's sort of like a journal for the md device itself. Now, the ReadyNAS Duo doesn't seem to use Linux md at all for implementing the RAID volume, but it does have an option to disable automatic resync in the event of an improper shutdown. And that leads to my question: how necessary is it to do a resync in such an event? If the Duo uses the same method as the Linux md driver, then obviously such a resync would rarely be necessary. But if it doesn't, then it would be the equivalent of using the md driver without a write-intent bitmap, and a resync would almost certainly be necessary in the event of an improper shutdown. Does the Duo (and maybe other units) use the equivalent of a write intent bitmap for RAID write operations?
My home networking consists of a slackware box, running iptables with a dual NIC. That's the firewall. I have a Netgear FSM 7352S, which is a level 3 switch, which I am currently just using as a switch. Clients are numerous PCs and a couple of networked printers.
The "firewall" machine is also a file server.
Here are the issues I could use some pointers on:
I'd like to assure that the VOIP adapters get priority, assuring QOS, particularly voice quality.
I'd like to provide reasonable priority for video streaming, such as hulu and other sources, that the kids use.
I'd like bulk data transfers (like backing up partitions) to the file server that runs iptables, and acts as the firewall/gateway for a cable internet connection. It would be good to be able to do this without impacting VOIP and video streaming.
I work for a company that rsyncs its media content to a datacentre daily for distribution via https to our clients. Unfortunately we've started to have problems with our rsync command/script.
It runs fine, but it seems to re-copy files that are already at the destination, of the same size and of the same date modified? We have hundreds of thousands of files, so this really cant happen. Our team here at the company use Windows XP to create the media, then put in on a the local suse server via SMB.
The code is
Code:
Both servers local and remote are Suse 10. The target/paths are irrelevant because they are correct and work.
Local server LS -L
Code:
For example
Remote server LS -L
Code:
As you can see local/remote files are the same, any ideas why this re-copy would happen? What's rsync playing at? We obviously just want it to sync new files and changes.
The requirements are:
SystemA must be 100% Identical to SystemB no matter what rsync runs into: symlinks, same size files with different dates, directories that no longer exist... etc.
What is your perfect rsync command?
I'm trying to put a variable into the rsync command but I just get some BS error about escaping - whatever that is suppose to mean.
This is my command:
Code:
rsync -rvvuc --exclude '/Output/*' ${OPTARGS} --delete "${SOURCE}scripts/" "${DEST}scripts/"
${OPTARGS} has this value:
--rsh='ssh -p 34'
Wont work. If I type that text directly, then it works. Other then using some if statement to have two different command calls, how can I make this work?
I'm going to be using this command to back up my files:
Should I change anything or is it ok?
I'm trying to understand rsync filters .. but it isn't goig to well.I want to rsync only files with a specific extension (.gz.des3). Could someone please tell me how to add such filter to the rsync commandline (not a seperate filter filer).
View 2 Replies View RelatedI have the following rsync command for making some backups:
Code:
rsync -r --progress --delete -H --numeric-ids -a --exclude=.gvfs /source /media/Backup
If I paste that in my Terminal, it will perform a backup of all the files and show me what's going on in the Terminal window. But how can I make that into a launcher? I have made a launcher on my desktop with that code in its Properties, but double-clicking it starts the rsync process (I can see HDD activity) but a Terminal window won't open.
I have a client that is using a windows based backup system called Retrospect and will be installing Linux to replace his windows server. Right now the way it works is on the XP pro workstation Norton 360 makes a backup to the server and then Retrospect make it backups every night. I want to add a linux based backup system that will give them a historical backup system that had XP pro client software that can backup open files and such I can do away with the Norton backup stuff and streamline this setup. I need it to have the capability to go back in time to restore something of an older date this historical. I was looking into:
[URL]
I have never used either one and was wondering what the linux community wold recommend between the two or if there is something better that has client software and free and does it have a GUI for my clients? I am leaning toward AMANDA after reading Jeremy's article:
[URL]
and there seams to be a webadmin gui as well.
[URL]
Why would rsync insert a user's home directory path in variable expansion when run via cron, but not when run manually. The gory details... Red Hat Enterprise Linux AS release 4 (Nahant Update 6) Linux 2.6.9-67.0.20.ELsmp The script (parts anyway, and simplified)...
Quote:
#!/bin/bash
. /home/bea/.bash_profile
echo rsyncloc=${rsyncloc} >> ${log}
[code]....
I received the following output from an rsync (3.0.0) command that was executed:
sending incremental file list
sent 77214 bytes received 484 bytes 155396.00 bytes/sec
total size is 254531170 speedup is 3275.90
What does "sending incremental file list" mean?
I have rsync error message for this command,
Quote:
rsync -avr --delete --exclude lampp/tmp/* /opt/* /backup/opt/
rsync -avr --delete --exclude data/session/* /var/www/html/dmcatholic/* /backup/dmcatholic/
The error message is here.
[code]....
sent 1917154 bytes received 384128 bytes 5731.71 bytes/sec total size is 12220966785 speedup is 5310.50 rsync error: some files could not be transferred (code 23) at main.c(892) [sender=2.6.8]
when i use rsync command to backup my image file , it shows the following error message.
bash: line 1: /usr/bin/rsync: Argument list too long
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: remote command could not be run (code 126) at io.c(463) [receiver=2.6.8]
The command which i used is rsync -avrl -e ssh cms@server:/data/cms/data/images/* /mnt/Backup/Intranet_cms_backup/images
I've been trying to create file sharing between my MacBook Pro and my Lucid Lynx box for time machine backups and media server purposes. I followed this guide:[URL]..Everything seems to work with these exceptions: I can see my LucidLynx box in my finder app in my Mac but only when I run these commands from Ubuntu:
Code:
sudo /etc/init.d/netatalk restart
sudo restart avahi-daemon
If I restart my LucidLynx box then I can't see anything in finder. I can't log into my LucidLynx box from finder. I don't get a bad username or password error it just tells me the connection failed. *Note if I do enter an incorrect username or password it WILL tell me it's incorrect. I've looked at this link below since some people have used it in theses forums but it's a bit dated[URL]..