Software :: CentOS 5 - Mirroring System Using Rsync
Mar 30, 2011
I believe it was rsync was the tool. I have a box running CentOS 5. It has a 250GB HD in it. I have another drive with the exact model. Currently it has a as a said a 250GB IDE drive. I want to shutdown the machine, install this other hard drive and set up a cron job that will backup my main drive at times that I set. This way if the main drive fails, I do not loose all the data and have to rebuild the server from scratch as I have been custom configuring it for years. I can't remember if there was an issue with the main drive being mounted to back it up or not. I have looked at some of the how's on rsync but they seem to only talk about using another server for this. If I shit down the box, install the new drive, and the box boots back up, is it going to ask about that drive or what do I need to do to get rsync going and does it partition the drive as such? And can I do it this way. This way if the main drive fails, I can just swap the drive and be on my marry way.
Using mrepo, i'm trying to pull down centos 5.6 updates from a mirror (say, mirrors.kernel.org or mirrors.usc.edu). i want to use rsync to do this so i can try to exclude packages i don't need on my servers (e.g. firefox, openoffice). when i try this, i get the error "@ERROR: Unknown module '5.6'".
The following rsync command is what mrepo uses when i run "mrepo -ugvvvv":
When i run that, i get the unknown module error.
But what i don't understand about that error is, sniffing out the rsync server's modules on the commandline shows me that module (and path?) should work. like so:
Anyone know what i'm doing wrong when trying to pull this mirror down via mrepo? (or even that long rsync command)
The rsync module "opensuse-full" which worked well so far seems currently broken. It tries to mirror a huge number of additional stuff (factory?) but fails with "permission denied". Any place where I could report this?Command:
I'm trying to mirror an apt-cacher-ng cache between two computers. I have apt-cacher-ng installed on my laptop, and I have another machine running apt-cacher-ng. In order to keep them both up to date with each other and to make sure all the computers have all the updates, I've been trying to find effective ways to keep them matched.
- Unison looks like what I want, it would delete files that are deleted, and it would add files that are added. (the assumption is if they're deleted from one, they'd be deleted from others). - Rsync seemed quite a bit easier, especially with the advanced permissions issues. Apt-cacher-ng uses a user called apt-cacher-ng.
Instead of giving root an ssh password, I wanted to just ssh as apt-cacher-ng. Then I can still get the files over the network, but without the root account being open. So I ran: passwd apt-cacher-ng and when I sshed, it looked like it was working until it logged me out (almost immediately). So that's not working. What am I missing? Maybe there's a better tool then rsync for this?
I have two web servers. One is active and one is in reserve. I keep the user data (web pages) in sync by running rsync every 10 minutes or so. This copies any changes from the active machine to the reserve machine. But, it's slow, only gets changes every 10 minutes, bogs down the disk, does strange things to files that are changing during the rsync process etc...
I want something that will automatically copy any changes from the active server to the reserve server as they are made. IE I hit 'save' on the active server, it copies the file to the reserve server. Simple!
I've been looking around and I see GFS which is really vastly more complicated than I need. I'm happy with read-only access on the reserve host, so I don't need distributed lock management.
I could theoretically implement this by setting inotify watchers on every file and running an SCP or rsync command when a file gets saved. So, it can't be that hard.
I do not need a true networked file system, as in something I mount over the network. I just want something to keep my disks in sync.
Blue sky thinking at the moment:I have a number of file servers, each at different sites. I would like to be able to make these appear as one, so that files on any server can be accessed from any site, and the user doesn't even see there are multiple servers.Obviously, the internet is slow, especially the upload speeds. So when a file is written the write ought to go to the server on the client's LAN - even if it was previously on another of the servers.However, for robustness, some sort of background mirroring is also wanted. If all the servers were left on and connected, they eventually end up all in sync. But this mirroring needs to be mindful of bandwidth usage; if someone writes a big file to their local server, copying that to the other servers can't interfere with normal internet usage.I think UnionFS or similar might be able to handle the unioning side, but not the mirroring stuff.
Is there any software that will do full mirroring between a local and a remote file system. I have a server (9.04) and a laptop (9.10). Each user has shared a shared directory on the server, and on the laptop. Updates to files may be done on either system. I want to keep both copies syncronised. Currently I use a script based on rsync (scheduled by cron) to keep the local and remote copies in sync.
The problem with this approach is that rsync only seems to be able to handle deletion of files if one file system is the master, which is not the case in my set-up. If I move a file to a different directory, rsync will reinstate the old file as well as copying the new one. I was hoping there was some software that could do proper mirroring between the 2 systems, but6 so far I cannot find anything.
I installed CentOS 5.4 on a virtual server to host the tool OpenVAS, which is used to execute vulnerability assessment.This tool is based on a master engine and thousand of plugins, each one doing a specific, little function.I wrote a cron that, every sunday, updates the list of these plugins to keep the system updated; this cron uses a tool's proprietary function that download the new plugins from a site using rsync.I was using the rsync 2.6.8-3.1 until last week, when I decided to install ALL the updates proposed by operating system itself (more than 170) and the whole list includes rsync 3.0.7-1.el5.rf (from rpmforge).
This monday, the log of the tool's function gave me a bad response about the impossibility to establish a connection with the remote server due to a connection timeout.Before doing the rollback of the system, I tried to understand what happened, so I started manually the function on a terminal and, together, I was watching the results using netstat -tuvnap, so I saw that there was an attempt of connection, between my server and the remote one, but the connection did not finish the three way handshake, because the state of connection was always "SYN_STAT".I tried to value the $RSYNC_PROXY variable, because there is a proxy between my server and the world, but the result was exactly the same, so I was forced to execute the rollback of the system, which works very well now!
I'm making my own yum repository - firstly so that all the machines I administer can be updatedvia the internal network, secondly so that I can test any updates on a spare machine before passing them on, and thirdly so that I can add my own repo for internal software.
I've created the necessary folders under my webserver, and used rsync to update them from my local CentOS mirror, following the instructions at [URL]
I notice it says to run "createrepo" on the base repository, created by copying the rpms from release DVD.
When I rsynced the updates repo, but I notice that the files in repodata are very small. In fact, having a look inside them, filelists.xml contains no file details. But if I run createrepo in the updates directory, filelists.xml gets lots of file details inside it.
I wondered if maybe the local mirror hadn't been updated properly, but checking against mirror.centos.org shows that has the same files.
how does the (real, live, CentOS) updates repo work when there is nothing in the filelists?
I have automated backups running on my ubuntu box using rsnapshot (rsync) basically following this tutorial. My concern is how to restore if everything is lost. Since it does not seem that I can backup the entire drive and that I have to choose individual folders and some like proc/ cause problems.
Currently in my rsnapshot.conf I have (see below). Is there a way to just clone the entire drive? Or should I not do this? Questions:
1, Can I backup in a way such that it is a clone of the drive so that it can be swapped with the current drive?
2, If not yes to clone, If I had a total drive failure would I install a basic ubuntu and then replace all the files with the backups?
I have a newly installed Dell Optiplex 755 with Ubuntu 10.04 (64-bit) and I am having serious issues with network copying. Whenever I start rsync och scp with larger amounts of data my system becomes practically unresponsive (all types of apps grey out) until those processes are completed. Cpu stays at below 10% and I have lots of free memory and such .
I regularly rsync directories to an external usb drive. Recently I've started getting errors which look: blah blah:Read-only file system (30)
The external usb drive has two partitions (ext3 and fat32) and I rsync to the ext3 one. The rsync command is rsync -ahi --delete /home/mike/some_dir /media/usb_drive
People have reported this error with faulty usb cables but everything was working fine until recently so it seem unlikely a usb cable would suddenly wear off.
When ever I transfer large files using cp, mv, rsync or dolphin the system will slow down to the point that it's unusable. It will sometime hang completely and not accept any input until the file is transferred. I have no clue what could be causing this problem but I know it shouldn't be happening.I am using Fedora 15 (2.6.40.3-0.fc15.x86_64) with KDE 4.6. have a Phenom II 955 processor, 6 GB of system ram and the OS and swap file is on an 80 GB SSD. Copying files in the SSD doesn't cause any problem, but moving files between my other two large HDDs causes the extreme slow down. Using htop I can see that my system load jumps to between 3 and 4, but my RAM and CPU usage stays low during the transfer. Here are two commands that take about 10 mins to run and make the system unusable while it's running. It usually transferring around 2-20GB worth of data during the transfers:
cp -a /media/data.1.5/backup/Minecraft/backups/* /media/data.0.5/backup/Minecraft/backups/ rsync -a /media/data.1.5/backup/ /media/data.0.5/backup/ /media/data.1.5/ is the mount point for a 1.5 TB internal SATA drive, and /media/data.0.5/ is the mount point for a 500 GB internal SATA drive.
I am trying to setup a Rsync backup method.I have multiple files and folders all OWNED by different users.Now I can read the files as root.But I want to make a user for RSYNC that can ONLY read all these files for copying them across.It requires to create a non password login and I dont want to do that with the root user ! I have tried to create a new user and use 'setfacl':setfacl -m u:someuser:r /var/test.But when I su to 'someuser' I cannot copy the files to my homedir. So I think it wont work for rsync also.
Need to know if there are any known errors allocating disk space using rsync. I have recently setup a small file backup server (2 500GB drives) running Centos 5.2, with a rsync daemon (via xinetd). However I have noticed that after about a week of usage, if I reboot, then the disk that rsync uses, has a lot of multiply claimed blocks and inode errors, such that I have to run fsck multiple times to fix. This has happened a couple of times now so I don't think that its random.
I am running version 5.5. I am using Grsync to back up 2 folders to an USB external 1.5 terabyte drive. Sometimes grsync backs the folders up correctly but most of the time, I need to unmount and then mount the drive to get it to work.
I'm using rsync to create a mirror of the data files on our main server every day. I've looked at the man page, and can't see it; can I get a listing of the files that have been changed on or added to the mirror when it's completed? Can it just log what it's doing to a file?
how rsync will handle disk images. Will rsync copy only the changed blocks of a vhd or a lun? This is what I've been told, but wouldn't this require overlaying a filesystem on the vhd? How would rsync handle copying a 500GB lun?
Thought I'd post it here because it's more server related than desktop... I have a script that does:
[Code]....
This is used to sync my local development snapshot with the live web server. There has to be a more compact way of doing this? Can I combine some of the rsyncs? Can I make the rsync set or keep the user and group affiliations? Can I exclude .* yet include .htaccess?
When I run rsync --recursive --times --perms --links --delete --exclude-from='Documents/exclude.txt' ./ /media/myusb/
where Documents/exclude.txt is
- /Downloads/ - /Desktop/books/
the files in those directories are still copied onto my USB.
And...
I used fetchmail to download all my gmail emails. When I run rsync -ar --exclude-from='/home/xtheunknown0/Documents/exclude.txt' ./ /media/myusb/ I get the first image at url.
I have a Linux box, running rsync, only problem is, I have to turn it off at night. What I want to do is turn it on and when it turns on, rsync will already be running in daemon mode rather then having to run rsync daemon. Does anyone know the solution? I'm sure its simple, I just cannot figure it out
I have a tiny shell script to rsync files between two servers and remove the source files.
This script works fine, when it has been initiated manually or even when the rsync command is executed on the command line.
But the same script doesn't work, when I try to automate it through crontab.
I am using 'abc' user to execute this rsync, instead of root, as root login to servers are restricted in all of our servers, by us.
As I mentioned earlier, manual execution works like charm!
When this rsync.sh is initiated through crontab, it runs the first command(chown abc.abc ...) perfectly without any issues. But the second line is not at all executed, and there is no log entry i can find at /mnt/xyz/folder/rsync.log.
