I want to use rsync in order to have a folder synced at startup with my fat32 partition. I figured out how to mount the fat32 partition automatically at startup but I'm failing with the rsync command.I came across this script but it works only the first time, when there is a new file to sync it fails.
View 3 RepliesThought I'd post it here because it's more server related than desktop... I have a script that does:
[Code]....
This is used to sync my local development snapshot with the live web server. There has to be a more compact way of doing this? Can I combine some of the rsyncs? Can I make the rsync set or keep the user and group affiliations? Can I exclude .* yet include .htaccess?
When I run rsync --recursive --times --perms --links --delete --exclude-from='Documents/exclude.txt' ./ /media/myusb/
where Documents/exclude.txt is
- /Downloads/
- /Desktop/books/
the files in those directories are still copied onto my USB.
And...
I used fetchmail to download all my gmail emails. When I run rsync -ar --exclude-from='/home/xtheunknown0/Documents/exclude.txt' ./ /media/myusb/ I get the first image at url.
I have a tiny shell script to rsync files between two servers and remove the source files.
This script works fine, when it has been initiated manually or even when the rsync command is executed on the command line.
But the same script doesn't work, when I try to automate it through crontab.
I am using 'abc' user to execute this rsync, instead of root, as root login to servers are restricted in all of our servers, by us.
As I mentioned earlier, manual execution works like charm!
When this rsync.sh is initiated through crontab, it runs the first command(chown abc.abc ...) perfectly without any issues. But the second line is not at all executed, and there is no log entry i can find at /mnt/xyz/folder/rsync.log.
I just tried to sync files from one server to another. After the sync process, I found the files are bigger than original ones.
I looked up the web and found someone mentions the rsync daemon. So I have to run the daemon on one server before I run the rsync?
The command I used is rsync --partial --progress -r source destination
use rsync to cp such files and dirs under /var/www/html/mydir directory but these two files(/dir4/1.html /dir4/2.html) cant rsync to dest mechine.
rsync configure file,below...
I'm using Ubuntu 10.04 LTS server and Postgresql 8.4. I have a .sh script that is run by cron every other hour. That works fine. The .sh script includes an rsync command that copies a postgresql dump .tar file to a remote archive location via ssh. That fails when run by cron; I think because it is (quietly) asking for the remote user's password (and not getting it). I set up the public/private ssh key arrangement. The script succeeds when run manually as the same user that the cron job uses, and does not ask for the password. I am able to ssh to the remote server from the source server (using the same username) and not get the password prompt (both directions), so why doesn't rsync work? I even put a .pgpass file in the root of that user's directory with that user's password, and the user/password are identical on both servers.
I think the problem is rsync is not able to use the ssh key correctly. I tried adding this to my script but it didn't help.
Code:
Here is the rsync command embedding in the .sh script.
Code:
Here is the cron entry:
Code:
I've bought a NAS (Western Digital My Book World Edition 1TB) which I want to use to make backups (preferably incremental) of some important files.
After much deliberation in finding suitable backup software it seems like good old rsync is the best thing for the job (backintime struggles to copy to remote locations?)
I have enabled SSH on the NAS config. I'm using the following command to do a test run on a small folder:
Code:
sudo rsync -azvv -e ssh /home/matt/Careers/ admin@192.168.1.100:/public/AutoBackup/
And I get the following error:
Code:
admin@192.168.1.100's password:
[Code].....
Anyone know where I'm going wrong here? I'm sure it's probably something simple but I can't crack it. I've tried variations on the destination folder such as admin@192.168.1.100:/AutoBackup/ without success.
I want to use rsync to synchronize some folders on my LAN. I have this working with two scripts; one runs at the beginning of my work session and gets the latest directory tree from the server, and the other runs at the end of the session to put any local changes back on the server. My "get" script looks something like this:
Code:
rsync -avuzb --backup-dir=/home/user/rsync_backup_dir --delete my_server:/home/bak/common_data/ /home/user/data
This works well, and with the "b" option any file that has been deleted from the master directory tree on the server will be deleted from the local machine and moved to the local backup directory. This is a safety measure to prevent the loss of files through a mistake (on my part).
The problem is the "put" script:
Code:
rsync -avuzb --backup-dir=/home/user/rsync_backup_dir --delete /home/user/data/ my_server:/home/bak/common_data
I want to run both scripts from the local machine, but the "put" script will not save deleted files to a backup directory. I tried using a remote backup directory like "my_server:/home/user/rsync_backup_dir" but this did not work. Is there a way to backup files deleted from a remote server from an rsync script run locally?
I saw in a magazine reference to using rsync to have identical copies of folders. This looks like something I could find useful as I have a large number of items in need of safe backup.
I have the folders on an old system on a home network and would like to copy these over to a USB Hard Drive.
Currently the folders reside on SFTP xxx.xxx.xxx.xxx and I wish to sync them to a USB port on my laptop.
I setup keyed ssh between two of my computers on my lan. It was working great. I used Ubuntu's Passwords and Encryption Keys tool to generate the key.
Yesterday I tried rsync'ing a grip of files to the ssh server and couldn't get it to work, I figured I was just messing the rsync command up. Today I just tried to log in using ssh and it just hangs there after I type the ssh command.
With verbosity on it stops at checking the blacklist file. Does this mean it decided my key is one of the badly generated ones? Why would it decide this now I thought it was supposed to catch that when generating the key.
Also I just checked that file it's trying to check doesn't even exist. Should it?
Code:
I'm currently trying to have crontab to automatically backup files from ramdisk. It works perfectly when I run it myself by simply cd:ing to scripts directory and type ./save_world.sh.
The problem is, that crontab DOES (at least it looks like it) run that command every one minute. /var/log/syslog does show it executing that line every one minute without any errors. I'm currently very confused what I did wrong here. I have tried rebooting, fiddling with crontab line, tried sudo crontab -e but nothing seems to work.
My script is called save_world.sh and it is located in /home/phoe/minecraft/rpg/
Code:
My crontab -e has one line and it is following:
Code:
I haven't determined any specific time yet, because I'm just trying to get it work first.
Snippet from /var/log/syslog:
Code:
I have a samba share to a windows 7 computer I do not know if I will be able to use backintime or not so I want to know how to have rsync do my backup.I read the man but I'm not sure if I understand the it.on same computer different hard drive to run every hour in a script. Leanne is windows 7 share and backup is the other hard drive in the computer rsync -arvRzEP /media/leanne /media/backup.
View 1 Replies View RelatedSo I just used rsync to backup about 400gb of data to my NAS. Look just over a day to complete, which is what I figured. I decided I should run rsync again to see how its going to handle comparing and only adding new files to the remote location. So I added a few new files and then ran the backup again. Well rsync is trying to do a complete copy of all of my original data, even though they have not changed.
Is there a way that I can tell rsync to compare the two directories and only add the new files and delete the ones that are no longer in the original location?
Here is command that I am running:
Code:
sudo rsync -azvv --progress --stats /media/sda1/multimedia/movies /home/codeblue/NAS_Share_Point
I've been trying to get my headless Ubuntu 9.10 server to back up files from my Windows XP box and onto a 1 TB Seagate FreeAgent Go USB drive (which is connected to the Ubuntu server). I've tried two different methods, both of which are behaving strangely and not quite working. I'm using SSH to access my Ubuntu server.
The /etc/fstab line for the USB drive looks like this:
Code:
/dev/sdb1 /media/usb-backup-windows ntfs uid=34,gid=34,umask=022,dirsync,sync 0 0
I'm using NTFS because I'd like to share this drive between Windows and Linux. The drive gets mounted fine and I can read/write files after booting up. The way I'd like for this to work is to have my Ubuntu box mount the Windows drive using CIFS. So I mounted the C drive using the following command:
Code:
smbmount //windowsxp/C$ /mnt/windowsxp/C -o directio,iocharset=utf8,noperm,nounix,ro,credentials=/home/user/.smb/passwords.conf
The mount works fine. I can browse directories under /mnt/windows/xp/C, read files, copy files to Ubuntu, etc. So now I have my USB drive mounted and the C drive on my Windows box mounted. Should be good to go, right? Unfortunately, after several minutes (this varies, sometimes it can go an hour or so) of copying files using rsync --archive /mnt/windowsxp/C/ /media/usb-backup-windows/C/ (the actual command I use has more options - not sure if that's important), the server locks up.
The SSH session dies and I can no longer ping it. The server will eventually start responding after several minutes, only to lock up again a few minutes later, and so on and so on. When it locks up, the following messages end up in /var/log/kern.log:
Code:
CIFS VFS: Unexpected lookup error -26
CIFS VFS: No response to cmd 46 mid 59789
CIFS VFS: Send error in read = -11
CIFS VFS: server not responding
I did some Googling on these messages and came across a suggestion to set /proc/fs/cifs/OplockEnabled to 0. I gave that a shot but it didn't make a difference. I also tried plugging in a mouse and noticed that I could make the server respond immediately after a lock up by moving the mouse. I have to move the mouse though - just leaving it plugged in without movement doesn't help. I have to wait for the hang to occur and then move it. Once I do that, things progress for another few minutes.
This got me thinking that I had a lack of entropy and the mouse movement was kicking things into gear. So I tried moving /dev/random to /dev/random-chaos, and created a symlink /dev/random that just pointed to /dev/urandom. This didn't work - same exact behavior. So why in the world does moving the mouse bring the server back and cause it to start responding, if only for a few more minutes until the next hang?
I then gave up on this approach and tried connecting to an rsync daemon running on my Windows box (using Cygwin), instead of using the CIFS mount point. After getting the config file right and figuring out how to run it as a service on Windows, I started getting files copied once again. However, after what seems to be about the same length of time (several minutes to an hour or so), the rsync connection dies and I get the following message in the Windows rsync log file:
Code:
2010/03/22 13:01:01 [4024] rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Connection reset by peer (104)
2010/03/22 13:01:01 [4024] rsync error: error in rsync protocol data stream (code 12) at io.c(1539) [sender=3.0.7]
The Windows box is running rsync 3.0.7 and Ubuntu is running 3.0.6, but they are both protocol 30. The rsync error log on Ubuntu doesn't help much - it also says "Connection reset by peer". I've tried this at least a dozen times and it always fails with these messages. It's weird because it's always 4 bytes, never anything different. I also noticed that /var/log/kern.log had the following messages, although they do not line up with the times that rsync died:
Code:
usb 1-5: reset high speed USB device using ehci_hcd and address 2
I did some Googling on this message and tried some stuff that worked for other people. I added dirsync and sync to /etc/fstab. I tried setting /sys/block/sdb/device/max_sectors to 64. Neither one of those made a difference. unloading the ehci_hcd module and dropping back to USB 1.0. However, 9.10 doesn't seem to have that module loaded, so I'm not exactly sure how to turn off USB 2.0 and just try 1.0. I'm not real enthused with that workaround anyway because I have at least 500 GB to copy.
I've kind of run out of ideas here. It's frustrating because the entire reason I bought the hardware and set up Ubuntu was to run backups. I'm not sure if my problem is a networking issue (CIFS VFS server not responding, connection reset by peer), a problem with running headless (wiggling the mouse temporarily prevents the hang), a USB device problem (reset high speed USB device messages), or something else entirely.
I am currently backing up my data but find that it takes way to long to do a rsync, it takes forever to just find the differences and transfer them.Out of 3 separate rsyncs the main one that is slow is my www.skins.be mirror directory which is 41GB and has 392,200 files, sorted into multiple directories. Which grows by around 100 every couple days.I think that something that would be able to track changes by inotify time on directories will speed it up since Picasa sure finds the changes fast when I open it and it is tracking over 26,200 pictures. I just don't know of a backup solution that does that.
View 4 Replies View RelatedI'm trying to learn how rsync works to backup my system. I tried:
Code:
rsync -azvv /home /media/Elements
I get a folder called home on my external hard drive but when I use ls -l to see the permissions they are all wrong.
On my /home folder the permissions for /nathan are
drwxr-xr-x 48 nathan nathan
The permissions on the backup /nathan folder are
drwx------ 1 nathan nathan
I also tried using the long version of -a which is -rlptgoD and that didn't work either. What do the 48 and 1 mean when I used ls -l? When I look in the /nathan folder the permissions are all screwed up too. A lot of the files are backed up as executable and the permissions are all screwed up. I also ran it with sudo, and that didn't work either. The permissions were still screwed up and ownership is messed up too.
This should be a quick one. I'm trying to backup a single directory and it's subdirectories on my Lucid Server to a freenas box across my network. This is what I'm using to do that rsync -r -a -v -z * --delete freenas: DSIBackups..It almost works perfectly except for one problem. When a file is deleted at the source, this command doesn't seem to delete it on the receiving end. I assumed that the --delete would do that but aparently not.
View 1 Replies View RelatedI launched a script using rsync and the option -delete-before, however the destination folder was the wrong one. I noticed it only seconds after but it was still too late. In less than 5 sec rsync deleted over 200gb of data on my external hard drive... It is not in my Trash nor on the HDD trash but it's not possible that the data were ereased in less than 5 sec! (I dont know how rsync handle deletions.) I know I am really stupid but is there a way to get back all my data (mainly HD movies in mkv, mp4 and some avi)
View 1 Replies View Relatedwhen rsync is finished the update, or in the meantime - i need to move the updated files to a different location - like date +%Y%m%d something or what ..the reason is, because of the development, i need the modified files, but all of them, not just the last one - so i have to store them daily, but i dont want to store the whole dir - just that few files which are updated does it make sense?
View 5 Replies View RelatedOK, here is the test code I am using: rsync -acrv /home/randy/Documents /media/USB2/testbackup. What should I put in the code so that when I delete something in the Documents folder, it will also be deleted in the testbackup folder when I perform a sync.
How can I have it so the code will check if USB2 is available, if it is, do the sync, if not, do nothing.
I want to synchronize sets of files (e.g. from or to flash memory). rsync is powerful, but --delete option is dangerous. Anyone know whether there's a way to do --delete interactively, i.e. get rsync or some near equivalent to ask (y/n, in a console window) before deleting?
View 9 Replies View RelatedI am running a headless Ubuntu server accessed through Webmin. The server is running 10.4.2 64 bit version. I have a number of cron jobs including a simple back-up job which is:
Code:
rsync -av /media/server/ /media/backup/backup/
All of the other jobs run fine but for some reason this job which is scheduled to run each day at midnight does not run. If I SSH into the server and run the job manually it works fine.
The output of 'crontab -l' is:
@daily rsync -av /media/server/ /media/backup/backup/
11 3 * * * /etc/webmin/package-updates/update.pl
@daily tar czf /media/backup/serversystem.bz2 --exclude=/proc --exclude=/lost+found --exclude=/backup.tar.bz2 --exclude=/mnt --exclude=/media --exclude=/sys /
@reboot inadyn --username ********** --password ********** --update_period_sec 600 --alias **********
@daily /etc/webmin/backup-config/backup.pl 12978522112842
The output of 'crontab -e' is:
Error opening terminal: unknown.
crontab: "/usr/bin/sensible-editor" exited with status 1
I'm using Ubuntu 10.10 64-bit and I'm a fairly new linux user. I've been trying to set up Grsync to to make incremental backups to my external harddrive but have some problems. In the file lists it looks like files are actually being copied but when I look at the harddrive none of the new files are actually there. As far as I can understand I'm not making a dry-run. I also found this thread: [URL] And I tried using the commands suggested in that post, with the same results.
View 9 Replies View RelatedI am running Ubuntu 10.04. I am transferring roughly 62 GB of data libraries to my 84 GB /home partition. I'm using rsync because scp kept stalling, and I had to restart it over and over. Things were going great until recently when it began to show an error: "failed: No space left on device (28)" These are the things I've done so far: Used the GUI to find out how much I have copied so far: 5,149,552 which take up 30.2 GB. df -h, it tells me that my /home partition is 56% used, and that I have 33 GB available. (42 GB used out of 78 GB with 33 GB available) Also, none of my other partitions are anywhere near 100%. the /home partition is the most-used and it's only a little over half-full. du -s in the directory where I'm copying all of this: it also returned 42206500. Additionally, when I try to save screen captures, it sometimes fails with a "device full" error. What's going on? Am I really out of space? Why doesn't it show me that I'm out of space?
Is there a hidden temp file that rsync uses that just got too full? I did a little research on wikipedia and it said that ext4 has a 64,000 directory limit. Could it be that I somehow broke that limit with all of these files? Solution: not enough inodes for the vast amount of subdirectories on hard drive. This wasn't an RSYNC problem, rather, a partition configuration issue. To check inode usage: df -i If you want to add any inodes, you will need to backup your partition and format it using mke2fs (man mke2fs). Be sure to change the respective inode setting.
I have successfully backed up my files using a script to a remote server with a log file output.However the log file is appended each time.I wish to have a different log file each time with date and time and have yet to figure that part out.
View 2 Replies View RelatedI have a Popcorn Hour that I'm going to put a 2TB internal HD into. I want to be able to synchronize the data from the Popcorn Hour to my desktop. My desktop has 2x1TB USB drives. Is there a way that I can get the 2x1TB drives to be seen as 1x2TB drive. My desktop is currently running mythbuntu.
View 1 Replies View RelatedI'm trying to set up rsync on a clean install of ubuntu server 9.10, with ubuntu-desktop installed.
In the end, I'd liek the 2 windows clients (running Vista and Win7) to syncronise with the server when they log on/off. I'd also like them to access a printer attached to the server.
I've got a bash script that uses rsync via SSH to back up data from a remote webserver.
This script works fine when I run it myself but when it runs as a cron job I get the following error:
Code:
2010/02/20 08:15:01 [6786] Unexpected remote arg: user@www.server.com:~/www
2010/02/20 08:15:01 [6786] rsync error: syntax or usage error (code 1) at main.c(1219)
[code]....
I am running this command,
Code:
rsync -vrl --progress --stats --perms --delete --exclude '*~' --exclude 'Music' --exclude 'server' --exclude 'Downloads' --exclude 'Ubuntu One' --exclude 'Videos' /home/dave/ /media/server/backup/desktop/dave
It transfers files that remain unchanged. The one that I notice every time is
Code:
.VirtualBox/HardDisks/anOS.vdi
I have several OSes installed in VirtualBox, so this takes a very long time.