Software :: Rsync Can't Copy Certain Files With Unusual Characters While Cp Can
Aug 20, 2010
On my Ubuntu box, I have a mounted windows share connected via gvfs called graphics. I want to backup everything on a nightly basis from graphics to backupserver/graphics . If I use rsync, it will not copy files that have parent directories with funky characters in them (but the directories themselves will be copied!). Everything else gets rysnced just fine.
graphics/test/macdir/picture.psd
...when rsynced over to ...
backupserver/graphics/
gives the error:
rsync: mkstemp "/home/administrator/.gvfs/drobo on x.x.x.x/linux_backups/graphics/test/ macdir/.picture.psd" failed: Operation not supported (95)
The directory macdir gets created but there is nothing in there. This happens for all files underneath dirs with funky names. cp -Rf works perfectly! Directory and child files all get copied over no matter how strange the characters get in the directory names.
View 7 Replies
ADVERTISEMENT
Jun 1, 2011
How can I use rsync to copy ONLY the my home folder (and nothing inside of it, just the folder name) to another machine. I've tried things like
Code:
rsync -av /path/to/src /path/to/dest/
or
Code:
rsync -av -f"+ */" -f"- *" /path/to/src /path/to/dest/
This last option recursively (through the -a switch) copies only folders, including all subfolders. If I try
Code:
rsync -v -f"+ */" -f"- *" /path/to/src /path/to/dest/
nothing is copied (not even my home folder.
View 9 Replies
View Related
Feb 9, 2010
I have recently purchased an external hard drive in order to backup my home partition. In my PC I have a "1.5T" drive with several partitions on it, containing OSes and the home partition. The home partition is 1.3T according to df, the external drive contains one partition that spans the entire disk,df reports it as 1.4T in size. Both partitions are ext3. When I use rsync to copy files from the home partition to the external partition, the external disk becomes full, despite the destination - supposedly - being larger than the source. I don't understand why copying files from one partition to a slightly bigger partition should need more space than on the source partition. Does anyone know what is happening ?
Details : I created the partition on the external drive with gparted; gparted reported it the already have several gigabytes in used space immediately after the partitions creation - I thought at the time that this must be normal. The home partition contains many files of all sorts, including lots of big audio and video files. If you are wondering, for all my important files this external disk is only secondary backup, as they are also backed up to the "internet".
These are the mount points :
/mnt/tmp/ : home partition, /dev/sdb6
/mnt/external/ : external partition, /dev/sdc1
I used rsync to copy the files, I know there are more efficient ways to do this, but I wanted to use the same command that I will subsequently run to sync the backup.
rsync -av --progress --stats --recursive --perms --links --delete /mnt/tmp/ /mnt/external/
Next I tried adding the --sparse switch, as I was wondering if the problem may come form sparse files. I don't know however if rsync would go back and shrink the sparse file by just adding the switch and executing the command. I also added --one-file-system, for good measure. Here is what I ran next :
rsync -av --progress --stats --sparse --one-file-system --recursive --perms --links --delete /mnt/tmp/ /mnt/external/
I tried an fsck on the home partition :
fsck -f /dev/sdb6
This is the output from the last rsync :
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: write failed on "abcd.avi": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.6]
[code]....
Looking at the destination after a partial copy seems to indicate that the problem is not symbolic links being "expanded". I have not checked the source filesystem for sparse files, nor the destination to see if these files could be larger there, as this does not seem trivial.
Here is some additional info :
$ df /mnt/tmp/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb6 1415342836 1414173740 369096 100% /mnt/tmp
[code]....
View 2 Replies
View Related
Apr 20, 2010
I've got quite a decent rsync script setup, however I'd like to invoke it whenever there's change to a file. My initial idea was to use find, however this has two major flaws - the first being my particular unix veriant cant understand -print0 which means this doesn't work, the second is that I'm not 100% sure how to put variables into quotation marks so ls can understand the target:
Code:
for i in `find /shares/ -mtime -1 -print`; do ls -ltr $i;done
View 14 Replies
View Related
Jun 14, 2011
I want to run rsync on server A to copy all files from Server B when they are newer than 7 days.(find . -mtime -7) I don't want to delete the files on Server B.
View 2 Replies
View Related
Sep 18, 2009
I just tried to sync files from one server to another. After the sync process, I found the files are bigger than original ones.
I looked up the web and found someone mentions the rsync daemon. So I have to run the daemon on one server before I run the rsync?
The command I used is rsync --partial --progress -r source destination
View 1 Replies
View Related
May 18, 2010
i need help in this issue how to find files with unusual size and with unusual names of EX : just dots, names ending with space(s),names containing shell wildcard characters, names containing non-ASCII (control) characters
View 3 Replies
View Related
Jul 21, 2010
use rsync to cp such files and dirs under /var/www/html/mydir directory but these two files(/dir4/1.html /dir4/2.html) cant rsync to dest mechine.
rsync configure file,below...
View 2 Replies
View Related
Sep 2, 2010
We're in the process of implementing an offsite backup of all our servers to a remote Linux server. We're using rsync over ssh.What I've found is that characters such as ±, ¶,´ and £ are replaced on the Linux server with underscores.I don't mind if it changes these characters in the filenames of documents, but when it renames a language pack from Espa±ol.clx to Espa_ol.clx, it could cause issues for us further down the line.
What do I need to do differently to make the special characters copy over correctly? For the initial sync which will take place locally, before the machine is moved offsite, I have SAMBA enabled. I am able to copy files from Windows to the Samba share, retaining the original filename, though it looks different in the Linux directory listing, i.e. t̻st becomes ĻstThese files get deleted by rsync when it runs, as it does not match the filenames.
View 1 Replies
View Related
Dec 6, 2010
I'm having to rebuild my home server following a failed upgrade from 8.04 to 10.04 . All data lives in /shared on this server, the contents of which are mirrored weekly to a USB HDD which is mounted at /backup, using rsync:
sudo rsync -av --progress --delete /shared /backup
I recovered the contents of the backup drive to the rebuild server's /shared directory using the cp command with the archive flag set to preserve ownership, timestamps etc. Everything looks fine to me. However, when I do a test rsync (adding -n to the command above) then it looks like rsync wants to recopy everything, and I'm at a loss to see why. For example, below is a test on the subdirectory /shared/backgrounds. The file attributes look identical:
Code:
chris@quadra:~$ ls -la /shared/backgrounds
total 8112
drwxr-xr-x 2 chris chris 4096 2009-04-12 11:06 .
[code]....
Is there some condition that rsync can detect that I can't see? Is rsync sensitive to the way the HDD is mounted? (the USB HDD is actually mounted at /media/backup, and /backup is a soft link to this.)
View 3 Replies
View Related
Jan 14, 2011
This command would copy the files to the local directory,find /mnt/nas -type f -ctime 1 -iname '*.avi' -exec rsync -av {} /mnt/Mythbuntu
View 1 Replies
View Related
May 27, 2010
I have a newly installed Dell Optiplex 755 with Ubuntu 10.04 (64-bit) and I am having serious issues with network copying. Whenever I start rsync och scp with larger amounts of data my system becomes practically unresponsive (all types of apps grey out) until those processes are completed. Cpu stays at below 10% and I have lots of free memory and such .
View 3 Replies
View Related
Jan 26, 2011
I just installed a new HD on my system with multiple HD's already. I have a drive with two versions of Ubuntu & would like to copy the complete drive to the new drive along with all the contents & partitions of the Ubuntu drive.
1 - Could I partition the new drive & just copy the contents using rsync?
2 -If I copy all the contents over could I just reinstall Grub & edit fstab & be good to go?
View 3 Replies
View Related
May 26, 2011
Using Ubuntu Server 10.04 LTS. I'm new to Ubuntu and testing Rsync. I successfully copied 3TB of data from a Win7 machine to an MDADM Raid5 array. All appears to be fine. Used a Win app for the copy. I then deleted a 250GB folder on the Raid5 array and recopied the data using Rsync. Rsync was executed via a Putty session on a WinXP machine. The source was an eSata attached drive (same drive used for the big 3TB copy) and the destination was the same Raid5 array. That copied just fine. I bit verified it with a Win7 app. Perfect.
I then used the following Rsync script to copy a single 26GB file from that same eSata drive back to (what I intended to be) the Raid5 array:
Code:
neil@ANTECUBSV:/mnt$ rsync -r -a -v -e ssh --delete /mnt/disk1/Test/ /mnt/Test
sending incremental file list
created directory /mnt/Test
./ C_VOL-S300-b001.spf
sent 3020267622 bytes received 34 bytes 50760800.94 bytes/sec
total size is 3019898880 speedup is 1.00
neil@ANTECUBSV:/mnt$ cd raid
Note that only about 3GB copied. No error messages were posted to the putty session. I made a mistake in the Rsync command, creating the Test folder directly in the mount folder rather than the Raid array, as I intended. That is a little strange, yes, but I would not think it would cause a partial copy? The /mnt folder is on my system drive, which had about 34GB available space before the copy, so comfortably would have had 6GB or so after.
The eSata disk is mounted as /mnt/disk1
The Raid5 array is mounted as /mnt/raid
I then recopied the file to the correct intended destination on the Raid5 array, which has about 400GB free space (plenty).
Code:
neil@ANTECUBSV:/mnt/raid$ rsync -r -a -v -e ssh --delete /mnt/disk1/Test/ /mnt/raid/Test
sending incremental file list
./
deleting 2010-07-05 Backyard Birds/Thumbs.db
deleting 2010-07-05 Backyard Birds/
C_VOL-S300-b001.spf
sent 11105775462 bytes received 34 bytes 50366328.78 bytes/sec
total size is 11104419840 speedup is 1.00
neil@ANTECUBSV:/mnt/raid$ df -h
Note that only about 11GB was copied, and this was confirmed with an ls -l command. Now I am correctly copying the file to the Raid array but it is still incomplete. I then copied the file back to the /mnt folder to see if the problem reproduces:
Code:
neil@ANTECUBSV:/mnt/raid$ rsync -r -a -v -e ssh --delete /mnt/disk1/Test/ /mnt/Test
sending incremental file list
created directory /mnt/Test
./
C_VOL-S300-b001.spf
sent 26327927554 bytes received 34 bytes 56558383.65 bytes/sec
total size is 26324713984 speedup is 1.00
neil@ANTECUBSV:/mnt/raid$ cd /mnt/test
This time I got my full 26GB file. Why I might be getting inconsistent results? This is quite troubling of course. I'd also be interested in basic a command line Linux diff app (that does file directory as well as bit level checking) if one is available.
View 9 Replies
View Related
Apr 10, 2010
I have a WD world book edition 1TB NAS drive, and just purchased an acomdata 1tb drive and connected it to the NAS via USB. If I recall I think the WD NAS has a ext_ or some type of linux filesystem on it, and the acomdata has a ntfs filesystem on it.
What I want to do is copy over certain directory trees of the NAS to the USB attached drive. I usually use MS synctoy to sync folders from my windows pc to the NAS drive, and MS richcopy to make the initial transfer from PC to NAS. For this operation though, since it is taking place entirely on the NAS and its connected drive, I thought that rsync would be the best option, and it is available on my NAS drive.
Last night I entered in rsync -avr /movies/* /usb1-1share1/ to copy the entire "movies" dir to the drive, which shows up as usb1-1share1 on the NAS drive. It copied most of the directory tree ok, but a lot of the folders were empty, so this morning I tried rsync -Carv --ignore-existing /movies/* /usb1-1share1/ to try and get all the files missed, without recopying the 24GB that did make it across. This also managed to copy a few more GB over, but not everything.
I am running the command from an ssh session on the NAS using putty on my PC, in as user "admin" which should have all rights over these folders. There is a bunch of errors in the command window like this: rsync: failed to set times on "/shares/usb1-1share1/movies/classics/fulldvd/First Blood DVD/.VTS_01_2.VOB.RxdjWZ": Operation not permitted (1)
I want to restart another session and get the files it missed, but I want to find out what I am doing wrong first. Should I be doing this as root user? am I missing some switches or just plain doing it all wrong?
View 3 Replies
View Related
Jun 15, 2011
I am trying to create a simple bash script to rsync some folders within a directory stucture. I am using wild cards, in the rsync source directory structure, but my command always fails. I believe it is the way I am using wild cards within my for loop. Here is my command ;
Code:
for seq in `cat test.txt` ; do rsync -nvP /folder/folder/folder/folder/folder/**/$seq /folder/folder/folder/ ; done This always fails, where if I do a ls to the destination, to test the path, it always works.
View 5 Replies
View Related
May 14, 2011
when I installed 13.37 I created a local copy of the entire stable tree (source/ and all the rest) just to have all that stuff around to browse offline.
Now, to instruct myself, I'm trying to use rsync to keep this stuff up to date. But I seem either to have misread the rsync man page or ... well, I don't know. I am issuing the following command and getting the results seen below:
Code:
View 3 Replies
View Related
Sep 1, 2010
I have a 160GB harddrive which I installed a F12, would like to upgrade to a bigger drive, but I hate to have to re-install everything.
Recommend a good disk copy utility? The utility should be able to not only copy files, but boot sector and everything. So I just need to make a copy, change my BIOS to boot from the new drive and run everything as before.
View 11 Replies
View Related
Jul 9, 2010
just installed ubuntu couple of days back on my netbook. I am still a beginner, enjoying my adventure exploring ubuntu. I have another desktop which runs on XP. I am able to access XP shared folders through my netbook(linux). However, i wanted to copy files from XP infact folders using TERMINAL in my netbook, not copy and paste using my mouse. Are there any commands for it?
View 1 Replies
View Related
Nov 12, 2010
how do i delete any single one of the files in my whole disk that say something along the lines to "chromium os"? i.m just wondering because i tried to install through virtual box and failed, and probably have three different partitions consisting of nothing.
View 3 Replies
View Related
Jun 1, 2010
I want to archive all .ctl files in a folder, recursively.tar -cf ctlfiles.tar `find /home/db -name "*.ctl" -print`The error message :tar: Removing leading `/' from member namestar: /home/db/dunn/j: Cannot stat: No such file or directorytar: 74.ctl: Cannot stat:No such file or directoryI have these files: /home/db/dunn/j 74.ctl and j 75. Notice the extra space. What if the files have other special characters? How do I archive these files recursively?
View 2 Replies
View Related
Sep 14, 2010
Can files/directories have Greek characters? If I selected English when installed Linux?
View 3 Replies
View Related
Apr 27, 2010
I hope you can help. I have a collection of spreadsheets with data that needs to be imported in to SQL. The data has been manually entered although there are portions where data has been copied and pasted from the web.
When converting these sheets to a CSV I get strange characters where it looks as though data has been copied and pasted. Is it possible to write a script (AWK?) to pull out these characters?
I guess the script will need to keep alpha characters, spaces, numerics and commas but nothing else. How easy is this to do?
View 7 Replies
View Related
Jul 19, 2010
I have a bunch of files that I need to rename, ordinarily this is pretty easy task. The problem here is that the file names have Chinese / Japanese characters. ie [$$$$$$$$].SOMETHING BLAH BLAH.ext Where all the "$$$$" are insert Chinese characters. The problem is that sed or perl doesn't seem to handle the Chinese characters correctly so using a regular expression like this 's/^[*.]//' which would normally work doesn't. From what I have read so far I believe these characters are double encoded UTF-8 (not 100% sure) which could be the problem. So far I've tried numerous different regex's as well as playing around with convmv to see if I could convert the filenames to just single encoded characters but I've had no luck.
View 1 Replies
View Related
Aug 25, 2009
How can you create a script to move or copy files from a main directory into multiple directories below the main directory.
View 7 Replies
View Related
Sep 9, 2011
My Source folder contains 424.8 GB in 502,474 files. My Destination folder was created fresh, and after the copy contains 394.0 GB in 486.514 files. I am running it as grsync with root authority. The only options are to preserve time, permissions, owner and group., and to produce a verose output and transfer progress. There are no exceptions specified to skip any files.
The rsync command is this:
rsync -r -t -p -o -g -v --progress -c -l -H -D -s /mnt/Backups/monthly.3/ /mnt/EX-Fantom/monthly.3/
I have run it again to give it a chance to get it right. Same result. The source is in an rsnapshot folder, but this is the first backup, the original, containing only whole files, not links.
View 3 Replies
View Related
Mar 2, 2010
I'm trying to using rsync to backup some files, about half a TB. It's now it a state where it keeps sending the same files everytime it runs. for example:
rsync -av /data/source/* user@host:/data/dest
sending incremental file list
source/file1.txt
source/file2.txt
I then verify those files are copied over. then the next time it runs it does the same thing
rsync -av /data/source/* user@host:/data/dest
sending incremental file list
source/file1.txt
source/file2.txt
any idea why it's getting stuck on these files? I've tried to wipe the whole dest directory out and start over but no luck.
View 2 Replies
View Related
Jul 27, 2011
I have two directories, dirA whicht contains N gb of data and dirB which is supposed to contain only the newest M gb of data from dirA. When files are added to dirA, they sould also be added to dirB, while the oldest files in dirB should be deleted.Is that possible with rsync? or any other software?
View 1 Replies
View Related
Mar 15, 2010
So I just used rsync to backup about 400gb of data to my NAS. Look just over a day to complete, which is what I figured. I decided I should run rsync again to see how its going to handle comparing and only adding new files to the remote location. So I added a few new files and then ran the backup again. Well rsync is trying to do a complete copy of all of my original data, even though they have not changed.
Is there a way that I can tell rsync to compare the two directories and only add the new files and delete the ones that are no longer in the original location?
Here is command that I am running:
Code:
sudo rsync -azvv --progress --stats /media/sda1/multimedia/movies /home/codeblue/NAS_Share_Point
View 9 Replies
View Related
Jun 9, 2011
I am running Ubuntu 10.04. I am transferring roughly 62 GB of data libraries to my 84 GB /home partition. I'm using rsync because scp kept stalling, and I had to restart it over and over. Things were going great until recently when it began to show an error: "failed: No space left on device (28)" These are the things I've done so far: Used the GUI to find out how much I have copied so far: 5,149,552 which take up 30.2 GB. df -h, it tells me that my /home partition is 56% used, and that I have 33 GB available. (42 GB used out of 78 GB with 33 GB available) Also, none of my other partitions are anywhere near 100%. the /home partition is the most-used and it's only a little over half-full. du -s in the directory where I'm copying all of this: it also returned 42206500. Additionally, when I try to save screen captures, it sometimes fails with a "device full" error. What's going on? Am I really out of space? Why doesn't it show me that I'm out of space?
Is there a hidden temp file that rsync uses that just got too full? I did a little research on wikipedia and it said that ext4 has a 64,000 directory limit. Could it be that I somehow broke that limit with all of these files? Solution: not enough inodes for the vast amount of subdirectories on hard drive. This wasn't an RSYNC problem, rather, a partition configuration issue. To check inode usage: df -i If you want to add any inodes, you will need to backup your partition and format it using mke2fs (man mke2fs). Be sure to change the respective inode setting.
View 9 Replies
View Related
