General :: Rsync To Copy Directory Tree From NAS Drive ?
Apr 10, 2010
I have a WD world book edition 1TB NAS drive, and just purchased an acomdata 1tb drive and connected it to the NAS via USB. If I recall I think the WD NAS has a ext_ or some type of linux filesystem on it, and the acomdata has a ntfs filesystem on it.
What I want to do is copy over certain directory trees of the NAS to the USB attached drive. I usually use MS synctoy to sync folders from my windows pc to the NAS drive, and MS richcopy to make the initial transfer from PC to NAS. For this operation though, since it is taking place entirely on the NAS and its connected drive, I thought that rsync would be the best option, and it is available on my NAS drive.
Last night I entered in rsync -avr /movies/* /usb1-1share1/ to copy the entire "movies" dir to the drive, which shows up as usb1-1share1 on the NAS drive. It copied most of the directory tree ok, but a lot of the folders were empty, so this morning I tried rsync -Carv --ignore-existing /movies/* /usb1-1share1/ to try and get all the files missed, without recopying the 24GB that did make it across. This also managed to copy a few more GB over, but not everything.
I am running the command from an ssh session on the NAS using putty on my PC, in as user "admin" which should have all rights over these folders. There is a bunch of errors in the command window like this: rsync: failed to set times on "/shares/usb1-1share1/movies/classics/fulldvd/First Blood DVD/.VTS_01_2.VOB.RxdjWZ": Operation not permitted (1)
I want to restart another session and get the files it missed, but I want to find out what I am doing wrong first. Should I be doing this as root user? am I missing some switches or just plain doing it all wrong?
when I installed 13.37 I created a local copy of the entire stable tree (source/ and all the rest) just to have all that stuff around to browse offline.
Now, to instruct myself, I'm trying to use rsync to keep this stuff up to date. But I seem either to have misread the rsync man page or ... well, I don't know. I am issuing the following command and getting the results seen below:
I have ubuntu 9.10 & windows 7 on my laptop. I have one 40GB Primary partition (C & 60 GB logical partition on my windows. I have given another 20GB for ubuntu.
I hibernated my windows7 & then started ubuntu 9.10. Then I copied a folder containing some PDF files from my ubuntu to that 60GB (D on windows. Then I rebooted the machine & choose windows from GRUB. the windows came up from hibernation but nowhere I was able to see that folder which I copied.
Since ubuntu supports (understands) NTFS file system it means when I copied that folder it should have updated the Directory Tree of NTFS on that 60GB (D but that folder is not shown.
When I restart the windows the folder appears (ofcourse because windows checks the file system again for consistency while in hibernation it does not).
I am trying to create a simple bash script to rsync some folders within a directory stucture. I am using wild cards, in the rsync source directory structure, but my command always fails. I believe it is the way I am using wild cards within my for loop. Here is my command ;
for seq in `cat test.txt` ; do rsync -nvP /folder/folder/folder/folder/folder/**/$seq /folder/folder/folder/ ; done This always fails, where if I do a ls to the destination, to test the path, it always works.
I just installed a new HD on my system with multiple HD's already. I have a drive with two versions of Ubuntu & would like to copy the complete drive to the new drive along with all the contents & partitions of the Ubuntu drive.
1 - Could I partition the new drive & just copy the contents using rsync?
2 -If I copy all the contents over could I just reinstall Grub & edit fstab & be good to go?
Are there any other archiving tools than tar that preserve Linux file permissions and user/group flags? I want to create archives of a directory tree on my ubuntu box. The directory trees are large, a first run with tar cvpzf archive.tgz /home/foo/bar yielded a 5GB archive. I also want to keep all permissions and other flags and special files.I'm fine with a 5GB archive, however to look inside that archive -- since it is a compressed tar archive -- the whole 5GB have to be decompressed first! (Or so it appears when opening it with the archive viewer -- I'm happy to be corrected.)So I need a way to "backup" a directory (tree) while preserving full filesystem attributes and right and creating an archive with an "index" that hasn't to be decompressed for browsing its contents. An archive is either a single file, or a (small) set of files that carries full and complete information within this file/s. That is, it can live on any filesystem (size permitting), it can be burnt onto a DVD, you can split it (after which the point of this question is really lost - but still), ...
This little fragment does the opposite of what I want and I am trying to figure out how to reverse it. I want to strip off the directory and echo just the filenames. I cannot predict how many levels deep the directory tree is.
I would like to overwrite files in a directory tree, recursively. The ones I would like to overwrite match the filename "x_alpha*.png" and have a size exactly 456 bytes. Is there any way to search for these recursively in a directory tree, and overwrite them with a reference file, for example "e:mydirgood.png"
I am using Windows 7, but I have UnxUtils, so I can use those too. What I am looking for is something like this, generated automatically: copy /y e:mydirgood.png e:mydiracx_alpha0023.png copy /y e:mydirgood.png e:mydirefgx_alpha0045.png copy /y e:mydirgood.png e:mydirhx_alpha0248.png
I have recently purchased an external hard drive in order to backup my home partition. In my PC I have a "1.5T" drive with several partitions on it, containing OSes and the home partition. The home partition is 1.3T according to df, the external drive contains one partition that spans the entire disk,df reports it as 1.4T in size. Both partitions are ext3. When I use rsync to copy files from the home partition to the external partition, the external disk becomes full, despite the destination - supposedly - being larger than the source. I don't understand why copying files from one partition to a slightly bigger partition should need more space than on the source partition. Does anyone know what is happening ?
Details : I created the partition on the external drive with gparted; gparted reported it the already have several gigabytes in used space immediately after the partitions creation - I thought at the time that this must be normal. The home partition contains many files of all sorts, including lots of big audio and video files. If you are wondering, for all my important files this external disk is only secondary backup, as they are also backed up to the "internet".
These are the mount points :
/mnt/tmp/ : home partition, /dev/sdb6 /mnt/external/ : external partition, /dev/sdc1
I used rsync to copy the files, I know there are more efficient ways to do this, but I wanted to use the same command that I will subsequently run to sync the backup.
Next I tried adding the --sparse switch, as I was wondering if the problem may come form sparse files. I don't know however if rsync would go back and shrink the sparse file by just adding the switch and executing the command. I also added --one-file-system, for good measure. Here is what I ran next :
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write failed on "abcd.avi": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.6]
Looking at the destination after a partial copy seems to indicate that the problem is not symbolic links being "expanded". I have not checked the source filesystem for sparse files, nor the destination to see if these files could be larger there, as this does not seem trivial.
Here is some additional info :
$ df /mnt/tmp/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdb6 1415342836 1414173740 369096 100% /mnt/tmp
I am performing a dry run using Rsync on 2 different boxes.While i'm doing that, Under destination directory, I want a specific directory x to be ignored for sync.Please let me know the exact pattern to ignore the directory.The current command I'm using is:rsync -avnc --delete $LOCAL_DIR $USERNAME@$DESTINATION_IP:$REMOTE_DIRunder DESTINATION_IP, I would want to ignore a particular directory under REMOTE_DIR.
I have a home storage of many drives which are seldom accessed, and extremely seldom written to. So, I made several RAID1 arrays with "write-mostly" drives in each one, so that all the drives spin down after a while, and if they are accessed, only one drive in each array has to spin up. This way, I hope to minimize the risk of losing anything due to mechanical shock or wear.
But it was not so easy. First, the write-mostly drives did spin up on every read request due to a bug in the kernel, so I managed to get the fix accepted into the newer kernels (which is my largest contribution to the Linux development so far. See [URL] .....) Now, if I read directly from the array, it works as expected (only the first drive spins up); but if I create a filesystem there (ext4) and read from it, the second drive still spins up anyway. (Does ext4 write something when it reads a file?.. I have it mounted with relatime, so AFAIK it shouldn't...)
Well, I thought, I'll just mount the filesystems read-only and remount them if I need to modify the data there, which is not very often, to say the least. Also, this way the drives don't have to spin up every system shutdown to unmount the filesystems; during a power outage, it used to take a lot of time for them to spin up one by one just to unmount, while my UPS had to survive long enough for them to finish. That's one more problem solved.
But I decided to improve even further. Sometimes I just want to see the contents of those arrays without even accessing some files (for example, when I accidentally click the wrong network drive button in Windows, or when I just want to see if I have something there or not). If I read the directory trees during startup and keep them cached, then those drives won't even have to spin up at those moments! The question is, is there a way to do it? I remember Linus boasting about our kernel being the best at caching the filesystem trees, and I know there is vmtouch [URL] .... which allows us to keep some files cached, but what about the directory tree itself?.. If I simply "find" all the files there, it seems to work, but they get evicted from cache sooner or later...
If transfer all files under a directory by rsync, what is the order that rsync determines to transfer the files one by one?At first it looked like rsync transfers files in alphabetical order, but later I found rsync skipped some files in the first sweep through the alphabetic order, and then went back to transfer files that were skipped in the first time and this time still in alphabetic order.
Is there a way to force rsync to not make directories in its destination directory; ie, to simply dump all of the files from the source directory directly into the destination without copying any of the folders that the files were originally in? I tried --no-dirs, but that seems to only be for empty directories.
Recently i am trying to check on the rsync speed for single file(2.4GB iso) directory ( 900MB directory with files inside ) When i run the rsync for single file: the speed i get is average 50MBps However, for a directory: average speed is 10MBps Is there any reason behind this ? i tried to google but unable to get the concept.
We have an rsync cron job set up to mirror all the files in a "..dashtdocsdocs" folder to the same folder on another server. It copies all the files over correctly and deletes any files in the "docs" directory that aren't in the sending directory, but it also deletes any files we put in the target directory's parent folder (..dashtdocs or other subfolders like ..dashtdocsimages) even though they've been excluded in the .rsync-filter file.
So for example server A has ..dashtdocsdocs and ..dashtdocsimages. Server B has ..dashtdocsdocs but if I manually copy the images folder over to ..dashtdocsimages, the images folder gets deleted from the target directory every time rsync runs.
I'd like to keep just the docs directory synched and update other folders manually, but they keep getting deleted. It looks to me like it's running a delete-excluded option, but that option wasn't used.
I've got a directory structure full of files. I want to convert them all into some other format, but I don't know how to keep the directory structure, is there a way so I can "set" which command to use on files instead of `cp`?
Why would rsync insert a user's home directory path in variable expansion when run via cron, but not when run manually. The gory details... Red Hat Enterprise Linux AS release 4 (Nahant Update 6) Linux 2.6.9-67.0.20.ELsmp The script (parts anyway, and simplified)...
i would like to find and backup all *.mp4 files from /Pictures and its sub-directories and move them to a single directory on a remote. I can find and move the files but I don't want the directory structure...just the files to be placed in the remote directory.
I have a user account (no root access) on a Debian system. My disk quota for my home directory is limited to 250MB, of which I've used 150MB.
I would like to install a program that requires 95MB of space (let's call it prog-5.1) into $HOME/local/bin. Because the source files take up 1GB, I've had to place them in the /tmp directory (to stay under my limit).
Unfortunately when I run the config script, I receive the following:
The reason for this is that /tmp is mounted in noexec mode, meaning I can't execute code in /tmp directory:
I'm wondering if anyone has any ideas about running the config script in my home directory, and somehow pointing it to configure the files in the /tmp/prog-5.1/ directory.
I found it easiest to embed narp into the gtk+-2.0 installation because all I needed to do was tweak the Makefile.am and Makefile.in files and let configure do the rest.This is a hack. I've failed at getting my own makefiles to work.
Would anyone point me in the right direction on how to properly make an installer for Linux? Also, what is the correct/common directory tree for Linux software packages? Any more suggestions on proceeding with this project using community standards?
I have a 1TB USB external drive, currently formatted as fat32. What I need to do is copy two folders and all their subfolders, totaling about 500GB, to that external drive. The USB drive will have to transfer back and forth between RHEL, Windows XP, and Mac OSX computers freely.What format should I go with on the USB drive, FAT32 or NTFS?What rsync switches should I use? I know I don't want to use -a because I don't want any permissions restored. I'm guessing I'll have to run rsync a couple times to fully get all the files, so I need to be able to cancel an rsync, then have it pick back up where it left off, not start over and recopy every file again.
The problem was finding an applet to browse directory tree directly in the Gnome panel similar to that in "Recent Documents". The thread was closed without any solution posted there and also no solution available elsewhere (I searched for over 2 hours) in the forum.
Search for "file-browser-applet" in the Synaptic Package Manager. The applet with exactly that name is what you need. It allows to browse and open files on your computer from the GNOME panel. Once installed, "Add to panel".
I'd like to know if it's possible to automatically mount, and fire up rsync to sync a USB drive with a directory? Specifically, I'd like to copy as much data as the drive can hold and only delete the oldest files if space is needed. I would assume I'd do something like this with a script, but my problem, is where to start.