General :: Rsync Speed For Single Files And Directory?
Mar 23, 2010
Recently i am trying to check on the rsync speed for single file(2.4GB iso) directory ( 900MB directory with files inside ) When i run the rsync for single file: the speed i get is average 50MBps However, for a directory: average speed is 10MBps Is there any reason behind this ? i tried to google but unable to get the concept.
If transfer all files under a directory by rsync, what is the order that rsync determines to transfer the files one by one?At first it looked like rsync transfers files in alphabetical order, but later I found rsync skipped some files in the first sweep through the alphabetic order, and then went back to transfer files that were skipped in the first time and this time still in alphabetic order.
We have an rsync cron job set up to mirror all the files in a "..dashtdocsdocs" folder to the same folder on another server. It copies all the files over correctly and deletes any files in the "docs" directory that aren't in the sending directory, but it also deletes any files we put in the target directory's parent folder (..dashtdocs or other subfolders like ..dashtdocsimages) even though they've been excluded in the .rsync-filter file.
So for example server A has ..dashtdocsdocs and ..dashtdocsimages. Server B has ..dashtdocsdocs but if I manually copy the images folder over to ..dashtdocsimages, the images folder gets deleted from the target directory every time rsync runs.
I'd like to keep just the docs directory synched and update other folders manually, but they keep getting deleted. It looks to me like it's running a delete-excluded option, but that option wasn't used.
i would like to find and backup all *.mp4 files from /Pictures and its sub-directories and move them to a single directory on a remote. I can find and move the files but I don't want the directory structure...just the files to be placed in the remote directory.
In reading the rsync man page and browsing a lot of websites, I ended up a bit confused, or maybe it was just too much eggnog. Anyway, to exclude a directory "videos" with everything in it, which is /home/user1/camera/videos and I'm rsyncing the whole user1 directory to an external drive
I want to copy a big file from my harddrive to a removable drive by rsync. For some other reason, the operation cannot complete in a single run. So I am trying to figure out how to use rsync to resume file copying from where was left last time.
I have tried use the option --partial or --inplace, but together with --progress, I found rsync with --partial or --inplace actually starts from the beginning instead of from what was left last time. Mannually early stopping rsync and checking the size of the received file also confirm what I found.
But with --append, rsync starts from what was left last time. I am confused as I saw on the manpage --partial, --inplace or --append seem to relate to resuming copying from what was left last time. Is someone able to explain their difference? Why --partial or --inplace do not work for resuming copying? Is it true that for resuming copying, rsync has to work with option --append?
Also if a partial file was left by mv or cp not by rsync, will rsync --append correctly resume the file copying?
I have a number of crash.log files scattered about my system and I would like to run a command to find all the crash.log files on the system and copy them to a single directory; each with a unique filename. For example, copy crash.log from ~/directory_1 , ~/directory_2 , ~/directory_3 and so on to ~/crash_logs/crash.log1 , ~/crash_logs/crash.log2 , ~/crash_logs/crash.log3 etc.
I am performing a dry run using Rsync on 2 different boxes.While i'm doing that, Under destination directory, I want a specific directory x to be ignored for sync.Please let me know the exact pattern to ignore the directory.The current command I'm using is:rsync -avnc --delete $LOCAL_DIR $USERNAME@$DESTINATION_IP:$REMOTE_DIRunder DESTINATION_IP, I would want to ignore a particular directory under REMOTE_DIR.
I have a WD world book edition 1TB NAS drive, and just purchased an acomdata 1tb drive and connected it to the NAS via USB. If I recall I think the WD NAS has a ext_ or some type of linux filesystem on it, and the acomdata has a ntfs filesystem on it.
What I want to do is copy over certain directory trees of the NAS to the USB attached drive. I usually use MS synctoy to sync folders from my windows pc to the NAS drive, and MS richcopy to make the initial transfer from PC to NAS. For this operation though, since it is taking place entirely on the NAS and its connected drive, I thought that rsync would be the best option, and it is available on my NAS drive.
Last night I entered in rsync -avr /movies/* /usb1-1share1/ to copy the entire "movies" dir to the drive, which shows up as usb1-1share1 on the NAS drive. It copied most of the directory tree ok, but a lot of the folders were empty, so this morning I tried rsync -Carv --ignore-existing /movies/* /usb1-1share1/ to try and get all the files missed, without recopying the 24GB that did make it across. This also managed to copy a few more GB over, but not everything.
I am running the command from an ssh session on the NAS using putty on my PC, in as user "admin" which should have all rights over these folders. There is a bunch of errors in the command window like this: rsync: failed to set times on "/shares/usb1-1share1/movies/classics/fulldvd/First Blood DVD/.VTS_01_2.VOB.RxdjWZ": Operation not permitted (1)
I want to restart another session and get the files it missed, but I want to find out what I am doing wrong first. Should I be doing this as root user? am I missing some switches or just plain doing it all wrong?
Is there a way to force rsync to not make directories in its destination directory; ie, to simply dump all of the files from the source directory directly into the destination without copying any of the folders that the files were originally in? I tried --no-dirs, but that seems to only be for empty directories.
Why would rsync insert a user's home directory path in variable expansion when run via cron, but not when run manually. The gory details... Red Hat Enterprise Linux AS release 4 (Nahant Update 6) Linux 2.6.9-67.0.20.ELsmp The script (parts anyway, and simplified)...
Long story short, I have a failing RAID3 array which is showing file corrupted blocks (and the RAID controller card is periodically not initializing). [URL].. I thought I had robust backups but as it turns out, my backup volumes seem to have been misplaced (don't ask ) so I have no viable backups. I'm trying to back up as much as I can before the inevitable impending catastrophic failure.
I must be doing something wrong because running rsync on my Unix FreeBSD/FreeNAS server (syncing to local USB drive) is really slooooow. Below you will see an example. A 500MB file took almost 10 minutes to sync to a local USB drive! I can FTP this file in a few minutes over my LAN.
Just ran a test and it took 20 seconds to FTP this file across my gigabit LAN where rsync took 10 minutes to perform a local copy)
I'd like to copy a file, say widgets/water.txt, to all subfolders in the folder widgets using a single command. So if the folder widgets has 10 subfolders like widgets/blue, widgets/green, etc. I'd like to copy water.txt to all of them with one command.
I tried the commands
cp water.txt ./*/water.txt cp water.txt ./*/
However these don't seem to work. The latter gives 'cp: omitting directory' errors.
I am trying to write a very simple script that will go to every subdirectory of a single directory and run a command (lets call it make_ndx).I know I can write this the long way with in a text document with something like:
cd /"the directory"/"the 1st subdirectory" make_ndx cd .. cd "the 2nd subdirectory" cd ..
Alternatively, I also tried: for i in 'find /path/somemorepath -type d -mindepth 1'; do cd $i; make_ndx -f *.gro; done which returns me with the error cd: find: no such file or directory. But if I run the find command by itself to test if I am calling the right directories, it gives me the exactly the output I am looking for. Any ideas? Should I just write the find results to a file and loop through the contents of the file (which seems a little bit like overkill) or am I just making a simple typographical mistake and I am just not seeing it?
I have two directories, dirA whicht contains N gb of data and dirB which is supposed to contain only the newest M gb of data from dirA. When files are added to dirA, they sould also be added to dirB, while the oldest files in dirB should be deleted.Is that possible with rsync? or any other software?
rsync -r -v -e ssh email@example.com:/usr/local/websites/* /usr/local/websites and each time I run it it copies everything - all files. I thought rsync was only supposed to copy files that had been added or modified.
I'm planing to copy a productive mysql innodb file from one server to another, and the file size is around 300GB. As the file is keeping changing all the time, I have to shutdown mysql instance and copy the large data file to other server as quickly as possible.I should have to find a way to speed up file copying ... I'm wondering whether there's a way to copy file block by block.If the destination side block has same content, then bypass it.
I'm using the command below to sync two directories. Problem is insted of deleting the files on the target directory it simply appends a ~ character at the end of the file name. Not sure why this is happening?I'd like to have all deletes on the source replicated on target.
I was going to do a rsync -r -a -z -v -p -e sshto move some files frome server to another, but then realized all I really need are files which have dates starting June 1, 2008 to current. Is there a way to have rsync only sync those files?he directory structure that's my source goes all the way back to 2004.
im on 10.10(desktop) and mdadm was v2.8.1 from 2008, very out of date so i tried 3.2.1 -> no change. mdadm raid1 read speeds are the same as single disk. note i used the tests in the disk utility benchmarking tool at first --these showed raid 5 atleast to be much better but when i tried dd reads raid5 dropped off with larger data to almost the same (slow) speed as raid1. compare:
using two partitions will be enough to show raid1 performs at single disk speed. I dont really want to use a 4 disk raid0 just to get the read speed i should be able to get with raid1 as i dont really care about the size loss. I would of course use raid10 but i have found this suffers from the same problem (achieve same read speed as 2 disk stripe). So whilst im shocked others aren't reporting this, unless there is some obscure reason why my system would give these results i think raid1 in not behaving as it should.
I was wondering if there is a way to tell rsync to only apply changes (delete, overwrite,create) only if all files in the file list transferred successfully.Just to clarify, this would essentially be putting a transaction around the transfer.