Recently i am trying to check on the rsync speed for single file(2.4GB iso) directory ( 900MB directory with files inside ) When i run the rsync for single file: the speed i get is average 50MBps However, for a directory: average speed is 10MBps Is there any reason behind this ? i tried to google but unable to get the concept.
View 2 Replies
What would be the easiest way to go about rsyncing the n newest files in a directory to a remote server?
View 2 Replies View RelatedIf transfer all files under a directory by rsync, what is the order that rsync determines to transfer the files one by one?At first it looked like rsync transfers files in alphabetical order, but later I found rsync skipped some files in the first sweep through the alphabetic order, and then went back to transfer files that were skipped in the first time and this time still in alphabetic order.
View 1 Replies View RelatedWe have an rsync cron job set up to mirror all the files in a "..dashtdocsdocs" folder to the same folder on another server. It copies all the files over correctly and deletes any files in the "docs" directory that aren't in the sending directory, but it also deletes any files we put in the target directory's parent folder (..dashtdocs or other subfolders like ..dashtdocsimages) even though they've been excluded in the .rsync-filter file.
Here's our current rsync .sh:
/usr/local/bin/rsync --stats -qPzrtpl --delete --password-file=/var/run/.appprodrsync --log-file=/export/home
/webuser/logs/rsync-extranet-log -FF /export/home/ appprodrsync@appprod::dprweb_extranet/ > /export/home/webuser/logs/rsync-extranet-output 2>&1
[code]....
So for example server A has ..dashtdocsdocs and ..dashtdocsimages. Server B has ..dashtdocsdocs but if I manually copy the images folder over to ..dashtdocsimages, the images folder gets deleted from the target directory every time rsync runs.
A: ..dashtdocsdocs
..dashtdocsimages
B: ..dashtdocsdocs
..dashtdocsimages (<-- gets deleted everytime rsync runs)
I'd like to keep just the docs directory synched and update other folders manually, but they keep getting deleted. It looks to me like it's running a delete-excluded option, but that option wasn't used.
i would like to find and backup all *.mp4 files from /Pictures and its sub-directories and move them to a single directory on a remote. I can find and move the files but I don't want the directory structure...just the files to be placed in the remote directory.
To find my files I use
rsync -r -a -v -e "ssh -l user" --delete --include '*/' --include '*.mp4' --exclude '*' /home/drew/Pictures/ remoteserver:/Users/drew/mp4
but this creates all the subdirectories
I also tried
find ~/Pictures -name "*.mp4" -exec rsync -r -a -v -e "ssh -l user" --delete {} remote:/Users/drew/mp4 ;
This works but takes forever
How can I use rsync to copy ONLY the my home folder (and nothing inside of it, just the folder name) to another machine. I've tried things like
Code:
rsync -av /path/to/src /path/to/dest/
or
Code:
rsync -av -f"+ */" -f"- *" /path/to/src /path/to/dest/
This last option recursively (through the -a switch) copies only folders, including all subfolders. If I try
Code:
rsync -v -f"+ */" -f"- *" /path/to/src /path/to/dest/
nothing is copied (not even my home folder.
In reading the rsync man page and browsing a lot of websites, I ended up a bit confused, or maybe it was just too much eggnog. Anyway, to exclude a directory "videos" with everything in it, which is /home/user1/camera/videos and I'm rsyncing the whole user1 directory to an external drive
[code]...
I want to copy a big file from my harddrive to a removable drive by rsync. For some other reason, the operation cannot complete in a single run. So I am trying to figure out how to use rsync to resume file copying from where was left last time.
I have tried use the option --partial or --inplace, but together with --progress, I found rsync with --partial or --inplace actually starts from the beginning instead of from what was left last time. Mannually early stopping rsync and checking the size of the received file also confirm what I found.
But with --append, rsync starts from what was left last time. I am confused as I saw on the manpage --partial, --inplace or --append seem to relate to resuming copying from what was left last time. Is someone able to explain their difference? Why --partial or --inplace do not work for resuming copying? Is it true that for resuming copying, rsync has to work with option --append?
Also if a partial file was left by mv or cp not by rsync, will rsync --append correctly resume the file copying?
I have a number of crash.log files scattered about my system and I would like to run a command to find all the crash.log files on the system and copy them to a single directory; each with a unique filename. For example, copy crash.log from ~/directory_1 , ~/directory_2 , ~/directory_3 and so on to ~/crash_logs/crash.log1 , ~/crash_logs/crash.log2 , ~/crash_logs/crash.log3 etc.
View 1 Replies View RelatedI am performing a dry run using Rsync on 2 different boxes.While i'm doing that, Under destination directory, I want a specific directory x to be ignored for sync.Please let me know the exact pattern to ignore the directory.The current command I'm using is:rsync -avnc --delete $LOCAL_DIR $USERNAME@$DESTINATION_IP:$REMOTE_DIRunder DESTINATION_IP, I would want to ignore a particular directory under REMOTE_DIR.
View 1 Replies View RelatedI just tried to sync files from one server to another. After the sync process, I found the files are bigger than original ones.
I looked up the web and found someone mentions the rsync daemon. So I have to run the daemon on one server before I run the rsync?
The command I used is rsync --partial --progress -r source destination
I have a WD world book edition 1TB NAS drive, and just purchased an acomdata 1tb drive and connected it to the NAS via USB. If I recall I think the WD NAS has a ext_ or some type of linux filesystem on it, and the acomdata has a ntfs filesystem on it.
What I want to do is copy over certain directory trees of the NAS to the USB attached drive. I usually use MS synctoy to sync folders from my windows pc to the NAS drive, and MS richcopy to make the initial transfer from PC to NAS. For this operation though, since it is taking place entirely on the NAS and its connected drive, I thought that rsync would be the best option, and it is available on my NAS drive.
Last night I entered in rsync -avr /movies/* /usb1-1share1/ to copy the entire "movies" dir to the drive, which shows up as usb1-1share1 on the NAS drive. It copied most of the directory tree ok, but a lot of the folders were empty, so this morning I tried rsync -Carv --ignore-existing /movies/* /usb1-1share1/ to try and get all the files missed, without recopying the 24GB that did make it across. This also managed to copy a few more GB over, but not everything.
I am running the command from an ssh session on the NAS using putty on my PC, in as user "admin" which should have all rights over these folders. There is a bunch of errors in the command window like this: rsync: failed to set times on "/shares/usb1-1share1/movies/classics/fulldvd/First Blood DVD/.VTS_01_2.VOB.RxdjWZ": Operation not permitted (1)
I want to restart another session and get the files it missed, but I want to find out what I am doing wrong first. Should I be doing this as root user? am I missing some switches or just plain doing it all wrong?
Is there a way to force rsync to not make directories in its destination directory; ie, to simply dump all of the files from the source directory directly into the destination without copying any of the folders that the files were originally in? I tried --no-dirs, but that seems to only be for empty directories.
View 1 Replies View Relateduse rsync to cp such files and dirs under /var/www/html/mydir directory but these two files(/dir4/1.html /dir4/2.html) cant rsync to dest mechine.
rsync configure file,below...
Why would rsync insert a user's home directory path in variable expansion when run via cron, but not when run manually. The gory details... Red Hat Enterprise Linux AS release 4 (Nahant Update 6) Linux 2.6.9-67.0.20.ELsmp The script (parts anyway, and simplified)...
Quote:
#!/bin/bash
. /home/bea/.bash_profile
echo rsyncloc=${rsyncloc} >> ${log}
[code]....
I'm trying to setup rsync to backup a remote directory to my local drive.
I cd to the directory that I want to pull the files to, then I enter:
rsync -vrtW account@remote.com:~/public_html
I enter the password then it starts running. I get all the files listed, but none of them actually transfer. What am I missing?
Long story short, I have a failing RAID3 array which is showing file corrupted blocks (and the RAID controller card is periodically not initializing). [URL].. I thought I had robust backups but as it turns out, my backup volumes seem to have been misplaced (don't ask ) so I have no viable backups. I'm trying to back up as much as I can before the inevitable impending catastrophic failure.
I must be doing something wrong because running rsync on my Unix FreeBSD/FreeNAS server (syncing to local USB drive) is really slooooow. Below you will see an example. A 500MB file took almost 10 minutes to sync to a local USB drive! I can FTP this file in a few minutes over my LAN.
Just ran a test and it took 20 seconds to FTP this file across my gigabit LAN where rsync took 10 minutes to perform a local copy)
[Code]...
I'd like to copy a file, say widgets/water.txt, to all subfolders in the folder widgets using a single command. So if the folder widgets has 10 subfolders like widgets/blue, widgets/green, etc. I'd like to copy water.txt to all of them with one command.
I tried the commands
Code:
cp water.txt ./*/water.txt
cp water.txt ./*/
However these don't seem to work. The latter gives 'cp: omitting directory' errors.
I am trying to write a very simple script that will go to every subdirectory of a single directory and run a command (lets call it make_ndx).I know I can write this the long way with in a text document with something like:
cd /"the directory"/"the 1st subdirectory"
make_ndx
cd ..
cd "the 2nd subdirectory"
cd ..
Alternatively, I also tried: for i in 'find /path/somemorepath -type d -mindepth 1'; do cd $i; make_ndx -f *.gro; done which returns me with the error cd: find: no such file or directory. But if I run the find command by itself to test if I am calling the right directories, it gives me the exactly the output I am looking for. Any ideas? Should I just write the find results to a file and loop through the contents of the file (which seems a little bit like overkill) or am I just making a simple typographical mistake and I am just not seeing it?
selectively archive the subdirectories of a single parent directory in which said subdirectories have creation year-date 2008 and older.
View 5 Replies View RelatedI'm trying to using rsync to backup some files, about half a TB. It's now it a state where it keeps sending the same files everytime it runs. for example:
rsync -av /data/source/* user@host:/data/dest
sending incremental file list
source/file1.txt
source/file2.txt
I then verify those files are copied over. then the next time it runs it does the same thing
rsync -av /data/source/* user@host:/data/dest
sending incremental file list
source/file1.txt
source/file2.txt
any idea why it's getting stuck on these files? I've tried to wipe the whole dest directory out and start over but no luck.
I have two directories, dirA whicht contains N gb of data and dirB which is supposed to contain only the newest M gb of data from dirA. When files are added to dirA, they sould also be added to dirB, while the oldest files in dirB should be deleted.Is that possible with rsync? or any other software?
View 1 Replies View Relatedrsync -r -v -e ssh root@nn.nn.nn.nn:/usr/local/websites/* /usr/local/websites and each time I run it it copies everything - all files. I thought rsync was only supposed to copy files that had been added or modified.
View 3 Replies View RelatedI'm planing to copy a productive mysql innodb file from one server to another, and the file size is around 300GB. As the file is keeping changing all the time, I have to shutdown mysql instance and copy the large data file to other server as quickly as possible.I should have to find a way to speed up file copying ... I'm wondering whether there's a way to copy file block by block.If the destination side block has same content, then bypass it.
View 4 Replies View RelatedI'm using the command below to sync two directories. Problem is insted of deleting the files on the target directory it simply appends a ~ character at the end of the file name. Not sure why this is happening?I'd like to have all deletes on the source replicated on target.
View 2 Replies View RelatedThis is a quick one, I don't have any problems yet, I just want to check that this is going to do what I think it is. I added the line
Code:
00 01 * * * rsync -avz --delete /local-storage /mnt/usbackup
to my crontab file, am I to understand that this will backup /local-storage to my external
[code]...
I was going to do a rsync -r -a -z -v -p -e sshto move some files frome server to another, but then realized all I really need are files which have dates starting June 1, 2008 to current. Is there a way to have rsync only sync those files?he directory structure that's my source goes all the way back to 2004.
View 5 Replies View Relatedim on 10.10(desktop) and mdadm was v2.8.1 from 2008, very out of date so i tried 3.2.1 -> no change. mdadm raid1 read speeds are the same as single disk. note i used the tests in the disk utility benchmarking tool at first --these showed raid 5 atleast to be much better but when i tried dd reads raid5 dropped off with larger data to almost the same (slow) speed as raid1. compare:
[code]....
using two partitions will be enough to show raid1 performs at single disk speed. I dont really want to use a 4 disk raid0 just to get the read speed i should be able to get with raid1 as i dont really care about the size loss. I would of course use raid10 but i have found this suffers from the same problem (achieve same read speed as 2 disk stripe). So whilst im shocked others aren't reporting this, unless there is some obscure reason why my system would give these results i think raid1 in not behaving as it should.
I have an external sata dock for hdd that give me a lot of error till linux decided to lower the speed of it to 1.5 then it start work well
View 2 Replies View RelatedI was wondering if there is a way to tell rsync to only apply changes (delete, overwrite,create) only if all files in the file list transferred successfully.Just to clarify, this would essentially be putting a transaction around the transfer.
View 3 Replies View Related
