General :: Rsync -z Vs Scp ... Which One Is Faster Over Network
Dec 3, 2010
I often have to transfer huge data over our LAN from one computer to another. The size of the files varies and can be somewhere from 2 GB to 50GB or more! The only accepted connection protocol between machines are ssh; and rsync and scp are the only options available for copying over network (unison is not installed). I usually use rsync with "-z" option to copy over network. if "rsync -z" is faster than "scp" for data transfer?
View 5 Replies
ADVERTISEMENT
May 9, 2010
I am currently backing up my data but find that it takes way to long to do a rsync, it takes forever to just find the differences and transfer them.Out of 3 separate rsyncs the main one that is slow is my www.skins.be mirror directory which is 41GB and has 392,200 files, sorted into multiple directories. Which grows by around 100 every couple days.I think that something that would be able to track changes by inotify time on directories will speed it up since Picasa sure finds the changes fast when I open it and it is tracking over 26,200 pictures. I just don't know of a backup solution that does that.
View 4 Replies
View Related
Oct 22, 2010
I use openvpn to connect otherwise isolated machines, and use samba to share filesystems across the vpn, which works just fine.But I recently discovered that copying files using rsync -e ssh is so much faster than copying from a mounted filesystem - like about 5 times faster.I've got comp-lzo enabled in both server and the client, at least I think I have, the directive is there in both the server.conf and the client.conf files, but how do I check that it's active?Does anyone know if I can make openvpn behave more like rsync, because copying is easier than rsyncing?
View 8 Replies
View Related
Apr 23, 2011
I'm using Scientific Linux on a laptop, connecting to a debian server on my LAN via a basic BT HomeHub router. When starting a file transfer with ftp or rsync from the laptop my connection is dropped, every time, after a couple of seconds. I have to reset my network adaptor and restart my network manager (wicd in this case) in order to get network connectivity back. I had a very similar problem before - [URL]. On this occasion my workaround was to replace NetworkManager with yast. I am now using SL though so no yast available, and I am not using NetworkManager any more anyway.
View 3 Replies
View Related
Jan 27, 2010
I recently built a small Intel Atom 330 based server for my home. I'm using the Vortexbox Fedora-based OS to run the server (primarily used as a media server). So far everything is working great. In addition to my media server, I've got a DLink DNS-321 NAS. I would like to setup a scheduled, incremental backup of my main server to the DLink NAS. I understand rsync is an excellent option, and am willing to undertake the task of setting it up on my server, but I am uncertain how to make it all happen with the DLink NAS. The NAS is very barebones, and I don't know if I can even install rsync on it. I don't even know if I can get to any kind of command line on the NAS box. 1. Can I mount the NAS drive on my main Linux server and then just run rsync on the server?
View 7 Replies
View Related
May 13, 2011
I made a script to backup file from each host with general password in local network. This script using SSH Pass and Rsync with this
syntax:
rsync --rsh="sshpass -p password ssh -l root" hostath destinationpath
Everything is okay under 9.10 version until I migrate to Ubuntu 11.04, there is always give an error:
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(541) [Receiver=3.0.7]
I am using bash version: GNU bash, version 4.2.8(1)-release (i686-pc-linux-gnu) and 2.6.38-8-generic kernel
View 3 Replies
View Related
Feb 24, 2011
Just tried to rsync /repositories/server:/monitoring for SLE 11.1 which is visible through http [URL], but when I browse through [URL] the SLE_11.1 directory isn't there.
View 3 Replies
View Related
Jul 9, 2010
A bunch of hosts all connected to each other.I'd like to have a directory where a group of hosts can dump files and all access, synchronising with each other their changes automatically and transparent to the user.How can I have it so that rsync updates both ways?
* host A asks host B for a list of files, modification dates, checksums
* then applies the rules to select which files it will fetch from B
--> if A owns the file then do not get modifications from B
--> files differ, then select newer file
* then fetches them
* and so on for each host...
View 1 Replies
View Related
Jun 14, 2010
I have an old laptop (PIII 800MHz, with 256 RAM) that I wish to use as my home server: it'll have to serve just two people, so I think that I'll be more than ok as for the RAM and the CPU.The issue is about data, because the internal hard disk is a 12GB, that is...ridicolous! I have more than 60GB of mixed storage and counting (images, videos and music) in an external usb hd. I could put the hd in my desktop pc just to serve the big files through ethernet or let it inside its usb box attached to the laptop.
The question is: which of these solutions will be the fastest? USB 1.0 attached to the server (laptop) or internal hard disk serving files via 10/100 ethernet to the laptop on demand?what about your experience? Is the difference based on a human notable scale?
View 3 Replies
View Related
Mar 13, 2011
Why is it faster to have more primary partitions when using Linux? Please give some real examples; I know some theoretical reasons but I don't understand them well.
View 2 Replies
View Related
Aug 19, 2010
I have a 4 * 1.5TB RAID5 disk array (software linux RAID, formatted with jfs) on my Fedora 12 system and want to expand it by adding another 1.5TB disk. I have added a drive to the system and conducted a simple performance check on it to make sure it was functioning properly:
Code:
# dd if=/tmp/bigfile.dat of=/dev/sdg1
5478774+1 records in
5478774+1 records out
2805132609 bytes (2.8 GB) copied, 168.77 s, 16.6 MB/s
But 16.6 MB/s is lousy. I ran an iostat -dmx 2 on this drive at the time of this lousy performance, and typical output was:
Code:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 1.50 0.00 258.50 0.00 16.25 0.00 128.74 0.17 0.66 0.36 9.30
sdg 0.00 3556.00 4160.00 32.00 16.25 16.00 15.76 135.73 35.11 0.24 100.05
(note that sda and sdb are a linux raid mirror set for the / filesystem that holds /tmp). I formatted the new drive (/dev/sdg1) with jfs and mounted it under /mnt2:
Code:
# jfs_mkfs /dev/sdg1
jfs_mkfs version 1.1.13, 17-Jul-2008
Warning! All data on device /dev/sdg1 will be lost!
Continue? (Y/N) y
[code].....
I would like to know what the performance will be like before I add the disk to the array because I don't want to wait for the whole array to be rebuilt before finding out my array is performing badly. The array is used as part of a mythtv system and has up to 6 simultaneous recordings running on it, so it needs to perform well.
View 6 Replies
View Related
Jul 27, 2010
i run ubuntu 10.04 on my old pc pentium 4, 500mb ram ,how can ido for make faster..
View 5 Replies
View Related
Jul 27, 2011
how can I make my system's booting faster. Is Linux support to increase it?
View 1 Replies
View Related
Aug 7, 2010
Does anyone have any info on the status of the Max Planck Institute server ftp5.gwdg.de? I maintain selected local copies of openSUSE updates and KDE etc. using an rsync:/ connection to this server but it seems to have been unavailable for the last couple of days at least - it may have been longer but I've had hardware problems on my home server recently.
View 9 Replies
View Related
May 27, 2010
I have a newly installed Dell Optiplex 755 with Ubuntu 10.04 (64-bit) and I am having serious issues with network copying. Whenever I start rsync och scp with larger amounts of data my system becomes practically unresponsive (all types of apps grey out) until those processes are completed. Cpu stays at below 10% and I have lots of free memory and such .
View 3 Replies
View Related
Mar 10, 2010
i am trying to transfer a file from my live linux machine to remote linux machine it is a mail server and single .tar.gz file include all data. but during transfer it stop working. how can i work and trouble shooot the matter. is there any better way then this to transfer huge 14 gb file over network,vpn,wan transfer. the speed is 1mbps,rest of the file it copy it.
rsync -avz --stats bkup_1.tar.gz root@10.1.1.22:/var/opt/bkup
[root@sa1 logs_os_backup]# less remote.log
Wed Mar 10 09:12:01 AST 2010
building file list ... done
bkup_1.tar.gz
deflate on token returned 0 (87164 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(274)
building file list ... done
code....
View 1 Replies
View Related
Aug 9, 2011
I find myself grepping the same codebase over and over. While it works great, each command takes about 10 seconds, so I am thinking about ways to make it faster.So can grep use some sort of index? I understand an index probably won't help for complicated regexps, but I use mostly very simple patters. Does an indexer exist for this case?EDIT: I know about ctags and the like, but I would like to do full-text search.
View 5 Replies
View Related
Dec 15, 2010
I'm using Linux Mint 8. However, internet browser speed is very slow as compared to using the same computer with Windows. In Linux, I've used Firefox 3.5 and Epiphany 2.28. Epiphany runs somewhat faster than Firefox so I mainly use it for the internet.
What I would like to know is if Epiphany browser 2.30 is noticably faster than 2.28? Should I go thru the trouble of upgrading to Mint 9 inorder to be able to use Epiphany 2.3 - as well as Chromium 8?
View 2 Replies
View Related
Feb 26, 2011
I am currently facing a problem with rsync (or maybe ssh, I don't know yet). I have daily backups that are done via cron using rsync from my computer (client) to a NAS. And systematically during these backups my computer completely freezes (no remote login possible, nothing), so I have to hard reboot. As you might imagine, I cannot do this every day. In /var/log/messages there is just a line like that :
Feb 26 17:23:53 eggplant rsyslogd: -- MARK --
right before the hang. The rsync worked well before and I don't really know what changed in my system config. In /var/log/rsyncd.log I sometimes (but not every time) I get :
2011/02/26 16:39:19 [13229] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(543) [Receiver=3.0.7]
I aldready update the default rsync from opensuse 11.3 to 3.0.7.7, but it does not help. How can I at least know what is going wrong? I am trying to track down the problem but don't know how to start. Last also a log of the rsync command (client) running ssh in verbose using the command "rsync -av --delete --timeout=180 -e "ssh -v -p xxxxxx -i /root/.ssh/rsync-key_eggplant" /mds root@xxxxxxxx:/volume1/autoBackup/eggplant":
OpenSSH_5.4p1, OpenSSL 1.0.0 29 Mar 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
[code]....
View 9 Replies
View Related
Apr 24, 2010
I've been working with Macs nowadays and noticed that programs load like "instantly" once it has been open and closed.
I don't know if MAC is pre-fetching them to somewhere but it is good to start Firefox right after you closed it.
what MAC actually does and how this could be implemented ? I have 6gb of ram on my laptop. Running Kubuntu 10.04 RC1.
View 3 Replies
View Related
Oct 25, 2010
I am using Red Hat Linux enterpriser 4 using two physical LAN cards.There are two different ISP internet lines coming into a single gateway computer. Is it possible that if in our one network we are using 2 MB bandwidth and from another network we are getting 2 MB bandwidth.(Two different Networks) and combine it into 4 MB. In bonding it combine the bandwidth or it do loadbalancing ?
The question is can we accumulate or add these two bandwidth in Linux machine and it sent total bandwidth 4 to the end users ?
Lan Card A
IP Address: 192.168.1.250 2MB Bandwidth
l
l
l
[Code]....
View 2 Replies
View Related
Nov 17, 2010
Thought I'd post it here because it's more server related than desktop... I have a script that does:
[Code]....
This is used to sync my local development snapshot with the live web server. There has to be a more compact way of doing this? Can I combine some of the rsyncs? Can I make the rsync set or keep the user and group affiliations? Can I exclude .* yet include .htaccess?
View 6 Replies
View Related
Jan 7, 2011
When I run rsync --recursive --times --perms --links --delete --exclude-from='Documents/exclude.txt' ./ /media/myusb/
where Documents/exclude.txt is
- /Downloads/
- /Desktop/books/
the files in those directories are still copied onto my USB.
And...
I used fetchmail to download all my gmail emails. When I run rsync -ar --exclude-from='/home/xtheunknown0/Documents/exclude.txt' ./ /media/myusb/ I get the first image at url.
View 9 Replies
View Related
Apr 12, 2011
I have a tiny shell script to rsync files between two servers and remove the source files.
This script works fine, when it has been initiated manually or even when the rsync command is executed on the command line.
But the same script doesn't work, when I try to automate it through crontab.
I am using 'abc' user to execute this rsync, instead of root, as root login to servers are restricted in all of our servers, by us.
As I mentioned earlier, manual execution works like charm!
When this rsync.sh is initiated through crontab, it runs the first command(chown abc.abc ...) perfectly without any issues. But the second line is not at all executed, and there is no log entry i can find at /mnt/xyz/folder/rsync.log.
View 6 Replies
View Related
Sep 18, 2009
I just tried to sync files from one server to another. After the sync process, I found the files are bigger than original ones.
I looked up the web and found someone mentions the rsync daemon. So I have to run the daemon on one server before I run the rsync?
The command I used is rsync --partial --progress -r source destination
View 1 Replies
View Related
Jul 21, 2010
use rsync to cp such files and dirs under /var/www/html/mydir directory but these two files(/dir4/1.html /dir4/2.html) cant rsync to dest mechine.
rsync configure file,below...
View 2 Replies
View Related
Dec 8, 2010
I'm using Ubuntu 10.04 LTS server and Postgresql 8.4. I have a .sh script that is run by cron every other hour. That works fine. The .sh script includes an rsync command that copies a postgresql dump .tar file to a remote archive location via ssh. That fails when run by cron; I think because it is (quietly) asking for the remote user's password (and not getting it). I set up the public/private ssh key arrangement. The script succeeds when run manually as the same user that the cron job uses, and does not ask for the password. I am able to ssh to the remote server from the source server (using the same username) and not get the password prompt (both directions), so why doesn't rsync work? I even put a .pgpass file in the root of that user's directory with that user's password, and the user/password are identical on both servers.
I think the problem is rsync is not able to use the ssh key correctly. I tried adding this to my script but it didn't help.
Code:
Here is the rsync command embedding in the .sh script.
Code:
Here is the cron entry:
Code:
View 6 Replies
View Related
May 4, 2011
I don't understand the results of a simple performance test I ran using two basic scripts (running on a high end server):
perfVar.zsh :
#!/bin/zsh -f
MYVAR=`cat $1`
for i in {1..10}
[code]...
Performance test result:
> time ./perfVar.zsh BigTextFile > /dev/null
./perfVar.zsh FE > /dev/null 6.86s user 0.32s system 100% cpu 7.177 total
> time ./perfCat.zsh BigTextFile > /dev/null
./perfCat.zsh FE > /dev/null 0.01s user 0.10s system 91% cpu 0.118 total
I would have thought that accessing a VARIABLE was way faster than reading a FILE on the file system... Why this result ?Is there a way to optimize the perfCat.zsh script by reducing the number of accesses to the file system ?
View 3 Replies
View Related
Feb 24, 2011
I am trying to run a rsync backup Script to backup my data from my HD to my NAS Drive (currently mounted as Samba share, but can change to NFS if that would improve things, if I can work out how!), but when I run the Script I get the following error on every file it tries to copy
Code:
As I mentioned I am mounting my Share from my NAS as Samba and below is the fstab entry
Code:
The rsync command with options is as follows
Code:
The error seems to be with the -a option, but I have no idea why.
View 7 Replies
View Related
May 26, 2010
i have a cheap-arse employer who doesn't provide any data backup for us. so i've setup my own very simple server with ubuntu 10.04 (no raid). from my linux desktop i want to backup to my file server. from my file server i configured samba to allow me to access my data from the many Windows PCs in the place. I also have a USB drive for when i go home or to another building.
most of my work happens on my linux desktop. so i use rsync to 'backup' from there to the file server. can i use rsync to keep my usb drive also sync'ed? in the same command?
when i access files on my file server from a Windows PC via samba, and change something, will rsync pick up that change on the server, and migrate the changes back to my linux desktop?
View 4 Replies
View Related
