General :: Files Seem To Take Up More Space In Destination After Rsync Copy?

Feb 9, 2010

I have recently purchased an external hard drive in order to backup my home partition. In my PC I have a "1.5T" drive with several partitions on it, containing OSes and the home partition. The home partition is 1.3T according to df, the external drive contains one partition that spans the entire disk,df reports it as 1.4T in size. Both partitions are ext3. When I use rsync to copy files from the home partition to the external partition, the external disk becomes full, despite the destination - supposedly - being larger than the source. I don't understand why copying files from one partition to a slightly bigger partition should need more space than on the source partition. Does anyone know what is happening ?

Details : I created the partition on the external drive with gparted; gparted reported it the already have several gigabytes in used space immediately after the partitions creation - I thought at the time that this must be normal. The home partition contains many files of all sorts, including lots of big audio and video files. If you are wondering, for all my important files this external disk is only secondary backup, as they are also backed up to the "internet".

These are the mount points :

/mnt/tmp/ : home partition, /dev/sdb6
/mnt/external/ : external partition, /dev/sdc1

I used rsync to copy the files, I know there are more efficient ways to do this, but I wanted to use the same command that I will subsequently run to sync the backup.

rsync -av --progress --stats --recursive --perms --links --delete /mnt/tmp/ /mnt/external/

Next I tried adding the --sparse switch, as I was wondering if the problem may come form sparse files. I don't know however if rsync would go back and shrink the sparse file by just adding the switch and executing the command. I also added --one-file-system, for good measure. Here is what I ran next :

rsync -av --progress --stats --sparse --one-file-system --recursive --perms --links --delete /mnt/tmp/ /mnt/external/

I tried an fsck on the home partition :

fsck -f /dev/sdb6

This is the output from the last rsync :

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: write failed on "abcd.avi": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.6]

[code]....

Looking at the destination after a partial copy seems to indicate that the problem is not symbolic links being "expanded". I have not checked the source filesystem for sparse files, nor the destination to see if these files could be larger there, as this does not seem trivial.

Here is some additional info :

$ df /mnt/tmp/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb6 1415342836 1414173740 369096 100% /mnt/tmp

[code]....

View 2 Replies


ADVERTISEMENT

General :: Rsync Deleting Files From Destination Directory?

Jul 14, 2011

We have an rsync cron job set up to mirror all the files in a "..dashtdocsdocs" folder to the same folder on another server. It copies all the files over correctly and deletes any files in the "docs" directory that aren't in the sending directory, but it also deletes any files we put in the target directory's parent folder (..dashtdocs or other subfolders like ..dashtdocsimages) even though they've been excluded in the .rsync-filter file.

Here's our current rsync .sh:

/usr/local/bin/rsync --stats -qPzrtpl --delete --password-file=/var/run/.appprodrsync --log-file=/export/home
/webuser/logs/rsync-extranet-log -FF /export/home/ appprodrsync@appprod::dprweb_extranet/ > /export/home/webuser/logs/rsync-extranet-output 2>&1

[code]....

So for example server A has ..dashtdocsdocs and ..dashtdocsimages. Server B has ..dashtdocsdocs but if I manually copy the images folder over to ..dashtdocsimages, the images folder gets deleted from the target directory every time rsync runs.

A: ..dashtdocsdocs
..dashtdocsimages
B: ..dashtdocsdocs
..dashtdocsimages (<-- gets deleted everytime rsync runs)

I'd like to keep just the docs directory synched and update other folders manually, but they keep getting deleted. It looks to me like it's running a delete-excluded option, but that option wasn't used.

View 2 Replies View Related

Fedora :: Use Rsync To Backup My Music Collection - Add Files To The Destination Drive?

Mar 6, 2010

I'm currently learning to use rsync to backup my music collection. I have a Firefox tab open to the rsync manual page(s) and have been reading man rsync and running experimental rsync operations.I've been doing this for the last 3-4 hours. I've used rsync for this purpose in the past with disastrous results. What was and is once again (due to a month and a half of file pruning) a 9000 file music collection had mysteriously grown to over 25,000 music files and 80GB of data! This was likely due to the fact that I didn't really know what I was doing with rsync and had never spent too much time learning about all the parameters, what their functions are and how they may relate to my goal.Here are the particulars:

* Source drive is a 500GB disk, /media/sata500/music/.

* Destination drive is a 250GB USB disk, /media/FreeAgent/music, connected to the same computer that houses the 500GB disk.

* I want to copy or backup files from /media/sata500/music to /media/FreeAgent/music.

* I do not want to create ANY duplicates of files that exist.

* I only want to add files to the destination drive if they are new on the source drive, like if I rip a CD and add the contents to the source. I want them copied over next time I run rsync.

Here's the rsync command in it's most recently used form, and probably very immature at this point.

Code:

rsync -t -r -vv --stats -i --log-file=/home/glenn/rsync.log /media/sata500/music/* /media/FreeAgent/music/

This appears to have copied all files and folders and I'm satisfied that my goal has been met with some success. To convince myself of this I ran the command and then once it was complete I added 2 new songs putting them in their respective folders on the source drive and ran the same command again. The resulting output was

[code]....

Two files transferred. Exactly what I want.Both folders now house 20,931 files and use 40.6GB. Identical as far as I can tell.What I'm concerned about are time stamps and play count data, etc. Anything that changes the original file. I don't want this data to cause a file to be transferred as I'm afraid that the new file will be created along side the old file of the same name thereby starting this whole music collection expansion thing all over again. I've invested a lot of time and effort to get it pruned down to where there are virtually no duplicates and albums are correct in that they contain the proper songs in the proper order.

View 14 Replies View Related

Software :: Copy Files From One Source To Multiple Destination Simultaneously?

Aug 5, 2010

i want to copy one or more files or directory from one drive to multiple drive simultaneously. It like a cloning a disk. But i dont like clone entire disk. i want to copy/clone only certine files or folder. if any one can know how to copy one source to multiple destination simultaneously.

View 3 Replies View Related

General :: RSYNC - Ignore A Directory On Destination Box?

Jul 28, 2011

I am performing a dry run using Rsync on 2 different boxes.While i'm doing that, Under destination directory, I want a specific directory x to be ignored for sync.Please let me know the exact pattern to ignore the directory.The current command I'm using is:rsync -avnc --delete $LOCAL_DIR $USERNAME@$DESTINATION_IP:$REMOTE_DIRunder DESTINATION_IP, I would want to ignore a particular directory under REMOTE_DIR.

View 1 Replies View Related

Software :: When Copy Files Between Two Samba Shares From Windows Machine \ Copies The File Through PC To The New Destination?

Apr 17, 2011

I recently replaced my windows fileserver with one running Ubuntu. One thing I've noticed (which is a annoying) is that when I copy files between two samba shares from my windows machine, it copies the file through my PC to the new destination. On windows shares it just did some sort of local copy (ie it took about 2 seconds) rather than 3-4 minutes. Is this the normal behaviour, is there any way around it on Linux

View 5 Replies View Related

Ubuntu :: 5,149,552 Files In 30.2 GB And RSYNC Out Of Space?

Jun 9, 2011

I am running Ubuntu 10.04. I am transferring roughly 62 GB of data libraries to my 84 GB /home partition. I'm using rsync because scp kept stalling, and I had to restart it over and over. Things were going great until recently when it began to show an error: "failed: No space left on device (28)" These are the things I've done so far: Used the GUI to find out how much I have copied so far: 5,149,552 which take up 30.2 GB. df -h, it tells me that my /home partition is 56% used, and that I have 33 GB available. (42 GB used out of 78 GB with 33 GB available) Also, none of my other partitions are anywhere near 100%. the /home partition is the most-used and it's only a little over half-full. du -s in the directory where I'm copying all of this: it also returned 42206500. Additionally, when I try to save screen captures, it sometimes fails with a "device full" error. What's going on? Am I really out of space? Why doesn't it show me that I'm out of space?

Is there a hidden temp file that rsync uses that just got too full? I did a little research on wikipedia and it said that ext4 has a 64,000 directory limit. Could it be that I somehow broke that limit with all of these files? Solution: not enough inodes for the vast amount of subdirectories on hard drive. This wasn't an RSYNC problem, rather, a partition configuration issue. To check inode usage: df -i If you want to add any inodes, you will need to backup your partition and format it using mke2fs (man mke2fs). Be sure to change the respective inode setting.

View 9 Replies View Related

General :: Run Rsync On Server A To Copy All Files From Server B When They Are Newer Than 7 Days

Jun 14, 2011

I want to run rsync on server A to copy all files from Server B when they are newer than 7 days.(find . -mtime -7) I don't want to delete the files on Server B.

View 2 Replies View Related

Ubuntu :: Use Rsync To Copy ONLY Home Directory And No Files?

Jun 1, 2011

How can I use rsync to copy ONLY the my home folder (and nothing inside of it, just the folder name) to another machine. I've tried things like

Code:

rsync -av /path/to/src /path/to/dest/

or

Code:

rsync -av -f"+ */" -f"- *" /path/to/src /path/to/dest/

This last option recursively (through the -a switch) copies only folders, including all subfolders. If I try

Code:

rsync -v -f"+ */" -f"- *" /path/to/src /path/to/dest/

nothing is copied (not even my home folder.

View 9 Replies View Related

Software :: Rsync Can't Copy Certain Files With Unusual Characters While Cp Can

Aug 20, 2010

On my Ubuntu box, I have a mounted windows share connected via gvfs called graphics. I want to backup everything on a nightly basis from graphics to backupserver/graphics . If I use rsync, it will not copy files that have parent directories with funky characters in them (but the directories themselves will be copied!). Everything else gets rysnced just fine.

graphics/test/macdir/picture.psd
...when rsynced over to ...
backupserver/graphics/

gives the error:

rsync: mkstemp "/home/administrator/.gvfs/drobo on x.x.x.x/linux_backups/graphics/test/ macdir/.picture.psd" failed: Operation not supported (95)

The directory macdir gets created but there is nothing in there. This happens for all files underneath dirs with funky names. cp -Rf works perfectly! Directory and child files all get copied over no matter how strange the characters get in the directory names.

View 7 Replies View Related

Ubuntu :: Use Scp In Order To Copy Files To Android Phone - Error "destination Does Not Exist"

Feb 14, 2011

I am trying to use scp in order to copy files from an ubuntu machine to my Android phone. I use connectbot to log in to my ubuntu. However, when I use scp I get an error as if the destination does not exist. The destination is /mnt/sdcard/download on myAndroid 2.2.

View 1 Replies View Related

Programming :: Shell Script To Copy Newly Changed Files With Rsync?

Apr 20, 2010

I've got quite a decent rsync script setup, however I'd like to invoke it whenever there's change to a file. My initial idea was to use find, however this has two major flaws - the first being my particular unix veriant cant understand -print0 which means this doesn't work, the second is that I'm not 100% sure how to put variables into quotation marks so ls can understand the target:

Code:

for i in `find /shares/ -mtime -1 -print`; do ls -ltr $i;done

View 14 Replies View Related

General :: Copy File Recursively Ignoring Destination Directory Structure?

Jul 8, 2011

I have the following content on the source directory:

source/foo.txt
source/bar/another_file.txt
source/bar2/and_another.txt

I want copy those files to a destination directory which, after copy, shall look like this:

destination/foo.txt
destination/another_file.txt
destination/and_another.txt

How can I do this? It seems that "cp" lacks such an option

View 1 Replies View Related

Server :: RSync Without Delete Any File On The Destination?

Feb 16, 2010

I'm using rsync for a backup-sript at the moment and want to keep all files. The files are always unique, so I want to rsync without delete any file on the destination.

I've tried with --no-delete and --max-delete=0 but nothing seens to work. Is there even a possibility to do so?

View 2 Replies View Related

General :: Copy Certain File Types Recursively While Maintaining File Structure On Destination?

Jun 14, 2011

I have just been bothered by a fairly small issue for some time now. I am trying to search (using find -name) for some .jpg files recursively. This is a Redhat environment with bash.

I get this job done though I need to copy ALL of them and put them in a separate folder BUT I also need to keep the order intact after copying.

For e.g - If I get a JPG file under /home/usr/new/1/ then the destination also needs to be /test/old/new/1/.

At the moment, I am simply putting all files under /test/old/ and I can't somehow get the later /new/1/ folder path created under /test/old/

I understand this could well be done using while OR if else loop, though if someone can just guide me with a hint, I would be really grateful.

I will complete the rest of the steps and was asking here since I am still not comfortable with the shell/bash scripts yet and planning to be really good at it over the next couple of months.

View 1 Replies View Related

General :: Copy File Whose Source Path Is Mentioned In File To Specified Destination

Jun 3, 2010

I had a situation in which the the path of the file to be copied is written in other file and I had to copy it using shell script..I can use cp $(cat /home/robert/location.txt) /media/sda1 on normal linux shell...But I am using buildroot script where $(cat /home/robert/location.txt) evaluate to nothing..is just blank..

View 1 Replies View Related

General :: Rsync To Copy Directory Tree From NAS Drive ?

Apr 10, 2010

I have a WD world book edition 1TB NAS drive, and just purchased an acomdata 1tb drive and connected it to the NAS via USB. If I recall I think the WD NAS has a ext_ or some type of linux filesystem on it, and the acomdata has a ntfs filesystem on it.

What I want to do is copy over certain directory trees of the NAS to the USB attached drive. I usually use MS synctoy to sync folders from my windows pc to the NAS drive, and MS richcopy to make the initial transfer from PC to NAS. For this operation though, since it is taking place entirely on the NAS and its connected drive, I thought that rsync would be the best option, and it is available on my NAS drive.

Last night I entered in rsync -avr /movies/* /usb1-1share1/ to copy the entire "movies" dir to the drive, which shows up as usb1-1share1 on the NAS drive. It copied most of the directory tree ok, but a lot of the folders were empty, so this morning I tried rsync -Carv --ignore-existing /movies/* /usb1-1share1/ to try and get all the files missed, without recopying the 24GB that did make it across. This also managed to copy a few more GB over, but not everything.

I am running the command from an ssh session on the NAS using putty on my PC, in as user "admin" which should have all rights over these folders. There is a bunch of errors in the command window like this: rsync: failed to set times on "/shares/usb1-1share1/movies/classics/fulldvd/First Blood DVD/.VTS_01_2.VOB.RxdjWZ": Operation not permitted (1)

I want to restart another session and get the files it missed, but I want to find out what I am doing wrong first. Should I be doing this as root user? am I missing some switches or just plain doing it all wrong?

View 3 Replies View Related

General :: Copying Files From Multiple Locations To One Destination

Feb 21, 2011

Suppose I have a tree structure like this:
/home/mahmood/sim/a/b/file1.cpp
/home/mahmood/sim/a/b/file2.h
/home/mahmood/sim/a/c/file3.txt
/home/mahmood/sim/d/file4.txt

How can I copy all of them to /home/mahmood/sim. So that when I run "ls" in /home/mahmood/sim, I see all files:
file1.cpp
file2.h
file3.txt
file4.txt

Can 'cp' search for all file and copy them in another folder?

View 3 Replies View Related

Programming :: Used The Output Of Search To Copy In Another Destination?

Dec 20, 2010

i want to used the output of search to copy in another destination examlpe. fine /home/* mtime -1 i want to used the output of the find command to copy an another destination

View 4 Replies View Related

General :: Renaming Files With Sed - Mv: Missing Destination File Operand After `$i'

Mar 20, 2011

Today i am trying to learn how to use sed. I set up a testing folder with the following files:

AAb.lol
AAc.lol
AAx.lol
test.sh

My goal is to create a script (test.sh) which renames all the files to their original name without AA. I want to end up with this:

b.lol
c.lol
x.lol
test.sh

sed seemed to be the perfect tool so i went ahead and created a script which i think should clear the job.

[Code].........

mv: missing destination file operand after `$i' From that 2nd line i can tell that $NewName is just empty. I also read something about sed needing the -e option for scripting purposes but i just don't understand it.

View 7 Replies View Related

General :: Scripting - Cp A List Of Files Which Starts With - Or Space Without - Or Space?

Oct 14, 2009

I have a list of files in a directory which starts with - or have space.

I need to copy them back without the space or -?

linux1[sim]% ls linux1[sim]% cp ./-* ./*

View 8 Replies View Related

Software :: Run Rsync To Download Files From A Server Without Rsync Daemon?

Sep 18, 2009

I just tried to sync files from one server to another. After the sync process, I found the files are bigger than original ones.

I looked up the web and found someone mentions the rsync daemon. So I have to run the daemon on one server before I run the rsync?

The command I used is rsync --partial --progress -r source destination

View 1 Replies View Related

Server :: Rsync Can Not Rsync Files With Include Filter

Jul 21, 2010

use rsync to cp such files and dirs under /var/www/html/mydir directory but these two files(/dir4/1.html /dir4/2.html) cant rsync to dest mechine.

rsync configure file,below...

View 2 Replies View Related

Debian :: Copy Anything It Says Enough Space Memory To Copy?

Jul 5, 2010

I am using windows xp and debian linux.In windows xp I am having around 25 gb offree memory but in linux if i copy anything it says enough space memory to copy

View 4 Replies View Related

General :: Rsync Would Keep Sending The Files Over And Over Again?

Mar 2, 2010

I'm trying to using rsync to backup some files, about half a TB. It's now it a state where it keeps sending the same files everytime it runs. for example:

rsync -av /data/source/* user@host:/data/dest
sending incremental file list
source/file1.txt
source/file2.txt

I then verify those files are copied over. then the next time it runs it does the same thing

rsync -av /data/source/* user@host:/data/dest
sending incremental file list
source/file1.txt
source/file2.txt

any idea why it's getting stuck on these files? I've tried to wipe the whole dest directory out and start over but no luck.

View 2 Replies View Related

General :: Rsync Only Newest N Gb Of Files?

Jul 27, 2011

I have two directories, dirA whicht contains N gb of data and dirB which is supposed to contain only the newest M gb of data from dirA. When files are added to dirA, they sould also be added to dirB, while the oldest files in dirB should be deleted.Is that possible with rsync? or any other software?

View 1 Replies View Related

General :: Rsync With Scp - Copies Everything - All Files

Jul 12, 2011

rsync -r -v -e ssh root@nn.nn.nn.nn:/usr/local/websites/* /usr/local/websites and each time I run it it copies everything - all files. I thought rsync was only supposed to copy files that had been added or modified.

View 3 Replies View Related

General :: Rsync N Newest Files In A Directory?

Nov 8, 2010

What would be the easiest way to go about rsyncing the n newest files in a directory to a remote server?

View 2 Replies View Related

General :: Rsync Not Deleting Files On Target

Sep 8, 2010

I'm using the command below to sync two directories. Problem is insted of deleting the files on the target directory it simply appends a ~ character at the end of the file name. Not sure why this is happening?I'd like to have all deletes on the source replicated on target.

View 2 Replies View Related

General :: Rsync, Crontab And Backing Up Files?

Apr 11, 2011

This is a quick one, I don't have any problems yet, I just want to check that this is going to do what I think it is. I added the line

Code:
00 01 * * * rsync -avz --delete /local-storage /mnt/usbackup
to my crontab file, am I to understand that this will backup /local-storage to my external

[code]...

View 4 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved