Software :: How To Control File Order In Rsync

Mar 30, 2011

I am looking for a way to control the order in which files (or more specifically, certain subtrees) are updated for a very large rsync update done daily. There is a finite time frame and finite bandwidth to do these updates, and they usually do NOT complete. I want to control the order so that more important files are copied earlier.

What I have been doing is breaking up the rsync runs into a few steps where the first step is a very limited set of subtrees using --include and/or --exclude, and each step thereafter is increasing the amount, until the last step is the whole tree to be updated. This is not working because the time it takes to transfer the file list is large (varying from 10 minutes to 40 minutes depending on step). I really need to get this back down to doing a single rsync run to reduce that wasted time.

I cannot ultimately split things because there are a lot of hardlinks that need to be retained (or else the destination space and bandwidth would be exceeded).

I was looking at rsync's batch mode. My thought was if I could get a list of what needs to be updated, I could then sort that list and do just 1 more rsync run to do the updates in the order I want. But it turns out the batch mode file is in some cryptic binary format. I really have no idea if that could have worked.

The order selection is not simply based on the names in the top directory, so I cannot simply just list those in the preferred order on the command line (otherwise, yes, that would control the order). If I do list directory names from various depths, that list gets flattened into the destination directory, which would not work to replicate the whole tree exactly.

View 6 Replies


ADVERTISEMENT

Debian :: Execute Command Mactime In Order To Control Changes Made To The File System?

Apr 23, 2011

I am trying to execute command mactime in order to control changes made to the file system, but I am getting an error. I am running the shell as root, and it is the first time I run the command in this system - Debian Squeeze up to date. The I/O is as follows:

# mactime 3/1/2011
cannot exec /bin/date: No such file or directory
cannot exec /bin/hostname: No such file or directory
cannot exec /bin/uname -n: No such file or directory
Cannot open /var/cache/tct/data/Amnesiac/body: No such file or directory

The first three lines of output are the ones that are worrying at the moment; I am not concerned with the last. Although error messages suggest the programs date, hostname and uname do not exist in /bin directory, they are available as shown by

[Code]...

View 3 Replies View Related

General :: Order Of Transfer All Files Under A Directory By Rsync

Sep 25, 2010

If transfer all files under a directory by rsync, what is the order that rsync determines to transfer the files one by one?At first it looked like rsync transfers files in alphabetical order, but later I found rsync skipped some files in the first sweep through the alphabetic order, and then went back to transfer files that were skipped in the first time and this time still in alphabetic order.

View 1 Replies View Related

OpenSUSE :: Control Load Order In KDE When Booting?

Feb 20, 2010

How do I control load order in KDE when booting? I would like to load networkmanager 1st and some widget like "remember the milk" last in order. I have a conflict between two ("Remember the milk" is bad coded - but only remote-syncing ToDo widget that I know). When RtM loads, if network is not ready, authentication fails. Then only way to connect it is to kill widget and place it again. whatever the reason, how do I control load order of apps/widgets in KDE?

View 1 Replies View Related

Hardware :: How To Control The Driver Order Be Loaded

Apr 4, 2011

How to control the driver order be loaded by linux.

For example: I have two ethernet card is A and B. Sometimes A be loaded first sometimes another. How to control A card be load first in every times?

View 5 Replies View Related

Ubuntu :: PCMan File Manager File/folder Numbers In Wrong Order?

Jul 25, 2011

I have recently switched to using LXDE on my PC and I am on the whole pretty pleased with it. However,PCMan is giving me a really odd problem. Some of the files/folders are being displayed in the wrong order where they contain numbers. They are being ordered by their first digit not the whole number.

[Code]...

View 5 Replies View Related

Ubuntu :: Find The Dependent Information Like That In The Control File Of .deb File?

Nov 17, 2010

As I know, lly includes a control file. Takelibfreetype6_2.3.9-5ubuntu0.4_i386.deb for example, its control file includes the following content:

Package: libfreetype6
Source: freetype
Version: 2.3.9-5ubuntu0.4

[code]...

View 2 Replies View Related

General :: Use Sort Command For Its Ascending Order And Descending Order?

Aug 18, 2010

in my text file of 5 columns 2 column is in DDMMYYYY format. (ie DATE OF BIRTH). how to use sort command for its ascending order and descending order.

Is the same can be use for the date format YYYYMMDD ?

View 3 Replies View Related

General :: RSYNC Error .tar.gz 14gb File / Transfer Huge 14 Gb File Over Network,vpn,wan Transfer?

Mar 10, 2010

i am trying to transfer a file from my live linux machine to remote linux machine it is a mail server and single .tar.gz file include all data. but during transfer it stop working. how can i work and trouble shooot the matter. is there any better way then this to transfer huge 14 gb file over network,vpn,wan transfer. the speed is 1mbps,rest of the file it copy it.

rsync -avz --stats bkup_1.tar.gz root@10.1.1.22:/var/opt/bkup

[root@sa1 logs_os_backup]# less remote.log
Wed Mar 10 09:12:01 AST 2010
building file list ... done
bkup_1.tar.gz
deflate on token returned 0 (87164 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(274)
building file list ... done
code....

View 1 Replies View Related

Debian :: Rsync And Different File Systems ?

Oct 21, 2010

I have 2 different mounts. One points to a local windows share(NTFS ->Samba) and the other one points to a PPTP VPN connection sharing(I belive that is NTFS too). I use "cifs" scheme in my fstab to mount these. And I use my Debian box to copy between these 2 mounts. I have started using Rsync for that purpose, I think that it works fine for now. My main problem is that it looks like Rsync cannot figure out if the files are same or not in source and target folders when I use these mounts. Most of the time Rsync copies the same files and folders over and over again even though those files and folders are on the target.

I am wondering if there is a way to make this scheme work? Being on a Vpn connection(slow) a Windows box, Rsync could have save a lot of my time if it could have recognized the files and folders that are same on both ends

View 4 Replies View Related

Ubuntu :: Rsync Log File With Data?

Jul 1, 2011

I have successfully backed up my files using a script to a remote server with a log file output.However the log file is appended each time.I wish to have a different log file each time with date and time and have yet to figure that part out.

View 2 Replies View Related

CentOS 5 :: Time Changes In Rsync Log File

Mar 18, 2010

I'm running rsync in daemon mode on CentOS 5.4, and I'm seeing inconsistent times reported in the log file.

View 2 Replies View Related

Ubuntu Servers :: Advanced Rsync - Make The Rsync Set Or Keep The User And Group Affiliations?

Nov 17, 2010

Thought I'd post it here because it's more server related than desktop... I have a script that does:

[Code]....

This is used to sync my local development snapshot with the live web server. There has to be a more compact way of doing this? Can I combine some of the rsyncs? Can I make the rsync set or keep the user and group affiliations? Can I exclude .* yet include .htaccess?

View 6 Replies View Related

Ubuntu :: Can't Rsync Gmail / Run Rsync --recursive --times --perms --links --delete?

Jan 7, 2011

When I run rsync --recursive --times --perms --links --delete --exclude-from='Documents/exclude.txt' ./ /media/myusb/

where Documents/exclude.txt is

- /Downloads/
- /Desktop/books/

the files in those directories are still copied onto my USB.

And...

I used fetchmail to download all my gmail emails. When I run rsync -ar --exclude-from='/home/xtheunknown0/Documents/exclude.txt' ./ /media/myusb/ I get the first image at url.

View 9 Replies View Related

Server :: Rsync Execution - With Crontab - Have Given Full Path To Rsync Too

Apr 12, 2011

I have a tiny shell script to rsync files between two servers and remove the source files.

This script works fine, when it has been initiated manually or even when the rsync command is executed on the command line.

But the same script doesn't work, when I try to automate it through crontab.

I am using 'abc' user to execute this rsync, instead of root, as root login to servers are restricted in all of our servers, by us.

As I mentioned earlier, manual execution works like charm!

When this rsync.sh is initiated through crontab, it runs the first command(chown abc.abc ...) perfectly without any issues. But the second line is not at all executed, and there is no log entry i can find at /mnt/xyz/folder/rsync.log.

View 6 Replies View Related

Software :: Run Rsync To Download Files From A Server Without Rsync Daemon?

Sep 18, 2009

I just tried to sync files from one server to another. After the sync process, I found the files are bigger than original ones.

I looked up the web and found someone mentions the rsync daemon. So I have to run the daemon on one server before I run the rsync?

The command I used is rsync --partial --progress -r source destination

View 1 Replies View Related

General :: Rsync - Password-file Not Working ?

May 23, 2011

I made a shell backup script that uses Rsync and I am trying to get rid of the password prompt because it will use a CRON to run. I have set my variable in my shell script at:

PASSWORD_FILE=rsync_password

And the password in that file only takes up 1 line.

However when I run (ignore $DESTINATION)

rsync -aRvz tmp $DESTINATION --password-file=$PASSWORD_FILE

It still gives me the prompt. How can I accomplish this? I cannot allow a prompt and I do not want to have to use keys.

View 1 Replies View Related

Ubuntu :: Rsync Warning - File Has Vanished ?

Mar 24, 2010

I'm always getting this error when I run the rsync command:

Code:

when I try to navigate the file:

Code:

it gives

Code:

View 2 Replies View Related

Ubuntu :: Rsync Sh File Don't Work With Cron

Mar 25, 2011

I have a backup sh file that I have been using for a long time. It has always worked. 2 Days back I switched to a different pc and now suddenly the script don't work.If I run it manually in the terminal it works. But when it execute with cron it doesn't copy any files to the backup destination. It starts but doesn't copy anything.Can someone help me as to why it works manually but not with cron ?

View 9 Replies View Related

Ubuntu :: Rsync Not Maintaining File Ownership?

May 16, 2011

I have an Ubuntu machine running NFS4 server and a plugapps (arch linux) machine connecting as the client. The plugbox is running an rsync job to backup the home directory from Ubuntu to a local USB HDD.

All of the files in the destination have owner nobody and group nobody.

Ubuntu /etc/exports:

Code:
/home 192.168.2.1/24(rw,sync,no_subtree_check,no_root_squash)
plugbox /etc/fstab

[Code].....

how I can mantain the file owners. I have the UID's and passwords sync'd between the two machines for both root and the user who's home dir is being backed up.

View 9 Replies View Related

General :: Rsync Command - For File Backup ?

Mar 23, 2011

I'm going to be using this command to back up my files:

Should I change anything or is it ok?

View 4 Replies View Related

Server :: RSync Without Delete Any File On The Destination?

Feb 16, 2010

I'm using rsync for a backup-sript at the moment and want to keep all files. The files are always unique, so I want to rsync without delete any file on the destination.

I've tried with --no-delete and --max-delete=0 but nothing seens to work. Is there even a possibility to do so?

View 2 Replies View Related

Software :: Transferring File Permissions Using Rsync

Feb 3, 2010

I am using rsync to backup data from a file server to a external device, the data is accessed from windows boxes via samba with their own usernames, user1, user2 etc... Rsync is failing to copy the file permissions when sending to the external device, I have tested sending to a ftp server and a usb hard drive and received the same error, see below:

Code:
root@Fileserver:~# rsync -avz --delete-after /shared/fileshare/ /backup/backup
building file list ... done
./
manager/
rsync: chown "/backup/backup/manager/cronman.txt" failed: Operation not permitted (1)
public/ .....
sent 339 bytes received 104 bytes 886.00 bytes/sec
total size is 4593 speedup is 10.37
rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]
When I backup to a local directory the permissions are copied correctly.

View 4 Replies View Related

Software :: Unable To Transfer File Using Rsync?

Dec 7, 2009

I am unable to transfer file using rsync. I have installed rsync and ssh package.. When I try to transfer file , its giving an error rsync -v -e ssh a usw3tvx@lalqau40:ssh:lalqau40: Name or service not knownrsync: connection unexpectedly closed (0 bytes received so far) [sender]rsync error: error in rsync protocol data stream (code 12) at io.c(463) [sender=2.6.8]I tried with simple format... nothing works...Even I couldnt ssh with hostname. However I can connect with Ip address... I

View 1 Replies View Related

Ubuntu Multimedia :: Can't Choose File Order On CD Burning?

Mar 18, 2010

I want to do is this: choose the order (other than alphabetical) of the files that I burn to a disc.Brasero doesn't allow it K3b doesn't allow it GnomeBaker doesn't allow it Unless I am missing something, I can't figure out any way other than renaming all my files with a single number and/or letter at the beginning of a file name to choose the order of files. This blows my mind it seems like the most "no duh" feature that everyone would need in any Cd burning program. if I am stupid or if the software developers are stupid. (because I really can't see any other options) This is seriously so important (to me), that I will abandon Linux and go back to Windows if I can't do it.

View 9 Replies View Related

Ubuntu :: Tweaking Nautilus' File Sorting Order?

Dec 31, 2010

I've recently switched from Windows to Ubuntu and have a question regarding tweaking the character order Nautilus uses for alphabetical sorting. In my music/graphics/etc folder hierarchies, I have used a hyphen at the start as a 'hack' to 'sticky' some folders above the rest for quicker access. This worked fine in Windows, but Nautilus ignores a hyphen in it's sorting calculations. Is there anyway, simple or complex to replicate this behaviour in Nautilus?

View 4 Replies View Related

Ubuntu :: Grub Boot Order No Menu.lst File?

Jan 21, 2011

I am trying to change the default boot in grub to the last OS, and cannot find the menu.lst file under the /boot/grub directory.

I have tried to use the locate command to find the menu.lst file, but it doesn't exist. I have grub 1.98 installed and was wondering if the file I need to edit might be under a different name.

I recently updated Ubuntu to 10.04 (lucid) Kernel linux 2.6.32-27-generic gnome 2.30.2

View 2 Replies View Related

Ubuntu :: File List Now In Reverse Alphabetical Order

May 26, 2011

I am running fully updated ubuntu 10.10 and now, since 2 days ago, when I open a file
in open office, gimp or gedit etc the directories and files now appear in reverse alphabetical order.

View 8 Replies View Related

Applications :: File Processing Based On Alphabetical Order?

Jan 22, 2009

In Linux, the files were processing based on timestamp. How to process the files based on alphabets? My application is in windows. Here I am processed the files based on alphabetical order. While coming to linux its coming wrong.

View 5 Replies View Related

General :: Sort .txt File By The Numerical Order Of Column?

Mar 22, 2010

I have this .txt file

3183630 R
3574216 W
5264994 M
2656889 A

I want to sort it by the numerical order of column. After sort, I wish to get output like

2656889 A
3183630 R
3574216 W
5264994 M

anyone knows how to use sort to do that? or any other command?

View 6 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved