Software :: How To Control File Order In Rsync
Mar 30, 2011
I am looking for a way to control the order in which files (or more specifically, certain subtrees) are updated for a very large rsync update done daily. There is a finite time frame and finite bandwidth to do these updates, and they usually do NOT complete. I want to control the order so that more important files are copied earlier.
What I have been doing is breaking up the rsync runs into a few steps where the first step is a very limited set of subtrees using --include and/or --exclude, and each step thereafter is increasing the amount, until the last step is the whole tree to be updated. This is not working because the time it takes to transfer the file list is large (varying from 10 minutes to 40 minutes depending on step). I really need to get this back down to doing a single rsync run to reduce that wasted time.
I cannot ultimately split things because there are a lot of hardlinks that need to be retained (or else the destination space and bandwidth would be exceeded).
I was looking at rsync's batch mode. My thought was if I could get a list of what needs to be updated, I could then sort that list and do just 1 more rsync run to do the updates in the order I want. But it turns out the batch mode file is in some cryptic binary format. I really have no idea if that could have worked.
The order selection is not simply based on the names in the top directory, so I cannot simply just list those in the preferred order on the command line (otherwise, yes, that would control the order). If I do list directory names from various depths, that list gets flattened into the destination directory, which would not work to replicate the whole tree exactly.
View 6 Replies
ADVERTISEMENT
Apr 23, 2011
I am trying to execute command mactime in order to control changes made to the file system, but I am getting an error. I am running the shell as root, and it is the first time I run the command in this system - Debian Squeeze up to date. The I/O is as follows:
# mactime 3/1/2011
cannot exec /bin/date: No such file or directory
cannot exec /bin/hostname: No such file or directory
cannot exec /bin/uname -n: No such file or directory
Cannot open /var/cache/tct/data/Amnesiac/body: No such file or directory
The first three lines of output are the ones that are worrying at the moment; I am not concerned with the last. Although error messages suggest the programs date, hostname and uname do not exist in /bin directory, they are available as shown by
[Code]...
View 3 Replies
View Related
Sep 25, 2010
If transfer all files under a directory by rsync, what is the order that rsync determines to transfer the files one by one?At first it looked like rsync transfers files in alphabetical order, but later I found rsync skipped some files in the first sweep through the alphabetic order, and then went back to transfer files that were skipped in the first time and this time still in alphabetic order.
View 1 Replies
View Related
Feb 20, 2010
How do I control load order in KDE when booting? I would like to load networkmanager 1st and some widget like "remember the milk" last in order. I have a conflict between two ("Remember the milk" is bad coded - but only remote-syncing ToDo widget that I know). When RtM loads, if network is not ready, authentication fails. Then only way to connect it is to kill widget and place it again. whatever the reason, how do I control load order of apps/widgets in KDE?
View 1 Replies
View Related
Apr 4, 2011
How to control the driver order be loaded by linux.
For example: I have two ethernet card is A and B. Sometimes A be loaded first sometimes another. How to control A card be load first in every times?
View 5 Replies
View Related
Jul 25, 2011
I have recently switched to using LXDE on my PC and I am on the whole pretty pleased with it. However,PCMan is giving me a really odd problem. Some of the files/folders are being displayed in the wrong order where they contain numbers. They are being ordered by their first digit not the whole number.
[Code]...
View 5 Replies
View Related
Nov 17, 2010
As I know, lly includes a control file. Takelibfreetype6_2.3.9-5ubuntu0.4_i386.deb for example, its control file includes the following content:
Package: libfreetype6
Source: freetype
Version: 2.3.9-5ubuntu0.4
[code]...
View 2 Replies
View Related
Aug 18, 2010
in my text file of 5 columns 2 column is in DDMMYYYY format. (ie DATE OF BIRTH). how to use sort command for its ascending order and descending order.
Is the same can be use for the date format YYYYMMDD ?
View 3 Replies
View Related
Mar 10, 2010
i am trying to transfer a file from my live linux machine to remote linux machine it is a mail server and single .tar.gz file include all data. but during transfer it stop working. how can i work and trouble shooot the matter. is there any better way then this to transfer huge 14 gb file over network,vpn,wan transfer. the speed is 1mbps,rest of the file it copy it.
rsync -avz --stats bkup_1.tar.gz root@10.1.1.22:/var/opt/bkup
[root@sa1 logs_os_backup]# less remote.log
Wed Mar 10 09:12:01 AST 2010
building file list ... done
bkup_1.tar.gz
deflate on token returned 0 (87164 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(274)
building file list ... done
code....
View 1 Replies
View Related
Oct 21, 2010
I have 2 different mounts. One points to a local windows share(NTFS ->Samba) and the other one points to a PPTP VPN connection sharing(I belive that is NTFS too). I use "cifs" scheme in my fstab to mount these. And I use my Debian box to copy between these 2 mounts. I have started using Rsync for that purpose, I think that it works fine for now. My main problem is that it looks like Rsync cannot figure out if the files are same or not in source and target folders when I use these mounts. Most of the time Rsync copies the same files and folders over and over again even though those files and folders are on the target.
I am wondering if there is a way to make this scheme work? Being on a Vpn connection(slow) a Windows box, Rsync could have save a lot of my time if it could have recognized the files and folders that are same on both ends
View 4 Replies
View Related
Jul 1, 2011
I have successfully backed up my files using a script to a remote server with a log file output.However the log file is appended each time.I wish to have a different log file each time with date and time and have yet to figure that part out.
View 2 Replies
View Related
Mar 18, 2010
I'm running rsync in daemon mode on CentOS 5.4, and I'm seeing inconsistent times reported in the log file.
View 2 Replies
View Related
Nov 17, 2010
Thought I'd post it here because it's more server related than desktop... I have a script that does:
[Code]....
This is used to sync my local development snapshot with the live web server. There has to be a more compact way of doing this? Can I combine some of the rsyncs? Can I make the rsync set or keep the user and group affiliations? Can I exclude .* yet include .htaccess?
View 6 Replies
View Related
Jan 7, 2011
When I run rsync --recursive --times --perms --links --delete --exclude-from='Documents/exclude.txt' ./ /media/myusb/
where Documents/exclude.txt is
- /Downloads/
- /Desktop/books/
the files in those directories are still copied onto my USB.
And...
I used fetchmail to download all my gmail emails. When I run rsync -ar --exclude-from='/home/xtheunknown0/Documents/exclude.txt' ./ /media/myusb/ I get the first image at url.
View 9 Replies
View Related
Apr 12, 2011
I have a tiny shell script to rsync files between two servers and remove the source files.
This script works fine, when it has been initiated manually or even when the rsync command is executed on the command line.
But the same script doesn't work, when I try to automate it through crontab.
I am using 'abc' user to execute this rsync, instead of root, as root login to servers are restricted in all of our servers, by us.
As I mentioned earlier, manual execution works like charm!
When this rsync.sh is initiated through crontab, it runs the first command(chown abc.abc ...) perfectly without any issues. But the second line is not at all executed, and there is no log entry i can find at /mnt/xyz/folder/rsync.log.
View 6 Replies
View Related
Sep 18, 2009
I just tried to sync files from one server to another. After the sync process, I found the files are bigger than original ones.
I looked up the web and found someone mentions the rsync daemon. So I have to run the daemon on one server before I run the rsync?
The command I used is rsync --partial --progress -r source destination
View 1 Replies
View Related
May 23, 2011
I made a shell backup script that uses Rsync and I am trying to get rid of the password prompt because it will use a CRON to run. I have set my variable in my shell script at:
PASSWORD_FILE=rsync_password
And the password in that file only takes up 1 line.
However when I run (ignore $DESTINATION)
rsync -aRvz tmp $DESTINATION --password-file=$PASSWORD_FILE
It still gives me the prompt. How can I accomplish this? I cannot allow a prompt and I do not want to have to use keys.
View 1 Replies
View Related
Mar 24, 2010
I'm always getting this error when I run the rsync command:
Code:
when I try to navigate the file:
Code:
it gives
Code:
View 2 Replies
View Related
Mar 25, 2011
I have a backup sh file that I have been using for a long time. It has always worked. 2 Days back I switched to a different pc and now suddenly the script don't work.If I run it manually in the terminal it works. But when it execute with cron it doesn't copy any files to the backup destination. It starts but doesn't copy anything.Can someone help me as to why it works manually but not with cron ?
View 9 Replies
View Related
May 16, 2011
I have an Ubuntu machine running NFS4 server and a plugapps (arch linux) machine connecting as the client. The plugbox is running an rsync job to backup the home directory from Ubuntu to a local USB HDD.
All of the files in the destination have owner nobody and group nobody.
Ubuntu /etc/exports:
Code:
/home 192.168.2.1/24(rw,sync,no_subtree_check,no_root_squash)
plugbox /etc/fstab
[Code].....
how I can mantain the file owners. I have the UID's and passwords sync'd between the two machines for both root and the user who's home dir is being backed up.
View 9 Replies
View Related
Mar 23, 2011
I'm going to be using this command to back up my files:
Should I change anything or is it ok?
View 4 Replies
View Related
Feb 16, 2010
I'm using rsync for a backup-sript at the moment and want to keep all files. The files are always unique, so I want to rsync without delete any file on the destination.
I've tried with --no-delete and --max-delete=0 but nothing seens to work. Is there even a possibility to do so?
View 2 Replies
View Related
Feb 3, 2010
I am using rsync to backup data from a file server to a external device, the data is accessed from windows boxes via samba with their own usernames, user1, user2 etc... Rsync is failing to copy the file permissions when sending to the external device, I have tested sending to a ftp server and a usb hard drive and received the same error, see below:
Code:
root@Fileserver:~# rsync -avz --delete-after /shared/fileshare/ /backup/backup
building file list ... done
./
manager/
rsync: chown "/backup/backup/manager/cronman.txt" failed: Operation not permitted (1)
public/ .....
sent 339 bytes received 104 bytes 886.00 bytes/sec
total size is 4593 speedup is 10.37
rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]
When I backup to a local directory the permissions are copied correctly.
View 4 Replies
View Related
Dec 7, 2009
I am unable to transfer file using rsync. I have installed rsync and ssh package.. When I try to transfer file , its giving an error rsync -v -e ssh a usw3tvx@lalqau40:ssh:lalqau40: Name or service not knownrsync: connection unexpectedly closed (0 bytes received so far) [sender]rsync error: error in rsync protocol data stream (code 12) at io.c(463) [sender=2.6.8]I tried with simple format... nothing works...Even I couldnt ssh with hostname. However I can connect with Ip address... I
View 1 Replies
View Related
Mar 18, 2010
I want to do is this: choose the order (other than alphabetical) of the files that I burn to a disc.Brasero doesn't allow it K3b doesn't allow it GnomeBaker doesn't allow it Unless I am missing something, I can't figure out any way other than renaming all my files with a single number and/or letter at the beginning of a file name to choose the order of files. This blows my mind it seems like the most "no duh" feature that everyone would need in any Cd burning program. if I am stupid or if the software developers are stupid. (because I really can't see any other options) This is seriously so important (to me), that I will abandon Linux and go back to Windows if I can't do it.
View 9 Replies
View Related
Dec 31, 2010
I've recently switched from Windows to Ubuntu and have a question regarding tweaking the character order Nautilus uses for alphabetical sorting. In my music/graphics/etc folder hierarchies, I have used a hyphen at the start as a 'hack' to 'sticky' some folders above the rest for quicker access. This worked fine in Windows, but Nautilus ignores a hyphen in it's sorting calculations. Is there anyway, simple or complex to replicate this behaviour in Nautilus?
View 4 Replies
View Related
Jan 21, 2011
I am trying to change the default boot in grub to the last OS, and cannot find the menu.lst file under the /boot/grub directory.
I have tried to use the locate command to find the menu.lst file, but it doesn't exist. I have grub 1.98 installed and was wondering if the file I need to edit might be under a different name.
I recently updated Ubuntu to 10.04 (lucid) Kernel linux 2.6.32-27-generic gnome 2.30.2
View 2 Replies
View Related
May 26, 2011
I am running fully updated ubuntu 10.10 and now, since 2 days ago, when I open a file
in open office, gimp or gedit etc the directories and files now appear in reverse alphabetical order.
View 8 Replies
View Related
Jan 22, 2009
In Linux, the files were processing based on timestamp. How to process the files based on alphabets? My application is in windows. Here I am processed the files based on alphabetical order. While coming to linux its coming wrong.
View 5 Replies
View Related
Mar 22, 2010
I have this .txt file
3183630 R
3574216 W
5264994 M
2656889 A
I want to sort it by the numerical order of column. After sort, I wish to get output like
2656889 A
3183630 R
3574216 W
5264994 M
anyone knows how to use sort to do that? or any other command?
View 6 Replies
View Related