I am looking for a way to control the order in which files (or more specifically, certain subtrees) are updated for a very large rsync update done daily. There is a finite time frame and finite bandwidth to do these updates, and they usually do NOT complete. I want to control the order so that more important files are copied earlier.
What I have been doing is breaking up the rsync runs into a few steps where the first step is a very limited set of subtrees using --include and/or --exclude, and each step thereafter is increasing the amount, until the last step is the whole tree to be updated. This is not working because the time it takes to transfer the file list is large (varying from 10 minutes to 40 minutes depending on step). I really need to get this back down to doing a single rsync run to reduce that wasted time.
I cannot ultimately split things because there are a lot of hardlinks that need to be retained (or else the destination space and bandwidth would be exceeded).
I was looking at rsync's batch mode. My thought was if I could get a list of what needs to be updated, I could then sort that list and do just 1 more rsync run to do the updates in the order I want. But it turns out the batch mode file is in some cryptic binary format. I really have no idea if that could have worked.
The order selection is not simply based on the names in the top directory, so I cannot simply just list those in the preferred order on the command line (otherwise, yes, that would control the order). If I do list directory names from various depths, that list gets flattened into the destination directory, which would not work to replicate the whole tree exactly.
I am trying to execute command mactime in order to control changes made to the file system, but I am getting an error. I am running the shell as root, and it is the first time I run the command in this system - Debian Squeeze up to date. The I/O is as follows:
# mactime 3/1/2011 cannot exec /bin/date: No such file or directory cannot exec /bin/hostname: No such file or directory cannot exec /bin/uname -n: No such file or directory Cannot open /var/cache/tct/data/Amnesiac/body: No such file or directory
The first three lines of output are the ones that are worrying at the moment; I am not concerned with the last. Although error messages suggest the programs date, hostname and uname do not exist in /bin directory, they are available as shown by
If transfer all files under a directory by rsync, what is the order that rsync determines to transfer the files one by one?At first it looked like rsync transfers files in alphabetical order, but later I found rsync skipped some files in the first sweep through the alphabetic order, and then went back to transfer files that were skipped in the first time and this time still in alphabetic order.
How do I control load order in KDE when booting? I would like to load networkmanager 1st and some widget like "remember the milk" last in order. I have a conflict between two ("Remember the milk" is bad coded - but only remote-syncing ToDo widget that I know). When RtM loads, if network is not ready, authentication fails. Then only way to connect it is to kill widget and place it again. whatever the reason, how do I control load order of apps/widgets in KDE?
I have recently switched to using LXDE on my PC and I am on the whole pretty pleased with it. However,PCMan is giving me a really odd problem. Some of the files/folders are being displayed in the wrong order where they contain numbers. They are being ordered by their first digit not the whole number.
i am trying to transfer a file from my live linux machine to remote linux machine it is a mail server and single .tar.gz file include all data. but during transfer it stop working. how can i work and trouble shooot the matter. is there any better way then this to transfer huge 14 gb file over network,vpn,wan transfer. the speed is 1mbps,rest of the file it copy it.
[root@sa1 logs_os_backup]# less remote.log Wed Mar 10 09:12:01 AST 2010 building file list ... done bkup_1.tar.gz deflate on token returned 0 (87164 bytes left) rsync error: error in rsync protocol data stream (code 12) at token.c(274) building file list ... done code....
I have 2 different mounts. One points to a local windows share(NTFS ->Samba) and the other one points to a PPTP VPN connection sharing(I belive that is NTFS too). I use "cifs" scheme in my fstab to mount these. And I use my Debian box to copy between these 2 mounts. I have started using Rsync for that purpose, I think that it works fine for now. My main problem is that it looks like Rsync cannot figure out if the files are same or not in source and target folders when I use these mounts. Most of the time Rsync copies the same files and folders over and over again even though those files and folders are on the target.
I am wondering if there is a way to make this scheme work? Being on a Vpn connection(slow) a Windows box, Rsync could have save a lot of my time if it could have recognized the files and folders that are same on both ends
I have successfully backed up my files using a script to a remote server with a log file output.However the log file is appended each time.I wish to have a different log file each time with date and time and have yet to figure that part out.
Thought I'd post it here because it's more server related than desktop... I have a script that does:
[Code]....
This is used to sync my local development snapshot with the live web server. There has to be a more compact way of doing this? Can I combine some of the rsyncs? Can I make the rsync set or keep the user and group affiliations? Can I exclude .* yet include .htaccess?
When I run rsync --recursive --times --perms --links --delete --exclude-from='Documents/exclude.txt' ./ /media/myusb/
where Documents/exclude.txt is
- /Downloads/ - /Desktop/books/
the files in those directories are still copied onto my USB.
And...
I used fetchmail to download all my gmail emails. When I run rsync -ar --exclude-from='/home/xtheunknown0/Documents/exclude.txt' ./ /media/myusb/ I get the first image at url.
I have a tiny shell script to rsync files between two servers and remove the source files.
This script works fine, when it has been initiated manually or even when the rsync command is executed on the command line.
But the same script doesn't work, when I try to automate it through crontab.
I am using 'abc' user to execute this rsync, instead of root, as root login to servers are restricted in all of our servers, by us.
As I mentioned earlier, manual execution works like charm!
When this rsync.sh is initiated through crontab, it runs the first command(chown abc.abc ...) perfectly without any issues. But the second line is not at all executed, and there is no log entry i can find at /mnt/xyz/folder/rsync.log.
I made a shell backup script that uses Rsync and I am trying to get rid of the password prompt because it will use a CRON to run. I have set my variable in my shell script at:
PASSWORD_FILE=rsync_password
And the password in that file only takes up 1 line.
I have a backup sh file that I have been using for a long time. It has always worked. 2 Days back I switched to a different pc and now suddenly the script don't work.If I run it manually in the terminal it works. But when it execute with cron it doesn't copy any files to the backup destination. It starts but doesn't copy anything.Can someone help me as to why it works manually but not with cron ?
I have an Ubuntu machine running NFS4 server and a plugapps (arch linux) machine connecting as the client. The plugbox is running an rsync job to backup the home directory from Ubuntu to a local USB HDD.
All of the files in the destination have owner nobody and group nobody.
how I can mantain the file owners. I have the UID's and passwords sync'd between the two machines for both root and the user who's home dir is being backed up.
I'm using rsync for a backup-sript at the moment and want to keep all files. The files are always unique, so I want to rsync without delete any file on the destination.
I've tried with --no-delete and --max-delete=0 but nothing seens to work. Is there even a possibility to do so?
I am using rsync to backup data from a file server to a external device, the data is accessed from windows boxes via samba with their own usernames, user1, user2 etc... Rsync is failing to copy the file permissions when sending to the external device, I have tested sending to a ftp server and a usb hard drive and received the same error, see below:
Code: root@Fileserver:~# rsync -avz --delete-after /shared/fileshare/ /backup/backup building file list ... done ./ manager/ rsync: chown "/backup/backup/manager/cronman.txt" failed: Operation not permitted (1) public/ ..... sent 339 bytes received 104 bytes 886.00 bytes/sec total size is 4593 speedup is 10.37 rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9] When I backup to a local directory the permissions are copied correctly.
I am unable to transfer file using rsync. I have installed rsync and ssh package.. When I try to transfer file , its giving an error rsync -v -e ssh a usw3tvx@lalqau40:ssh:lalqau40: Name or service not knownrsync: connection unexpectedly closed (0 bytes received so far) [sender]rsync error: error in rsync protocol data stream (code 12) at io.c(463) [sender=2.6.8]I tried with simple format... nothing works...Even I couldnt ssh with hostname. However I can connect with Ip address... I
I want to do is this: choose the order (other than alphabetical) of the files that I burn to a disc.Brasero doesn't allow it K3b doesn't allow it GnomeBaker doesn't allow it Unless I am missing something, I can't figure out any way other than renaming all my files with a single number and/or letter at the beginning of a file name to choose the order of files. This blows my mind it seems like the most "no duh" feature that everyone would need in any Cd burning program. if I am stupid or if the software developers are stupid. (because I really can't see any other options) This is seriously so important (to me), that I will abandon Linux and go back to Windows if I can't do it.
I've recently switched from Windows to Ubuntu and have a question regarding tweaking the character order Nautilus uses for alphabetical sorting. In my music/graphics/etc folder hierarchies, I have used a hyphen at the start as a 'hack' to 'sticky' some folders above the rest for quicker access. This worked fine in Windows, but Nautilus ignores a hyphen in it's sorting calculations. Is there anyway, simple or complex to replicate this behaviour in Nautilus?
I am trying to change the default boot in grub to the last OS, and cannot find the menu.lst file under the /boot/grub directory.
I have tried to use the locate command to find the menu.lst file, but it doesn't exist. I have grub 1.98 installed and was wondering if the file I need to edit might be under a different name.
I recently updated Ubuntu to 10.04 (lucid) Kernel linux 2.6.32-27-generic gnome 2.30.2
I am running fully updated ubuntu 10.10 and now, since 2 days ago, when I open a file in open office, gimp or gedit etc the directories and files now appear in reverse alphabetical order.
In Linux, the files were processing based on timestamp. How to process the files based on alphabets? My application is in windows. Here I am processed the files based on alphabetical order. While coming to linux its coming wrong.
