Software :: Need To Transfer Huge Number Of Files - Good FTP Program?
Oct 26, 2009
I need to transfer some massive amount of data (2.5terrabyte, many files, directory structure) to a embedded raid-box which has a minimal linux on it (some custom distro from western digital). We tried rsync (version 2.6.7), but it crashes because the filelist is too big for the ram available (fixed in later versions of rsync, but I don't know how to update, it's not debian based and there are no compiler tools). We tried nfs, but the max bandwidth produced is around 1 mb/sec (cpu bound?), so it'd take around 3 weeks this way. Samba has problems with big files (and we have some 20gb files in there).
SCP isn't installed, and would probably also be cpu bound due to encryption I think. So the only option left would be ftp, we're currently trying using ncftp with the command "put -R /path/to/data/" , but it's been running for over an hour, eating up most of the ram, and not using any bandwidth. I think it is still building a file list or something. FTP already worked for a single 20gb file with acceptable bandwidth of about 12mb/sec. Does anyone know a better ftp program (for console) that can start transferring some data or at least display an estimated time for the copy-preperation?
View 8 Replies
ADVERTISEMENT
Mar 10, 2010
i am trying to transfer a file from my live linux machine to remote linux machine it is a mail server and single .tar.gz file include all data. but during transfer it stop working. how can i work and trouble shooot the matter. is there any better way then this to transfer huge 14 gb file over network,vpn,wan transfer. the speed is 1mbps,rest of the file it copy it.
rsync -avz --stats bkup_1.tar.gz root@10.1.1.22:/var/opt/bkup
[root@sa1 logs_os_backup]# less remote.log
Wed Mar 10 09:12:01 AST 2010
building file list ... done
bkup_1.tar.gz
deflate on token returned 0 (87164 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(274)
building file list ... done
code....
View 1 Replies
View Related
Apr 30, 2010
Can someone recommend a good program for editing MP3 files? I need to shorten/fade in/fade out some tracks. Cool Edit Pro does nothing via VirtualBox!
View 4 Replies
View Related
Aug 25, 2011
I have recently rented a server which is running Ubuntu linux. I can connect to it through ssh. I've written a Java application that I would like to run on the server 24/7. How do I transfer the files from my computer to the server so I can run the program from the server?
View 11 Replies
View Related
Oct 20, 2010
I have two servers, one has an empty / and the other has a subdirectory with a large number (4 gig) with many, many files. I need a way to transfer the files en masse from the server with the large number of files to the one that is essentially blank.I don't have space on the used host to simply gzip all the files. I've googled this and see that there may be some combination of tar and/or gzip that will let me do this with some sort of redirection.
I really need and example line of how this can be accomplished. If my explanation seems rather sparse, I can supply more details.
View 3 Replies
View Related
Feb 13, 2010
Seldom use windows, but need to transfer huge music file from Linux to XP
View 3 Replies
View Related
Sep 12, 2010
Lastb often shows me a huge list of attempted ssh logins.Such as this excerpt:
Code:
admin ssh:notty Sat Sep 11 23:47 - 23:47 (00:00) 184-154-37-12.Huge-DNS.COM
root ssh:notty Sat Sep 11 23:47 - 23:47 (00:00) 184-154-37-12.Huge-DNS.COM
[code]....
View 14 Replies
View Related
Aug 21, 2010
I'm writing a user ranking module for a site. This ranking depends on some criterias and it's possible to set or unset any one of these criterias in order to consider them in calculating the user rank or not. And here's the way I've implemented the ranking calculation :
when I set one or more of the criterias to be considered in ranking , for each user in the system I insert one record for each criteria , for example : if I have 2 criterias and both are set and consider that I have two users , I'll have :
Ranking table
--------------
username | criteria | to_be_added | score
--------------------------------------------------
user1 | criteria1 | 1 | 0
[code]....
It means I just set the to_be_added field to 1 for all of them and leave the calculation of score for each criteria for each user to the time the user logins so that to prevent doing all these calculations at once , because there are a huge number of users ... But there is one problem , if I want to show for example the best user (based on the highest score) , the result can't always be true because some users might not logged in at that time and their score might be zero .
View 1 Replies
View Related
Jul 7, 2011
So I noticed today that my machines root hard drive had almost no space on it.
I did a disk usage analyzer and I found out my var/log folder is 95GB.
The large logs are:
can I just delete them? also how can I stop this from happening again?
View 1 Replies
View Related
Jan 29, 2016
I installed Debian on my laptop and now I had warning that my log files inside var are getting out of hands...Following files...
36G daemon.log
48G daemon.log.1
41G kern.log
55G kern.log.1
31G messages
42G messages.1
8.2G syslog
17G syslog.1
How can I clear this and set up properly so they don't take so much space?
View 6 Replies
View Related
Nov 19, 2010
how can i delete my log files? They are 131.2 GB! And i need space on my pc . And is it ok to delete it ?
View 6 Replies
View Related
Feb 12, 2011
I've started using the huge.s kernel and when i try to compile packages slackware complains about kernel headers but all i see is the smp header files on the slackware discs ?
View 2 Replies
View Related
Mar 10, 2010
I'm using fedora 12 64 bit and im looking for a program i can autotune spoken words. Does anyone have any suggestions? Ive heard of Autotalent but currently it doesn't support 64 bit linux.
View 2 Replies
View Related
Mar 5, 2010
Does anyone know of a good reminder program?
I want some sort of pop-up or message display for things like appointments, due dates, etc. Maybe send a notice to email. Preferably something that will go out about two weeks or more.
Doesn't have to be fancy or even gui based.
View 7 Replies
View Related
Mar 11, 2010
Anyone know of a good ftp program or best way to do this on ubuntu 9.10?
View 9 Replies
View Related
Jul 16, 2010
Ok, Noobe to Ubuntu, recently installed Lucid, did all the updates, Got my UT 99 loaded and working natively with TeamSpeak 3 (yea, even the sound works now) ... I really like the setup I have now. Can some one recommend a good backup program to use that will "completely" backup my current system "as installed"? I have installed "Back in Time", and it seems really easy to use, but I'm looking for something similar to (sorry have to say it "Windows System Restore").
I've tweaked and tweaked, updated my ALSA drivers, working openAL and got the latest ATI driver installed for my ATI 5770 ... even got a Xfi linux beta working from Creative (that was fun).. Something user friendly too, I'm learning, but still very new to Linux/commandline/Ubuntu.
View 9 Replies
View Related
Sep 1, 2011
Best one I've found so far is Glom, but it's extremely hard to use, anyone know of one better?
View 4 Replies
View Related
Jan 22, 2009
I am running Ubuntu 8.04 and 8.10. I want a good GUI encryption/decryption program.
View 5 Replies
View Related
Jul 25, 2010
I have a 60GB partition with / and home on it. I logged on yesterday and it gave me a warning saying that I had only 1.9 GB of disk space left. I ignored it for a day and assumed that i had too many videos and pics.But the next day i had not added any files or downloaded any software but i had 0B left. I used the disk usage analyser and found that 33GBs came from /var/log. It was from two log files. syslog and daemon.log 16.5GB each!! I opened them up and i found that this line of text was repeated hnundreds of thousands of times.
Code:
Jul 22 19:32:36 aulenback-desktop ntfs-3g[5315]: Failed to decompress file: Value too large for defined data type
[code]...
View 3 Replies
View Related
Jul 27, 2011
i have about 2 TB of 700mb avi files as data on disc want to spread it across two 2TB ext usb drives (sata 3.5 inside the housing) obviously i have to rip them to the laptop and then move to the ext hdd (omg laborious little task) am i better doing the ripping in meerkat or in a windows machine? files need to be accessible by W7, XP, and meerkat to vlc player. what should i format the discs to?
View 5 Replies
View Related
Dec 22, 2010
I am facing a strange problem in my server, One of my filesystem shows as 3.1G when I execute df -h command and the utilization shows as 83%, but when I cd to the directory /usr/local I could not find any huge files in that filesystem and I have searched for hidden files as well,
groupserver:~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda9 3.1G 2.5G 532M 83% /usr/local
groupserver:/usr/local # du -sh *
0 bin
93M abinav
[Code]...
View 2 Replies
View Related
Aug 24, 2011
Can anyone recommend a good and free UML program? I've been googling and the only thing that really caught my attention is Visual Paradigm.
View 1 Replies
View Related
Apr 23, 2010
What is a Good Lightscribe Labeling Program for Ubuntu 9.04 64-bit? I installed LIGHTSCRIBE SIMPLE LABELER but it's the biggest piece of crap program, you can't even set the Font Size, all it allows you to do is set the Font. I'm not a Linux Expert so I need a program that is easy to install and does not require compiling or complex manual installation with multiple command line text commands (unless I can cut and paste). I'm still confused by the complex Linux directory structure I have no idea where or what my home directory is or how to get to it from a terminal window / command line prompt.
View 4 Replies
View Related
Jul 30, 2010
I am looking for remote desktop software that will allow me to not only view this particular desktop from miles away but send files back and fourth between the two computers. any ideas?
View 9 Replies
View Related
Jul 22, 2010
What's a good cad program that works with Linux?
Is it possible to do any cad drawings on Linux?
I ve given up trying to run auto cad on Linux no matter how i try it just wont work properly and crashes and I lose what I have done.
Are there any other options apart from QCAD?
View 2 Replies
View Related
Oct 29, 2010
I'm looking for a fast way to verify a copy of a folder with 150Gigs of data, in 33 files. Some of the files are a few kb, while a few are 20-30Gigs. I've done a file count, which is quick, but doesn't verify that all the files are intact. I tried running md5sum on them, which works, but will probably take as long as copying the files in the first place. Diff works too, but is slow too.
View 1 Replies
View Related
Jun 24, 2011
I have a wav file bigger than 8GB. i recorded it on a windows PC. unfortunately wav files cant be bigger than 2GB. somehow i got a file that is almost 9GB. I tried to chop the file under ubuntu into smaller pieces to open it part by part. i used gnome split to divide the file and made 10 parts out of it. now i have these parts of the data which i cant read with no program except for gnome split to merge them together again - which would only bring me to the beginning of my problem. so my question is: is there any other way to open/ split&open a wav file of that size or maybe a way to open the splitted file partially?
View 3 Replies
View Related
Apr 12, 2010
how to create the huge.s kernel files on the slackware disks? or at least direct me to a post if there is the same question. I currently rsync my files to Alien BOB's script, and i use syslinux to install from my usb stick. i was wanting to install using a later kernel just for testing purposes. (i.e 2.6.34-rc3 as of this writing)
View 9 Replies
View Related
Jan 14, 2010
What program is good to install to partition my PC after installation (I installed without a partition). Also, any quick tips in partitioning? Lastly, I have an old Ubuntu installed together w/XP on another PC. I want to install Xubuntu to replace the old Ubuntu, using the same partition. Is that possible?
View 2 Replies
View Related
Mar 21, 2010
Anyone know any good word programs with a good grammar checker? i tried language tool with OOo but it is still buggy lags to find grammar, and unless the sentence is quite simple, it misses alot. tried lightproof too, it has yet to find any error, even in hyphenation which is what it is specifically meant to do. This would be nice to have so I can ween myself entirely off windows.
View 2 Replies
View Related