Software :: Need To Transfer Huge Number Of Files - Good FTP Program?

Oct 26, 2009

I need to transfer some massive amount of data (2.5terrabyte, many files, directory structure) to a embedded raid-box which has a minimal linux on it (some custom distro from western digital). We tried rsync (version 2.6.7), but it crashes because the filelist is too big for the ram available (fixed in later versions of rsync, but I don't know how to update, it's not debian based and there are no compiler tools). We tried nfs, but the max bandwidth produced is around 1 mb/sec (cpu bound?), so it'd take around 3 weeks this way. Samba has problems with big files (and we have some 20gb files in there).

SCP isn't installed, and would probably also be cpu bound due to encryption I think. So the only option left would be ftp, we're currently trying using ncftp with the command "put -R /path/to/data/" , but it's been running for over an hour, eating up most of the ram, and not using any bandwidth. I think it is still building a file list or something. FTP already worked for a single 20gb file with acceptable bandwidth of about 12mb/sec. Does anyone know a better ftp program (for console) that can start transferring some data or at least display an estimated time for the copy-preperation?

View 8 Replies


ADVERTISEMENT

General :: RSYNC Error .tar.gz 14gb File / Transfer Huge 14 Gb File Over Network,vpn,wan Transfer?

Mar 10, 2010

i am trying to transfer a file from my live linux machine to remote linux machine it is a mail server and single .tar.gz file include all data. but during transfer it stop working. how can i work and trouble shooot the matter. is there any better way then this to transfer huge 14 gb file over network,vpn,wan transfer. the speed is 1mbps,rest of the file it copy it.

rsync -avz --stats bkup_1.tar.gz root@10.1.1.22:/var/opt/bkup

[root@sa1 logs_os_backup]# less remote.log
Wed Mar 10 09:12:01 AST 2010
building file list ... done
bkup_1.tar.gz
deflate on token returned 0 (87164 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(274)
building file list ... done
code....

View 1 Replies View Related

Ubuntu Multimedia :: Recommend Good Program For Editing MP3 Files?

Apr 30, 2010

Can someone recommend a good program for editing MP3 files? I need to shorten/fade in/fade out some tracks. Cool Edit Pro does nothing via VirtualBox!

View 4 Replies View Related

Fedora Servers :: Transfer Files From Computer - Run Program?

Aug 25, 2011

I have recently rented a server which is running Ubuntu linux. I can connect to it through ssh. I've written a Java application that I would like to run on the server 24/7. How do I transfer the files from my computer to the server so I can run the program from the server?

View 11 Replies View Related

General :: Transfer Large Number Of Files Host To Host

Oct 20, 2010

I have two servers, one has an empty / and the other has a subdirectory with a large number (4 gig) with many, many files. I need a way to transfer the files en masse from the server with the large number of files to the one that is essentially blank.I don't have space on the used host to simply gzip all the files. I've googled this and see that there may be some combination of tar and/or gzip that will let me do this with some sort of redirection.

I really need and example line of how this can be accomplished. If my explanation seems rather sparse, I can supply more details.

View 3 Replies View Related

Ubuntu :: Transfer Huge Music File To XP?

Feb 13, 2010

Seldom use windows, but need to transfer huge music file from Linux to XP

View 3 Replies View Related

Security :: Huge Number Attempted Ssh Logins?

Sep 12, 2010

Lastb often shows me a huge list of attempted ssh logins.Such as this excerpt:

Code:
admin ssh:notty Sat Sep 11 23:47 - 23:47 (00:00) 184-154-37-12.Huge-DNS.COM
root ssh:notty Sat Sep 11 23:47 - 23:47 (00:00) 184-154-37-12.Huge-DNS.COM

[code]....

View 14 Replies View Related

Programming :: Implement User Ranking In Php With A Huge Number Of Users?

Aug 21, 2010

I'm writing a user ranking module for a site. This ranking depends on some criterias and it's possible to set or unset any one of these criterias in order to consider them in calculating the user rank or not. And here's the way I've implemented the ranking calculation :

when I set one or more of the criterias to be considered in ranking , for each user in the system I insert one record for each criteria , for example : if I have 2 criterias and both are set and consider that I have two users , I'll have :

Ranking table
--------------
username | criteria | to_be_added | score
--------------------------------------------------
user1 | criteria1 | 1 | 0

[code]....

It means I just set the to_be_added field to 1 for all of them and leave the calculation of score for each criteria for each user to the time the user logins so that to prevent doing all these calculations at once , because there are a huge number of users ... But there is one problem , if I want to show for example the best user (based on the highest score) , the result can't always be true because some users might not logged in at that time and their score might be zero .

View 1 Replies View Related

Ubuntu :: Huge /var/log Files

Jul 7, 2011

So I noticed today that my machines root hard drive had almost no space on it.

I did a disk usage analyzer and I found out my var/log folder is 95GB.

The large logs are:

can I just delete them? also how can I stop this from happening again?

View 1 Replies View Related

Debian :: 8.0 Jessie - Var Log Files Getting Huge

Jan 29, 2016

I installed Debian on my laptop and now I had warning that my log files inside var are getting out of hands...Following files...

36G daemon.log
48G daemon.log.1
41G kern.log
55G kern.log.1
31G messages
42G messages.1
8.2G syslog
17G syslog.1

How can I clear this and set up properly so they don't take so much space?

View 6 Replies View Related

Ubuntu :: Delete HUGE Log Files?

Nov 19, 2010

how can i delete my log files? They are 131.2 GB! And i need space on my pc . And is it ok to delete it ?

View 6 Replies View Related

Slackware :: Huge.s No Header Files ?

Feb 12, 2011

I've started using the huge.s kernel and when i try to compile packages slackware complains about kernel headers but all i see is the smp header files on the slackware discs ?

View 2 Replies View Related

Fedora :: Looking For A Good Autotuning Program?

Mar 10, 2010

I'm using fedora 12 64 bit and im looking for a program i can autotune spoken words. Does anyone have any suggestions? Ive heard of Autotalent but currently it doesn't support 64 bit linux.

View 2 Replies View Related

OpenSUSE :: Need Good Reminder Program

Mar 5, 2010

Does anyone know of a good reminder program?

I want some sort of pop-up or message display for things like appointments, due dates, etc. Maybe send a notice to email. Preferably something that will go out about two weeks or more.

Doesn't have to be fancy or even gui based.

View 7 Replies View Related

Ubuntu Networking :: Get Good Ftp Program?

Mar 11, 2010

Anyone know of a good ftp program or best way to do this on ubuntu 9.10?

View 9 Replies View Related

Ubuntu :: Good OS Backup Program?

Jul 16, 2010

Ok, Noobe to Ubuntu, recently installed Lucid, did all the updates, Got my UT 99 loaded and working natively with TeamSpeak 3 (yea, even the sound works now) ... I really like the setup I have now. Can some one recommend a good backup program to use that will "completely" backup my current system "as installed"? I have installed "Back in Time", and it seems really easy to use, but I'm looking for something similar to (sorry have to say it "Windows System Restore").

I've tweaked and tweaked, updated my ALSA drivers, working openAL and got the latest ATI driver installed for my ATI 5770 ... even got a Xfi linux beta working from Creative (that was fun).. Something user friendly too, I'm learning, but still very new to Linux/commandline/Ubuntu.

View 9 Replies View Related

Ubuntu :: Want A Good Database Program

Sep 1, 2011

Best one I've found so far is Glom, but it's extremely hard to use, anyone know of one better?

View 4 Replies View Related

Ubuntu :: Need Good GUI Encryption Program

Jan 22, 2009

I am running Ubuntu 8.04 and 8.10. I want a good GUI encryption/decryption program.

View 5 Replies View Related

Ubuntu :: HUGE Syslog And Daemon.log Files?

Jul 25, 2010

I have a 60GB partition with / and home on it. I logged on yesterday and it gave me a warning saying that I had only 1.9 GB of disk space left. I ignored it for a day and assumed that i had too many videos and pics.But the next day i had not added any files or downloaded any software but i had 0B left. I used the disk usage analyser and found that 33GBs came from /var/log. It was from two log files. syslog and daemon.log 16.5GB each!! I opened them up and i found that this line of text was repeated hnundreds of thousands of times.


Code:
Jul 22 19:32:36 aulenback-desktop ntfs-3g[5315]: Failed to decompress file: Value too large for defined data type

[code]...

View 3 Replies View Related

Ubuntu :: Moving A Huge Amount Of Files?

Jul 27, 2011

i have about 2 TB of 700mb avi files as data on disc want to spread it across two 2TB ext usb drives (sata 3.5 inside the housing) obviously i have to rip them to the laptop and then move to the ext hdd (omg laborious little task) am i better doing the ripping in meerkat or in a windows machine? files need to be accessible by W7, XP, and meerkat to vlc player. what should i format the discs to?

View 5 Replies View Related

Red Hat / Fedora :: Can't Find Any Huge Files In That Filesystem

Dec 22, 2010

I am facing a strange problem in my server, One of my filesystem shows as 3.1G when I execute df -h command and the utilization shows as 83%, but when I cd to the directory /usr/local I could not find any huge files in that filesystem and I have searched for hidden files as well,

groupserver:~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda9 3.1G 2.5G 532M 83% /usr/local
groupserver:/usr/local # du -sh *
0 bin
93M abinav

[Code]...

View 2 Replies View Related

Fedora :: Can Anyone Recommend A Good And Free UML Program?

Aug 24, 2011

Can anyone recommend a good and free UML program? I've been googling and the only thing that really caught my attention is Visual Paradigm.

View 1 Replies View Related

Ubuntu :: Good Lightscribe Program For 9.04 64bit

Apr 23, 2010

What is a Good Lightscribe Labeling Program for Ubuntu 9.04 64-bit? I installed LIGHTSCRIBE SIMPLE LABELER but it's the biggest piece of crap program, you can't even set the Font Size, all it allows you to do is set the Font. I'm not a Linux Expert so I need a program that is easy to install and does not require compiling or complex manual installation with multiple command line text commands (unless I can cut and paste). I'm still confused by the complex Linux directory structure I have no idea where or what my home directory is or how to get to it from a terminal window / command line prompt.

View 4 Replies View Related

Ubuntu :: Good Remote Desktop Program?

Jul 30, 2010

I am looking for remote desktop software that will allow me to not only view this particular desktop from miles away but send files back and fourth between the two computers. any ideas?

View 9 Replies View Related

Software :: Need Good Cad Program That Works With System

Jul 22, 2010

What's a good cad program that works with Linux?
Is it possible to do any cad drawings on Linux?
I ve given up trying to run auto cad on Linux no matter how i try it just wont work properly and crashes and I lose what I have done.
Are there any other options apart from QCAD?

View 2 Replies View Related

Ubuntu :: Md5sum - Fast Way To Verify Huge Files

Oct 29, 2010

I'm looking for a fast way to verify a copy of a folder with 150Gigs of data, in 33 files. Some of the files are a few kb, while a few are 20-30Gigs. I've done a file count, which is quick, but doesn't verify that all the files are intact. I tried running md5sum on them, which works, but will probably take as long as copying the files in the first place. Diff works too, but is slow too.

View 1 Replies View Related

Ubuntu Multimedia :: Opening Huge Wave Files?

Jun 24, 2011

I have a wav file bigger than 8GB. i recorded it on a windows PC. unfortunately wav files cant be bigger than 2GB. somehow i got a file that is almost 9GB. I tried to chop the file under ubuntu into smaller pieces to open it part by part. i used gnome split to divide the file and made 10 parts out of it. now i have these parts of the data which i cant read with no program except for gnome split to merge them together again - which would only bring me to the beginning of my problem. so my question is: is there any other way to open/ split&open a wav file of that size or maybe a way to open the splitted file partially?

View 3 Replies View Related

Slackware :: Create The Huge.s Kernel Files On The Disks?

Apr 12, 2010

how to create the huge.s kernel files on the slackware disks? or at least direct me to a post if there is the same question. I currently rsync my files to Alien BOB's script, and i use syslinux to install from my usb stick. i was wanting to install using a later kernel just for testing purposes. (i.e 2.6.34-rc3 as of this writing)

View 9 Replies View Related

Ubuntu :: Which Program Is Good To Install To Partition My PC After Installation

Jan 14, 2010

What program is good to install to partition my PC after installation (I installed without a partition). Also, any quick tips in partitioning? Lastly, I have an old Ubuntu installed together w/XP on another PC. I want to install Xubuntu to replace the old Ubuntu, using the same partition. Is that possible?

View 2 Replies View Related

Ubuntu :: Need Word Program With Good Grammar Checker

Mar 21, 2010

Anyone know any good word programs with a good grammar checker? i tried language tool with OOo but it is still buggy lags to find grammar, and unless the sentence is quite simple, it misses alot. tried lightproof too, it has yet to find any error, even in hyphenation which is what it is specifically meant to do. This would be nice to have so I can ween myself entirely off windows.

View 2 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved