I need to download some very large files (circa 75 GB) from a remote server via SFTP. I've been using SFTP via the command line on my Linux netbook. Around halfway through, the transfer stops and says "stalled." Can anybody recommend a reliable way to download these files?
View 1 RepliesI am looking for a file sharing program to install on my dedicated server that will allow me to upload large MP3 files and allow my clients to download them. these files are recordings of counseling sessions for families who are seeking help for their children.
What I am looking for is similar to the system this company uses [URL].
i am trying to delete multiple files from sftp with non-interactive way as per the desire ext. But i am not able to get the proper extension of the files for removing files.
lftp -u ${user},${passwd} sftp://${ip}<<EOF
cd ${path}
rm *${ext}
bye
EOF
how I can randomly write / create a 1 GB file in bash to test disk / network i/o? I was told I could use the 'dd' command but I don't know if there are some better ways and or what the 'dd' command looks like.
View 7 Replies View Relatedin my middle of script i am using tar command to tar some 1000 images and size of each image is 5MB.
all the images are provided as argument as tar -cvf file.tar <all images as argument>
but my tar file file.tar does not contain all the images.
How do I download all the files form here: [URL]. I am on freeBSD 7.0 and I tried wget with the -r switch and it gives me URL's only. Maybe this is simply not an ftp site I don't know. How I can download all those files with the same directory structure.
View 5 Replies View RelatedI've a directory containing around 2.8 lacs of files. I want to move them to another directory.If I use cp or mv then I get an error 'argument list too long'. If I write a script like
for file in ls *; do
cp {source} to {destination}
done
then because of ls command , its performance degrades.How can I do this?
I want to transfer an arbitrarily large file (say >20GB) between 2 servers. I have several considerations:
Must use port 22 (ssh) because of firewall restrictions
Cannot tax the CPU (production server)
Memory efficiency
Would prefer a checksum check but that could be done manually
Time is not of the essence
Server A and Server B are on the same private network (sharing a switch) and data security is not a concern, Server A and Server B are not on the same network and transfer will be via the public internet so data security is a concern, My first thought was using nice on an scp command with a non-CPU-intensive cypher (blowfish?). But I thought I'll refer to the SU community for recommendations.
I am facing problem in copying a large number of file 18 lakh (18,000,000) files from my personal hardisk to another hardisk each file is very small and size of folder is around 3.95 GB copying files using copy given by Windows is frustrating and I am not even able to compress file its giving me error that its not readable.And problem is I am not able to open this drive in Linux it showing me error there saying do diskchk in Windows and Windows disk check is also not able to repair this drive and goes into some mode unsolvable.Is there any way to open disk with error to open in Windows and if not any way I can copy data faster?ERROR: Disk labled EDU is corrupt go to windows and chkdsk /f there and reboot into window 2 times.
View 3 Replies View RelatedI understand that chroot is usually used to provide security, however, for my issue, security is a big don't care. I am very new to using chroot and don't fully understand how the chroot'd env works.
problem: Trying to use a vendor supplied cross compile environment. The environment runs as a chroot'd env and works just fine. I have a large number of additional modules that I wish to compile in the chroot'd environment. FYI, these modules are also (succesfully) compiled for other targets not using chroot'd env's. Copying the source files into the the chroot environment is not an option (don't have hours to wait for copies to finish and it would break the make system). Having them live in the environment is also not an option (the chroot build is a tiny part of the build process and we cannot revamp our entire source tree to accommodate it).
I am looking for a way to have the compiler in the chroot'd env have access to a path that is outside of the env and typically higher up in the same path that holds the chroot'd env. I have tried soft links (they don't work as expected). Hard links only work for single files and there are 10's of thousands of files that would need to be linked. I am not sure how I would go about exporting the additional files and then mounting the exported files in the chroot'd env (or if that would even work).
I am looking for a way to search for large numbers in text files and print the nearby lines.
For example if I had a text file like:
Event: 11
blah: 3
blah: 41 bleh: 19
Event: 2
blah: 31
[Code].....
I have two NASes. I work off of one, and the other is used as a backup. As I have it set up now, it's slow. Running a backup takes a week. Even for 7 TB, with 1,979,407 files, this seems a bit outlandish,particularly as both systems are RAID-5 and the network is all gigabit. I've been digging about in the rsync man pages, and I really don't understand what differentiates the various topologies.Right now, all the processing is being done on the backup NAS, which has the main volume from the main NAS mounted locally over SMB. I suspect that the SMB overhead is killing me, particularly when dealing with lots of files.
I think what I need is to set up rsync on the main nas as a daemon, and then run a local rsync client to connect to it, which would hopefully allow me to completely avoid the whole SMB-in-the-middle affair, but aside from mentioning that it's there, I can find very little information on why one would want to use the daemon mode for rsync.
Here's my current rsync command line: rsync -r -progress --delete /cifs/Thecus/ /mnt/Storage/input? Is there a better way/tool to do this? Edit:Ok, to address the additional questions: The "Main" NAS is a Thecus N7700. I have additional modules installed that give me SSH, and it has rsync, but it's not in the $PATH, and I havn't figured out how to edit the local $PATH in a way that persists between reboots. The "Backup" NAS is a DIY affair, built around a 1.6Ghz Via Mobo with a Adaptec Hardware RAID card. It's running CentOS 5 with a full desktop environment. It's the hardware I'm running rsync from. (Gigabit is through a additional PCI card).
Further Edit: Ok, got rsync over SSH working (thanks, lajuette!).I had to do a bit of tweaking on my command line, I'm running rsync with the args:rsync -rum --inplace --progress --delete --rsync-path=/opt/bin/rsync sys@10.1.1.10:/raid/data/Storage /mnt/Storage (Note: I'm specifically not using -a, because I want to change the ownership to the local account, to not freak-out SELinux)
For my research I have some very large files that are basically millions of lines of ten columns of numbers. These files can be up to 5 GB in size. Recently I noticed that when I made a copy of one of my files, some exclamation points appeared in it where there should not be any: in front of random numbers throughout the file. Making another copy of the file would result in exclamation points in front of different numbers in different parts of the file. Doing this many times has given me up to four exclamation points in different parts of the file. Sometimes the file copies just fine without producing any extraneous exclamation points.Additionally, I have occasionally seen a "^K" where there should be a newline (the data that should have been on the next line was instead on the previous line with a ^K in front of it) in copies that I have made of my files. I don't know if this is related or not.
View 7 Replies View RelatedI have a very simple php web application deployed on linux (centOS4) machine. It creates a file and stores the file in /tmp folder on my linux machine. The path for this file is specified in the href attribute of the link. Ideally when we click this link the download manager should pop up so that the file can be downloaded on client machine.
When i access this website remotely from my window xp machine on firefox it downloads the file properly but when i run on internet explorer (i have IE7 on my windows XP) and click the link, the download manager does'nt pop's up. even when i right-click that link and select save as, an error message pop's up saying "file path not found". possibly IE is not able to determine the linux file path .so how do i work around this. is there some specific way for specifying the linux file paths to be downloaded by IE?
As a Windows user, I generated a pair of DSA keys from CoreFTP Lite and sent it to a third party that runs an SFTP server. They told me that a valid DSA key needs to have ssh-dsa at the start and the username@systemname at the end. CoreFTP generated neither the ssh-dsa header nor the username@systemname footer. I tried with WinSCP and it didn't generate them either. Is there a difference between how SFTP works between Windows and Linux? If I put a useraccount@systemname at the end of the text will it work? How would the Linux system validate that my system is called "systemname"? If it can't validate, what is the purpose of adding it?
View 2 Replies View RelatedI am using the diff command with the -r option, to compare a large number of files and files in subdirectories. My main interest is to find out which files have been changed, and not what the actual changes are, and since a lot of files has been changed, it would be a lot easier to view the file names only. Is there and option for diff that might do this, or does there exist a similar tool/command that could do the job?
View 1 Replies View RelatedI've discovered that Dolphin seems to lose random files when copying many large folders.
I first noticed this a few months ago when I tried to copy my music library from one folder to another on the same HDD. It consisted of around 600 folders and 6500 files. During the copy there were no errors but after the copy I found that some of the newly copied folders were missing files. I put it down to human error or a glitch.
Yesterday I tried to copy 13 folders containing rips of some of my DVDs. Each folder basically had one film of either 700MB or 1.4GB. Again no errors showed up during the copy but I found 3 of the newly copied folders were empty.
It's not so critical with music or films but I can't afford to lose work data like this.
Has anyone experienced or seen a similar problem with Dolphin? I'm going to have to do some more extensive testing but this is not good.
The first time I noticed the problem I was running KDE4.3.4 (I think) and now the latest was with KDE4.4.0.
I want to automate this using script.How to automate it?
File1:
s.no# 1 name:aaaaaa
city:abcd
[code]...
How do you transfer files on a Linux machine with ftp,sftp and scp disabled onto a remote machine
View 2 Replies View Relatedi have a vsftpd server running well but i want to make/force all users to use sftp and not just ftp is this possible?
View 1 Replies View Relatedacl FILE_MP3 urlpath_regex -i .mp3$ http_access deny FILE_MP3. I HAVE SET THIS RULE; ACL rule in Squid to block downloading of .mp3 files
But I don't understand the purpose of "".mp3$ here the ""? even without it ("") I am able to block downloading od mp3 files and what is purpose "$" ich at the end?
I have two servers, one has an empty / and the other has a subdirectory with a large number (4 gig) with many, many files. I need a way to transfer the files en masse from the server with the large number of files to the one that is essentially blank.I don't have space on the used host to simply gzip all the files. I've googled this and see that there may be some combination of tar and/or gzip that will let me do this with some sort of redirection.
I really need and example line of how this can be accomplished. If my explanation seems rather sparse, I can supply more details.
I would like to transfer files from my PC to my account at a sftp server, and I don't know how to do it.
My PC is running with:
User: User1
Address: 10.0.2.3
My sftp account is:
User: SFTPUser1
Address: sftp-server
I can access the sftp server with the command:
Code:
sftp SFTPUser1@sftp-server
The sftp server doesn't answer to ssh requests.
How can I transfer files to the sftp server?
I want to access my files on my home computer when I'm away (school, vacation etc). I set up a scheduled task to send to send me my external IP address by email because I have a dynamic IP and dont want a static url. (dyndns) I planned to use sftp for an encrypted connection, but I dont know where to start. How can I set up my computer to accept incoming requests, but without any security issues?
Ubuntu 10.10 Desktop 32-bit
openssh-server installed
Can i block on debian that user can not download any files from host via winscp or other SFTP client ?
View 4 Replies View RelatedWhere is the config file for the sftp bit? At the mo it shows all the hiddenfiles (dot) and I don't want it too. Don't laugh, I have just configured my proftp for this, and realised, hang on this isn't the program that dishs out sftp!
View 1 Replies View RelatedProblem is with files greater than 4G onto dual layer or BD disks.mkisofs crashes. I believe the problem to be when this requires udf filesystem and disk does not get written to. As yesterday version is suse11.3 patched to date. k3b writes standard dvd disks ok. I am seeing a lot of searches saying this is because of cdrkit rather than cdrtools, can I replace this easily? Is this the case? Tried setting the k3b options when burning to udf same error. Other searches show k3b has fixed probs with these issues, so appears to point to underlying mkisofs stuff full error log in yesterdays post if it helps. I will also try on ubuntu to see if it works there
View 1 Replies View RelatedI always wanted cp to have a progress bar for large files. I came across this:[URL]... I just wonder, how could you install it as an Arch package? Is it possible?
View 9 Replies View RelatedI currently using RPM version 4.4.2.3. It fails to build with files larger than 2 GB and file sets larger than 3 GB. I have files larger than 2 GB and sets larger than 3 GB.Is there a work around, possibly a switch or option for RPM, that will ignore this limitation?
View 8 Replies View RelatedI am using xfce desktop and terminal.
1) How do i check to see what is the default FTP Client used by Slackware12.2 when i type:
$ftp
2) How do i get ftp to download large files? > 2gb from ftp server elektroni.phys.tut.fi or any ftp server.