Ubuntu Servers :: System Crash When Copying Large File
Jun 15, 2010
I am having a bit of a problem with my Ubuntu Server 10.04 install. I think it might be a kernel problem. Basically, what happens is when I copy a large file (a 160GB disk image) to my drive (>60GB) the system consistently crashes after about 60GB of the file is transferred. It doesn't matter if I am sending the file using cifs, or over SSH. Checking syslog (paste dump here), it seems these flush errors always appear shortly before the crash occurs. The destination filesystem is a hardware RAID 10 array with 2TB of space. It is formatted as EXT4.
I am trying to copy a file from a network resource on my Local Area Network, which is about 4.5 GB. I copy this file through GNOME copying utilities by first going to Places --> Network and then selecting the Windows Share on another computer on my network. I open it and start copying the file to my FAT32 drive by Ctrl+C and Ctrl+V. It copies well up-to 4 GB and then it hangs.
After trying it almost half a dozen times I got really annoyed and left it hung and went to bed. Next morning when I checked a message box saying "file too large to write" has appeared.
I am very annoyed. I desperately need that file. It's an ISO image and it is not damaged at all, it copies well to any windows system. Also, I have sufficient space on the drive in which I am trying to copy that file.
I was just copying a large (50GB) file from one mounted partition to another mounted partition (a USB drive), but before the operation completed, my root filesystem, on a separate partition, filled up.Because it filled up I also couldn't get past the login when I rebooted. I think this is because there is no room to load temporary files. I'm expanding the root partition to temporarily fix this. how can I avoid my root file system filling up when copying a massive file between mounted partitions? the file is being cached in root during the transfer.
I'm planing to copy a productive mysql innodb file from one server to another, and the file size is around 300GB. As the file is keeping changing all the time, I have to shutdown mysql instance and copy the large data file to other server as quickly as possible.I should have to find a way to speed up file copying ... I'm wondering whether there's a way to copy file block by block.If the destination side block has same content, then bypass it.
This has happened several times now, with 9.10 and 10.04. I back up my photos periodically to external drives, using Nautilus. At the next attempted login Gnome won't start and sometimes gives power manager incorrect installation error.
First time this happened I was stumped and eventually did a clean install. Second time, I found advice elsewhere in this forum to solve this by emptying root trash, which did the trick. This time, however, root trash has nothing in it and 2 users trash were insignificant (I emptied them all anyway with rm -r). Tried looking for enormous directories but couldn't find a smoking gun. I would rather not end up doing another clean install - a painful and extreme solution. I'm continuing to look for solutions to the immediate problem, but my question really is, what causes this and how do I prevent it in the future? I've run Computer Janitor regularly and ran apt-get clean but no help. Should I do all my large scale copying from terminal? I'm not a total noob, but close.
I am trying to copy a 7.3gb .iso file to an 8gb USB stick and I get the following error when it hits 4.0gb
Error while copying "xxxxxx.iso". There was an error copying the file into /media/6262-FDBB. Error splicing file: File too large The file is to be used by a windows user, and I'm just trying to do a simple copy, not a burn to USB or anything fancy. Using 10.4.1.LTS, AMD Dual Core, all latest patches.
I'm trying to create an Ubuntu Server file server that will handle large file transfers (up to 50gb) from the LAN with Windows clients. We've been using a Windows server on our LAN on the file transfers will occasionally fail... though the server is used for other services as well.
The files will be up to 50gb. My thoughts are to create a VLAN (or separate physical switch) to ensure maximum bandwidth. Ubuntu server will be 64bit with 4tb of storage in a RAID 5 config.
basically want i want to do is copy my whole file system to a different hard drive, then reconfigure my partitions and copy it back. then reconfigure grub.
the reason i want to do this is when on dual boot i gave it only 70gb of space and now i want to add 300 more. and since the 300gb of space is a primary partition and this is a secondary i cant extend them or combine them.
so what i want to do is. sudo cp -rP / /home/me/sshfs-folder
Is there a file system that both Mac OSX 10.5 and linux can read/write for large files (like 4gb files)? My desktop is Ubuntu and I run most from there, but I want to back up my MacBook and linux box on the same external hard drive. Seems there are some (paid) apps for Mac that will mount NTFS but I'm wondering if there is just a shared files ystem that will work for both.
I am running Ubuntu 10.10 (amd64) on an Intel Atom 525 motherboard. I have a 5-bay SATA enclosure connected to my system through a Syba SD-SATA2-2E2I PCI card (uses the SIL3124 chipset). Under small to moderate load (transfer speed: ~25 Mb/sec), this setup works perfectly. Under heavy load (transfer speed: ~40 - 70 Mb/sec), the system crashes. By crash I mean I loose SSH connectivity, I can't ping the system, etc. The only solution is a hard reset. Initially, I thought this was a SAMBA problem so I logged into the system and tried doing a copy directly from the SATA enclosure to the system disk (connected directly to the motherboard). After 100 GB copied (at ~75 Mb/sec), the system crashed. My question: I am fairly new to Linux and Ubuntu. How would I go about diagnosing this problem?
I want to copy a file system with dd. Most of the forums and how-to's warn that the file system must be unmounted. Otherwise, data could be changing and inconsistent or, at best, there may be open inodes, and the copied file system would need to be recovered the first time it is mounted. However, is it sufficient if the file system is mounted read-only? Or does a file opened with read-only still result in an open inode?
I'm experiencing connection problem when transferring a large file from Windows 7 (Home Premium) to my Ubuntu 11.04. The transfer starts, but after a couple of seconds the connection drops and all the shares are unavailable. I'm also unable to connect to the server over ssh, and the only thing I can do to restore the connection is to reboot the server. The strange part is that this was never a problem a couple of weeks ago, and I've not done anything to the setup on either machines besides installing security updates.
I've a directory containing around 2.8 lacs of files. I want to move them to another directory.If I use cp or mv then I get an error 'argument list too long'. If I write a script like
for file in ls *; do cp {source} to {destination} done
then because of ls command , its performance degrades.How can I do this?
I am facing problem in copying a large number of file 18 lakh (18,000,000) files from my personal hardisk to another hardisk each file is very small and size of folder is around 3.95 GB copying files using copy given by Windows is frustrating and I am not even able to compress file its giving me error that its not readable.And problem is I am not able to open this drive in Linux it showing me error there saying do diskchk in Windows and Windows disk check is also not able to repair this drive and goes into some mode unsolvable.Is there any way to open disk with error to open in Windows and if not any way I can copy data faster?ERROR: Disk labled EDU is corrupt go to windows and chkdsk /f there and reboot into window 2 times.
Every once in a while on a computer I'm ssh'd into, I will accidentally type "cat largefile.txt" and my screen will start rushing with text for the next 10 minutes. I'm always working in a screen session, so my current solution is to just log out and then log back in, and since it can go 100X faster when I'm logged out, it'll finish in the short time it takes me to type my password in again. Is there a better way? Either involving the fact I'm in a screen session? Or a way to do this within SSH? What doesn't work: detaching from the screen session (doesn't respond until file is done outputting) trying command to move to a different window in the screen session (also doesn't respond) typing ctrl+C to kill cat command (also doesn't respond, probably because the command is done and the buffers just have to catch up).
I just bought a HP 3085dx laptop with an intel 5100 agn wireless card. The problem: copying a big file over the wireless to a gigabit hardwired to the router computer only gives an average 3.5MB/Second transfer rate. If I do the same copy from my wireless-n macbook pro to the same computer. I get a transfer rate of about 11MB/sec. Why the big difference? I noticed the HP always connects to the 2.4 GHZ band instead of the 5GHZ bands...
[jerry@bigbox ~]$ iwconfig wlan0 wlan0 IEEE 802.11abgn ESSID:"<censored>" Mode: Managed Frequency:2.412 GHz Access Point: 00:24:36:A7:27:A3 Bit Rate=0 kb/s Tx-Power=15 dBm Retry long limit: 7 RTS thr: off Fragment thr:off Power Management: off Link Quality=70/70 Signal level=-8 dBm Noise level=-87 dBm Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0 Tx excessive retries:0 Invalid misc:0 Missed beacon:0
I am not getting any errors. I don't know why the bit rate is not known. My airport extreme base station typically reports that the 'rate' for the hp is typically 250~300MBi and about the same for the MacBook Pro. The hp is about 6 inchs away from the base station. Is there anyway to get the rascal to go mo'faster? Is there anyway to get the rascal to use the 5GHZ band.
For my research I have some very large files that are basically millions of lines of ten columns of numbers. These files can be up to 5 GB in size. Recently I noticed that when I made a copy of one of my files, some exclamation points appeared in it where there should not be any: in front of random numbers throughout the file. Making another copy of the file would result in exclamation points in front of different numbers in different parts of the file. Doing this many times has given me up to four exclamation points in different parts of the file. Sometimes the file copies just fine without producing any extraneous exclamation points.Additionally, I have occasionally seen a "^K" where there should be a newline (the data that should have been on the next line was instead on the previous line with a ^K in front of it) in copies that I have made of my files. I don't know if this is related or not.
I'm running squeeze on i686, my last update was this morning (9/9/2010). Running the stock kernel 2.6.32-5-686
My PC keeps crashing when copying files. The monitor displays the desktop, but the mouse doesn't move, nothing responds and I can't even ssh in to debug it.
This happens when copying files using nautilus and the command line. I'm trying to copy files from my local hard disk to an nfs drive which is mounted in my system and has been working fine for years.
how to debug this after a hard reboot?
This has only just started happening, last night was the first time.
I am dual booting XP and Ubuntu 10.04, but in the future I will be getting a new machine and I will only be running Ubuntu and won't have access to iTunes. Because I have an iPod Touch, I have been trying to find workarounds for syncing everything that iTunes took care of in the past. One problem I have is managing movies. I have looked through various media players/iPod management tools (Amarok, Rhythmbox, gtkpod) and I am using Rhythmbox to sync my music and and attempting to use gtkpod to sync my movies.
gtkpod is able to sync songs (Tested with a few minute test clip) and short *.mp4 files (15mb I know for sure from test). I am unable, however, to get it to sync a movie (~700mb) I am able to drag it onto my iPod in gtkpod, but when I try to save the changes and write the files, it hangs at "Copying Tracks" at 0%. It eventually crashes during the couple times I have tried to wait it out. So this being my situation, my question is, is there a size limit to the *.mp4 files I can sync to my iPod Touch via gtkpod? is there any other tools that you know of that I can sync videos to my iPod with?
I've discovered that Dolphin seems to lose random files when copying many large folders.
I first noticed this a few months ago when I tried to copy my music library from one folder to another on the same HDD. It consisted of around 600 folders and 6500 files. During the copy there were no errors but after the copy I found that some of the newly copied folders were missing files. I put it down to human error or a glitch.
Yesterday I tried to copy 13 folders containing rips of some of my DVDs. Each folder basically had one film of either 700MB or 1.4GB. Again no errors showed up during the copy but I found 3 of the newly copied folders were empty.
It's not so critical with music or films but I can't afford to lose work data like this.
Has anyone experienced or seen a similar problem with Dolphin? I'm going to have to do some more extensive testing but this is not good.
The first time I noticed the problem I was running KDE4.3.4 (I think) and now the latest was with KDE4.4.0.
I m having a RHEL-5 sever.ABC directory size is 57GB after taking backup in the same disk with name ABC.bkp showing 56GB. i used below command to copy/backup. # cp -r ABC ABC.bkp (different sizes after copying)..I checked both the directory sizes by #du -sh <ABC> and du -ks <ABC.bkp>In both GB and KB there is lots of difference (200mb). why this will happen in copying? what is the solution for above question? what is the correct way of copying 1dir to newdir exactly?
I'm planning to add 1tb sata disk to my lovely file-server under ubuntu 10.10,what i want is use this disk as additional storage for network user,indows and ubuntu?I mean when my ubuntu server down (worse case) I can easily take out the disk from ubuntu machine and plug in on windows machine
When I try to copy PDF files from one folder to another folder, it give me this error: "Error while copying "2004-SNUG-Europe-paper_...log_DPI_with_SystemC.pdf". There was an error copying the file into /media/CCDCE66BDCE64F70/Backup Master/Heterogeneous_cosimulation/Documentation" "Error splicing file: Input/output error" What is the reason of this error and how can this be fixed?
I have a 7.2 GB file (VMWare virtual machine file) that I am trying to copy from its original location to the another folder OR to external hard drive...each time I try to do this, I always get the following error after the copying process reach 'exactly' 1.4 GB
Error reading from file input/output error
And I have to either Cancel or Skip
I've tried to split the files to smaller pieces but the idea didn't work as I still get the same error whenever I try to compress/ split or do any operation with this file. how I can copy this file?
I use release 11.2 and this one works very well. I try new release 11.3 and my system crash when I wont to start system on runlevel 5. There is Sempron 2 GHz CPU and nvidia fx550 graphics card. Keyboard PS2 does not responds but mouse on USB work well. When I start "safe" boot option system work well. What I can to do ? What I must change
I am using RHEL 5.I have a very large test file which cannot be opened in vi.The content of the file has some 8000 lines.I need to view ten lines between 5680 to 5690.How can i view these particular lines in a large file.what is command and option i need to use.
I am having problems with scp during a backup operationI added a ps -ef before and after the scp operation used during the backup.The backup is a script to backup a Zimbra ServerI am including the code segment that I am having problems
Code: # DRCP Section. To scp newly created archives to a remote system if [ "$DRCP" = "yes" ]
I've already asked this in the mythbuntu and didn't get an answer there so I'm trying here. OK just added another 2Tb WD drive to my mdadm controled RAID5 array, and the reshape is finished:-
Code:
Code: Mar 28 18:28:36 alpha2 kernel: [104357.343421] md: md1: reshape done. Mar 28 18:28:36 alpha2 kernel: [104357.525114] RAID5 conf printout: Mar 28 18:28:36 alpha2 kernel: [104357.525119] --- rd:4 wd:4 Mar 28 18:28:36 alpha2 kernel: [104357.525122] disk 0, o:1, dev:sda2
