Fedora :: Heavy Software Raid I/o Causes System Lockup
May 9, 2011
Here is the scenario: I load hundreds of h264/ac3 video clips from my HD video camera onto a four disk software raid stripe under Fedora 14 running the 2.6.38.3-15.rc1.fc15.x86_64 kernel. I then run a python script I've written to do batch process transcoding on the clips so that I can then edit them in cinelerra. FWIW, the transcode command is: nice ffmpeg -i infile.MTS -y -deinterlace -s hd720 -r 29.97 -acodec pcm_s16be -ar 48000 -ac 2 -vcodec mjpeg -qscale 2 out.mov. The python script batches these commands until they are all completed.
The system is capable of 16 concurrent threads and I set up 8 concurrent run queues to do the processing. Occasionally (about one in every 7 runs), the system will hard lockup during the batch. The X screens are still visible, but the mouse and keyboard are dead and the onscreen clocks freeze. The network io led still flashes but the hard disk io led is on solid at that point, leading me to theorize a kernel bug in the software raid handling. I am not able to remote log in and the system must be then hard rest with the reset button. After the reset there is no useful information in the system logs. this is a pretty uber SMP machine: two westmere quad core cpus, 24GB of ram and a four WD disk software stripe array running under the Intel ICH10 controller. I also have a RocketRaid controller installed but no disks are currently attached to it, and I disabled its bios in the setup. Additional disks under the ICH10 are an SSD system disk, and a sata insertion caddy that I usually have occupied with a disk I do backups to. So, all 6 channels on the ICH are usuallt occupied. General output from hdparm -i is:
Where does one look in terms of diagnostics after a system lockup and hard reset? For example, one of my computers, running Fedora 14 i686, has taken to locking up quite a bit lately. My only way out is a hard reset. Now, once the thing is booted back up I'd like to try to figure out where the problem lies. Where do I look? Will any of the log files reveal anything relevant to me? If so, what sort of things would I be looking for. Keywords?
This same box also locked up when I had Fedora 14 x86_64 installed. I had read that there were some issues with the newer 64 bit kernels that were causing lockups and / or extremely high loads. This is why I switched back to the 32 bit version, 2 days ago to be exact. The box gets little use. It's a toy really and not my main "daily use" computer. It is, however, running folding@home. Maybe that's where the problem is. About 15 minutes ago I called home and had my wife reset the box, then I logged in from work and stopped the folding service. I'll have to see if it locks up without folding running. Folding was running on this box when I had the 64 bit version installed also.
I have a problem with my Debian squeeze system. After I have run some CPU-intensive application for a while the system becomes sluggish. Ok, maybe one would expect that. But. The system remains sluggish even after that process has been killed and then top shows high cpu usage from unrelated processes and a high %sy percentage as well. I don't really know where to begin solving this problem except I've checked that this is specifically related to (at least) CPU use as opposed to heavy memory use or IO (by running an application with low mem/no io but high cpu use). I am running the latest standard kernel from squeeze, 2.6.32-5-686. Also, thing is, I remember having similar problems with other Linux distros before, where for example gzipping a large file would cause the same. I did not have similar problems on {Net,Open}BSD, so a hardware fault seems an unlikely reason to me.
I have been encountering lots of heavy system cpu usage. It is sporadic, last for about 10 minutes, and brings everything else to crawl (latency of even typing characters into an editor or a terminal is painfully slow). My computer is an i7 quad, and all 8 logical cores fire at near 100% during these periods. I can see the usage clearly with htop. The red bars in htop indicate system use (I have attached a screen shot for an example). However, as there is no process per se, I have no clue what is causing it.
It is something I have to fix as the computer becomes unusable periodically throughout the day.I One thing to add, I am using the computer as a ltsp server on ubuntu, attaching to it using two thin clients. There is always heavy ethernet traffic through the second ethernet card. I don't know if this is related. However, I do think all this problem began when I installed the ltsp and switched to using a thin client.
I have recently begun contributing to the Folding@Home distributed computing project. Due to oversights in the application's design and to process the most advantageous work units, I need to run this command line app through wine. The application is very CPU/RAM intensive, but it does not put much load on the hard drive. It is multi-threaded and uses up to 100% of all 6 cores in my machine. I run it with a "nice" level of 19 with hopes that it won't slow down my normal desktop use.
Here is the problem... when running this setup, my system will frequently freeze for a fraction of a second and then resume. With the more complicated work units, this can happen 1-2 times per minute. During the freeze, the mouse will stop moving, videos stop playing, music pauses, etc. The system is completely unresponsive. However, each instance only lasts for a very short time. Since this happens so often, my productivity is negatively impacted (and it's very annoying).
I previously ran the native Linux version of F@H without this problem, but that was also processing much less complicated calculations. I have tried with wine 1.2.1 and 1.3.5 with the same results. The application does not have problems running on Windows. It has been suggested that the current Linux CPU scheduler is to blame, but is there anything I can do to resolve this now or work around it?
I use the Futura font as my application/desktop/titlebars font, and I think it looks beautiful. However, its bold version is much too dark for my liking and it really spoils the look of my desktop when it's used in applications (Nautilus' sidebar and Libre Office's menubar are the main offenders), and what I would much rather is if the system would use Futura's heavy weight option instead of bold, but after searching gconf and for configuration files I can't work out where or how I would do this.
The following are two extracts from /var/log/messages.
Code: Aug 10 17:29:52 darkstar sshd[11675]: reverse mapping checking getaddrinfo for 116.214.25 -66.del.tulipconnect.com [116.214.25.66] failed - POSSIBLE BREAK-IN ATTEMPT! Aug 10 17:29:52 darkstar sshd[11675]: Invalid user plcmspip from 116.214.25.66 Aug 10 17:29:52 darkstar sshd[11675]: Failed password for invalid user plcmspip from 116. 214.25.66 port 40032 ssh2
I've just installed OpenSuse 11.3 on my two Sony Vaio VPCF11Z1E machines (which have a Geforce GT 330M graphics card).Both machines appear to setup correctly and behave normally, but at seemingly random intervals (of the order of magnitude of 15-30 minutes) the whole system locks up for no readily apparent reason. I can still move the mouse pointer around the screen, but I cannot click on anything, and the whole keyboard is unresponsive (including any reasonable ctrl-alt-xxx which might normally help). The only way to get out of this seems to be to press and hold the power button to shut the whole machine off (which is not ideal!)
I am not sure of which LibreOffice version I am using because I can't open the ****ed program. So here's what's happening: I start LibreOffice, my screen looks unusual (it tiles a section of the screen that is the same size as the LibreOffice window (I think) across both of my monitors) and my system locks up. I am unable to use Ctrl+Alt+Backspace, Ctrl+Alt+Delete, Ctrl+Alt+F1-F8, or any other keyboard combination. No mouse click registers and the cursor acts as if the mouse is in one location (if the system locked up with my cursor over a link, the cursor stays like that) BUT the mouse still moves. Also, if I am playing music when it locks up, I can continue to hear the music and the music continues to shuffle through my playlist.
I am able to access my machine normally through ssh and the only way I've been able to fix this is to hold down the power button or pull the power. I have not identified what action causes the system to lock up, the only thing I've noticed is that LibreOffice is open every time this happens (It's happened no matter if Writer or Calc is open). The system almost always locks up upon startup of LibreOffice if the recovery window pops up, otherwise it locks up at some point during my LibreOffice session no matter what. I have not touched LibreOffice on my other machine's openSUSE installation, so I don't know how reproducible the problem is. Let me know if you need any extra information or if there are any diagnostic commands I could run.
I have a E-MU 0202 USB soundcard that I want to use as a input for my Debian Squeeze 6.0.1. Stock squeeze (kernel 2.6.32-5-686, version 2.6.32-31) has alsa-driver 1.0.21 as kernel modules and alsa-utils 1.0.23. The E-MU needs 1.0.23+ alsa driver, with 1.0.21 it loads in alsa but is not functional (no input) So I tried to install the 1.0.23 modules with module assistant: m-a update && m-a prepare m-a a-i alsa I left all conf-files (like /etc/modprobe.d/alsa-base.conf) standard. When I now connect the E-MU device, my whole system becomes unresponsive (to keyboard/mouse), and I get the following messages in my console: BUG: soft lockup - CPU#0 stuck for 61s! [modprobe:1292] Also, when I start top before connecting, I can see modprobe process taking 100% for a few minutes before system freezes completely. It did generate an oops once: [URL].... Does anybody here have any ideas how to get this working?
Running updated 11.04 on a older Dell Precision 360n. Been having random system freezes for the past month that requires a hard restart (REISUB doesn't work) each time. Sometimes the caps lock and scroll lock LEDs blink, other times not. sometimes it happens during boot, other times during normal usage. There hasn't been anything in any of the logs (except once, which I'll post below). This happened running 10.04 and 10.10 as well. I've run PC-check a few times and all hardware diagnostics have passed. Until today, i've been able to boot and run Trinity Rescue Kit 3.4 to mount the disk and examine the logs (and do backups), but now TRK won't boot and spews kernel panics when loading.
Here is my system: I have dell poweredge 1950 PERC 6 with 300 GB raid system. It has two disks of each 300GB RAID mirrored system. I have few applications and data that reached around 280GB. As you know, poweredge 1950 we can have only two disk.
They are not mission critical. Hence, I wanted to remove the raid system and use as a non-raid system. By doing it, The applications and data can grow upto 600GB. I do not want to loose the data and setup. I am not so clear about RAID system and its conversion.
I have implemented LVM to expand the /home partition. I would like to add 2 more disks to the system and use raid 5 for those two disks plus the disk used for /home. Is this possible? If so, do I use type fd for the two new disks and use type 8e for the existing LVM /home disk? Or do I use type fd for all of the raid disks?
After first installing Fedora on my computer, I was having a problem with my programs hanging for about 10-20 seconds, and then starting working again. After a while I discovered that every time it happened, there where some disk activity, and when the disk activity stopped, the programs started working again. The problem also occurred, when I was copying large files. Then the programs would hang until the copy was finished.After some googling around I tried to change the scheduler for SDA from cfq ro deadline, which seemed to solve the problem.
However the problem still seems to occur, if I do something that requires heavy network activity. If I copy a large file from one computer to another or download something using Vuze, all programs becomes unusable until network activity stops again. have 2 computers, which both seems to have the same problem (AMD X2 3800+ and Intel Core I5 750). They are running Fedora 12 and 13 both 64 bit. Both are using Asus motherboards with onboard LAN
I have two 1TB hard drives in a RAID 1 (mirroring) array. I would like to add a third 1TB drive and create a RAID 5 with the 3 drives for a 2TB system. I have ubuntu installed on a separate drive. Is it possible to convert my RAID 1 system to a RAID 5 without losing the data? Is there a better solution?
I upgraded from F10 to F12 using preupgrade. The upgrade itself completed with no errors, but I'm unable to boot afterward.
Symptoms:...Grub starts, the initramfs loads, and the system begins to boot. After a few seconds I get error messages for buffer i/o errors on blocks 0-3 on certain dm devices (usually dm0 and dm2, but I can't get a shell to figure out what those are). An error appears from device-mapper that it couldn't read an LVM snapshot's metadata. I get the message to press "I" for interactive startup. UDEV loads and the system tries to mount all filesystems. Errors appear stating that it couldn't mount various LVM partitions. Startup fails due to the mount failure, the system reboots, and the steps repeat.
Troubleshooting done:...I have tried to run preupgrade again (the entry is still in my grub.conf file). The upgrade environment boots, but it fails to find the LVM devices and gives me a question to name my machine just like for a fresh install. I also tried booting from the full install DVD, but I get the same effect. Suspecting that the XFS drivers weren't being included, I have run dracut to create a new initramfs, making sure the XFS module was included. I have loaded the preupgrade environment and stopped at the initial GUI splash screen to get to a shell prompt. From there I can successfully assemble the raid arrays, activate the volume group, and mount all volumes -- all my data is still intact (yay!). I've run lvdisplay to check the LVM volumes, and most (all?) appear to have different UUIDs than what was in /etc/fstab before the upgrade -- not sure if preupgrade or a new LVM package somehow changed the UUIDs. I have modified my root partition's /etc/fstab to try calling the LVM volumes by name instead of UUID, but the problem persists (I also make sure to update the initramfs as well). From the device-mapper and I/O errors above, I suspect that either RAID or LVM aren't starting up properly, especially since prior OS upgrades had problems recognizing RAID/LVM combinations (it happened so regularly that I wrote a script so I could do a mkinitrd with the proper options running under SystemRescueCD with each upgrade).
I have tried booting with combinations of the rootfstype, rdinfo, rdshell, and rdinitdebug parameters, but the error happens so early in the startup process that the messages quickly scroll by and I just end up rebooting.
System details:4 1-TB drives set up in two RAID 1 pairs. FAT32 /boot partition RAIDed on the first drive pair. Two LVM partitions -- one RAIDed on the second drive pair and one on the remainder of the first drive pair. Root and other filesystems are in LVM; most (including /) are formatted in XFS.
I've made some progress in diagnosing the issue. The failure is happening because the third RAID array (md2) isn't being assembled at startup. That array contains the second physical volume in the LVM volume group, so if it doesn't start then several mount points can't be found.
The RAID array is listed in my /etc/mdadm.conf file and identified by its UUID but the Fedora 12 installer won't detect it by default. Booting the DVD in rescue mode does allow the filesystems to be detected and mounted, but the RAID device is set to be /dev/md127 instead of /dev/md2.
The arrays are on an MSI P35 motherboard (Intel ICH9R SATA chipset) but I'm using LInux software RAID. The motherboard is configured for AHCI only. This all worked correctly in Fedora 10.
I have Seamonkey set to play a .wav file when mail arrives. Since the latest update to pulseaudio, whenever mail arrives part of the sound plays, then one of my processor cores goes to 100% running pulseaudio. When I kill that process, the system returns to normal. This happens every time a sound is played.
I have a fancy new Quad Core Xeon machine here that I'm trying to install F10 x64 on. The installation went fine. When it reboots the first time, though, I get the dreaded Error Loading OS.
So I dug up my bookmark of the awesome Grub HOWTO here and followed the steps to install Grub manually. Alas, I get the same result after installing it. Even tried doing a makeactive (hd0) just in case, but no luck.
I did more searching and searched for "raid install" but didn't find anything helpful. So here I am... hoping somebody has some insight.
I was recently requested to try and convert a running system to RAID level 1. I did not succeed in this task. However, I am still interested in accomplishing this task in a test environment.What I did was create a RAID device with a device missing e.g.
Once the RAID was up and running I created the /etc/mdadm.conf file so that it would activate the RAID device on boot. Once the device was created I copied all the data from the running root filesystem to the /dev/md0 device. The only directory that I didn't copy was the /proc directory. When I rebooted the machine I had a kernel panic and the system couldn't find /dev/root. I would greatly appreciate it if someone could provide some information/tips regarding the problem.
I'm not sure if this helps and I'm not sure how to tell anyone what I found, but was a problem in Fedora 10 and 11 were VM's running in KVM would lockup due to (disk or network) IO activity. The problem silently vanished as soon as I installed Fedora 12 as my VM. I upgraded my VM to Fedora 13 and the problem re-surfaced. Oddly, no other distributions seem to have this problem any more judging from Google. My configuration:Host = x86_64, 4 cores, Fedora 11VM = i686, 2 virtual cores, Fedora 10, 11, or 13 (Fedora 12 works)What worked for me was using only one core in my VM. Things have been stable for a few days now
I have a box that doesn't have a Raid controller or a software raid running currently. I would like to make it a RAID 1. Since it seems there isn't any IDE RAID controllers hardly around, I have another HD that is the exact model as the drive currently in the box running CentOS. Can I some how add the second drive and get the box to mirror from here own out? The box gets really hot and I want to be ready for a HD failure.
I have a new Fedora 11 install with all updates working on my: Dell XPS M1330 laptop with nVidia GeForce 8400M GS
I wanted to install compiz fusion on Gnome so I followed these step by step instructions from:
Fedora refused to boot, locking up during the process.
Here is what happens...
After updates I had 3 options at boot up:
A) If I choose the TOP one (newest) it will boot up to a black screen and nothing else, I get a white block which moves with mouse. I cant get a shell and any key I hit just echoes back...ctr-alt-del will result in normal shutdown process displaying.
If I watch the services start (F1) I see all -OK- except: nvidia.ko: Driver already enabled
The boot continues with all OK until it gets to: STARTING atd: at which time the screen flashes for about 7 seconds and locks up.
B) If I choose MIDDLE boot option and watch the boot process I see most services start OK except:
-[OK]
Checking for module nvidia.ko [FAILED] nvidia.ko for kernel 2.6.30.8-64.fc11.i686.PAE was not found [WARNING] The nvidia driver will not be enabled until found [WARNING] Driver already disabled
-[OK]
Then the services continue on with [OK] until a few seconds later the screen halts with a mess of colors and symbols on screen and system stops booting. Any key pressed simply echoes back with no other result. Again ctrl-alt-del refreshes screen and shutdown of services proceeds normally.
I am having a problem with lockups on a new FC12 box (dual core 3 ghz, 4 gb memory,nvidia 8400gs). This happens reliably when replaying MythTV videos, but also randomly at other times using other apps (I suspect also boot but I can't be sure; it just occasionally stalls part way through the boot display).
Usually but not always this is accompanied by a kernel panic (caps lock + scroll lock lights flash on keyboard).
I did a core dump with kdump and it reports: Thread 1 (<main task>): Cannot access memory at address 0xffff880028025b70I am in the process of running memtest86+ right now. It's been through several passes without errors. I'm going to let it run some more, but if that is dependable it's looking like the RAM sticks are not the problem.
I am having a problem with lockups on a new FC12 box (dual core 3 ghz, 4 gb memory,HDPVR,nvidia 8400gs). This happens reliably when replaying MythTV videos, but also randomly at other times using other apps (I suspect also boot but I can't be sure; it just occasionally stalls part way through the boot display).
Usually but not always this is accompanied by a kernel panic (caps lock + scroll lock lights flash on keyboard).
I did a core dump with kdump and it reports:
Thread 1 (<main task>): Cannot access memory at address 0xffff880028025b70
I am in the process of running memtest86+ right now, and it's been through several passes without errors. I know it needs to run more, but given the reliability of this problem with MythTV video playback I am wondering if the problem could be in the video card memory.
Does anyone know of a linux (or bootable) tester for video memory? All I am able to find are some things for windoze like this.
Also is there any way to track that address from the core dump back to a physical location?
I've been running F@H on my system for a while now, these specs:
ASUS Rampage Formula Q9450 @ 3.6Ghz 4GB DDR2 8800GT 512MB Fedora 12
Running various different nvidia drivers, with working 3D, and running the latest nvidia drivers with cuda for GPU2 folding + SMP folding.
I decided to add a 9800GT to the machine for more folding PPD, and I've run into some issues.
Firstly there were problems starting X, it would hang as it started or even before it started, so I switched to RPMFusion's repo drivers which have the VGA_ARB patch applied. This now gets me into X and working 2D.
The problem I have is, although the system is completely stable running folding on all four cores, SMP, 24/7, as soon as I start anything involving 3D (such as glxgears) it will run for 3 seconds then hardlock the machine. I mean seriously lock it, no switching to terminals, and an ssh session into it from another PC dies.
Its the same with any other driver that will boot into X with the VGA_ARB patch (talking nvidia binary here).
Both cards work perfectly individually, AND both cards work, including full 3D in Windows 7 when together, which would tend to rule out a hardware fault.
I've tried to install Fedora 11, both 32 and 64 on my main machine.It could not install as it stops on the first install window. I've already filed a bug but really haven't seen any feed back yet.The bug has something to do with Anaconda and the Raid array but I really can't tell.
I have an Intel Board (see signature). I am running intel raid software under W7 currently.It works fine. But, I'm wondering, when I attempt to install F!!, is my current raid set-up causing problems? Do I need to get rid of the intel raid software and use a Fedor/Linux raid program to manage the raid array??
when trying to choose the Linux partition, I found that it would just hang at a blank screen. Looking further into the problem, I had booted with the Linux Rescue and checked the /boot/grub/grub.conf file. It appeared to be that the menu was set to hidden. I got rid of that line, and checked the boot arguments. It still had acpi=off set with the kernel and everything else seems to be in order. Root is set to root (hd0,2) which should be correct.
I have (had) Debian Testing running on a 250GB IDE hard drive, partitioned normally.
I also have 4x 1TB drives in a raid 5 using mdadm, and 2x 500GB drives in a raid 1 also with mdadm.
I put the two arrays in lvm using:
I then used "lvcreate" to make storage/backup 300GB, and the rest went to storage/media (approx. 2TB usable). I put an xfs filesystem on both and mounted them.
All was working fine until the system drive shorted out and died on me this morning. As far as I can tell, all my other drives and everything else is fine. I do a daily rsnapshot of the filesystem, which of course is residing on storage/backup (stupid, I know). So I have full backups of everything, but I'll have to put a new hard drive in and reinstall Debian before I can restore everything.
I've reinstalled before and simply reassembled mdadm arrays and remounted them before with no problems, but this is the first time I've used lvm, so I'm not sure what I have to do to restore everything. Is it as simple as reinstalling the system then doing a:
