Fedora :: On Boot: Mdadm Failed To Start Raid: Input/Output Error?
Jul 31, 2010
I have a strange issue with my RAID5 array - it worked fine for a month, a couple days ago it didn't start on boot with mdadm reporting "Input/Output error" - I didn't panic restarted my computer, same error. Then opened a Disk utility and it reported State: Not running, partially assembled - don't know why, I've pressed Stop RAID Array and started it again, voila - it reported State: Running - I've checked components list and there was nothing wrong with it. So I run Check Array utility, waited almost 3 hours to finish it and it worked since than, till today's morning - I've started my computer, and here we go, same error.
See screenshots:
This is an initial state just after computer startup:
This is after I stop and start RAID5:
This is a components list:
I can see nothing wrong there yet not sure why mdadm fails on boot. I do not really like the windows solution I guess, when I check my array again, it will work fine again, but it then can fail in the same way without known reason.
i want to use linux but i am having some issues getting it to work on my system. my desktop is a p4 3.4gig with 512 DDR. When I insert my ubuntu CD it starts loading but then i get this screen/message:
Busybox v1.15.3(ubuntu 1:1.15.3-1ubuntu5) built-in shell(ash) Enter 'help' for list of built in commands (initramfs) mount:mounting/dev/loop0 on// filesystem.squashfs failed input/output error Can not mount /dev/loop0 (/cdrom/casper/filesystem/squashfs) on //filesystem.squashfs if I put the same disc into my HP laptop it boots up ubuntu fine. (however it won't allow me to get online, but that is a different issue).
I have a raid5 on 10 disk, 750gb and it have worked fine with grub for a long time with ubuntu 10.04 lts. A couple of days ago I added a disk to the raid, growd it and then resized it.. BUT, I started the resize-process on a terminal on another computer, and after some time my girlfriend powered down that computer! So the resize process cancelled in the middle and i couldn't acess any of the HDDs so I rebooted the server.
Now the problem, the system is not booting up, simple black with a blinking line. Used a rescue CD to boot it up, finised the resize-process and the raid seems to be working fine so I tried to boot normal again. Same problem. Rescue cd, updated grub, got several errors: error: unsupported RAID version: 0.91. I have tried to purge grub, grub-pc, grub commmon, removed /boot/grub and installed grub again. Same problem.
I have tried to erased mbr (# dd if=/dev/null of=/dev/sdX bs=446 count=1) on sda (ide disk, system), sdb (sata, new raid disk). Same problem. Removed and reinstalled ubuntu 11.04 and is now getting error: no such device: (hdd id). Again tried to reinstall grub on both sda and sdb, no luck. update-grub is still generating error about raid id 0.91 and is back on a blinking line on normal boot. When you'r resizeing a raid MDADM changed the ID from 0.90 to 0.91 to prevent something that happend happened. But since I have completed the resize-process MDADM have indeed changed the ID back to 0.90 on all disks.
I have also tried to follow a howto on a similar problem with a patch on [URL] But I cant compile, various error about dpkg. So my problem is, I cant get grub to work. It just gives me a blinking line and unsupported RAID version: 0.91.
I just had a whole 2TB Software RAID 5 blow up on me. I rebooted my server, which i hardly ever do and low and behold i loose one of my raid 5 sets. It seems like two of the disks are not showing up properly.. What i mean by that is the OS picks up the disks, but it doesnt see the partitions.
I ran smartct -l on all the drives in question and they're all in good working order.
Is there some sort of repair tool i can use to scan the busted drives (since they're available) to fix any possible errors that might be present.
Here is what the "good" drive looks like when i use sfdisk:
Quote:
sudo sfdisk -l /dev/sda Disk /dev/sda: 121601 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sda1 0+ 121600 121601- 976760001 83 Linux /dev/sda2 0 - 0 0 0 Empty
This is what I get when following the instructions to fix grub. I type in "grub-install /dev/sda" and that's the error I get.I'm trying to put Ubuntu or Mint on a Dell Mini 9 with 16gb ssd, but no matter what I do I cannot achieve this goal
My dad is out of town for a while, so I've been using his computer to rip my DVDs with Handbrake. He has Windows Vista installed, so I did a full installation of Fedora 13 on a thumb drive and just run everything off of that. However, when I went to delete some of the encoded videos after I had moved them to my computer, I got a system error that said 'can't delete files: I/O error.' Now, if I try to boot up Windows, his computer is stuck on a black screen checking a bunch of dlls. He'll be back in a couple of weeks, so I'd like to have it fixed by then.
This is message I get when I try and start itmdadm: /dev/md0 assembled from 2 drives - not enough to start the arrayBelow is the information I've collected about any help on how I can get the raid back up and going to I can get the data off of it would be awesome
I even threw some video DVDs at it to make sure it wasn't the disc.
Code:
[pickens@acer1 Videos]$ dd if=/dev/sr0 of=POTC.iso dd: reading `/dev/sr0': Input/output error 5088+0 records in
[code]....
I am getting the same thing on my laptop running Mandriva, oddly enough. Two different drives, two different computers, two different distros and multiple DVDs.
When running rsnapshot tonight it failed with a bunch of
cannot remove `file': Input/output error
errors.
I navigated to the directory and tried to delete it manually and I got the same errors. I tried with 'sudo rm directory'. How else can I delete this directory? What would cause a situation like this?
I'm either missing the obvious or have four identically defective tape drives. Each is a Seagate or IBM Travan drive (yes, I know they're old and I need to access the data; they were in Win boxes) using QIC-80/DC-2120 media. I've been trying to solve this using three Fedora boxes (different versions) and one Debian machine.
The OS's recognize the drives and they show up as /dev/st0.
With no media in the drives, I get a status report; with media inserted, I get the same input/output errors.
Examples (all as root):
With tape inserted:
With no tape in drive:
With tape inserted
Everything I've found on Google suggests that the drives are defective.
I'm trying to install Wine on Fedora 13, I tried installing it from the "Add/Remove Software Repository" but the installation fails, I tried installing it from the Terminal using the following command as root: "yum install wine"
I booted my Fedora 13 live CD on my Dell server, and configured networking and SSHd. On my desktop I logged into the live cd and started copying over a big file (1.5 GB) (i.e. copying from the desktop computer to the server). After copying about half the fil, the download fails with an error message about the filesystem being read-only.I've seen this exact same behavior on both i386 and x86_64 of the same live CD, but don't know why it's happening. I've also had the same issue when copying small files. In /var/log/messages these messages are repeated many times:
Code: Dec 20 12:32:23 localhost kernel: Buffer I/O error on device dm-0, logical block 684075 Dec 20 12:32:23 localhost kernel: lost page write due to I/O error on dm-0
My home-backup server, with 8*2TB disks won't boot anymore. Two disks failed at the same time and i rebuilt the raid 6 array without any problem, but now i can't boot the os. I'm using ubuntu server, 10.10. I've made screens of the displays to don't copy everything here. The problem at the boot:
And the Grub config: It's not a production server, but i would like to have it online. I've tried for the lasts 2 days (just a couple hours a day) but without success. I was suggested to do "mount -o remount,rw /" and than edit /etc/fstab, but it get the file don't exist error.
Tried rebooting with "Option" key and choosing Linux, but no difference.I am trying this on a Mac G4 with Ubuntu 10.10 installed on its own drive.There is another drive with Mac OSX 10.2.8.
I am running a 14 disk RAID 6 on mdadm behind 2 LSI SAS2008's in JBOD mode (no HW raid) on Debian 7 in BIOS legacy mode.
Grub2 is dropping to a rescue shell complaining that "no such device" exists for "mduuid/b1c40379914e5d18dddb893b4dc5a28f".
Output from mdadm: Code: Select all # mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Wed Nov 7 17:06:02 2012 Raid Level : raid6 Array Size : 35160446976 (33531.62 GiB 36004.30 GB) Used Dev Size : 2930037248 (2794.30 GiB 3000.36 GB) Raid Devices : 14
[Code] ....
Output from blkid: Code: Select all # blkid /dev/md0: UUID="2c61b08d-cb1f-4c2c-8ce0-eaea15af32fb" TYPE="xfs" /dev/md/0: UUID="2c61b08d-cb1f-4c2c-8ce0-eaea15af32fb" TYPE="xfs" /dev/sdd2: UUID="b1c40379-914e-5d18-dddb-893b4dc5a28f" UUID_SUB="09a00673-c9c1-dc15-b792-f0226016a8a6" LABEL="media:0" TYPE="linux_raid_member"
[Code] ....
The UUID for md0 is `2c61b08d-cb1f-4c2c-8ce0-eaea15af32fb` so I do not understand why grub insists on looking for `b1c40379914e5d18dddb893b4dc5a28f`.
**Here is the output from `bootinfoscript` 0.61. This contains alot of detailed information, and I couldn't find anything wrong with any of it: [URL] .....
During the grub rescue an `ls` shows the member disks and also shows `(md/0)` but if I try an `ls (md/0)` I get an unknown disk error. Trying an `ls` on any member device results in unknown filesystem. The filesystem on the md0 is XFS, and I assume the unknown filesystem is normal if its trying to read an individual disk instead of md0.
I have come close to losing my mind over this, I've tried uninstalling and reinstalling grub numerous times, `update-initramfs -u -k all` numerous times, `update-grub` numerous times, `grub-install` numerous times to all member disks without error, etc.
I even tried manually editing `grub.cfg` to replace all instances of `mduuid/b1c40379914e5d18dddb893b4dc5a28f` with `(md/0)` and then re-install grub, but the exact same error of no such device mduuid/b1c40379914e5d18dddb893b4dc5a28f still happened.
[URL] ....
One thing I noticed is it is only showing half the disks. I am not sure if this matters or is important or not, but one theory would be because there are two LSI cards physically in the machine.
This last screenshot was shown after I specifically altered grub.cfg to replace all instances of `mduuid/b1c40379914e5d18dddb893b4dc5a28f` with `mduuid/2c61b08d-cb1f-4c2c-8ce0-eaea15af32fb` and then re-ran grub-install on all member drives. Where it is getting this old b1c* address I have no clue.
I even tried installing a SATA drive on /dev/sda, outside of the array, and installing grub on it and booting from it. Still, same identical error.
I have been having this problem for the past couple days and have done my best to solve it, but to no avail. I am using mdadm, which I'm not the most experienced in, to make a raid5 array using three separate disks (dev/sda, dev/sdc, dev/sdd). For some reason not all three drives are being assembled at boot, but I can add the missing array without any problems later, its just that this takes hours to sync. Here is some information:
Recently updated to kernel 2.5.32.11-99 and upon restart, failed to boot. Get stuck at a recovery prompt asking for root password or Ctrl-D to continue (which reboots system). I worked around this problem by reverting to the previous kernel 2.6.32.9-70. From what I can tell, with the newer kernel there is a problem with device enumeration of software raid volumes.
I installed Fedora 12 and performed the normal updates. Now I can't reboot and get the following console error message.
ERROR: via: wrong # of devices in RAID set "via_cbcff jdief" [1/2] on /dev/sda ERROR: removing inconsistent RAID set "via_cbcff jdief" ERROR: no RAID set found No root device found Boot has failed, sleeping forever.
120GB Sata HDD - Primary OS drive 3 x 1.0TB Sata HDD - Raid 5 array
This is on a C2D MSI P35 Platinum board. Anyway, did a fresh install of F12 on the 120GB, which I had problems with - Anaconda refused to see the drive. Fedora Live could see it fine, and it was listed as an 'nvidia_raid_member' - no idea why, but I completely erased the disc under the Live CD and proceeded to install F12.
Once F12 was installed, I loaded up mdadm to re-activate my Raid 5 array, using 'sudo mdadm --assemble --uuidthe uuid) - and it started with only 2 of the 3 drives. My /dev/sdb drive did not activate into the array, due to what mdadm said was a mismatched UUID. Ok, so I erased /dev/sdb, intending to rebuild the array. Erased /dev/sdb, and then attempted 'sudo mdadm --add /dev/md0 /dev/sdb' and I get this error: "mdadm: Cannot add disks to a 'member' array, perform this operation on the parent container" - I can find NO information on this error message.
[Code].....
I don't believe the hard drives are connected in the exact same order they were in before - I disconnected everything in the system and blew it out (it was pretty dusty)
Got a little problem after the install of Fedora 12. First there was not problem in opening the raid-device, after i tryed to automount it with crypttab and fstab im not longer able to open it. Here some outputs code...
Today went from jaunty to koala then to lucid. The package manager doesnt work nothing works in terminal for apt-get or anything getting the message ---->
Quote:Error benchmarking: helper exited with exit code 1: Error reading 27099136 bytes at 2509258752 from /dev/sda: Input/output errorthis issue is also preventing me from making a copy/backup of the partition with gpartedwhich i decided i should do since i had a scare when booting todayunable to mount / i said fix itthen it said /var was not readyi waited it became readyi am running 10.04.2 would upgrading the kernel to get trim support fix it?
Code: sudo fsck -t ext4 /dev/sda1 fsck from util-linux-ng 2.17.2 e2fsck 1.41.14 (22-Dec-2010) System: clean, 247383/983040 files, 1394162/3907803 blocks Solution: drive is under warranty and it needs to be used this is not software problem
I have dir in my HD that is very important to me,but when I try cd commad to go in the dirI got this message:cannot access `./Lab': Input/output erroralso when use ls to view the currently dir and the other dir's also I got this message:"ls: cannot access Lab: Input/output errorLab"with touch:touch: cannot touch `Lab': Input/output errorwith mkdir:mkdir: cannot create directory `Lab': Input/output error
We are running an C++ application in AWS EC2 instance (CentOS 5.4) mounted with an EBS Volume (say /mymountpoint). We do more simultaneous writes to the EBS Volume from our application. But at some point we get 'ERROR: Input/output error'. After this, 'ls -l /mymountpoint' command itself fails with the i/o error. The filesystem which we use for the EBS Volume is xfs. I unmounted the drive and done xfs_check and again mounted the drive. Now, everything seems to be working fine. But the issue still persists everytime when we do simultaneous writes.
I believe the following details will be useful, [root@domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release CentOS release 5.4 (Final) .....
Output of dmesg : SGI XFS with ACLs, security attributes, large block numbers, no debug enabled SGI XFS Quota Management subsystem Filesystem "sdh": Disabling barriers, not supported by the underlying device XFS mounting filesystem sdh Ending clean XFS mount for filesystem: sdh ..... Starting XFS recovery on filesystem: sdh (logdev: internal) Ending XFS recovery on filesystem: sdh (logdev: internal) The XFS utilities are in v2.9.4
I rebuilt my box with Fedora 14. installed httpd, mysql php etc. when I tried to start my httpd it was giving me error about finding the root document and server name I was able to fix temp the issue with the name by placing ServerName 192.168.1.105:80 in the httpd.conf file and the root document somehow was fixed also. but I am still can't start the httpd and no error message except that it failed. I also disabled Selinix but still no luck.
