Ubuntu Servers :: Hdd Hiccups - Wait At 99%, Iostat Util At 99% For NO Reason?
Jan 28, 2010
I use this machine as a samba server with one small IDE hdd for system and one large SATA hdd (1.5TB) hooked via 4xSATA PCI card. The machine has 1.5GB RAM, and is also to run 2 ktorrent clients inside two Xvncs.The problem is, that even when ktorrent is doing nothing I observe A LOT of hdd activity to the point where movies or even mp3s stored on this server played via samba on a windows machine "stutter". E.g. smplayer will repeat 5 second piece a few times before moving on to the next piece... and it goes on for a good 10 minutes, once it starts.Even browsing directories is slow to the point where it takes 5-20 seconds to show the content in Total Commander (equivalent of Midnight Commander for windows).I am not sure how I can track what is really happening. Why would ktorrent clients create a massive I/O when there's virtually no traffic to/from them? (I have total of 1kB/s down and 10kB/s up while taking the masurements below).Or is there something wrong with the hdd? I had to send back the first one I got, it had plenty of bad sectors (this one does not though, as far as I can tell no data loss occured, just performance sucks).Here's some diagnostic data, please let me know if there's anything else I should check.
Core 2 Duo E4600 2GB DDR2 RAM (1 stick) Intel ICH10R based motherboard (tried an ICH9R aswell) 4-port SATA controller (PCI Sil 3114) O/S: Ubuntu Desktop x64 10.04 LTS (using 'desktop' because I like having a remote desktop)
The Storage Setup Disks: Assorted selection of 9 disk. 750GB, 1000GB and 1500GB Seagate and Western Digital disks. The disks are joined through a standard LVM2 configuration. I don't know the LVM term, but normally you'd call it a JBOD setup. On that LVM device, I've put a cryptsetup device, made with the LUKS tools (aes-xts-plain 256) On the cryptsetup device, I've created and mounted an EXT4 partition.
All in all, a completely standard LVM2 and LUKS setup, running EXT4. After a reboot, I proceed to unlock my cryptsetup encryption device, and then mount the EXT4 partition. All is well, the mount is accessible and everything looks fine. I then try to send a file to the mount, via Samba. After a few hundred MB written, the I/O wait goes berserk. It stays at 50% (dual core setup remember). The system becomes unresponsive to network commands (can't browse samba) for about 5-10 minutes. When it finally responds, the I/O wait is gone and everything is now fine. I can write and read hundreds of GB's of data without any issues at all. I can benchmark and stress all disks perfectly fine and no logs are showing disk errors.
I tried monitoring my disks with 'iostat -d 2' while the I/O wait was happening, and there is some slight Blk_read/s activity on 1 disk at a time. First for example /dev/sda is showing a little Blk_read/s acitivty, then it jumps to the next disk, and when every disk has show that slight Blk_read/s activity (500-800 or so) the problem is gone and the I/O wait is no more. I've tried changing motherboards, switching disks around on the controllers, checking individual disks, replacing disks and I've tried different versions of Ubuntu. The problem however persists. I could see it being a network issue, possibly a driver issue. But since the NIC is a standard RTL8111 on-board it seems unlike that the problem wouldn't be more widespread since this NIC is litterally being used everywhere. I did change my motherboard, so a faulty NIC seems unlikely twice in a row.
I have a start and stop script registered with chkconfig for a fuse mounted drive. I can /sbin/service scriptname start and /sbin/service scriptname stop with no issues.
The stop script unmounts the drives and then stops the service with a function to wait until the pidof service is gone. My drives are showing up as dirty on reboot, which should mean that my script that functions normally in cli, is running out of time and possible being ignored by service after a certain wait period when shutdown -h now is issued. System is a remote server so I can not view the output during shutdown. How can I increase the wait before /sbin/service issues a [FAILED] if that is the case on shutdown? The startup script runs, however the shutdown script does not...the script does work though with the service scriptname stop command.
I'm using Ubuntu 10.04 and have it hooked up to my HDTV. My GPU is an 8600 GT and I am using the current nvidia driver. Mother board is GIGABYTE GA-MA78GM-S2H, CPU is AMD 4850e, using 2x1G ram 800MHz. Everything I have installed on it comes straight through Synaptic. Smplayer seems to work fine with the VDPAU selected (cpu usage is nice and low during HD playback), but there is a slight problem that really annoys. Every 2-3 seconds the picture seems to have the hiccups, its as though it gets stuck on one frame for a split second and then jumps to the next.
we have 22 servers Dell (20 Poweredge R210 + 2 Poweredge R310) with CentOS 5.5 x86_64 (kernel 2.6.18-194.26.1.el5). Half of these are on different rack. During the day, some server reboot without any reason and the messages log file had no evidence of what happened.
From /var/log/messages: ... Dec 8 14:10:02 hadoop011 snmpd[3029]: Received SNMP packet(s) from UDP: [172.27.1.21]:50825 Dec 8 14:10:02 hadoop011 snmpd[3029]: Connection from UDP: [172.27.1.21]:50825 Dec 8 14:15:26 hadoop011 syslogd 1.4.1: restart.
[Code].....
Some minutes after the reboot, messages log this error: Dec 8 14:18:14 hadoop011 Server Administrator: Instrumentation Service EventID: 1404 Memory device status is critical Memory device location: DIMM_A3 Possible memory module event cause:Multi bit error encountered.
Is it possible that 20% (circa) of 60 memory bank are broken? By the way, memory bank have Lifetime warranty from Kingston. If I reboot again the server affected by this error, the error disappear from messages log file and from OpenManage Web interface.
Is it possible that is a CentOS bug? I had search some relative bugs on Bugzilla but without any results.
I have a RHEL5 server that hosts an apache SSL proxy and about 20 tomcat instances. As of late we've had latency issues on the system that I can't pin down. In trying to diagnose whether the local HD is being over-utilized, I started gathering disk utilization stats using iostat and sar. For Sar, I'm using the "tps" metric, and for IOStat I'm combining reads and writes per second for the raw disk device, sda. When I put the stats into excel, the profile of the graphed data points match up for the most part, but sar is reporting the values for the same data points as being many magnitudes higher. Can anybody give me a hint as to why one tool would report the same data differently when (as far as I know) both of them pull their disk I/O stats from the same place?
I am trying to set-up a home server using Ubuntu 9.10 server edition. My PC has 4 x 500Gb SATA drives which I will use for storage. I also have a 80Gb IDE drive which I want to install OS on. During installation the 80Gb IDE drive shows as sde. When doing the install I chose to install to whole disk (sde), all appears to go well. When I reboot all I get is "GRUB loading, please wait.... Error 15" Is the problem caused by my drive being sde and not sda? I have also tried partitioning sde as /. /boot and swap, still get Error 15.
I have network drive which connected to my wireless router via usual LAN cable. I mount this network drive using Samba to my Kubuntu laptop filesystem. When I play music (MP3, Flac) from that drive, it sometimes hiccups - music stops for few seconds and then resumes. I usually use VLC, but MPlayer and Audacious have same results. I tried to increase buffer in my VLC -> ALSA settings. I increased it from default 300 to 5000ms, still same result. I have very similar issue on same laptop when booted to WinXP. The only player which does not have hiccups in XP is KMPlayer. But it does not want to work in Kubuntu at all.
I need to write a wrapper script for iostat functionality . in solaris unix i have iostat but in linux i don't have iostat , so I need to write a wrapper script which does all the functionality of iostat . i am entirely new to scripting
Trying to understand grep,sed,awk but maybe its too early for me and also i suspect iostat is not the correct program for exactly what i'm looking for...The goal is to print only the current read and write speeds of the disk, represented in a numerical value with two separated commands. So for example when writing a file to disk from an external disk, the value reflects the speed of the process.
"iostat -dk sda" prints Code: Linux 2.6.38 (Infidel) 07/30/2011 _x86_64_ (4 CPU)
I ran into this problem myself, and I've seen that others have had the same problem when updating from Karmic to Lucid via apt-get. There is a bug in apt for Lucid, and it has been noted on one of the Ubuntu lists
This is a simple how-to and a work-around to the upgrading with apt-get:
1) Update your sources.list file to replace all instances of 'karmic" to 'lucid' : $ sudo sed -i 's/karmic/lucid/g' /etc/apt/sources.list
2) Run apt-get update: $ sudo apt-get update
3) Run a regular upgrade; $ sudo apt-get upgrade -y
Here is where the problem comes in. Apt is looking for the util-linux package, which is part of upstart-job, but someone apparenlty forgot to write the code to tell apt-about this (whoops!)
So, you now need to install upstart-job by itself. By skipping this step you will get the error message when you try to run step 5: "E: Could not perform immediate configuration on 'util-linux'.Please see man 5 apt"
4) $ sudo apt-get install upstart-job
5) Now you can run apt-get dist-upgrade: $ sudo apt-get dist-upgrade -y
I've found that my skype stopped working with recent -current updates ( around beta ... ). It was looking for a libuuid.so.1 library - on checking I found the 64-bit version but no 32-bit version - quite strange as this has been working fine since my original slackware64 13.0 installation. Anyhow, I got the 32-bit util-linux package, did the 32-bit package convert thing and skype is now working:
1. convertpkg-compat32 -i util-linux-ng-2.17.2-i486-1.txz 2. un-archive 3. copy the libuuid lib to /lib and soft-link in /usr/lib
I didn't install the whole compat package as this is not really necessary I think.
I'm looking for suggestions for a command line utility than can generate a contact sheet (e.g. jpg etc) from a video? I'm mainly interested in starting from FLV files, but i guess the solution will probably work for other formats with appropriate codecs etc. So, for example, if i have a 30 minute video, every 2 minutes through it i'd like it to generate a thumbnail and then put 15 of these in a 5 x 3 grid in a single jpg. On Windows,Media Player Classic Homecinem will do this through its ave Thumbnails command.
Ideally i'd like to do this from the command line/script in a debian server environment, rather than a desktop GUI. It doesn't have to be a single command, if a chain of steps could do it.
My ubuntu stops when mounting system hdd. The screen display the following messages :
mountall:/etc/fstab: No such file or directory fsck from util-linux-ng 2.16 WARNING: couldn't open /etc/fstab: No such file or directory init: mountall main process (545) terminated with status 1 General error mounting filesystems. A maintenance shell will now be started. CONTROL-D will terminate this shell and re-try. udevd[560]: can not read '/etc/udev/rules.d/z80_user.rules. Ubuntu: clean, 474879/24231936 files/28016581/96898047 blocks root@i7:~# exit_
I suspect the disk manager pysdm that i had just installed today and it had crash during the previous session. The /etc/fstab file does not exist anymore and i cant rename the fstab.bak because the disk is read-only even for my root user
I read recently that, if I install the 10.04 RC on my laptop I dont have to worry about reinstalling the 10.04 'official' version that comes out on the 29th.
I have a small question.We use a backup program, that start an sepparat process with the name SIDB.When the server get an shutdown command. The normale back-up process is stoped. But the SIDB is not stopped correctly. But the shutdown proces should wait till the SIDB proces is stoped, and then proced with the shutdown process
How do I create a command to launch a program and then have the terminal wait for a specified time and then move on to the next command?I'm wanting to create a startup script, and I need program B to wait until program A has finished loading up.
I applied all updates to my Kubuntu 10.04 installation, and shutdown and went to bed. Booted up this morning and found: [code]...
This error is returned if the module load command is used before loading a Multiboot kernel. It only makes sense in this case anyway, as GRUB has no idea how to communicate the presence of location of such modules to a non-Multiboot-aware kernel. lets re-install GRUB. So I booted up my live CD, and ran:
Same error. So I started looking around, and most complaints of this particular error seem to be dual boot environments, which I found rather odd, as I have never dual booted this machine.....
I'm having trouble with a bash script. Does anyone know why this doesn't work?
Code: nautilus ./ & wait $!
I'm writing a script which will extract a series of .rar files, present the extracted files to the user in nautilus so they may modify them, then when the user closes nautilus, the modified files are packed back into the archive.
I've installed both Fedora 15 x64 and Fedora 14 x64 on an HP Proliant DL380 G4 with the HP SmartArray SCSI controller, 2x146GB UW320 SCSI drives.The server was running RH v4 with a hardware RAID 1 with uptime of 400+ days with no issues.I broke the HW RAID and made 2x volumes and software RAID. I have another identical server with 4x 146GB UW320 SCSI drives with Fedora 12 x64 and SW RAID, so I figured 14/15 should be the same process.What happens is the server runs great for a while, but when I leave it for a few hours, it becomes non-responsive. I've re-installed 15 and 14 multiple times now and also installed NRPE, which shows load increasing to the point to where the server is non-responsive. It hits high load levels (as reported by NRPE). The load levels appear to be increasing over time. Such as last night:
log off at midnight, load average 0/0/0 2:03 am - WARNING - load average: 9.91, 8.85, 6.75 3:03 am - WARNING - load average: 16.91, 15.86, 14.47
I'm running home server on Debian stable with DHCP, DNS, Mail, VDR, Filesharing and my Weatherstation as main services. The filesharing is used to mount homes at clients. The machine features an Athlon BE-2300, 3GB RAM, GB-LAN, 1TB and 1.5TB SATA HDD plus HDDs for backups. Mainboard has an NVIDIA chipset with
Code: nVidia Corporation MCP65 SATA Controller (rev a3) The primary disks are running in RAID1 + LVM.
I have a certain process (or processes) and I want to wait for them to finish before launching another one. Do I have to make a while loop checking the process for this task or is there a ready command for it?
The application I am using writes a report file (ascii), but I am not able to print it, because from inside the application I have no access to the shell and so cannot invoke any printing command (lpr, lp, ...). Is there a tool/daemon which waits for a file in a given directory, takes it, and sends it to a printer?
I am trying to learn C++.I implemented a simple archive program, and I am in a situation in which the user is prompted by a menu to make a choice.So I have some cout instruction to illustrate the possible choices and then
int choice; cin>>choice;
and everything works fine.I introduced this code in a "while" loop that checks wether the choice made by the user is valid or not:
bool check=true; int choice; while(check) { cin>>choice; if(the choice is valid) {...;check=false} else cout<<"please make an other choice" }
What is happening is that if by mistake the user introduces a character in place of a number, the loop repeats indefinitely because the program, when it get to the "cin" instruction, does not pauses to wait for a new input.
