CentOS 5 Server :: Kswapd0 Using 100 Percent CPU - Drives Up System Load
Oct 20, 2010
Our CentOS 5.5 server has an intermittent problem where kswapd0 begins using 100% CPU, driving the system load to 20-30 and higher, and eventually crashes the server. The problem seems to be triggered by an intensive Java process (Lucene indexing), but only once every month or two. Lucene reindexing normally runs every 15 minutes without a problem. When the problem happens, there is still plenty of free RAM (as measured by /usr/bin/free's "buffers/cache" value, which is 1.5GB) and free swap space. The server is running MediaWiki 1.15 with the standard CentOS Apache, PHP, MySQL. My intuition is that this is a kernel/swapper bug.
Kernel info:
$ uname -a
Linux myhost 2.6.18-194.11.1.el5 #1 SMP Tue Aug 10 19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
Memory info:
Normal memory usage:
$ free
total used free shared buffers cached
Mem: 3090688 2769932 320756 0 258416 1531780
-/+ buffers/cache: 979736 2110952
Swap: 2097144 8212 2088932
Memory usage 30 seconds before a crash:
$ free
total used free shared buffers cached
Mem: 3090688 3050764 39924 0 113772 1309972
-/+ buffers/cache: 1627020 1463668
Swap: 2097144 179480 1917664
When I try to install anything recently, I was getting errors about "No Space". I noticed that the root drive (/dev/sda1) has 100% usage which I'm not sure how that suddenly happened.
There was a powerloss recently and I wondered if some serious corruption had occurred. Since I'm checking the root drive, I had fsck run after a restart:
Code: sudo shutdown -F -r now
FSCK went to work, briefly, and the logs (/var/logs/checkfs and /var/logs/checkroot) remain empty. Speaking of log files, I had a look at all of them and they take up a mere 32MB, so that's not the issue...
Using Code: du -h I know that: /var uses 1.2 GB /root uses 100 K /usr uses 1.4 GB /tmp is empty /home has 35 MB
Have already ran apt-get clean. How can I figure out what is taking up so much room? How can I go about figuring out what is huge and is safe to remove?
I am running centos 5. So far, it gives no problem but just yesterday, when it reported "no free space" for file writing, I try to remove some file as usual. Unfortunately this time no matter how much files I had deleted, it just keep showing no available space for doing so.
Result from df: [root@LSMSVR ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 1.2G 269M 879M 24% / /dev/hda6 4.8G 138M 4.4G 4% /tmp /dev/hda5 19G 2.4G 16G 14% /usr /dev/hda3 48G 12G 34G 25% /var /dev/hda2 379G 365G 0 100% /home /dev/hda1 99M 15M 80M 16% /boot tmpfs 180M 0 180M 0% /dev/shm ow to recover the lost space in /home?
The last days I try to find a good and easy solution to keep new installed Servers with same basic configuration. For example every new Installed Server should have:
- phpmyadmin installed and configured - munin installed and configured - mysql server with some predefined users and passwords - same users created with existing ssh keys. also ~/.ssh/authorized_keys file with pub key inside
At the moment I use some bash script to create the users and sync the keys. Basically I need to install predefined packages and remove some other packages configure some config files (or copy working configs). I have looked at cfengine and puppet Server. And I am asking my self if this tools are overkill for this task.
I just installed CentOS 5.5 on my machine and the installation appeared to complete successfully (it said it was successful). When I rebooted and tried to get into my new CentOS, my system completely locked up during the startup. According to the progress bar, it got stuck on the first-run configuration. I have not yet been able to boot into the OS. I am attempting to create a dual-boot system. I already have Windows XP installed on a separate hard drive. The GRUB loader works fine and I can choose either OS to boot into, it's just the CentOS won't finish booting. Windows is completely unaffected.Since I'm assuming the problem stems from the installation, I'll list the steps I followed.
1. Obtained the .iso image from a network drive at work (I am installing on my work machine). The image is dated May 17, 2010.
2. Burned the image to a DVD.
3. Booted from the DVD and chose to install using the graphical interface.
4. Checked the DVD. The installer verified that CentOS could be installed from it.
5. Picked my installation and keyboard languages.
6. Chose to create a custom layout for my partitions. On my second hard drive, I created the following partitions:
- Swap (8196MB, twice my system RAM) - ext3 (100MB, mounted to /boot) - ext3 (remaining drive space, mounted to /)
7. Picked my timezone (did not use UTC since Windows will handle setting the system time)
8. Set my root password. You don't get to know ;)
9. Did not choose to install any additional packages besides the KDE desktop. I wasn't sure what I'd need so I checked the option to customize later.
10. The installation than started and 15 minutes later, it told me it had succeeded and to remove the DVD and reboot.
11. Upon rebooting, I let the GRUB loader boot into CentOS (side note, I'd prefer if Windows was the default OS but that's something I should be able to Google on my own).
12. The startup looks like it's going fine until it gets to the first-run configuration, at which point the entire system locks up and requires a hard reboot. I've tried several times since and it always locks up at the same point.
I tore down my old ftp server running FreeNAS because it had some issues. I decided to add a couple more drives and install CentOS on it instead. So now I have 4x 2TB drives that I set up yesterday.OS is installed on a usb flash drive exclusively while I chose to set up the 2TB drives in two stripes and then mirrored, for RAID0+1 instead of RAID1+0.
/dev/md0 = drive 1+2 in RAID0 /dev/md1 = drive 3+4 in RAID0 /dev/md2 = md0+md1 in RAID1
Well, the sync time for these two mirrors of blank drives is almost done, having taken some 15+ hours.I chose this setup to have a single 3.6TB file system mirrored to another 3.6TB for "backup". As this is only going to be a headless ftp server, I don't need massive redundancy/backup. A mirror should suffice if I need to pull some files.So now I'm second guessing myself. I'm wondering if it is even worth it to stripe 2 drives into a single volume with such large sync times. Since data xfer will be over internet, I don't require the massive file writing capacity that would be preferable if it were in my home as a NAS. So, the "performance" increase of RAID0 is not strictly mandatory.
I'm wondering if I should just wipe the drives and set them up in JBOD or LVM instead of RAID0 arrays.In case of boot drive failure, I'd be booting a livecd to xfer files elsewhere or recover the boot drive. So, what difference would I see if I were livebooting to md0, md1, md2 for data viewing as opposed to livebooting to an LVM or JBOD array?As an aside, the drives are WD20EARS, which are 4k sectors which I aligned to start at sector 2048 in fdisk.
There is a tool appeared in repository called ktune; The purpose is to adjust some sysctl.conf settings to improve server speed on servers with heavy load. What is this tool for if one can achieve the same with the configuration file added to system startup? Or ktune is just such file?
After about 14 hours of running constantly, top shows that MyDNS is using 99 to 100% CPU. This is insane. There isn't any hints of why this is happening in the deamon logs.
Code: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24158 nobody 20 0 29120 3344 788 R 99 0.2 1148:24 mydns Heck, it only has 11 domains to handle.
I am attempting to upgrade a system from 4.7 to 5.2 using a (now) DVD drive attached to the onboard IDE. Originally I had tried using a remote NFS image and a USB stick but I thought maybe there was a problem with the image. I can get up to the point of the installation of selecting the keyboard for the system and then it freezes and never goes any further. It doesn't appear to be a kernel panic since I can still switch between consoles.
I've got an MSI K9NGM2-FID with 14 drives in it. It serves as a file server for our backup server. It's got a secondary 4 port Silicon Image SII 3114 SATA card using the sata_sil module, and an old IDE Promise FastTrak TX2000. Technically I could have 16 drives but the 750W PS is walking the fine line on tripping it's self-breaker with the 14 drives and 7 fans. I would like to NOT have to disconnect all of this to do the upgrade.
I thought maybe that running the install using the "noprobe" option would help so it didn't detect and load the modules for the Silicon Image or the Promise cards and detect all of the drives but it still gets stuck on the step after selecting the keyboard. The installation info console and the dmesg console don't really provide any useful information. The installation console says:
INFO : moving (1) to step welcome INFO : moving (1) to step language INFO : moving (1) to step keyboard INFO : moving (1) to step findrootparts
And the last lines of the dmesg console says:
<6>device-mapper: multipath: version 1.0.5 loaded <6>device-mapper: multipath round-robin: version 1.0.0 loaded <6>device-mapper: multipath emc: version 0.0.3 loaded
Is there a hidden "debug" option that will turn on a lot of extra logging?
Nagios had alerted me that the server had a very high load average exceeding the critical level (17+), when logging onto the server I found that all 4GB of the swap was in use despite the fact that there was 15GB+ of free memory (and that's not even including memory from cache and buffers!) Because it seems all heavily used pages were being stored in swap, the I/O wait on the server became very high, and 4 kswapd daemons were taking up nearly 100% available CPU. This did coincide with an error reported by Bacula during a backup job while changing to a bad tape...
From /var/log/bacula.log: Code: 10-Dec 02:11 bacula-sd JobId 1898: End of medium on Volume "4097" Bytes=434,170,000,000 Blocks=217,084 at 10-Dec-2010 02:11. 10-Dec 02:11 bacula-sd JobId 1898: 3307 Issuing autochanger "unload slot 4097, drive 0" command. 10-Dec 02:12 bacula-sd JobId 1898: 3301 Issuing autochanger "loaded? drive 0" command. 10-Dec 02:12 bacula-sd JobId 1898: 3302 Autochanger "loaded? drive 0", result: nothing loaded. 10-Dec 02:12 bacula-sd JobId 1898: 3304 Issuing autochanger "load slot 4096, drive 0" command. 10-Dec 02:13 bacula-sd JobId 1898: 3305 Autochanger "load slot 4096, drive 0", status is OK. 10-Dec 02:13 bacula-sd JobId 1898: Volume "4096" previously written, moving to end of data. 10-Dec 03:51 bacula-sd JobId 1898: Error: Unable to position to end of data on device "Tape-1" (/dev/IBMtape0n): ERR=dev.c:1384 read e rror on "Tape-1" (/dev/IBMtape0n). ERR=Input/output error.
10-Dec 03:51 bacula-sd JobId 1898: Marking Volume "4096" in Error in Catalog. 10-Dec 03:51 bacula-sd JobId 1898: 3307 Issuing autochanger "unload slot 4096, drive 0" command. 10-Dec 03:58 bacula-sd JobId 1898: 3301 Issuing autochanger "loaded? drive 0" command. 10-Dec 03:58 bacula-sd JobId 1898: 3302 Autochanger "loaded? drive 0", result: nothing loaded. 10-Dec 03:58 bacula-sd JobId 1898: 3304 Issuing autochanger "load slot 4098, drive 0" command. 10-Dec 03:58 bacula-sd JobId 1898: 3305 Autochanger "load slot 4098, drive 0", status is OK. 10-Dec 03:59 bacula-sd JobId 1898: Wrote label to prelabeled Volume "4098" on device "Tape-1" (/dev/IBMtape0n) 10-Dec 03:59 bacula-sd JobId 1898: New volume "4098" mounted on device "Tape-1" (/dev/IBMtape0n) at 10-Dec-2010 03:59. At the same time, these messages starting occuring in /var/log/messages:
Code: Dec 10 03:51:47 07 kernel: Mem-info: Dec 10 03:51:47 07 kernel: Node 0 DMA per-cpu: Dec 10 03:51:47 07 kernel: cpu 0 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 0 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 1 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 1 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 2 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 2 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 3 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 3 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 4 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 4 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 5 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 5 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 6 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 6 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 7 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 7 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: Node 0 DMA32 per-cpu: Dec 10 03:51:47 07 kernel: cpu 0 hot: high 186, batch 31 used:162 Dec 10 03:51:47 07 kernel: cpu 0 cold: high 62, batch 15 used:48 Dec 10 03:51:47 07 kernel: cpu 1 hot: high 186, batch 31 used:0 Dec 10 03:51:47 07 kernel: cpu 1 cold: high 62, batch 15 used:0 Dec 10 03:51:47 07 kernel: cpu 2 hot: high 186, batch 31 used:0 Dec 10 03:51:47 07 kernel: cpu 2 cold: high 62, batch 15 used:0 Dec 10 03:51:47 07 kernel: cpu 3 hot: high 186, batch 31 used:18 Dec 10 03:51:47 07 kernel: cpu 3 cold: high 62, batch 15 used:0 Dec 10 03:51:47 07 kernel: cpu 4 hot: high 186, batch 31 used:159 Dec 10 03:51:47 07 kernel: cpu 4 cold: high 62, batch 15 used:56 ... Dec 10 03:51:47 07 kernel: Node 3 HighMem per-cpu: empty Dec 10 03:51:47 07 kernel: Free pages: 732052kB (0kB HighMem) Dec 10 03:51:47 07 kernel: Active:4232128 inactive:3071288 dirty:158210 writeback:0 unstable:0 free:183320 slab:256840 mapped-file:289545 mapped-anon:3805487 pagetables:13063 Dec 10 03:51:47 07 kernel: Node 0 DMA free:10796kB min:4kB low:4kB high:4kB active:0kB inactive:0kB present:10356kB pages_scanned:0 all_unreclaimable? yes Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 3512 9067 9067 Dec 10 03:51:47 07 kernel: Node 0 DMA32 free:213332kB min:2500kB low:3124kB high:3748kB active:1794108kB inactive:1463220kB present:3596296kB pages_scanned:64 all_unreclaimable? no Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 0 5555 5555 Dec 10 03:51:47 07 kernel: Node 0 Normal free:41028kB min:3952kB low:4940kB high:5928kB active:3409444kB inactive:1471120kB present:5688320kB pages_scanned:0 all_unreclaimable? no Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 0 0 0 Dec 10 03:51:47 07 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 0 0 0 Dec 10 03:51:47 07 kernel: Node 1 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no ... Well to cut a long story short, I fixed the problem by disabling the swap partition with 'swapoff'. After about 30 mins all the swap was freed and the server went back to normal. I don't dare reactivate the swap partition and unfortunately as this is a live server which currently has no fail over, I can't reboot either
Server Spec: 4 * Dual-Core AMD Opteron(tm) Processor 8214 32GB DDR2 ECC RAM RHEL 5.5, 2.6.18-194.11.3.el5 SMP x86_64 Running many KVM VMs (All CentOS x64) and kksmd is used. bacula-dir Version: 5.0.0 IBM Tape Drive using lin_tape module version 1.34.0 according to modinfo
And before anybody asks # sysctl vm.swappiness vm.swappiness = 10
We had to reboot a server in the middle of a production day due to 99% iowait (lots of processes in deep sleep waiting for disk iops). That's never happened to us before. It had been 363 days since the last fsck, so it started automatically on reboot. It hung at 4.8% on a 2TB LVM2 volume for about an hour. I killed the fsck and rebooted the server. The second time, it went past that point and is currently at about 62%. First, what causes e2fsck to hang like that? Second, is there any danger in killing e2fsck, rebooting, and starting it again?
how the bash script should look to copy huge directory with multiple sub-folders to a new place place while checking load and stopping for several seconds if load reached lets say 3 or 4 ? I only know the simple command cp -r /dir/allfiles /dir/newplace However would like to copy over 30 000 files which will cause me a high load.
This is my first time to test piranha and I can't understand couple things: 1. how to setup public floating IP address between active and backup piranha node? 2. does piranha nodes have to run on public IP? 3. how to connect active piranha node with backup nodes? 4. does piranha only support http and ftp? 5. do I need to create common login for load monitor so that piranha could login to real servers and check the load ? 6. what is the hardware requirement for piranha to run on heavy loading site? I am using VMs all the time. I am using web gui interface with minor file editing, but I prefer done it with GUI. Currently running Centos 5.6 x86_64.
I am trying to setup a ventrilo server. I want it to load the server automatically on system boot so it is running at all times. I have downloaded the server files and can manually start it, but I do not know where/how to make it start automatically.
I am using Ubuntu11.11 on my Samsung netbook. Since my installation, I am meeting a problem from time to time that the process kswapd0 leads to high data exchange of my solid-state hard-disk, and freezes the entire system which forces me to restart my computer. This problem occurs even though kswapd0 occupies only 28% CPU resource.
recently i setup new LAMP server , after some days faced a strange problem? suddenly the server load goes to 100 or 80 but there is no wearied process running? the normal lod is between 0.5 to 1.5? the server have 2 hdd on hardware raid 0
I am using KVM and created four guest Operating systems on it.The server host is Ubuntu 10.04.I am using 4 websites in a reverse proxy environment.One of our website is running on CentOS VM.Right now there is no traffic on the website static HTML pages.I do not have any clue as why it was taking longer time to be accessed.
My sid running fine for months until recently. It is now freezed more frequently. Brief looking at process I found kswapd0 constantly consume 50% cpu right after booting in, even when swap usage is only 3%.
I use suse 11.2 86_64 on a comp with Nvidia Geforce 8300 on the motherboard using 512 Mb from main memory, AMD X2 Dual Core 4600+ RAM 8 GB DDR2 Dual channel and have following problem - from time to time when play a video (streaming or playing a file with any video player - vlc, smplayer, kaffeine etc.) the sound suddenly disappears short afterwards everything freezes even mouse, the hard disk work as wild and after 5 minutes or so the players quit and everything comes alive again. Long time I could not see the process responsible for that because even screen freezes, once I had the luck and noticed a process kswapd0 appearing in time of such an event. I googled and there were similar complying connected with that daemon. I read it will be a good idea to minimize the amount of swap, which in my case is 2 Gb , I unmounted the entire swap and restarted and the OS remains only on RAM 7,5 GB . The event appeared again and there were significant different, it obviously was shorter in time (about a minute) and mouse freezes for short period of time, but again the player quits on the end of the event. Is there a way that the process could be managed in some way as it is very annoying appearing all of a sudden with no warning or something ? Of course if that memory shifting or cleaning is the main problem, again it happens only in case player works and not every time, it is random.
so I setup a raid ten system and I was wondering what that difference between the active and spare drives is ? if I have 4 active drives then 2 the two stripes are then mirrored right?
I am building a home server that will host a multitude of files; from mp3s to ebooks to FEA software and files. I don't know if RAID is the right thing for me. This server will have all the files that I have accumulated over the years and if the drive fails than I will be S.O.L. I have seen discussions where someone has RAID 1 setup but they don't have their drives internally (to the case), they bought 2 separate external hard drives with eSata to minimize an electrical failure to the drives. (I guess this is a good idea)I have also read about having one drive then using a second to rsync data every week. I planned on purchasing 2 enterprise hard drives of 500 MB to 1 GB but I don't have any experience with how I should handle my data
I have some trouble with my hp workstation xw4400 with an ATI FireGL 3300 and the X-Server. There was no problem with the graphical install routine and the graphical boot works fine. But when the system changes into the login-screen, then it results in a black screen and a more or less frozen system.
I seem to be having a strange problem configuring Piranha to load balance (Direct route) 2 ports across 2 w2k3 servers in a test environment. What is strange is that 1 of the ports are working fine but the other port doesn't work. I've read many how-to and after many frustrating hours I disabled the firewall, iptables and arptables services and one of the ports are load balanced across the 2 real servers. Here's the environment.
[Code]....
I can telnet from the client to the realserves on both ports and it's works. When I telnet to the VIP only one port gets through and the other gives me "could not open connection to host port 32777 : connect failed. The configuration in Piranha for one port is the same as the other. I can't help but think that some other configuration for port 32777 was missed.
I have a weird performance issue with a centos 5 running a nfs server and a rh8 client. I think the fact that it is rh8 client should be downplayed. It is just that with rh8 client the performance degradation seems more clear. See test details below OS in server is Centos 5 x86_64 kernel 2.6.18-92.1.22.el5
1Gb connection between machines File to test over NFS is a 1GB file. First of all I wanted to measure how the network alone performs while using NFS. So in the server side I run a "cat" command on the 1GB file to /dev/null. Please note that the disk read speed is about 98MBs. At this point the file system has the 1GB file cached in memory. In the client side a "cat" on the same file gives me a speed of about 113MBs. It seems then that the bottleneck in this instance is the network and it is very close to nominal speed. So the network performance is really good. (BTW I know that the server got that file from cache because a vmstat or iostat shows no disk activity.)
The second test is reading from disk with no caching involve. In the server I flushed the 1GB file from the memory. For instance by reading another 5GB file and I repeat the same thing as above in the client (a cat on the 1GB file). Now, the server has to go to disk.(vmstat or iostat shows the disk activity). However the performance, now, is about 20MBs, I was expecting something closer so 90MBs. (since the reading speed in the server in the first test showed 98MBs).
This second test was repeated for ext2, ext3, xfs with no significant differences. A similar test using a RH8 NFS server and client gets me close to 60MBs for a 1GB file not cache by the file system in the serverSince network speeds and disk read speeds are not the bottlenecks ... what or where is the limiting factor then?
I run a webserver with centos 5 and like to change hardware. I run a 1u supermicro 6014t server with 4 *500gb raid 6 and like to downgrade to a smaler but more efficient server. The problem is this. I'dd like to transfer all the content and the whole OS to the new system, but how do I do that
I have (had) Debian Testing running on a 250GB IDE hard drive, partitioned normally.
I also have 4x 1TB drives in a raid 5 using mdadm, and 2x 500GB drives in a raid 1 also with mdadm.
I put the two arrays in lvm using:
I then used "lvcreate" to make storage/backup 300GB, and the rest went to storage/media (approx. 2TB usable). I put an xfs filesystem on both and mounted them.
All was working fine until the system drive shorted out and died on me this morning. As far as I can tell, all my other drives and everything else is fine. I do a daily rsnapshot of the filesystem, which of course is residing on storage/backup (stupid, I know). So I have full backups of everything, but I'll have to put a new hard drive in and reinstall Debian before I can restore everything.
I've reinstalled before and simply reassembled mdadm arrays and remounted them before with no problems, but this is the first time I've used lvm, so I'm not sure what I have to do to restore everything. Is it as simple as reinstalling the system then doing a:
I'm trying to create new RAM image file to get my server load raid1 module upon start, I was following redhat documentation & it suggested to use the following command mkinited --with=raid1 inited-raid1-$(uname -r).img $(uname -r) However after running this command I'm getting this message No Kernel available for 'inited-2.6.18-128.el5"
