Server :: How To Determine Swap Space Usage For Processes
Jan 10, 2011
On one of my servers the "free" command tells me that a lot of swap space are in use. What I'd like to do is to determine which processes have been swapped out. I tried issuing "top" and sort by the "swap" column, but this doesn't seem to provide correct values - when performing the same excersize on another server with close to no pages swapped out, the sum when adding the swap value for each process greatly exceeds the swap usage reported by "free". So how do I go about determining the swap space used for individual processes?
I have a linux server top reports about 9GB of swap used:But I cannot figure where's it use swap, some google results said that top - O commad follow by p will show swap usage by process. But as shown in the above image, taking a brief sum of the SWAP column shows that > 10GB of swap is used, so where does the 9GB figure for swap usage come from? Top reports that about 96492kb of ram is used by buffers. Is there anything I can do to utilize this, instead of using swap?
For some reason, if I leave my Linux box running for several days, the swap space and RAM slowly fill up until my system is so slow that it takes around 15 seconds just to open a new tab if Firefox (Iceweasel, specifically). I have 512GB RAM and almost a gig of swap; how on earth does it fill up so much? Even if I close all my programs, there's still over 600MB swap used and all RAM is full. I've included a screenshot of 'top' running just about two minutes after I closed all my running programs.
(Before I closed it, I had only 71MB swap free.) I know that Linux is supposed to make good usage of RAM, but isn't this over the top? Is there a way to force it to use only required memory with no or little extras kept in RAM? Just thought I'd add in the fact that I'm running Xfce as opposed to KDE or GNOME in an attempt to have a smoother running system on my old hardware. Also, what's the "VIRT" column?
Just noticed from the "top" command that one of my least heavily used box is swapping excessivly by a program called setroubleshootd. Following is the top section of the "top" command sorted by Swap used for both boxes. Also tried checking it out to see if there's a "service setroubleshootd restart" but when I checked the status I got the following.
Code: [root]# service setroubleshootd status setroubleshootd: unrecognized service Lightly loaded box with lots of swapping
I am running Apache 2.2.3 on a CentOS release 5.3 (Final) with 100 Sites. I've notice that Apache is making my server Swap around 200 MBs. "http://www.xxx.yyy.zzz/server-status" doesn't show me too much to, so I am looking the behavior of specific httpd process. ProcessID "18753" is the one for "http://www.xxx.yyy.zzz/server-status" in my browser.
This command show me (In KBs) how much virtual memory is that specific process using: # /etc/init.d/httpd start # grep Private_Dirty /proc/18753/smaps | awk '{ print $2 }' | xargs ruby -e 'puts ARGV.inject { |i, j| i.to_i + j.to_i }' 3012 ... Running this command a lot of times it gives me the same output, but suddenly... # grep Private_Dirty /proc/18753/smaps | awk '{ print $2 }' | xargs ruby -e 'puts ARGV.inject { |i, j| i.to_i + j.to_i }' 21708
Something make that process (and all the others httpd process too) to use a lot more memory!
Part of my httpd.conf: # Timeout 120 KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 3
My server is running Mysql 5.1.34, vsftpd 2.0.5, BIND 9.3.4-P1 (as slave). I couldn't found anything running in the specific time that httpd processes start to use that much memory.
My monitor tools indicate that the percentage of swap used is more than the ideal , so I want to calculate which process is consuming from the swap. I tried the following but it didn't help be 100%
#ps -eo vsz,rss,pid,args | sort -n
How to accurately calculate which process consuming from swap, just the swap no ram + swap!
Nagios had alerted me that the server had a very high load average exceeding the critical level (17+), when logging onto the server I found that all 4GB of the swap was in use despite the fact that there was 15GB+ of free memory (and that's not even including memory from cache and buffers!) Because it seems all heavily used pages were being stored in swap, the I/O wait on the server became very high, and 4 kswapd daemons were taking up nearly 100% available CPU. This did coincide with an error reported by Bacula during a backup job while changing to a bad tape...
From /var/log/bacula.log: Code: 10-Dec 02:11 bacula-sd JobId 1898: End of medium on Volume "4097" Bytes=434,170,000,000 Blocks=217,084 at 10-Dec-2010 02:11. 10-Dec 02:11 bacula-sd JobId 1898: 3307 Issuing autochanger "unload slot 4097, drive 0" command. 10-Dec 02:12 bacula-sd JobId 1898: 3301 Issuing autochanger "loaded? drive 0" command. 10-Dec 02:12 bacula-sd JobId 1898: 3302 Autochanger "loaded? drive 0", result: nothing loaded. 10-Dec 02:12 bacula-sd JobId 1898: 3304 Issuing autochanger "load slot 4096, drive 0" command. 10-Dec 02:13 bacula-sd JobId 1898: 3305 Autochanger "load slot 4096, drive 0", status is OK. 10-Dec 02:13 bacula-sd JobId 1898: Volume "4096" previously written, moving to end of data. 10-Dec 03:51 bacula-sd JobId 1898: Error: Unable to position to end of data on device "Tape-1" (/dev/IBMtape0n): ERR=dev.c:1384 read e rror on "Tape-1" (/dev/IBMtape0n). ERR=Input/output error.
10-Dec 03:51 bacula-sd JobId 1898: Marking Volume "4096" in Error in Catalog. 10-Dec 03:51 bacula-sd JobId 1898: 3307 Issuing autochanger "unload slot 4096, drive 0" command. 10-Dec 03:58 bacula-sd JobId 1898: 3301 Issuing autochanger "loaded? drive 0" command. 10-Dec 03:58 bacula-sd JobId 1898: 3302 Autochanger "loaded? drive 0", result: nothing loaded. 10-Dec 03:58 bacula-sd JobId 1898: 3304 Issuing autochanger "load slot 4098, drive 0" command. 10-Dec 03:58 bacula-sd JobId 1898: 3305 Autochanger "load slot 4098, drive 0", status is OK. 10-Dec 03:59 bacula-sd JobId 1898: Wrote label to prelabeled Volume "4098" on device "Tape-1" (/dev/IBMtape0n) 10-Dec 03:59 bacula-sd JobId 1898: New volume "4098" mounted on device "Tape-1" (/dev/IBMtape0n) at 10-Dec-2010 03:59. At the same time, these messages starting occuring in /var/log/messages:
Code: Dec 10 03:51:47 07 kernel: Mem-info: Dec 10 03:51:47 07 kernel: Node 0 DMA per-cpu: Dec 10 03:51:47 07 kernel: cpu 0 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 0 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 1 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 1 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 2 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 2 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 3 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 3 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 4 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 4 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 5 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 5 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 6 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 6 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 7 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 7 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: Node 0 DMA32 per-cpu: Dec 10 03:51:47 07 kernel: cpu 0 hot: high 186, batch 31 used:162 Dec 10 03:51:47 07 kernel: cpu 0 cold: high 62, batch 15 used:48 Dec 10 03:51:47 07 kernel: cpu 1 hot: high 186, batch 31 used:0 Dec 10 03:51:47 07 kernel: cpu 1 cold: high 62, batch 15 used:0 Dec 10 03:51:47 07 kernel: cpu 2 hot: high 186, batch 31 used:0 Dec 10 03:51:47 07 kernel: cpu 2 cold: high 62, batch 15 used:0 Dec 10 03:51:47 07 kernel: cpu 3 hot: high 186, batch 31 used:18 Dec 10 03:51:47 07 kernel: cpu 3 cold: high 62, batch 15 used:0 Dec 10 03:51:47 07 kernel: cpu 4 hot: high 186, batch 31 used:159 Dec 10 03:51:47 07 kernel: cpu 4 cold: high 62, batch 15 used:56 ... Dec 10 03:51:47 07 kernel: Node 3 HighMem per-cpu: empty Dec 10 03:51:47 07 kernel: Free pages: 732052kB (0kB HighMem) Dec 10 03:51:47 07 kernel: Active:4232128 inactive:3071288 dirty:158210 writeback:0 unstable:0 free:183320 slab:256840 mapped-file:289545 mapped-anon:3805487 pagetables:13063 Dec 10 03:51:47 07 kernel: Node 0 DMA free:10796kB min:4kB low:4kB high:4kB active:0kB inactive:0kB present:10356kB pages_scanned:0 all_unreclaimable? yes Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 3512 9067 9067 Dec 10 03:51:47 07 kernel: Node 0 DMA32 free:213332kB min:2500kB low:3124kB high:3748kB active:1794108kB inactive:1463220kB present:3596296kB pages_scanned:64 all_unreclaimable? no Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 0 5555 5555 Dec 10 03:51:47 07 kernel: Node 0 Normal free:41028kB min:3952kB low:4940kB high:5928kB active:3409444kB inactive:1471120kB present:5688320kB pages_scanned:0 all_unreclaimable? no Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 0 0 0 Dec 10 03:51:47 07 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 0 0 0 Dec 10 03:51:47 07 kernel: Node 1 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no ... Well to cut a long story short, I fixed the problem by disabling the swap partition with 'swapoff'. After about 30 mins all the swap was freed and the server went back to normal. I don't dare reactivate the swap partition and unfortunately as this is a live server which currently has no fail over, I can't reboot either
Server Spec: 4 * Dual-Core AMD Opteron(tm) Processor 8214 32GB DDR2 ECC RAM RHEL 5.5, 2.6.18-194.11.3.el5 SMP x86_64 Running many KVM VMs (All CentOS x64) and kksmd is used. bacula-dir Version: 5.0.0 IBM Tape Drive using lin_tape module version 1.34.0 according to modinfo
And before anybody asks # sysctl vm.swappiness vm.swappiness = 10
Before I start a flame war, I'd like to qualify my question with...I have a boatload of ram and a VERY thin install.(CLI openSuse 11.4-64) If I'm running the most baseline, text-only-install...and the whole system install is like 2GB or less, and I have 8GB of ram (which I could easily upgrade to 16). At install time...do I really need a swap partition at all? What purpose could a swap serve if I have that much ram in such a trimmed down environment?
Lucid on an Acer Travelmate800.Can anyone tell me why I have 0k for swap space? I allocated swap which I can see in my Disk Utility's 'volumes' display.
I'm running into a problem where my system is running out of disk space on the root partition, but I can't figure out where the runaway usage is. I've had a stable system for a couple of years now, and it just ran out of space. I cleaned some files up to get the system workable again, but can't find the big usage area, and I'm getting conflicting results.For example, when I do a df it says I'm using 44GB out of 58 GB:
Code: [root@Zion ~]# df -h Filesystem Size Used Avail Use% Mounted on
I'm interested in figuring out which programs on my machine are using swap, and how much each is using. I realize this can probably be done with top, but I am having trouble figuring how how.
What I've tried:
Start top Press f (add column) Press p (SWAP colum)
This adds a SWAP column, but the data doesn't seem to be correct. Top lists Firefox as using 582m of swap, but the header simultaneously reports that 0k of swap is being used.
I have a question. I want to monitor - CPU usage daily - RAM usage daily - Harddisk Space - top processes - hardware failure
What commands do I need to run to output the result to a log file? I know there are solutions both paid and free, but my company does not allow. they want linux built in commands or methods to do it. I do not know bash scripting. I know some commands like "df -h" to monitor harddisk space but not sure on the other stuffs.
I have to monitor the %MEM that processes are using, so I use the ps and sort command like this ....
ps -aux | sort -nk4
So this will sort by column 4 so I can see the all the processes that are using the highest %MEM. What I really want is to be able to just display the process name and the %MEM next to it. how to do this?
Is there a command to check specific processes that's using the most IO/disk usage? I know sar and ps but I want more specific details on IO on individual processes
I'm using slackware about a month now and two days ago I checked to see the usage of my RAM and I saw that there was no swap, no used, no total, nothing! (how can this be?) swap -s returned nothing, I checked fstab and there was swap there so I entered the line about swap:
I believe it' s correct. I checked after restart with "free" and the total was ok but used is 0. I copied about 5 GB to see what would happen and still nothing. RAM was nearly full but still no swap used!
We have several clusters used for high performance computing. The cluster nodes have 48GB memory and on each there is 2GB swap available. What I would like to avoid is the usage of swap as long as there is enough free memory or empty the swap as soon as there is a free memory. I thought that this could be achieved by setting the swapiness to 0, but although it helped a bit, it still didn't solve the problem completely. Even with swapiness set to 0, my memory usage looks like this:
I have a netbook and one of the suggested optimizations was to reduce the use of swap.I've noticed on my desktop and conventional laptop that it is quite usual for the swap file to be used even when memory usage is low (25% of available memory).I'm looking for some advise on the best way to set up a conventional desktop or laptop as my understanding is that the use of a swap file will slow the system responsiveness down. Can I use a version of the above to improve performance and reduce swap usage?
I have Ubuntu 10.04 installed on my PC with 1GB of RAM and 3GB of Swap partition. But the machine gets hung or reboots itself when the usage is even marginally higher. This is the output of "free -m" for Swap:
Swap: 2908 11 2897
I have tried increasing the swappiness to 80, and this not made any difference. I believe the RAM is taking up all the load and none of the pages are going in to the swap. Hence the slow response and the frequent hung system. I know this is an old PC but Windows XP (installed on another HDD) runs way better on the same configuration.
No matter what I do, the SWAP usage remains zero all the time. Is there anything I am missing? In the following case (see attached screenshot) I am running make on the kernel that I downloaded. The CPU usage is understandable. But what is the use of Swap when its never used. [URL]. I am on Kubuntu natty. My partitions are as follows:
I am running Ubuntu 10.04 amd64. So I have just returned from a week vacation to see that my / is full. When I had left, the ~80G partition was only consuming around 40G of space. I feel like this is a bug and i haven't tried a custom kernel yet to see if it is kernel level. Here is all the good information though to help debug:
Code: mount # mount /dev/mapper/pdc_cfefedbci3 on / type ext4 (rw,noatime,user_xattr,errors=remount-ro) proc on /proc type proc (rw,noexec,nosuid,nodev)
[Code].....
I am using a fakeraid with dm-raid on a raid0 for my /dev/mapper discs. I just wonder if anyone else is getting this error or if it is a problem with ext4.
You would think it would be easy to do this (and essential to maximizing and predicting the usage of tapes), but apparently this isn't so. The program MT(1) actually had some commands that give the block positition (if supported by the drive) from which the remaining space could be deduced (even if you couldn't predict exactly how much space the next archive would take up). However I'm using MT(1L) which no longer has such commands.
I am using LVM2 and have shrinked my /home partition and extended my / partition but I'm not sure if I used all the free space when growing my / partition. How can I find out? I prefer using the terminal if there is a graphical way to do this but I would like to know both ways if there are two ways.
I was trying to have a go at installing MAC on my dell alongside Ub and W7. I copied the grub2 to the first part of my sda5 so as to put the mac loader in MBR.I have since repaired grub2 back to MBR.But now my beloved Ubuntu starts up then continuosly increases RAM usage till 90% then SWAP 90% even when no application is running!!
