Nagios had alerted me that the server had a very high load average exceeding the critical level (17+), when logging onto the server I found that all 4GB of the swap was in use despite the fact that there was 15GB+ of free memory (and that's not even including memory from cache and buffers!) Because it seems all heavily used pages were being stored in swap, the I/O wait on the server became very high, and 4 kswapd daemons were taking up nearly 100% available CPU. This did coincide with an error reported by Bacula during a backup job while changing to a bad tape...
From /var/log/bacula.log: Code: 10-Dec 02:11 bacula-sd JobId 1898: End of medium on Volume "4097" Bytes=434,170,000,000 Blocks=217,084 at 10-Dec-2010 02:11. 10-Dec 02:11 bacula-sd JobId 1898: 3307 Issuing autochanger "unload slot 4097, drive 0" command. 10-Dec 02:12 bacula-sd JobId 1898: 3301 Issuing autochanger "loaded? drive 0" command. 10-Dec 02:12 bacula-sd JobId 1898: 3302 Autochanger "loaded? drive 0", result: nothing loaded. 10-Dec 02:12 bacula-sd JobId 1898: 3304 Issuing autochanger "load slot 4096, drive 0" command. 10-Dec 02:13 bacula-sd JobId 1898: 3305 Autochanger "load slot 4096, drive 0", status is OK. 10-Dec 02:13 bacula-sd JobId 1898: Volume "4096" previously written, moving to end of data. 10-Dec 03:51 bacula-sd JobId 1898: Error: Unable to position to end of data on device "Tape-1" (/dev/IBMtape0n): ERR=dev.c:1384 read e rror on "Tape-1" (/dev/IBMtape0n). ERR=Input/output error.
10-Dec 03:51 bacula-sd JobId 1898: Marking Volume "4096" in Error in Catalog. 10-Dec 03:51 bacula-sd JobId 1898: 3307 Issuing autochanger "unload slot 4096, drive 0" command. 10-Dec 03:58 bacula-sd JobId 1898: 3301 Issuing autochanger "loaded? drive 0" command. 10-Dec 03:58 bacula-sd JobId 1898: 3302 Autochanger "loaded? drive 0", result: nothing loaded. 10-Dec 03:58 bacula-sd JobId 1898: 3304 Issuing autochanger "load slot 4098, drive 0" command. 10-Dec 03:58 bacula-sd JobId 1898: 3305 Autochanger "load slot 4098, drive 0", status is OK. 10-Dec 03:59 bacula-sd JobId 1898: Wrote label to prelabeled Volume "4098" on device "Tape-1" (/dev/IBMtape0n) 10-Dec 03:59 bacula-sd JobId 1898: New volume "4098" mounted on device "Tape-1" (/dev/IBMtape0n) at 10-Dec-2010 03:59. At the same time, these messages starting occuring in /var/log/messages:
Code: Dec 10 03:51:47 07 kernel: Mem-info: Dec 10 03:51:47 07 kernel: Node 0 DMA per-cpu: Dec 10 03:51:47 07 kernel: cpu 0 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 0 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 1 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 1 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 2 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 2 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 3 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 3 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 4 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 4 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 5 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 5 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 6 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 6 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 7 hot: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: cpu 7 cold: high 0, batch 1 used:0 Dec 10 03:51:47 07 kernel: Node 0 DMA32 per-cpu: Dec 10 03:51:47 07 kernel: cpu 0 hot: high 186, batch 31 used:162 Dec 10 03:51:47 07 kernel: cpu 0 cold: high 62, batch 15 used:48 Dec 10 03:51:47 07 kernel: cpu 1 hot: high 186, batch 31 used:0 Dec 10 03:51:47 07 kernel: cpu 1 cold: high 62, batch 15 used:0 Dec 10 03:51:47 07 kernel: cpu 2 hot: high 186, batch 31 used:0 Dec 10 03:51:47 07 kernel: cpu 2 cold: high 62, batch 15 used:0 Dec 10 03:51:47 07 kernel: cpu 3 hot: high 186, batch 31 used:18 Dec 10 03:51:47 07 kernel: cpu 3 cold: high 62, batch 15 used:0 Dec 10 03:51:47 07 kernel: cpu 4 hot: high 186, batch 31 used:159 Dec 10 03:51:47 07 kernel: cpu 4 cold: high 62, batch 15 used:56 ... Dec 10 03:51:47 07 kernel: Node 3 HighMem per-cpu: empty Dec 10 03:51:47 07 kernel: Free pages: 732052kB (0kB HighMem) Dec 10 03:51:47 07 kernel: Active:4232128 inactive:3071288 dirty:158210 writeback:0 unstable:0 free:183320 slab:256840 mapped-file:289545 mapped-anon:3805487 pagetables:13063 Dec 10 03:51:47 07 kernel: Node 0 DMA free:10796kB min:4kB low:4kB high:4kB active:0kB inactive:0kB present:10356kB pages_scanned:0 all_unreclaimable? yes Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 3512 9067 9067 Dec 10 03:51:47 07 kernel: Node 0 DMA32 free:213332kB min:2500kB low:3124kB high:3748kB active:1794108kB inactive:1463220kB present:3596296kB pages_scanned:64 all_unreclaimable? no Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 0 5555 5555 Dec 10 03:51:47 07 kernel: Node 0 Normal free:41028kB min:3952kB low:4940kB high:5928kB active:3409444kB inactive:1471120kB present:5688320kB pages_scanned:0 all_unreclaimable? no Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 0 0 0 Dec 10 03:51:47 07 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Dec 10 03:51:47 07 kernel: lowmem_reserve[]: 0 0 0 0 Dec 10 03:51:47 07 kernel: Node 1 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no ... Well to cut a long story short, I fixed the problem by disabling the swap partition with 'swapoff'. After about 30 mins all the swap was freed and the server went back to normal. I don't dare reactivate the swap partition and unfortunately as this is a live server which currently has no fail over, I can't reboot either
Server Spec: 4 * Dual-Core AMD Opteron(tm) Processor 8214 32GB DDR2 ECC RAM RHEL 5.5, 2.6.18-194.11.3.el5 SMP x86_64 Running many KVM VMs (All CentOS x64) and kksmd is used. bacula-dir Version: 5.0.0 IBM Tape Drive using lin_tape module version 1.34.0 according to modinfo
And before anybody asks # sysctl vm.swappiness vm.swappiness = 10
I've got a question on free disk space. I'm currently running CentOS 5.5 on in Xenserver virtual environment. We've had an issue with disk space. My question is as follows: - from a ssh connection i run df -h this gives the value of 90% used leaving me with 9GB. If I use system monitor via a VNC connection the free disk space value is 20GB free on the same volume. Which one is correct? I do use SNMP to monitor the same volume and should alert me when < 10% is free I know this works as I set the alert threshold to < 90% I get an alert.
I have 160gb laptop. i installed vista in c primary partition which is 25gb and installed ubuntu in d primary partition which is 20gb. A remainig for my data. Now i tried to install CENT OS by formatting ubuntu. I inserted CENT OS DVD and restarted and i selected to delete my /dev/sda2 which is showing 20480mb and it shown me free space. but i tried to add partion /boot of 100mb it got added. but, when i am trying to add / of 3000mb in the remaining 20380mb free space it showing an error message that no free space is available.
I'm creating a bash script to check how much free space is left in /var directory then, if it hits a certain threshold, delete certain files with numbers for extensions (e.g. fileA.1, fileA.2 fileA.3, and fileA.4, fileB.1, fileB.2 fileB.3, and fileB.4 ). Here's a snippet from my script:
[Code]...
If I use a * as a wildcard for the number extension, the script fails. Maybe regex would work here, but I'm not particularly accomplished at it. Or some other construct.
I just used dd to clone a linux partition to a new hard drive, it had 800mb left on the old hard drive, after dd, new hard drive lists 1.29/1.3 terabytes full. Is this what happens by default in dd? How can I fix this?
I have a server running CentOS 5.3 (Final) Kernel version is:
2.6.18-128.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 athlon i386 GNU/Linux
The output from df -h is as follows:
Filesystem Size Used Avail Use% Mounted on /dev/sda2 9.5G 3.7G 5.4G 41% / /dev/sda5 4.6G 456M 3.9G 11% /var
[code]....
As you can see, /home claims to be 100% full - but yet there is actually 18Gb free? I seem to recall this could be something to do with running out of inode space?
I have a RHEL server and it's /boot only has 7MB free space on it, 122MB total size. Below is what's in the folder.Is there anything i should do to clean it up?
I'm new to fedora 13 and I have been through a few installs already with a 12TB raid. Fedora is installed on a separate 250GB drive. I've mounted the 12TB drive as a single share and I'm capturing large video files (12-90GB each) to the raid in a Samba Share across the network. The system runs great for about three days and then I start getting warning messages that "the volume filesystem root has only 1.9GB of disk space remaining" then another later 205MB etc until it eventually fills to 100% and then locks the machine. If I reboot I get a Gnome error and can't login. The only solution has been to reinstall fedora again from scratch.
Each time I allocate more space for root. My current partition is 65G in size. The raid shows only 5.1TB of space used and it shows 7.2TB of free space. The raid share shows as being mounted in /media. Root shows that it will be full at 5.2TB, and I'm almost there, so I'm probably looking at another install in just a short while when it freezes again. I've read reinstall and make a larger root partition, but I'm not sure how big that must be to avoid this problem in the future. Also, is there a limitation on the size that root can be? my question stems from the fact that I have over 7TB of free space but somehow the root is reporting as 100% full at only to 5.1TB.
just some time ago, my /usr partition's used space is started to increase rapidly, and currently it reached 17.5GB. We put /usr as a separate partition (/dev/sda2)
[root@linux root]# fdisk -l Disk /dev/hda: 40.0 GB, 40020664320 bytes 255 heads, 63 sectors/track, 4865 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/hda1 * 1 13 104391 83 Linux /dev/hda2 1276 4864 28828642+ f Win95 Ext'd (LBA) /dev/hda3 14 395 3068415 83 Linux /dev/hda4 396 526 1052257+ 82 Linux swap /dev/hda5 1276 3187 15358108+ 7 HPFS/NTFS /dev/hda6 3188 3249 497983+ 8e Linux LVM /dev/hda7 3250 3311 497983+ 8e Linux LVM
Here /dev/hda5 taken of How much capacity for NTFS (need space in MB).
I have a pc with windows on it, about 90% of the hard drive is full. I want to install dual boot ubuntu with ubuntu using about 70% of the hard drive, do I need to manually create space, or can I just set during the install will ubuntu just over-write that much. I don't care about the files I have under windows.
I am running centos 5. So far, it gives no problem but just yesterday, when it reported "no free space" for file writing, I try to remove some file as usual. Unfortunately this time no matter how much files I had deleted, it just keep showing no available space for doing so.
Result from df: [root@LSMSVR ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 1.2G 269M 879M 24% / /dev/hda6 4.8G 138M 4.4G 4% /tmp /dev/hda5 19G 2.4G 16G 14% /usr /dev/hda3 48G 12G 34G 25% /var /dev/hda2 379G 365G 0 100% /home /dev/hda1 99M 15M 80M 16% /boot tmpfs 180M 0 180M 0% /dev/shm ow to recover the lost space in /home?
I'm looking for a free backup solution how work in client-server in both environments Linux(server) and Windows(client). in my case, i want to give a disk space quota in my Linux server for each remote windows client.
I was trying to install Fedora 13, on to my laptop. I have 30 GB of unallocated space in extended partition. When trying to install Fedora 13, I got stuck, as the installer says that there is no free space for installation.can convert the unallocated space into free space.
i used gddrescure to clone an 80gb harddrive and this is the result ROFL.i guess you can only do this making sure the target drive is the same size, you see i didnt know lol so..i now have THIS problem.can anyone tell me how to turn my unallocated space into a usable 'free' space? i could play with gparted right now but i dont wanna do anything wrong, so if theres anyone who can tell me how to do this.
I have red hat linux server and it has mysql installed whenever i write on terminal command mysql -u root it shows error "ERROR 2002 (HY000): can't connect to local MYSQL server through '/var/lib/mysql/mysql.sock (111) "
And another problem is that it is showing 0 byte free space istaed of freeing the space. it may seems that both problems are dependent on each other.
i made space by shrinking my window partition and so i have unallocated and would like to add to sda2 to have more space. Check out this pic. How can i do this?
I have assigned 4G for my "/" directory, on slacware 10.2, and have not installed the GUI either. I am not sure what files to look for that have been growing over time that has completely depleted my space. Think it would be log files, but don't know where to find them.
I installed Ubuntu for a new server for a while (about one month long), then I logged in for configurations everyday. Today after I logged in system showed
Code: Usage of /: 93.8% of 35.76 GB so I used df -hl for query detail and system showed Code: Filesystem Size Used Avail Used% Mounted on /home/myuser/.Private 36G 34G 501M 99% /home/myuser
What happen with my server, Virus or not? Why that directory is so big? Is there necessary files? Can I resolve and how to do it?
I just install Ubuntu 9.10 on my flesh drive with capacity 4 Gb free space now is 300 Mb but i have found that some folder have colon FREE space and quite big some folders have free space more than 1 gb))here is a question 1: HOW COULD I GET THIS FREE SPACE? does exist any folders which are not needed for system and i can delete?