General :: Interpreting Output Of Memory Tools In RHEL 5.3?

I have been doing a lot of research on the web to piece together the components I need to understand what is probably a very simple thing. I've read up on buffers/cache vs. physical memory and using free -m and top and understanding that output, and reviewed Red Hat's article on virtual memory. I am hoping to get some assistance in putting it all together to understand what's happening in my situation because so far I have a lot of dangling factoids and no glue. I have also referenced the thread http://www.linuxquestions.org/questi...memory-309767/ but I didn't find my answer there.

My company is running a cluster of physical web servers (they are clones so running the same OS, with same RAM and applications installed). I'll be referring to a single system although the question applies to all because they're all showing similar output. The Multi Router Traffic Grapher is showing a steady swap usage for 3-4 months now. The average usage for today is 81%. There is, as far as I know, no performance hit because of it, but a developer noticed this and wanted to know what was going on, and I'd using this opp to learn. The system is Linux version 2.6.18-128.7.1.el5, running JBOSS, postgres, Apache, and Java. Here is the output from the memory commands I have used to try and understand what's going on.

Code:

top - 21:19:27 up 159 days, 7:43, 1 user, load average: 0.52, 0.61, 0.64
Tasks: 144 total, 1 running, 143 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 8.0%sy, 0.0%ni, 91.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

[code]....

So my interpretation of this output tells me that my system has 4GB of RAM installed, 23MB of free "physical" RAM, and 1.35GB of RAM in cache (according to the second line of free -m, which represents free physical memory and useable mapped memory if I understand correctly). I have a 2GB swap partition, 1.5GB of which is currently being used, and according to the MRTG is the average that has been in use since January. So the swap size isn't decreasing. Java (JBOSS) is using the most physical RAM at 1.6GB.

My question is: if I have 1.35GB of RAM cached, why is my system using all of that swap space and not using any of the cached RAM? I thought I understood that Linux is supposed to either re-use unmodified pages in cache or send older pages to disk when more RAM was needed. It seems like my system is always sending pages to disk, which I guess would normally mean I just need more RAM but, to have close to the same swap size consistently seems like there must be more to it.