Server :: RHEL Rel 5.4 Freezes With Large Jobs?

We're running
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.4 (Tikanga)
$ uname -r
2.6.18-164.11.1.el5

It hosts an Apache/2.2.3 web server. We also run apache-tomcat-5.5.23. Most of our programs are mod_perl. Sometimes our users input over-sized data sets, or queries that generate too much output. (I realize that we should try to prevent them from doing that, but right now I'm looking for a more general solution.)

When a large job runs it can 'freeze' our system. The system becomes unresponsive to everything, including command line commands. Sometimes it unfreezes after a while. Once, in this situation I was able to create a high-priority shell. ps reported:

[Code]...

Observing the machine, I see at least one very busy disk. I suspect that some high priority system process (perhaps kswapd) is using all the cpus, preventing anything else from running. Unfortunately, I cannot find much info on kswapd, or debuggging this problem.