Ubuntu Security :: Methods To Store Password For An Encrypted Filesystem?

I've created encryption systems on servers, but nearly always I have stored the password somewhere on the machine itself. The file is always 0600 to the relevant user, but a systematic analysis of my system could easily find the scripts that invoke decryption and discover the password. (The most blatant example of this is mounting SMB shares with the "-o credential_file" option where both the username and password are plain-text. In the cases where I've used this, the security of the share hasn't particularly mattered.)

Soon I might be faced with storing "patient health information" (PHI in the healthcare world) whose privacy is heavily regulated by the provisions of the US law called HIPAA. I've been thinking about creating an encrypted partition to hold the PHI, but I need a highly fault-tolerant method for obtaining the key from a different machine than tha server itself. At first, I thought about running a script using scp and shared keys to copy the key from the remote, use it to decrypt the partition, then erase it. I'd like to be able to do this with a pipe; otherwise I'll write the key in a non-persistent location like /dev/shm.

I need more than one machine to make this work to ensure I can obtain the key when needed (like at boot). One solution is to place copies of the key on multiple servers and try each of them until I find it. A more elegant solution would place the key in a DNS TXT record. I suspect I could use LDAP for this as well, but OpenLDAP and I have never really been on speaking terms. So does this make sense? I presume I can write a bash script to do all this at boot. Most of what will be stored in this partition is the PostgreSQL database in /var/lib/pgsql and perhaps some other files.

My understanding of encrypted file systems is that they are only encrypted when unmounted. When mounted they must be as visible to the operating system as an unencrypted partition. I suppose you could apply encryption to every single disk transaction, but that would require knowing the key all the time, and would seem to add a lot of overhead.