...
At Heartland Community College, we use a free app called Lamba Probe to monitor our heap. It's just another application that can sit along side your portal and portlets. View a live demo by using "demo" for username and password. Essentially it is a glorified tomcat manager application that allows you to see stats on database connection pools, threads, http and ajp connections and memory usage. Obviously it isn't going to be much help when tomcat hangs because it itself is dependent on tomcat, but it is a great tool for performance monitoring. Just drop the application into your webapps directory and by default, any accounts with the manager role in tomcat-users.xml will be able to log in. Of course it is very easy to change the role being used for authentication or configure the application to use a custom tomcat realm to perform authentication against ldap. This application has provided us additional information needed to tune our JVMs. Be sure to add the following java option to tomcat -Dcom.sun.management.jmxremote to enable the detailed memory breakdown.
Update: PsiProbe ( mentioned on parent page) is the community driven fork of Lamba Probe
We are also running a custom linux script that automatically runs every 30 minutes and doing a screen scrap of the tomcat manager app status page for memory levels. We can then graph this data to get trend infomation to help us predict when we should restart the JVM.