uPortal Heap Tuning

Reasoning

In preparation for the Fall 2006 semester UW Madison was looking at ways to improve portal performance. The initial effort was focused at tuning the JVM heap for the way uPortal uses memory.

Preparation

Reading

Before starting to make changes to the heap config following white papers were read to understand the options available for the heap and garbage collectors.

• Java Tuning White Paper
  • A general overview of tuning strategies for the JVM.
• Tuning Garbage Collection with Java 5
  • Details about the GC options for Java 5.
• Understanding Concurrent Mark Sweep Garbage Collector Logs
  • Provides good explanations of all that info the GC logs.

Testing

We used a jMeter load test script to test each change made to the JVM configuration. The test consisted of the script logging into the portal, visiting one other tab (chosen at random) and logging out. Each test latest 2 hours and was capped at a throughput of 2 requests per second. The throughput cap only applied to the initial view of the home tab after login and the random tab visit.

Break down of Configuration

1. Memory configuration
   JAVA_OPTS="-server"
   JAVA_OPTS="$JAVA_OPTS -Xms1280m"
   JAVA_OPTS="$JAVA_OPTS -Xmx1280m"
   JAVA_OPTS="$JAVA_OPTS -XX:NewSize=640m"
   JAVA_OPTS="$JAVA_OPTS -XX:MaxNewSize=640m"
   JAVA_OPTS="$JAVA_OPTS -XX:SurvivorRatio=5"
   JAVA_OPTS="$JAVA_OPTS -XX:TargetSurvivorRatio=90"
   JAVA_OPTS="$JAVA_OPTS -XX:MaxTenuringThreshold=12"
   JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC"
   JAVA_OPTS="$JAVA_OPTS -XX:+CMSIncrementalMode"
   JAVA_OPTS="$JAVA_OPTS -XX:+CMSIncrementalPacing"
   JAVA_OPTS="$JAVA_OPTS -XX:+CMSParallelRemarkEnabled"
   JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
   JAVA_OPTS="$JAVA_OPTS -XX:PermSize=64m"
   JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=64m"
   JAVA_OPTS="$JAVA_OPTS -XX:+UseTLAB"

1. Enable class unloading (needed with ConcMarkSweepGC
   JAVA_OPTS="$JAVA_OPTS -XX:+CMSClassUnloadingEnabled"
   JAVA_OPTS="$JAVA_OPTS -XX:+CMSPermGenSweepingEnabled"

1. debug and memory tweaks to avoid Hotspot Compiler Failure
   JAVA_OPTS="$JAVA_OPTS -XX:+PrintCompilation"
   JAVA_OPTS="$JAVA_OPTS -XX:CodeCacheMinimumFreeSpace=2M"
   JAVA_OPTS="$JAVA_OPTS -XX:ReservedCodeCacheSize=64M"
   JAVA_OPTS="$JAVA_OPTS -XX:CompileCommandFile=/my/portal/bin/hotspot_compiler"

1. Enable JMX Remote Monitoring
   JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"
   JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=9000"
   JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false"
   JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.password.file=/my/portal/bin/jmxremote.password"
   JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.access.file=/my/portal/bin/jmxremote.access"

1. turn on some debug for GC
   JAVA_OPTS="$JAVA_OPTS -verbose:gc"
   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
   JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
   JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
   JAVA_OPTS="$JAVA_OPTS -Xloggc:/my/portal/logs/portal/gc.log"

1. Enable remote debugging port
   JAVA_OPTS="$JAVA_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n"