JVM Configuration and Tuning

Recommended Core Settings

Java Settings

These are some recommended baseline settings and flags for Java 7 on a 6GB system.  For higher traffic systems, suggest 5GB to 6GB of heap on an 8GB system.  uPortal can run with a 2GB heap on a 4GB system (convenient for dev) and even less for local development, but it will spend significantly more time garbage collecting and cannot handle sudden large spikes in traffic.  With a 2GB heap or smaller it is recommended to not set NewSize and MaxNewSize and leave the defaults.

# Print computed flags after startup
-XX:+PrintCommandLineFlags

# Memory Configuration
-server -d64                   # switch to 64bit server mode
-Xms4096m                      # Set initial heap sizes
-Xmx4096m                      # Set max heap size
-XX:NewSize=2048m              # Set initial new size to 1/2 the initial heap
-XX:MaxNewSize=2048m           # Specify the max new size, generally allow the JVM to use up to 1/2 the max heap
-XX:MaxGCPauseMillis=250       # GC Pauses up to 250ms aren't bad, have ergonomics try and keep pauses below this level
-XX:+UseConcMarkSweepGC        # Force CMS for the collector
-XX:+CMSIncrementalMode        # Use incremental mode since minor CPU overhead is better than potential pauses. Useful if system has 1 or 2 processors.
-XX:+UseAdaptiveGCBoundary     # Lets the JVM adjust the young/old ratio

#-XX:+UseParNewGC              # ParNewGC is the default when using the CMS collector
-XX:+CMSClassUnloadingEnabled  # Enable perm-gen class unloading (needed with UseConcMarkSweepGC to GC PermGen)

# Set a large enough perm gen.  You can track how much you are using with jvisualvm, jconsole, or looking at the output in gc.log
-XX:PermSize=250m
-XX:MaxPermSize=400m

Java 8 has deprecated option '-XX:+CMSIncrementalMode' and based on my reading it was created for single or dual proc systems to relinquish the processor periodically (see http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#icms).  With modern multi-CPU and multi-core architectures it doesn't seem to me the option is needed.  However I have not seen any load testing that verifies that assumption (anyone willing to check it out?) so I have currently left the option in as recommended but I strongly suspect it should be removed.

JVM Logging Options

The JVM can be configured to log details about garbage collection and classloading that may be helpful when analyzing portal performance.  It is also recommended for production systems.  It can be analyzed using https://github.com/chewiebug/GCViewer

# GC Logging
-verbose:gc
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:+PrintTenuringDistribution
-Xloggc:${TOMCAT_HOME}/logs/gc.log

# Optional Classloading Logging
-verbose:class

JVM Monitoring Options

# Enable JMX Remote Monitoring
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.password.file=/my/password
-Dcom.sun.management.jmxremote.access.file=/my/access

JVM Debugging Options

# Debuging
-Xdebug
-Xrunjdwp:transport=dt_socket,address=7000,server=y,suspend=n

Socket connection timeout values

Networks and target systems can be unreliable.  A significant use of the portal is to consume data from other systems and present it to the user.  To guard against network or dependent system (databases, RSS feeds, RESTful services, etc.) issues, socket connections must have a timeout to prevent a portal or portlet worker thread from hanging indefinitely.  Though most portal and portlet code defines socket connection timeouts to guard against hanging indefinitely, it is recommended to also define some JVM-wide settings to protect all external connections.  For an Oracle JVM you do this with the following.  Adjust values to suit your preferences.

# Set connection timeout values to insure a poorly performing target system doesn't cause portlet or DB threads to hang
-Dsun.net.client.defaultConnectTimeout=10000 -Dsun.net.client.defaultReadTimeout=600000

For a more detailed discussion, refer to http://jasig.275507.n4.nabble.com/Hung-portlets-td4661265.html#a4661270

Misc Notes

One site reported having -Djsse.enableSNIExtension=false caused a calendar RSS connection to an https URL to return "java.net.SocketException: Connection reset". Seems like it should not but removing that option fixed their issue.