Download the source code

Introduction

uPortal provides a number of interesting statistics through JMX, such as RenderAverage, UserSessionCount, etc. You can browse these statistics using clients like jconsole. However, for monitoring we need a custom Java client. This project is a first attempt at polling these statistics and alerting when they breach certain limits.

Features

Semi-intelligent Email Alerts

For every statistic you track you can add a simple alert that will fire when certain conditions are met. For example, if RenderAverage is over 7 seconds for 10 out of the last 15 polls, an alert is sent (The script actually reads back in your log file to check the previous results). See the JavaDoc and code in the JmxPoll.java file. There are other methods that let you set limits as a percentage of other jmx values, for example, if the activeCount is X% of the maxThreads (for our thread pool).

Alerts are turned off during certain times. This is actually hardcoded in the JmxUtil.isAlertAllowed method to deny alerts between 12am and 7am. (Like I said the code could use some refactoring; this should probably be in build.properties)

Alerts are also queued and sent every X minutes (so your inbox doesn't explode when something goes wrong)

Statistics Log

Every time the script is run it logs the results.

Columns: Timestamp, RenderAverage, ThreadCount, UserSessionCount, GuestSessionCount, (next 4 columns are numActive, numIdle for jdbc/PortalDb and another db), HeapMemory, NonHeapMemory, SystemLoadAverage, tomcatThreadPool.activeCount

2009-11-09 13:59        823     139     394     37      0       2       0       3       454/1820        146/304 0.27    1
2009-11-09 14:00        933     139     396     38      0       2       0       3       466/1820        146/304 0.26    1
2009-11-09 14:01        914     142     388     39      0       2       0       3       400/1820        146/304 0.19    0
2009-11-09 14:02        905     142     381     37      0       2       0       3       415/1820        146/304 0.07    0

Setup

Tomcat Setup

Create these files in $TOMCAT/conf:

Add the following CATALINA_OPTS

Code Configuration

1. Extract the source somewhere
2. Copy build.properties.in to build.properties and edit appropriately
   1. Specify your username and password you defined above, change log paths, set your email address, etc.
3. Look over and Edit CalPolyJMXStats/src/main/java/edu/calpoly/jmx/clients/PortalJmxStats.java
   1. You probably don't have a jdbc/WarehouseDB, so remove those lines or you'll get an error.
4. mvn package
5. ant
   1. This runs ant lib, which copies Maven runtime dependencies to the lib directory (Requires Maven Ant Tasks)
6. Run portalJmxStats.sh to test
7. You probably want to run the sh script on a cronjob with a line like this:

* * * * * /deploy/scripts/jmxStats/portalJmxStats.sh

Notes

The StudentPay stuff is another Java application at CalPoly. You can just ignore it. You might consider adding your own clients and creating the appropriate sh scripts for them.