Monitoring at UWE
UWE IT Services has bought into a few commercial solutions for monitoring systems such as BMC software. However, the myUWE (uPortal) team still primarily checks the health of our systems using home-grown scripts. Despite this basic approach, our team is usually the first to spot when Eddie's in the space time continuum...
Heartbeat Monitoring
At UWE, we use a single PC in the corner of the office with a wall-mounted monitor which we use to monitor various urls - this runs a scheduled task every 10 minutes which calls a wsh script written in vbs. The script makes HTTP GET or POST (in cases where basic auth credentials are required) requests to several urls, waits for responses within 30 seconds, and checks for a specific string in each response.
These responses are collated and recorded as OK (timely response containing expected string), UX (unexpected: timely response but without expected string) or DN (down: no response within 30 seconds).
This is saved as XML and pushed out to a webserver where it is served as our 'green screen' using XSLT:
Individual boxes represent each service - green for OK, yellow for UX and red for DN. Any service which is defined as a production service in the monitoring script will also turn the background yellow, or red, thereby giving higher visibility to the issue.
In addition, emails and SMS messages are triggered by changes of state (OK becoming UX etc).