StatsRecorder

StatsRecorder prior to uPortal 2.5.1

StatsRecorder prior to uPortal 2.5.1 was a static service which would lookup and delegate to a backing IStatsRecorder implementation via portal.properties.

StatsRecorder as of uPortal 2.5.1

Bugs in 2.5.1

Alas, the StatsRecorder refactoring for uPortal 2.5.1 shipped with two minor but annoying bugs in the ConditionalStatsRecorder implementation. These have since been fixed in CVS and uPortal 2.5.2 will include the fixes.
See the relevant JIRA issues: ConditionalStatsRecorder.recordChannelDefinitionModified() broadcasts wrong event and ConditionalStatsRecorder.recordChannelTargeted() test for wrong flag.

As of uPortal 2.5.1, StatsRecorder is a Static Cover for a Spring-configured IStatsRecorder instance.

Expected advantages to be gained from this implementation include:

  • a design and implementation that is easier to understand, configure, and use
  • a more powerful and flexible implementation that allows multiple stats recorders, etc. Much of what is currently configured via static singleton, global configuration becomes instance configuration. it becomes possible to have four stats recorders each differently configured some operating in new threads etc.
  • Backwards-compatibility: IStatsRecorder API does not change. portal.properties driven configuration continues to work.

The Static Cover

The StatsRecorder static class becomes a "static cover", the legacy static way that code gets a handle to the IStatsRecorder instance declared as a Spring bean named "statsRecorder". Of course, anything that's being wired together in Spring need not use the static cover and can instead have its IStatsRecorder needs supplied via dependency injection.

The static cover tries to use the Spring bean. If it doesn't find the bean, then it falls back on the 2.5.0 behavior of using portal.properties configuration. If that doesn't work either, it falls back on recording no stats.

Implementing the legacy behaviors

Thread firing

Before uPortal 2.5.1, StatsRecorder used portal.properties properties to configure which IStatsRecorder methods, if any, it would propogate. It also used portal.properties to configure a thread pool and would fire a thread to execute actual event propogation. This allows, for instance, the statistics recording for channel rendering to happen outside of the thread actually trying to render channels and to manage channel rendering. Quite possibly necessary for recording channel rendering. Quite possibly overkill for recording user login, which is already an expensive operation anyway such adding some simple stats recording won't make a noticable difference.

Whether threads need to be fired can depend upon how expensive it is to do what you're trying to do with the stats. Logging probably doesn't need new threads. Writing to a slow backing database or calling out to a remote web service might.

In uPortal 2.5.1, the thread firing behavior moved out of StatsRecorder and into a particular IStatsRecorder wrapper implementation, ThreadFiringStatsRecorder.

Conditional event propogation

Before uPortal 2.5.1, StatsRecorder conditions event propogation on portal.properties properties and subsequent static singleton configuration of StatsRecorderSettings. This patch introduces a new interface, IStatsRecorderFlags, for which there are two implementations - a simple JavaBean implementation, and an implementation backed by the static singleton StatsRecorderSettings. A new IStatsRecorder wrapper implementation, ConditionalStatsRecorder, applies an IStatsRecorderFlags instance to decide which method calls to propogate to the wrapped IStatsRecorder. So, to achieve the legacy behavior, we use a ConditionalStatsRecorder wrapper configured to use the SettingsBackedStatsRecorderFlagsImpl.

New features

One requested feature for stats recording is that of being able to have multiple stats recorders. uPortal 2.5.1 supports this – and further allows each recorder to be differently configured if desired. The ListStatsRecorder is an IStatsRecorder which delegates to child IStatsRecorders.

Wiring examples

Each of these examples are what you could add to applicationContext.xml to configure an IStatsRecorder instance.

Legacy case: don't wire it

Get the legacy behavior by not wiring
<!-- 
 | no change to applicationContext.xml or to portal.properties 
 | or to any other configuration file is required to get the behavior of uPortal 2.5.0 
 +-->

If you don't declare any bean named "statsRecorder", then the StatsRecorder static cover will fall back on discovering StatsRecorder configuration from portal.properties. This is pursuant to the goal of "backwards compatibility".

In uPortal 2.6.0, as implemented in the current uPortal 2 HEAD, the legacy behavior goes away and failing to declare an IStatsRecorder instance via Spring configuration will result in no stats recording.

Doing nothing

Declaring a do-nothing stats recorder
<bean name="statsRecorder"
      class="org.jasig.portal.services.stats.DoNothingStatsRecorder"/>

Note that no factory is required - think of Spring as the ultimate Factory.

A logging example

Logging some IStatsRecorder events

<!--
 | The parent bean is the Conditional wrapper because first we want to filter down to
 | the events we're actually going to log.
 +-->
<bean name="statsRecorder"
    class="org.jasig.portal.services.stats.ConditionalStatsRecorder">
    <property name="flags">
        <!--
         | This JavaBean lets us configure which events we'd like the Conditional 
         | wrapper to propogate.  The flags default to false so we need only declared those
         | methods we would like to propogate.
         +-->
        <bean class="org.jasig.portal.services.stats.StatsRecorderFlagsImpl">
            <property name="recordChannelRendered" value="true"/>
        </bean>
    </property>

    <!-- 
     | The Conditional stats recorder will call this target when its condition is fulfilled, that is
     | when the method call is recordChannelRendered().  IStatsRecorder calls other than 
     | recordChannelRendered will have no effect, that is, are filtered away 
     | by the Conditional stats recorder. 
     +-->
    <property name="targetStatsRecorder">
        <!-- 
         | The target of the Conditional is the thread firing wrapper so that we'll use separate
         | threads to perform the actual logging. 
         +-->
        <bean class="org.jasig.portal.services.stats.ThreadFiringStatsRecorder">
            <!-- initial thread pool size -->
            <constructor-arg value="5"/>
            <!-- maximum thread pool size -->
            <constructor-arg value="15"/>
            <!-- thread priority -->
            <constructor-arg value="5"/>

            <!--
             | The target of the threads we fire is this LoggingStatsRecorder instance.
             +-->
            <property name="targetStatsRecorder">
                <bean class="org.jasig.portal.services.stats.LoggingStatsRecorder"/>
            </property>

        </bean>
    </property>
</bean>

This example responds only to the recordChannelRendered() IStatsRecorder method, executing the LoggingStatsRecorder using threads from a pool separate from the threads used for other aspects of uPortal.

Wiring in your custom stats recorder

Declaring your own stats recorder
<bean name="statsRecorder"
      class="edu.yale.its.portal.services.stats.YaleStatsRecorder"/>

Multiple stats recorders

An ambitious example involving multiple stats recorders

<!--
 | The parent bean is the Conditional wrapper because first we want to filter down to
 | the events we're actually going to use.
 +-->
<bean name="statsRecorder"
    class="org.jasig.portal.services.stats.ConditionalStatsRecorder">
    <property name="flags">
        <!--
         | This JavaBean lets us configure which events we'd like the Conditional 
         | wrapper to propogate.  The flags default to false so we need only declared those
         | methods we would like to propogate.
         +-->
        <bean class="org.jasig.portal.services.stats.StatsRecorderFlagsImpl">
            <property name="recordChannelRendered" value="true"/>
            <property name="recordFolderAddedToLayout" value="true"/>
        </bean>
    </property>

    <!-- 
     | The Conditional stats recorder will call this target when its condition is fulfilled, that is
     | when the method call is recordChannelRendered().  IStatsRecorder calls other than 
     | recordChannelRendered will have no effect, that is, are filtered away 
     | by the Conditional stats recorder. 
     +-->
    <property name="targetStatsRecorder">
        <!-- 
         | Here we're firing new threads.  If we're going to use a large list of stats recorders,
         | recording stats could take awhile so we want this to be undertaken in threads from the
         | pool for this purpose rather than engaging core channel rendering, session management, or
         | Servlet Container threads in this project.  We need to let go of the current thread to let it
         | get back to the work of rendering the response to the user.
         +-->
        <bean class="org.jasig.portal.services.stats.ThreadFiringStatsRecorder">
            <!-- initial thread pool size -->
            <constructor-arg value="5"/>
            <!-- maximum thread pool size -->
            <constructor-arg value="15"/>
            <!-- thread priority -->
            <constructor-arg value="5"/>

            <!--
             | The target of the threads we fire is the List recorder implementation, which
             | will delegate to our configured list of stats recorders.
             +-->
            <property name="targetStatsRecorder">
                <bean class="org.jasig.portal.services.stats.ListStatsRecorder">
                   <property name="children">
                       <list>
                           <!-- here we declare the IStatsRecorders we'd like to run. -->
                           <bean class="org.jasig.portal.services.stats.LoggingStatsRecorder"/>
                           <bean class="org.jasig.portal.services.stats.PrintingStatsRecorder"/>
                           <!-- 
                            | for example, you might use some of the included recorders 
                            | as well as a custom recorder implementation
                            +-->
                           <bean class="edu.someschool.DatabaseStatsRecorder">
                               <!--
                                | If your stats recorder needs a DataSource, you can inject it.
                                +--> 
                               <property name="dataSource">
                                   <bean 
                                       class="org.springframework.jndi.JndiObjectFactoryBean">
		                       <property 
                                           name="jndiName" 
                                           value="java:comp/env/jdbc/myDatasource" />
	                           </bean>
                              </property>
                           </bean>
                       </list>
                   </property>
                </bean>
            </property>

        </bean>
    </property>
</bean>

The future of StatsRecorder

Andrew's post to jasig-dev

My and others existing efforts in uP 2.5 wrt StatsRecorder have been to enable Spring-configuration of the IStatsRecorder implementation. This is evolutionary improvement allowing chaining and conditionalization of stats recording and, modulo some silly bugs I introduced in the glue code that have since been found and fixed, makes it easier to share IStatsRecorder implementations, use multiple IStatsRecorders at once, and declaratively turn on and off different kinds of stats recording without proliferating portal.properties properties or hacking the Java source code directly. Very modest incremental progress.

My intent wasn't to enshrine the current IStatsRecorder API as the way stats recording ought to be done forever more, but rather to make an already existing API more usable. The driving use case was "being able to have more than one stats recorder" and I think making it Spring-pluggable with a way to configure which events are propogated to which IStatsRecorder instances, and for which new threads are spawned, was a reasonable way to achieve that.

As has been noted on this list before, a monolithic IStatsRecorder API with methods for all the stats has some shortcomings. It's "monolithic" – if you want to write some code that's very concerned about session-related statistics and doesn't care about channel rendering, you're stuck with an object that has a bunch of useless methods unrelated to what you're trying to do. Extending a base class mitigates but doesn't fully resolve the API clutter. As you note, it's not clear how to expand the statistics being recorded to locally record more kinds of statistics. Do you hack the interface locally to add more methods, and thereby break compatibility with other IStatsRecorders? While that approach has worked for the schools that have done it, modeling this with Event objects and registration of Listeners to handle those events provides a more naturally expandable event recording environment than does modeling these statistics events as methods in an interface.

Yuji wrote:

I have been looking into adding stats to be recorded using the StatsRecorder. These are events specific to our local deployment so they are not something that should involve org.jasig.portal classes directly.

However, looking at 2.5.0 and peeking at 2.5.1, the thing that disturbed me is that it looks like adding new events entails adding to the IStatsRecorder interface rather than having the "Events" specified declaratively.

Is there any thought of refactoring IStatsRecorder to use declarative configuration for the recorded events?

Now, I am no Pattern or AOP guru, but it would seem to be something that Spring is well-suited to handle.

Likely. Spring ApplicationContexts support publishing events and registering to listen for events. Presumably what is currently accomplished via IStatsRecorder could instead be accomplished by the components currently invoking StatsRecorder instead publishing events to the ApplicationContext, and the IStatsRecorder implementations would be replaced by listeners registered with the ApplicationContext. New kinds of events are then accomodated by new publishers and listeners.

Worth some research to see how hard this is to implement sufficiently "backwards-compatibly" to be achievable in 2-5-patches. Presumably the StatsRecorder static cover would convert method calls to Event objects it publishes to the ApplicationContext, and we provide a way to wrap legacy IStatsRecorders in new Listeners and register them, and we're good.

I don't know of existing efforts to make this architectural change.

Andrew