[11:27:20 CST(-0600)] <jwennmacher1> EricDalquist: Good morning. Drew suggested I contact you about working on some statistics reporting.
[11:28:25 CST(-0600)] <EricDalquist> hi
[11:28:27 CST(-0600)] <EricDalquist> awesome
[11:29:21 CST(-0600)] <jwennmacher1> I am at a very early stage; I was just glancing at the statistics that are being collected right now. Initial observation is there are two sets currently collected not being reported on. Portlet/folder added/deleted/removed from layout. I was thinking maybe of starting with one of those (probably portlet). Thoughts?
[11:30:10 CST(-0600)] <EricDalquist> well, let me do a little overview of the whole stats/aggregation/reporting system
[11:30:51 CST(-0600)] <EricDalquist> opening up uPortal source just a minute ....
[11:33:09 CST(-0600)] <EricDalquist> so forgive me if you're familiar with parts of this already but I figure a full picture is good
[11:33:32 CST(-0600)] <EricDalquist> uportal uses an extension of the spring application context event apis
[11:33:38 CST(-0600)] <jwennmacher1> yep
[11:33:56 CST(-0600)] <EricDalquist> one of the event handlers sticks the events onto a concurrent queue
[11:34:08 CST(-0600)] <EricDalquist> and a background thread periodically flushes them out to the db via JpaPortalEventStore
[11:34:16 CST(-0600)] <EricDalquist> these are what we call raw events
[11:34:21 CST(-0600)] <EricDalquist> and are actually stored as JSON CLOBs
[11:35:32 CST(-0600)] <EricDalquist> then there is a background process that runs on one machine in the cluster that periodically aggregates that raw event data into some form that is easier to report on / process
[11:35:46 CST(-0600)] <EricDalquist> PortalEventProcessingManagerImpl is essentially the entry point for all of that logic
[11:36:45 CST(-0600)] <EricDalquist> that uses all instances of IPortalEventAggregator it finds in the app context, so we have sort of a pluggable api for doing this aggregation work
[11:37:26 CST(-0600)] <EricDalquist> right now we have aggregators that track: concurrent users, logins (unique & total), tab renders, & portlet executions
[11:37:40 CST(-0600)] <EricDalquist> this code is really finicky to write
[11:37:45 CST(-0600)] <EricDalquist> and VERY performance sensitive
[11:38:01 CST(-0600)] <EricDalquist> since you need to make sure the aggregator can handle processing the data faster than it is created
[11:39:08 CST(-0600)] <EricDalquist> for example looking at our logs at UW right now
[11:39:13 CST(-0600)] <EricDalquist> the aggregator is falling behind a bit
[11:39:20 CST(-0600)] <EricDalquist> Aggregated 10000 events created at 16.4745 events/second between 2012-12-11T09:54:51.359-06:00 and 2012-12-11T10:04:59.202-06:00 in 1108885ms - 9.0181 e/s a 0.5474x speedup.
[11:39:27 CST(-0600)] <EricDalquist> but we track A LOT of data
[11:39:41 CST(-0600)] <EricDalquist> and this seems to happen as our DB index statistics slowly get out of data
[11:39:46 CST(-0600)] <EricDalquist> out of date*
[11:39:54 CST(-0600)] <EricDalquist> so just something to keep in mind
[11:40:44 CST(-0600)] <EricDalquist> then there is the reporting piece
[11:40:59 CST(-0600)] <EricDalquist> we currently have two of those, LoginTotalsStatisticsController and ConcurrentUsersStatisticsController
[11:41:15 CST(-0600)] <EricDalquist> these are what let us get nice graphs/reports out of the aggregated data
[11:41:24 CST(-0600)] <EricDalquist> ok … does that all make sense?
[11:42:05 CST(-0600)] <jwennmacher1> yes. Good to know, especially the performance requirement