[11:27:20 CST(-0600)] <jwennmacher1> EricDalquist: Good morning. Drew suggested I contact you about working on some statistics reporting.
[11:28:25 CST(-0600)] <EricDalquist> hi
[11:28:27 CST(-0600)] <EricDalquist> awesome
[11:29:21 CST(-0600)] <jwennmacher1> I am at a very early stage; I was just glancing at the statistics that are being collected right now. Initial observation is there are two sets currently collected not being reported on. Portlet/folder added/deleted/removed from layout. I was thinking maybe of starting with one of those (probably portlet). Thoughts?
[11:30:10 CST(-0600)] <EricDalquist> well, let me do a little overview of the whole stats/aggregation/reporting system
[11:30:51 CST(-0600)] <EricDalquist> opening up uPortal source just a minute ....
[11:33:09 CST(-0600)] <EricDalquist> so forgive me if you're familiar with parts of this already but I figure a full picture is good
[11:33:32 CST(-0600)] <EricDalquist> uportal uses an extension of the spring application context event apis
[11:33:38 CST(-0600)] <jwennmacher1> yep
[11:33:56 CST(-0600)] <EricDalquist> one of the event handlers sticks the events onto a concurrent queue
[11:34:08 CST(-0600)] <EricDalquist> and a background thread periodically flushes them out to the db via JpaPortalEventStore
[11:34:16 CST(-0600)] <EricDalquist> these are what we call raw events
[11:34:21 CST(-0600)] <EricDalquist> and are actually stored as JSON CLOBs
[11:35:32 CST(-0600)] <EricDalquist> then there is a background process that runs on one machine in the cluster that periodically aggregates that raw event data into some form that is easier to report on / process
[11:35:46 CST(-0600)] <EricDalquist> PortalEventProcessingManagerImpl is essentially the entry point for all of that logic
[11:36:45 CST(-0600)] <EricDalquist> that uses all instances of IPortalEventAggregator it finds in the app context, so we have sort of a pluggable api for doing this aggregation work
[11:37:26 CST(-0600)] <EricDalquist> right now we have aggregators that track: concurrent users, logins (unique & total), tab renders, & portlet executions
[11:37:40 CST(-0600)] <EricDalquist> this code is really finicky to write
[11:37:45 CST(-0600)] <EricDalquist> and VERY performance sensitive
[11:38:01 CST(-0600)] <EricDalquist> since you need to make sure the aggregator can handle processing the data faster than it is created
[11:39:08 CST(-0600)] <EricDalquist> for example looking at our logs at UW right now
[11:39:13 CST(-0600)] <EricDalquist> the aggregator is falling behind a bit
[11:39:20 CST(-0600)] <EricDalquist> Aggregated 10000 events created at 16.4745 events/second between 2012-12-11T09:54:51.359-06:00 and 2012-12-11T10:04:59.202-06:00 in 1108885ms - 9.0181 e/s a 0.5474x speedup.
[11:39:27 CST(-0600)] <EricDalquist> but we track A LOT of data
[11:39:41 CST(-0600)] <EricDalquist> and this seems to happen as our DB index statistics slowly get out of data
[11:39:46 CST(-0600)] <EricDalquist> out of date*
[11:39:54 CST(-0600)] <EricDalquist> so just something to keep in mind
[11:40:44 CST(-0600)] <EricDalquist> then there is the reporting piece
[11:40:59 CST(-0600)] <EricDalquist> we currently have two of those, LoginTotalsStatisticsController and ConcurrentUsersStatisticsController
[11:41:15 CST(-0600)] <EricDalquist> these are what let us get nice graphs/reports out of the aggregated data
[11:41:24 CST(-0600)] <EricDalquist> ok … does that all make sense?
[11:42:05 CST(-0600)] <jwennmacher1> yes. Good to know, especially the performance requirement
[11:43:13 CST(-0600)] <EricDalquist> so for a place to start
[11:43:28 CST(-0600)] <EricDalquist> I think what may actually be the best spot are the reporting portlets
[11:43:46 CST(-0600)] <EricDalquist> we are currently collecting a ton of data about tab renders and portlet executions
[11:44:01 CST(-0600)] <jwennmacher1> yes. I see there are aggregators already written for portlet execution and tab mapping. I haven't checked to see if they are used yet. Would these be good candidates to consider since some of the foundation work appears to be present? I'm still somewhat of a newbie; I've done a bit of portlet work but not uPortal yet. I only have a few days to contribute before I'm off to another project for a while.
[11:44:27 CST(-0600)] <jwennmacher1> Reporting portlets are what Drew and I discussed.
[11:45:06 CST(-0600)] <EricDalquist> yeah these would be the best place to start
[11:45:14 CST(-0600)] <EricDalquist> I'd probably start with tabs first
[11:45:20 CST(-0600)] <EricDalquist> as they have the simpler of the two data models
[11:45:32 CST(-0600)] <jwennmacher1> Sounds good.
[11:46:05 CST(-0600)] <jwennmacher1> Have the aggregators had adequate performance testing or will I need to be concerned about that?
[11:46:25 CST(-0600)] <EricDalquist> yeah they have had a lot of performance testing
[11:46:28 CST(-0600)] <EricDalquist> many hours with a profile
[11:46:32 CST(-0600)] <EricDalquist> profiler*
[11:46:42 CST(-0600)] <EricDalquist> what I would recommend is starting with LoginTotalsStatisticsController
[11:46:45 CST(-0600)] <EricDalquist> copying that
[11:47:03 CST(-0600)] <EricDalquist> and reworking it to work against the TabRenderAggregationDao
[11:47:45 CST(-0600)] <EricDalquist> so the tab renders have one more "dimension" than logins to
[11:48:06 CST(-0600)] <EricDalquist> logs have: date&time & group
[11:48:17 CST(-0600)] <EricDalquist> logins have* (sorry for all the typos this morning)
[11:48:31 CST(-0600)] <EricDalquist> tab renders have: date&time, group & tab name
[11:48:38 CST(-0600)] <EricDalquist> so that is a little bit of added complexity
[11:49:20 CST(-0600)] <EricDalquist> ConcurrentUsersStatisticsController and LoginTotalsStatisticsController are good examples to get you started though
[11:49:47 CST(-0600)] <EricDalquist> the reporting portlet should just "auto detect" any other controllers that implement BaseStatisticsReportController
[11:49:52 CST(-0600)] <EricDalquist> and show it in the report list
[11:50:13 CST(-0600)] <EricDalquist> so just a copy and paste of LoginTotalsStatisticsController and then reworking for tab renders will be a good first step
[11:50:42 CST(-0600)] <EricDalquist> once you get that working and are more comfortable we can talk about additional report uis
[11:52:20 CST(-0600)] <EricDalquist> since tab renders track render count and then a bunch of data about the render time: sum of squares, population variance, geometric mean, sum of logs, mean, variance, standard deviation, max, min, and sum
[11:52:26 CST(-0600)] <EricDalquist> so lots of time to render data
[11:52:43 CST(-0600)] <EricDalquist> which could turn into some interesting reports
[11:52:46 CST(-0600)] <EricDalquist> even non-graph reports
[11:53:05 CST(-0600)] <EricDalquist> like for the portlet execution side (which tracks the same timing data) we'd love to have a "slowest portlet" report
[11:53:21 CST(-0600)] <EricDalquist> like I can log into the portal and see which portlets are taking the longest to render over the last 5 minutes
[11:53:44 CST(-0600)] <EricDalquist> ok … I think I'm done with my wall of text
[11:53:50 CST(-0600)] <EricDalquist> I'll be around all day/week
[11:54:03 CST(-0600)] <EricDalquist> so just poke me if you have questions or even want to chat about report ideas
[11:57:11 CST(-0600)] <jwennmacher1> Thanks. good idea on slowest portlet. For tabs what are the 'groups' you mentioned as another dimension? It's same as normal groups (everyone, students, etc.)?
[11:57:47 CST(-0600)] <EricDalquist> yes, but to insulate from portal config changes the event aggregation has its own group, tab and portlet lookup tables
[11:58:23 CST(-0600)] <EricDalquist> AggregatedGroupLookupDao, AggregatedTabLookupDao, AggregatedPortletLookupDao
[11:58:40 CST(-0600)] <EricDalquist> these capture the group/tab/portlet data from the primary uPortal daos the first time it is seen
[11:58:48 CST(-0600)] <EricDalquist> and the stats data actuall refers to these
[11:58:54 CST(-0600)] <EricDalquist> this is to that if say a tab or portlet is deleted
[11:58:59 CST(-0600)] <EricDalquist> you don't lose the stats data about it
[11:59:18 CST(-0600)] <EricDalquist> note that you may well run into areas where there are missing APIs
[11:59:30 CST(-0600)] <EricDalquist> like no way to get a list of all the tabs in the lookup dao
[11:59:38 CST(-0600)] <EricDalquist> this is simply due to nothing needing that api yet
[11:59:49 CST(-0600)] <EricDalquist> so you or I will need to add those APIs when you find the holes
[12:01:22 CST(-0600)] <jwennmacher1> Ahh I see what you mean about insulating. Gotcha. Thanks for the overview. That helps me quite a bit.
[12:01:44 CST(-0600)] <jwennmacher1> I'm sure I'll have tons of questions as I dig into it 
[12:04:18 CST(-0600)] <EricDalquist>
[12:06:42 CST(-0600)] <EricDalquist> drewwills: you have a few minutes to talk about person diretory?
[12:06:48 CST(-0600)] <EricDalquist> on a more abstract level?
[12:30:34 CST(-0600)] <drewwills> i will EricDalquist, sure
[12:31:04 CST(-0600)] <EricDalquist> so looking at PD on a higher level with various features
[12:31:12 CST(-0600)] <EricDalquist> what do you think of the current sql/ldap query templating
[12:31:28 CST(-0600)] <EricDalquist> where you stick a in where you want the search/restrictions to appear
[13:21:14 CST(-0600)] <drewwills1> sorry EricDalquist... i was on a call
[13:21:26 CST(-0600)] <EricDalquist> no problem
[13:21:59 CST(-0600)] <drewwills1> the issue i run into with that is that sometimes i need more flexibility
[13:22:22 CST(-0600)] <EricDalquist> yeah
[13:22:25 CST(-0600)] <EricDalquist> that is my thought as well
[13:22:29 CST(-0600)] <drewwills1> i may need to "select foo from bar where netId = "
[13:22:29 CST(-0600)] <EricDalquist> I'm not sure what a solution is though
[13:22:58 CST(-0600)] <EricDalquist> since that works great for simple queries but doesn't work for attribute sources than can have a more flexible search done
[13:23:07 CST(-0600)] <drewwills1> one sec...
[13:23:07 CST(-0600)] <EricDalquist> I'm open to all ideas here
[13:26:27 CST(-0600)] <drewwills1> i've used this approach, and it's attractive in some ways: https://gist.github.com/4261275
[13:26:42 CST(-0600)] <drewwills1> more complex, i guess, but flexible
[13:27:13 CST(-0600)] <EricDalquist> ok
[13:27:31 CST(-0600)] <EricDalquist> so the idea is that attribute source has a fixed set of query attributes
[13:27:47 CST(-0600)] <EricDalquist> you always have to ask for data with username=X
[13:29:04 CST(-0600)] <EricDalquist> so maybe the solution is just a variety of options when configuration attribute sources
[13:29:15 CST(-0600)] <EricDalquist> we have some like this that use named parameters
[13:29:19 CST(-0600)] <drewwills1> that's fine, but you might want to do lower(username) = x
[13:29:50 CST(-0600)] <EricDalquist> oh I was just saying that the incoming PD query would be for username=X
[13:29:57 CST(-0600)] <drewwills1> ah yes
[13:30:00 CST(-0600)] <EricDalquist> and X gets inserted for :username
[13:30:07 CST(-0600)] <drewwills1> yep, totally
[13:30:46 CST(-0600)] <drewwills1> but something like this alows you to "decorate" the SQL where the rubber hits the road around username = x
[13:31:05 CST(-0600)] <EricDalquist> ok
[13:31:20 CST(-0600)] <drewwills1> you might even want to do "where :grad_year > 2014"
[13:31:40 CST(-0600)] <drewwills1> use cases with cascading DAOs
[13:32:26 CST(-0600)] <drewwills1> brb... lunch
[13:33:54 CST(-0600)] <EricDalquist> ok
[14:39:23 CST(-0600)] <EricDalquist> drewwills: you around?
[14:57:48 CST(-0600)] <drewwills> yep
[14:57:55 CST(-0600)] <EricDalquist> so more PD stuff
[14:58:06 CST(-0600)] <EricDalquist> do you ever see the need for more complex person queries?
[14:58:07 CST(-0600)] <drewwills> ok
[14:58:18 CST(-0600)] <EricDalquist> right now we have this terrible API
[14:58:27 CST(-0600)] <drewwills> +1
[14:58:29 CST(-0600)] <EricDalquist> where you give it a map of attribute name>value pairs
[14:58:40 CST(-0600)] <EricDalquist> and each source figures out how it will handle those
[14:58:49 CST(-0600)] <EricDalquist> do we really need something more expressive?
[14:59:15 CST(-0600)] <drewwills> maybe... can you throw out an example?
[14:59:17 CST(-0600)] <EricDalquist> like something where you could do (firstName=Jane && (lastName=Doe || lastName=Smith))
[14:59:33 CST(-0600)] <drewwills> hmmm... ppossibly
[14:59:53 CST(-0600)] <drewwills> that would suppost full-featured directory searching, for example
[15:00:04 CST(-0600)] <EricDalquist> note that something like that really only works for attribute sources that support a more free-form query building
[15:00:18 CST(-0600)] <EricDalquist> sources that are using direct named attribute replacement wouldn't really fit here
[15:00:22 CST(-0600)] <EricDalquist> they would either have to be ignored
[15:00:38 CST(-0600)] <EricDalquist> or queried once per result after getting results from the more flexible sources
[15:00:42 CST(-0600)] <drewwills> instead of passing attr/value pairs, you could pass criteria objects
[15:00:49 CST(-0600)] <EricDalquist> right
[15:01:18 CST(-0600)] <EricDalquist> I'd probably model something after: http://static.springsource.org/spring-ldap/site/apidocs/org/springframework/ldap/filter/package-frame.html
[15:01:34 CST(-0600)] <drewwills> looking
[15:02:28 CST(-0600)] <drewwills> sure... not dissimilar to PAGS and DLM evaluators actually
[15:02:35 CST(-0600)] <EricDalquist> yeah
[15:02:41 CST(-0600)] <EricDalquist> it is essentially the LDAP filter syntax
[15:02:42 CST(-0600)] <drewwills> i wonder if we could consolidate even
[15:02:45 CST(-0600)] <EricDalquist> in java object form
[15:02:58 CST(-0600)] <EricDalquist> so all of this is coming from a potential chunk of time I have from UW to do a big refactoring of PD
[15:03:09 CST(-0600)] <EricDalquist> our motivations are around better query support
[15:03:15 CST(-0600)] <EricDalquist> and better concurrent support
[15:03:28 CST(-0600)] <EricDalquist> we're starting to see login slowdowns due to the large number of sources
[15:04:10 CST(-0600)] <EricDalquist> I'll be transcribing this into the wiki once I have better design bits
[15:04:14 CST(-0600)] <EricDalquist> but here are my notes so far: https://gist.github.com/4262128
[15:04:20 CST(-0600)] <drewwills> hmmm... it's nice that we have WI at the bleeding edge to work out all the kinks 
[15:04:25 CST(-0600)] <EricDalquist>
[15:04:36 CST(-0600)] <EricDalquist> nothing like 100k logins a day to make sure its working right 
[15:04:45 CST(-0600)] <EricDalquist> the idea that we greatly simplify the attribute sources
[15:04:51 CST(-0600)] <EricDalquist> have a spring XML namespace for configuration
[15:05:01 CST(-0600)] <EricDalquist> and a big chunk of "core" code that does most of the heavy lifting
[15:05:13 CST(-0600)] <drewwills> yeah this would probably be a good match for a spring xsd
[15:05:18 CST(-0600)] <EricDalquist> I'm also hoping we can get rid of the explicit dao ordering and stuff
[15:05:56 CST(-0600)] <EricDalquist> just configure a buncvh of attribute sources
[15:06:05 CST(-0600)] <EricDalquist> optionally ordering and merge strategies
[15:06:34 CST(-0600)] <EricDalquist> and the core logic should be able to figure out which attribute queries it can run in parallel
[15:06:42 CST(-0600)] <EricDalquist> and which need to wait for others
[15:06:57 CST(-0600)] <drewwills> i help probably 4+ schools a year start a portal... persondirs is always a challenge
[15:07:08 CST(-0600)] <EricDalquist> yeah
[15:07:15 CST(-0600)] <EricDalquist> and that is why I'm bugging you for input
[15:07:23 CST(-0600)] <EricDalquist> pain points and ideas around them are great
[15:07:25 CST(-0600)] <drewwills> yeah
[15:07:30 CST(-0600)] <EricDalquist> to make sure we avoid/fix those issues
[15:07:44 CST(-0600)] <drewwills> here's one item of specific input...
[15:08:24 CST(-0600)] <drewwills> default for queryType should be OR imho
[15:08:32 CST(-0600)] <drewwills> or should have been i guess
[15:08:55 CST(-0600)] <drewwills> but that may not even be relevant in the updated api
[15:09:23 CST(-0600)] <EricDalquist> so if I query for (foo=x, bar=y) default to get results for (foo==x || bar ==y)
[15:09:40 CST(-0600)] <EricDalquist> ok
[15:09:44 CST(-0600)] <EricDalquist> so some concerns I have about the more complex query language
[15:09:49 CST(-0600)] <EricDalquist> what do we do when:
[15:10:11 CST(-0600)] <EricDalquist> query: (lastName == Smith && isStudent=Y)
[15:10:18 CST(-0600)] <EricDalquist> and we have two attribute sources
[15:10:30 CST(-0600)] <EricDalquist> one supports both lastName and isStudent attributes
[15:10:37 CST(-0600)] <EricDalquist> the other only supports lastName
[15:10:52 CST(-0600)] <drewwills> more often what i see is client code that take 1 search term and applies it to all the querry attrs
[15:10:59 CST(-0600)] <drewwills> user submits foo
[15:11:17 CST(-0600)] <drewwills> translates to username=foo, firstName=foo, lastName=foo
[15:11:27 CST(-0600)] <EricDalquist> and we can get much more complex queries and weird "which attribute sources answer)"
[15:11:35 CST(-0600)] <EricDalquist> ah
[15:11:36 CST(-0600)] <drewwills> it's better if the default is a hit if any one of those matches foo
[15:12:30 CST(-0600)] <EricDalquist> right
[15:12:47 CST(-0600)] <EricDalquist> so in that case running a query for (username=foo || firstName=foo || lastName=foo)
[15:12:54 CST(-0600)] <drewwills> exactly
[15:13:03 CST(-0600)] <EricDalquist> the thing i'm not sure about is what to do when you have multiple attribute sources
[15:13:29 CST(-0600)] <EricDalquist> and the incoming query references attributes that one of the attributes sources can't query on
[15:13:37 CST(-0600)] <EricDalquist> if we always do OR queries that isn't a problem
[15:13:55 CST(-0600)] <EricDalquist> but if we ever do ANDs it is
[15:14:12 CST(-0600)] <drewwills> yep, that makes sense
[15:15:04 CST(-0600)] <EricDalquist> and that is where I'm kind of stuck for ideas right now
[15:15:10 CST(-0600)] <drewwills> so perhaps the DAOs themselvs should not even have a queryType == AND | OR
[15:15:33 CST(-0600)] <drewwills> maybe we move to a criteria model, and let the client code handle that detail
[15:15:53 CST(-0600)] <EricDalquist> by a criteria model you mean like that LDAP filter API?
[15:16:25 CST(-0600)] <drewwills> yes... and instead it could have a loosy-goosey/strict setting
[15:16:41 CST(-0600)] <drewwills> as to whether it will forgive missing attrs
[15:16:51 CST(-0600)] <drewwills> default is probably strict
[15:17:14 CST(-0600)] <EricDalquist> so by default if the source doesnt support any attribute in the query
[15:17:18 CST(-0600)] <EricDalquist> that source isn't queried
[15:17:44 CST(-0600)] <drewwills> well maybe not
[15:17:46 CST(-0600)] <EricDalquist> part of me is thinking that I need to break this up more
[15:17:47 CST(-0600)] <EricDalquist> hrm
[15:17:49 CST(-0600)] <EricDalquist> oh!
[15:17:53 CST(-0600)] <EricDalquist> so here is an idea
[15:18:03 CST(-0600)] <drewwills> consider (username == 'foo' | lastName == 'bar')
[15:18:04 CST(-0600)] <EricDalquist> we break up "searchable sources" and "attribute sources"
[15:18:16 CST(-0600)] <drewwills> if you only support username, that's cool
[15:18:31 CST(-0600)] <drewwills> but if it's (username == 'foo' & lastName == 'bar')
[15:18:51 CST(-0600)] <drewwills> you must support both... unless (perhaps) strict=false
[15:19:03 CST(-0600)] <EricDalquist> ok
[15:19:28 CST(-0600)] <drewwills> i'm not convinced the strict thing is necessary... just a thought
[15:20:37 CST(-0600)] <EricDalquist> yeah
[15:21:09 CST(-0600)] <EricDalquist> I'm wondering if we really just need that search vs source seperation
[15:21:31 CST(-0600)] <EricDalquist> so attribute sources where you can configure a "query expression placeholder" the that we have now
[15:21:35 CST(-0600)] <EricDalquist> we use those in searches
[15:21:50 CST(-0600)] <EricDalquist> and sources that use named parameters
[15:22:13 CST(-0600)] <EricDalquist> we don't treat as searchable
[15:22:13 CST(-0600)] <EricDalquist> hrm
[15:22:14 CST(-0600)] <EricDalquist> no
[15:22:18 CST(-0600)] <EricDalquist> that doesn't work either
[15:22:19 CST(-0600)] <EricDalquist> ok
[15:22:24 CST(-0600)] <EricDalquist> I'm going to go write up more examples
[15:37:14 CST(-0600)] <EricDalquist> so building on your strict/loose idea
[15:37:36 CST(-0600)] <EricDalquist> maybe "loose" does client side filtering
[15:37:43 CST(-0600)] <EricDalquist> so for a query of (firstName=Jane && (isStudent=Y || lastName=Doe))
[15:38:16 CST(-0600)] <EricDalquist> lets say we have sourceA that can query on all three attributes, we got ahead and run that query as is
[15:38:27 CST(-0600)] <EricDalquist> then we have sourceB that only supports firstName and lastName
[15:38:40 CST(-0600)] <EricDalquist> for B we run (firstName=Jane && (lastName=Doe))
[15:39:18 CST(-0600)] <EricDalquist> then during the merge process we apply that isStudent=Y filter in code