[11:27:20 CST(-0600)] <jwennmacher1> EricDalquist: Good morning. Drew suggested I contact you about working on some statistics reporting.

[11:28:25 CST(-0600)] <EricDalquist> hi

[11:28:27 CST(-0600)] <EricDalquist> awesome

[11:29:21 CST(-0600)] <jwennmacher1> I am at a very early stage; I was just glancing at the statistics that are being collected right now. Initial observation is there are two sets currently collected not being reported on. Portlet/folder added/deleted/removed from layout. I was thinking maybe of starting with one of those (probably portlet). Thoughts?

[11:30:10 CST(-0600)] <EricDalquist> well, let me do a little overview of the whole stats/aggregation/reporting system

[11:30:51 CST(-0600)] <EricDalquist> opening up uPortal source just a minute ....

[11:33:09 CST(-0600)] <EricDalquist> so forgive me if you're familiar with parts of this already but I figure a full picture is good

[11:33:32 CST(-0600)] <EricDalquist> uportal uses an extension of the spring application context event apis

[11:33:38 CST(-0600)] <jwennmacher1> yep

[11:33:56 CST(-0600)] <EricDalquist> one of the event handlers sticks the events onto a concurrent queue

[11:34:08 CST(-0600)] <EricDalquist> and a background thread periodically flushes them out to the db via JpaPortalEventStore

[11:34:16 CST(-0600)] <EricDalquist> these are what we call raw events

[11:34:21 CST(-0600)] <EricDalquist> and are actually stored as JSON CLOBs

[11:35:32 CST(-0600)] <EricDalquist> then there is a background process that runs on one machine in the cluster that periodically aggregates that raw event data into some form that is easier to report on / process

[11:35:46 CST(-0600)] <EricDalquist> PortalEventProcessingManagerImpl is essentially the entry point for all of that logic

[11:36:45 CST(-0600)] <EricDalquist> that uses all instances of IPortalEventAggregator it finds in the app context, so we have sort of a pluggable api for doing this aggregation work

[11:37:26 CST(-0600)] <EricDalquist> right now we have aggregators that track: concurrent users, logins (unique & total), tab renders, & portlet executions

[11:37:40 CST(-0600)] <EricDalquist> this code is really finicky to write

[11:37:45 CST(-0600)] <EricDalquist> and VERY performance sensitive

[11:38:01 CST(-0600)] <EricDalquist> since you need to make sure the aggregator can handle processing the data faster than it is created

[11:39:08 CST(-0600)] <EricDalquist> for example looking at our logs at UW right now

[11:39:13 CST(-0600)] <EricDalquist> the aggregator is falling behind a bit

[11:39:20 CST(-0600)] <EricDalquist> Aggregated 10000 events created at 16.4745 events/second between 2012-12-11T09:54:51.359-06:00 and 2012-12-11T10:04:59.202-06:00 in 1108885ms - 9.0181 e/s a 0.5474x speedup.

[11:39:27 CST(-0600)] <EricDalquist> but we track A LOT of data

[11:39:41 CST(-0600)] <EricDalquist> and this seems to happen as our DB index statistics slowly get out of data

[11:39:46 CST(-0600)] <EricDalquist> out of date*

[11:39:54 CST(-0600)] <EricDalquist> so just something to keep in mind

[11:40:44 CST(-0600)] <EricDalquist> then there is the reporting piece

[11:40:59 CST(-0600)] <EricDalquist> we currently have two of those, LoginTotalsStatisticsController and ConcurrentUsersStatisticsController

[11:41:15 CST(-0600)] <EricDalquist> these are what let us get nice graphs/reports out of the aggregated data

[11:41:24 CST(-0600)] <EricDalquist> ok … does that all make sense?

[11:42:05 CST(-0600)] <jwennmacher1> yes. Good to know, especially the performance requirement

[11:43:13 CST(-0600)] <EricDalquist> so for a place to start

[11:43:28 CST(-0600)] <EricDalquist> I think what may actually be the best spot are the reporting portlets

[11:43:46 CST(-0600)] <EricDalquist> we are currently collecting a ton of data about tab renders and portlet executions

[11:44:01 CST(-0600)] <jwennmacher1> yes. I see there are aggregators already written for portlet execution and tab mapping. I haven't checked to see if they are used yet. Would these be good candidates to consider since some of the foundation work appears to be present? I'm still somewhat of a newbie; I've done a bit of portlet work but not uPortal yet. I only have a few days to contribute before I'm off to another project for a while.

[11:44:27 CST(-0600)] <jwennmacher1> Reporting portlets are what Drew and I discussed.

[11:45:06 CST(-0600)] <EricDalquist> yeah these would be the best place to start

[11:45:14 CST(-0600)] <EricDalquist> I'd probably start with tabs first

[11:45:20 CST(-0600)] <EricDalquist> as they have the simpler of the two data models

[11:45:32 CST(-0600)] <jwennmacher1> Sounds good.

[11:46:05 CST(-0600)] <jwennmacher1> Have the aggregators had adequate performance testing or will I need to be concerned about that?

[11:46:25 CST(-0600)] <EricDalquist> yeah they have had a lot of performance testing

[11:46:28 CST(-0600)] <EricDalquist> many hours with a profile

[11:46:32 CST(-0600)] <EricDalquist> profiler*

[11:46:42 CST(-0600)] <EricDalquist> what I would recommend is starting with LoginTotalsStatisticsController

[11:46:45 CST(-0600)] <EricDalquist> copying that

[11:47:03 CST(-0600)] <EricDalquist> and reworking it to work against the TabRenderAggregationDao

[11:47:45 CST(-0600)] <EricDalquist> so the tab renders have one more "dimension" than logins to

[11:48:06 CST(-0600)] <EricDalquist> logs have: date&time & group

[11:48:17 CST(-0600)] <EricDalquist> logins have* (sorry for all the typos this morning)

[11:48:31 CST(-0600)] <EricDalquist> tab renders have: date&time, group & tab name

[11:48:38 CST(-0600)] <EricDalquist> so that is a little bit of added complexity

[11:49:20 CST(-0600)] <EricDalquist> ConcurrentUsersStatisticsController and LoginTotalsStatisticsController are good examples to get you started though

[11:49:47 CST(-0600)] <EricDalquist> the reporting portlet should just "auto detect" any other controllers that implement BaseStatisticsReportController

[11:49:52 CST(-0600)] <EricDalquist> and show it in the report list

[11:50:13 CST(-0600)] <EricDalquist> so just a copy and paste of LoginTotalsStatisticsController and then reworking for tab renders will be a good first step

[11:50:42 CST(-0600)] <EricDalquist> once you get that working and are more comfortable we can talk about additional report uis

[11:52:20 CST(-0600)] <EricDalquist> since tab renders track render count and then a bunch of data about the render time: sum of squares, population variance, geometric mean, sum of logs, mean, variance, standard deviation, max, min, and sum

[11:52:26 CST(-0600)] <EricDalquist> so lots of time to render data

[11:52:43 CST(-0600)] <EricDalquist> which could turn into some interesting reports

[11:52:46 CST(-0600)] <EricDalquist> even non-graph reports

[11:53:05 CST(-0600)] <EricDalquist> like for the portlet execution side (which tracks the same timing data) we'd love to have a "slowest portlet" report

[11:53:21 CST(-0600)] <EricDalquist> like I can log into the portal and see which portlets are taking the longest to render over the last 5 minutes

[11:53:44 CST(-0600)] <EricDalquist> ok … I think I'm done with my wall of text

[11:53:50 CST(-0600)] <EricDalquist> I'll be around all day/week

[11:54:03 CST(-0600)] <EricDalquist> so just poke me if you have questions or even want to chat about report ideas

[11:57:11 CST(-0600)] <jwennmacher1> Thanks. good idea on slowest portlet. For tabs what are the 'groups' you mentioned as another dimension? It's same as normal groups (everyone, students, etc.)?

[11:57:47 CST(-0600)] <EricDalquist> yes, but to insulate from portal config changes the event aggregation has its own group, tab and portlet lookup tables

[11:58:23 CST(-0600)] <EricDalquist> AggregatedGroupLookupDao, AggregatedTabLookupDao, AggregatedPortletLookupDao

[11:58:40 CST(-0600)] <EricDalquist> these capture the group/tab/portlet data from the primary uPortal daos the first time it is seen

[11:58:48 CST(-0600)] <EricDalquist> and the stats data actuall refers to these

[11:58:54 CST(-0600)] <EricDalquist> this is to that if say a tab or portlet is deleted

[11:58:59 CST(-0600)] <EricDalquist> you don't lose the stats data about it

[11:59:18 CST(-0600)] <EricDalquist> note that you may well run into areas where there are missing APIs

[11:59:30 CST(-0600)] <EricDalquist> like no way to get a list of all the tabs in the lookup dao

[11:59:38 CST(-0600)] <EricDalquist> this is simply due to nothing needing that api yet

[11:59:49 CST(-0600)] <EricDalquist> so you or I will need to add those APIs when you find the holes

[12:01:22 CST(-0600)] <jwennmacher1> Ahh I see what you mean about insulating. Gotcha. Thanks for the overview. That helps me quite a bit.

[12:01:44 CST(-0600)] <jwennmacher1> I'm sure I'll have tons of questions as I dig into it

[12:04:18 CST(-0600)] <EricDalquist>

[12:06:42 CST(-0600)] <EricDalquist> drewwills: you have a few minutes to talk about person diretory?

[12:06:48 CST(-0600)] <EricDalquist> on a more abstract level?

[12:30:34 CST(-0600)] <drewwills> i will EricDalquist, sure

[12:31:04 CST(-0600)] <EricDalquist> so looking at PD on a higher level with various features

[12:31:12 CST(-0600)] <EricDalquist> what do you think of the current sql/ldap query templating

[12:31:28 CST(-0600)]

<EricDalquist> where you stick a

Unknown macro: {0}

in where you want the search/restrictions to appear

[13:21:14 CST(-0600)] <drewwills1> sorry EricDalquist... i was on a call

[13:21:26 CST(-0600)] <EricDalquist> no problem

[13:21:59 CST(-0600)] <drewwills1> the issue i run into with that is that sometimes i need more flexibility

[13:22:22 CST(-0600)] <EricDalquist> yeah

[13:22:25 CST(-0600)] <EricDalquist> that is my thought as well

[13:22:29 CST(-0600)]

<drewwills1> i may need to "select foo from bar where netId =

Unknown macro: {username}

"

[13:22:29 CST(-0600)] <EricDalquist> I'm not sure what a solution is though

[13:22:58 CST(-0600)] <EricDalquist> since that works great for simple queries but doesn't work for attribute sources than can have a more flexible search done

[13:23:07 CST(-0600)] <drewwills1> one sec...

[13:23:07 CST(-0600)] <EricDalquist> I'm open to all ideas here

[13:26:27 CST(-0600)] <drewwills1> i've used this approach, and it's attractive in some ways: https://gist.github.com/4261275

[13:26:42 CST(-0600)] <drewwills1> more complex, i guess, but flexible

[13:27:13 CST(-0600)] <EricDalquist> ok

[13:27:31 CST(-0600)] <EricDalquist> so the idea is that attribute source has a fixed set of query attributes

[13:27:47 CST(-0600)] <EricDalquist> you always have to ask for data with username=X

[13:29:04 CST(-0600)] <EricDalquist> so maybe the solution is just a variety of options when configuration attribute sources

[13:29:15 CST(-0600)] <EricDalquist> we have some like this that use named parameters

[13:29:19 CST(-0600)] <drewwills1> that's fine, but you might want to do lower(username) = x

[13:29:50 CST(-0600)] <EricDalquist> oh I was just saying that the incoming PD query would be for username=X

[13:29:57 CST(-0600)] <drewwills1> ah yes

[13:30:00 CST(-0600)] <EricDalquist> and X gets inserted for :username

[13:30:07 CST(-0600)] <drewwills1> yep, totally

[13:30:46 CST(-0600)] <drewwills1> but something like this alows you to "decorate" the SQL where the rubber hits the road around username = x

[13:31:05 CST(-0600)] <EricDalquist> ok

[13:31:20 CST(-0600)] <drewwills1> you might even want to do "where :grad_year > 2014"

[13:31:40 CST(-0600)] <drewwills1> use cases with cascading DAOs

[13:32:26 CST(-0600)] <drewwills1> brb... lunch

[13:33:54 CST(-0600)] <EricDalquist> ok

[14:39:23 CST(-0600)] <EricDalquist> drewwills: you around?

[14:57:48 CST(-0600)] <drewwills> yep

[14:57:55 CST(-0600)] <EricDalquist> so more PD stuff

[14:58:06 CST(-0600)] <EricDalquist> do you ever see the need for more complex person queries?

[14:58:07 CST(-0600)] <drewwills> ok

[14:58:18 CST(-0600)] <EricDalquist> right now we have this terrible API

[14:58:27 CST(-0600)] <drewwills> +1

[14:58:29 CST(-0600)] <EricDalquist> where you give it a map of attribute name>value pairs

[14:58:40 CST(-0600)] <EricDalquist> and each source figures out how it will handle those

[14:58:49 CST(-0600)] <EricDalquist> do we really need something more expressive?

[14:59:15 CST(-0600)] <drewwills> maybe... can you throw out an example?

[14:59:17 CST(-0600)] <EricDalquist> like something where you could do (firstName=Jane && (lastName=Doe || lastName=Smith))

[14:59:33 CST(-0600)] <drewwills> hmmm... ppossibly

[14:59:53 CST(-0600)] <drewwills> that would suppost full-featured directory searching, for example

[15:00:04 CST(-0600)] <EricDalquist> note that something like that really only works for attribute sources that support a more free-form query building

[15:00:18 CST(-0600)] <EricDalquist> sources that are using direct named attribute replacement wouldn't really fit here

[15:00:22 CST(-0600)] <EricDalquist> they would either have to be ignored

[15:00:38 CST(-0600)] <EricDalquist> or queried once per result after getting results from the more flexible sources

[15:00:42 CST(-0600)] <drewwills> instead of passing attr/value pairs, you could pass criteria objects

[15:00:49 CST(-0600)] <EricDalquist> right

[15:01:18 CST(-0600)] <EricDalquist> I'd probably model something after: http://static.springsource.org/spring-ldap/site/apidocs/org/springframework/ldap/filter/package-frame.html

[15:01:34 CST(-0600)] <drewwills> looking

[15:02:28 CST(-0600)] <drewwills> sure... not dissimilar to PAGS and DLM evaluators actually

[15:02:35 CST(-0600)] <EricDalquist> yeah

[15:02:41 CST(-0600)] <EricDalquist> it is essentially the LDAP filter syntax

[15:02:42 CST(-0600)] <drewwills> i wonder if we could consolidate even

[15:02:45 CST(-0600)] <EricDalquist> in java object form

[15:02:58 CST(-0600)] <EricDalquist> so all of this is coming from a potential chunk of time I have from UW to do a big refactoring of PD

[15:03:09 CST(-0600)] <EricDalquist> our motivations are around better query support

[15:03:15 CST(-0600)] <EricDalquist> and better concurrent support

[15:03:28 CST(-0600)] <EricDalquist> we're starting to see login slowdowns due to the large number of sources

[15:04:10 CST(-0600)] <EricDalquist> I'll be transcribing this into the wiki once I have better design bits

[15:04:14 CST(-0600)] <EricDalquist> but here are my notes so far: https://gist.github.com/4262128

[15:04:20 CST(-0600)] <drewwills> hmmm... it's nice that we have WI at the bleeding edge to work out all the kinks

[15:04:25 CST(-0600)] <EricDalquist>

[15:04:36 CST(-0600)] <EricDalquist> nothing like 100k logins a day to make sure its working right

[15:04:45 CST(-0600)] <EricDalquist> the idea that we greatly simplify the attribute sources

[15:04:51 CST(-0600)] <EricDalquist> have a spring XML namespace for configuration

[15:05:01 CST(-0600)] <EricDalquist> and a big chunk of "core" code that does most of the heavy lifting

[15:05:13 CST(-0600)] <drewwills> yeah this would probably be a good match for a spring xsd

[15:05:18 CST(-0600)] <EricDalquist> I'm also hoping we can get rid of the explicit dao ordering and stuff

[15:05:56 CST(-0600)] <EricDalquist> just configure a buncvh of attribute sources

[15:06:05 CST(-0600)] <EricDalquist> optionally ordering and merge strategies

[15:06:34 CST(-0600)] <EricDalquist> and the core logic should be able to figure out which attribute queries it can run in parallel

[15:06:42 CST(-0600)] <EricDalquist> and which need to wait for others

[15:06:57 CST(-0600)] <drewwills> i help probably 4+ schools a year start a portal... persondirs is always a challenge

[15:07:08 CST(-0600)] <EricDalquist> yeah

[15:07:15 CST(-0600)] <EricDalquist> and that is why I'm bugging you for input

[15:07:23 CST(-0600)] <EricDalquist> pain points and ideas around them are great

[15:07:25 CST(-0600)] <drewwills> yeah

[15:07:30 CST(-0600)] <EricDalquist> to make sure we avoid/fix those issues

[15:07:44 CST(-0600)] <drewwills> here's one item of specific input...

[15:08:24 CST(-0600)] <drewwills> default for queryType should be OR imho

[15:08:32 CST(-0600)] <drewwills> or should have been i guess

[15:08:55 CST(-0600)] <drewwills> but that may not even be relevant in the updated api

[15:09:23 CST(-0600)] <EricDalquist> so if I query for (foo=x, bar=y) default to get results for (foo==x || bar ==y)

[15:09:40 CST(-0600)] <EricDalquist> ok

[15:09:44 CST(-0600)] <EricDalquist> so some concerns I have about the more complex query language

[15:09:49 CST(-0600)] <EricDalquist> what do we do when:

[15:10:11 CST(-0600)] <EricDalquist> query: (lastName == Smith && isStudent=Y)

[15:10:18 CST(-0600)] <EricDalquist> and we have two attribute sources

[15:10:30 CST(-0600)] <EricDalquist> one supports both lastName and isStudent attributes

[15:10:37 CST(-0600)] <EricDalquist> the other only supports lastName

[15:10:52 CST(-0600)] <drewwills> more often what i see is client code that take 1 search term and applies it to all the querry attrs

[15:10:59 CST(-0600)] <drewwills> user submits foo

[15:11:17 CST(-0600)] <drewwills> translates to username=foo, firstName=foo, lastName=foo

[15:11:27 CST(-0600)] <EricDalquist> and we can get much more complex queries and weird "which attribute sources answer)"

[15:11:35 CST(-0600)] <EricDalquist> ah

[15:11:36 CST(-0600)] <drewwills> it's better if the default is a hit if any one of those matches foo

[15:12:30 CST(-0600)] <EricDalquist> right

[15:12:47 CST(-0600)] <EricDalquist> so in that case running a query for (username=foo || firstName=foo || lastName=foo)

[15:12:54 CST(-0600)] <drewwills> exactly

[15:13:03 CST(-0600)] <EricDalquist> the thing i'm not sure about is what to do when you have multiple attribute sources

[15:13:29 CST(-0600)] <EricDalquist> and the incoming query references attributes that one of the attributes sources can't query on

[15:13:37 CST(-0600)] <EricDalquist> if we always do OR queries that isn't a problem

[15:13:55 CST(-0600)] <EricDalquist> but if we ever do ANDs it is

[15:14:12 CST(-0600)] <drewwills> yep, that makes sense

[15:15:04 CST(-0600)] <EricDalquist> and that is where I'm kind of stuck for ideas right now

[15:15:10 CST(-0600)] <drewwills> so perhaps the DAOs themselvs should not even have a queryType == AND | OR

[15:15:33 CST(-0600)] <drewwills> maybe we move to a criteria model, and let the client code handle that detail

[15:15:53 CST(-0600)] <EricDalquist> by a criteria model you mean like that LDAP filter API?

[15:16:25 CST(-0600)] <drewwills> yes... and instead it could have a loosy-goosey/strict setting

[15:16:41 CST(-0600)] <drewwills> as to whether it will forgive missing attrs

[15:16:51 CST(-0600)] <drewwills> default is probably strict

[15:17:14 CST(-0600)] <EricDalquist> so by default if the source doesnt support any attribute in the query

[15:17:18 CST(-0600)] <EricDalquist> that source isn't queried

[15:17:44 CST(-0600)] <drewwills> well maybe not

[15:17:46 CST(-0600)] <EricDalquist> part of me is thinking that I need to break this up more

[15:17:47 CST(-0600)] <EricDalquist> hrm

[15:17:49 CST(-0600)] <EricDalquist> oh!

[15:17:53 CST(-0600)] <EricDalquist> so here is an idea

[15:18:03 CST(-0600)] <drewwills> consider (username == 'foo' | lastName == 'bar')

[15:18:04 CST(-0600)] <EricDalquist> we break up "searchable sources" and "attribute sources"

[15:18:16 CST(-0600)] <drewwills> if you only support username, that's cool

[15:18:31 CST(-0600)] <drewwills> but if it's (username == 'foo' & lastName == 'bar')

[15:18:51 CST(-0600)] <drewwills> you must support both... unless (perhaps) strict=false

[15:19:03 CST(-0600)] <EricDalquist> ok

[15:19:28 CST(-0600)] <drewwills> i'm not convinced the strict thing is necessary... just a thought

[15:20:37 CST(-0600)] <EricDalquist> yeah

[15:21:09 CST(-0600)] <EricDalquist> I'm wondering if we really just need that search vs source seperation

[15:21:31 CST(-0600)]

<EricDalquist> so attribute sources where you can configure a "query expression placeholder" the

Unknown macro: {0}

that we have now

[15:21:35 CST(-0600)] <EricDalquist> we use those in searches

[15:21:50 CST(-0600)] <EricDalquist> and sources that use named parameters

[15:22:13 CST(-0600)] <EricDalquist> we don't treat as searchable

[15:22:13 CST(-0600)] <EricDalquist> hrm

[15:22:14 CST(-0600)] <EricDalquist> no

[15:22:18 CST(-0600)] <EricDalquist> that doesn't work either

[15:22:19 CST(-0600)] <EricDalquist> ok

[15:22:24 CST(-0600)] <EricDalquist> I'm going to go write up more examples

[15:37:14 CST(-0600)] <EricDalquist> so building on your strict/loose idea

[15:37:36 CST(-0600)] <EricDalquist> maybe "loose" does client side filtering

[15:37:43 CST(-0600)] <EricDalquist> so for a query of (firstName=Jane && (isStudent=Y || lastName=Doe))

[15:38:16 CST(-0600)] <EricDalquist> lets say we have sourceA that can query on all three attributes, we got ahead and run that query as is

[15:38:27 CST(-0600)] <EricDalquist> then we have sourceB that only supports firstName and lastName

[15:38:40 CST(-0600)] <EricDalquist> for B we run (firstName=Jane && (lastName=Doe))

[15:39:18 CST(-0600)] <EricDalquist> then during the merge process we apply that isStudent=Y filter in code