[13:10:37 CST(-0600)] <jwennmacher> EricDalquist: Good afternoon. From your feedback I started the first report (total tab renders). The form lists all the standard fields (start/end date, interval, groups) and I added tabs.
[13:10:53 CST(-0600)] <EricDalquist> great
[13:11:05 CST(-0600)] <jwennmacher> What should the tab names display? Currently in the selection I have 'tab name' / 'fragment owner'
[13:11:45 CST(-0600)] <jwennmacher> In the graph legend I have 'group/fragment name/tab name'.
[13:11:51 CST(-0600)] <jwennmacher> (I'll need to make them consistent)
[13:12:00 CST(-0600)] <EricDalquist> hrm
[13:12:29 CST(-0600)] <EricDalquist> I wish we could just do "tab name"
[13:12:32 CST(-0600)] <jwennmacher> I'm not sure what format would be most logical to the admins; do they need to know the fragment name?
[13:12:36 CST(-0600)] <EricDalquist> but lots of places are going to have duplicate tab names
[13:12:40 CST(-0600)] <EricDalquist> with different fragment owners
[13:12:46 CST(-0600)] <jwennmacher> I was thinking the same
[13:13:09 CST(-0600)] <EricDalquist> if space wasn't an issue we could do "Tab Name (fragmentowner)"
[13:13:17 CST(-0600)] <EricDalquist> but I'd worry that would take up too much ui space
[13:13:23 CST(-0600)] <EricDalquist> so maybe just fragmentowner for now?
[13:13:28 CST(-0600)] <EricDalquist> hrm
[13:13:36 CST(-0600)] <EricDalquist> but that breaks for fragments with multiple tabs
[13:13:37 CST(-0600)] <EricDalquist> 
[13:13:44 CST(-0600)] <EricDalquist> "Tab Name (fragmentowner)" I guess?
[13:13:52 CST(-0600)] <jwennmacher> Sounds good
[13:15:22 CST(-0600)] <jwennmacher> You have two very long CATCH_ALL fragment owners that show up in the selection box. CATCH_ALL_MISSING_TAB_OWNER and CATCH_ALL_PERSONAL_TAB_OWNER. I was thinking they could change to 'Missing owner' and 'Personal tab' or something like that
[13:15:32 CST(-0600)] <EricDalquist> yeah
[13:15:49 CST(-0600)] <EricDalquist> perhaps just stick some strings in Messages.properties for those?
[13:16:43 CST(-0600)] <jwennmacher> Or should the db be changed? They are currently coming straight from the DB.
[13:17:02 CST(-0600)] <EricDalquist> well I had those because I wanted something with very little chance of having a conflict
[13:19:21 CST(-0600)] <jwennmacher> OK I can look for those strings and replace with message property values. At least it aids internationalization
[13:19:29 CST(-0600)] <EricDalquist> yup
[13:19:41 CST(-0600)] <EricDalquist> figure that the deployer will have locale specific real tab names
[13:20:19 CST(-0600)] <jwennmacher> So for the graph legend do we want tabname (fragname) - groupname?
[13:21:45 CST(-0600)] <jwennmacher> If you are capturing the graph for to insert as a visual I could see having the groupname present. Its a bit wordy though. I was wondering if we needed the group name on the legend
[13:27:06 CST(-0600)] <EricDalquist> yeah, I'd assume you'd want it
[13:27:10 CST(-0600)] <EricDalquist> though perhaps you add some logic
[13:27:18 CST(-0600)] <EricDalquist> and if they only select one group we don';t include it
[13:27:24 CST(-0600)] <EricDalquist> or maybe include the group in the graph title instead
[13:28:35 CST(-0600)] <jwennmacher> I assume you'd want to be able to select multiple tab names.
[13:28:38 CST(-0600)] <EricDalquist> yes
[13:28:42 CST(-0600)] <EricDalquist> just like you do with groups right now
[13:29:00 CST(-0600)] <jwennmacher> With groups I only see 'Everyone'. I was going to look into why I don't see others.
[13:29:10 CST(-0600)] <EricDalquist> that is the only data you have aggregations for
[13:29:31 CST(-0600)] <EricDalquist> take a look at uportal-war/src/main/data/default_entities/event-aggregation/default.event-aggregation.xml
[13:29:42 CST(-0600)] <EricDalquist> you can tweak that file to aggregate more data
[13:29:52 CST(-0600)] <EricDalquist> and then use the import/export portlet to import the udpated file
[13:30:24 CST(-0600)] <EricDalquist> that would actually be another great task after the reporting portlets are done
[13:30:28 CST(-0600)] <EricDalquist> is an admin portlet
[13:30:32 CST(-0600)] <EricDalquist> to manage the data configured via that file
[13:51:14 CST(-0600)] <jwennmacher> OK got multiple groups. MUCH better. Thanks 
[13:52:22 CST(-0600)] <EricDalquist> nice
[13:52:42 CST(-0600)] <EricDalquist> so in theory the reporting portlets should only ever show you options where there is actually data
[13:53:02 CST(-0600)] <jwennmacher> That's pretty cool.
[13:57:01 CST(-0600)] <jwennmacher> I'll finish up this report and check it in UP-3625 for you to take a look at. I should have a few days before my next project to help out a bit more. What do you recommend I target next?
[13:57:35 CST(-0600)] <EricDalquist> awesome
[13:57:52 CST(-0600)] <EricDalquist> well a portlet execution report in the same model would be good
[13:58:09 CST(-0600)] <EricDalquist> and then perhaps the more specific "slow portlet" stuff
[13:58:19 CST(-0600)] <EricDalquist> we've talked about more of a "portal status" dashboard portlet
[13:58:46 CST(-0600)] <EricDalquist> you could have a portlet that shows concurrent users, top 5 slowest portlets
[13:58:51 CST(-0600)] <EricDalquist> tab render times
[13:59:02 CST(-0600)] <EricDalquist> load that data via a resource request & ajax with a timer
[13:59:10 CST(-0600)] <EricDalquist> and you would have a nice portlet to watch and monitor your portal
[13:59:14 CST(-0600)] <drewwills> sounds like some great progress james & eric... i'll be excited to see it
[13:59:19 CST(-0600)] <EricDalquist>
[13:59:20 CST(-0600)] <EricDalquist> me too
[13:59:50 CST(-0600)] <drewwills> basically all the schools I'm working with were very happy about stats hitting
[14:01:32 CST(-0600)] <jwennmacher> Those are some great ideas. I'll finish up and start working on the portlet execution report as it is more bite-sized.
[14:01:38 CST(-0600)] <EricDalquist> yup
[14:01:48 CST(-0600)] <EricDalquist> well and for the dashboard we can figure out a way to start small there
[14:02:14 CST(-0600)] <EricDalquist> I should also be able to pull this stuff into UWs portal fairly quickly
[14:02:19 CST(-0600)] <EricDalquist> so we can test with some big data sets
[14:02:48 CST(-0600)] <jwennmacher> sounds good
[14:03:15 CST(-0600)] <EricDalquist> like we have 5.8 million rows in UP_TAB_RENDER_AGGR
[14:04:09 CST(-0600)] <EricDalquist> and 34.4 million in UP_PORTLET_EXEC_AGGR
[14:04:12 CST(-0600)] <EricDalquist>
[14:04:34 CST(-0600)] <EricDalquist> I wish we had the time to really figure out if there was a NoSQL solution that would be easy for uPortal deployers to setup and work with
[14:04:56 CST(-0600)] <EricDalquist> but I have a feeling that introducing something like that, while great for data processing, would be bad for ease of deployment
[14:08:08 CST(-0600)] <jwennmacher> I'm not very familiar with the noSQL benefits. How would you use it? In its simplest form I understand it's basically a key-value store. How would you use it for this application?
[14:08:21 CST(-0600)] <EricDalquist> so the stats in this case are not very relational
[14:08:26 CST(-0600)] <EricDalquist> the raw stats especially
[14:08:31 CST(-0600)] <EricDalquist> it is more a big log file
[14:08:41 CST(-0600)] <EricDalquist> we just use a DB because we know deployers will have a central db
[14:08:46 CST(-0600)] <EricDalquist> that multiple portal instances can talk to
[14:09:11 CST(-0600)] <EricDalquist> by not having the true relational constraints we could in theory get much better performance and data storage profiles
[14:09:42 CST(-0600)] <EricDalquist> the aggregate data would be a little different, right now that is kind of relational
[14:09:53 CST(-0600)] <EricDalquist> but it wouldn't be hard to turn into a key-value setup either
[14:10:09 CST(-0600)] <EricDalquist> and again have potentially much better performance in both processing and searching
[14:11:18 CST(-0600)] <EricDalquist> who knows ... hibernate 5 should have decent nosql support
[14:11:28 CST(-0600)] <EricDalquist> so maybe at that point we can look at is an optional config thing
[14:11:47 CST(-0600)] <EricDalquist> like places with big data (like uw) could switch to a nosql provider
[14:11:59 CST(-0600)] <EricDalquist> and smaller deployers can just do SQL as they don't have the data volumes
[14:14:44 CST(-0600)] <jwennmacher> makes sense. Thanks
[14:15:00 CST(-0600)] <EricDalquist> it would give us a good buzzword though!
[14:15:13 CST(-0600)] <jwennmacher>
[15:23:51 CST(-0600)] <jwennmacher> EricDalquist: I'd like to get your feedback on something to make sure I'm going in the right direction.
[15:23:56 CST(-0600)] <EricDalquist> ok
[15:24:27 CST(-0600)] <jwennmacher> I'm making the modifications to allow for multiple tabs to be selected in the render tab report form.
[15:25:38 CST(-0600)] <jwennmacher> I think the current design doesn't allow for multiple tabs to be selected and included in the query.
[15:26:06 CST(-0600)] <EricDalquist> ah the search api?
[15:27:58 CST(-0600)] <jwennmacher> Yes. What I'm thinking is I need to modify the BaseStatsticsReportController method createAggregationsQueryKey to return a Set<K> instead of just a <K> and iterate through the sets, or I could modify TabRenderAggregationKey to have a Set<AggreatedTabMapping> instead of just AggregatedTabMapping.
[15:28:13 CST(-0600)] <jwennmacher> I think the former solution makes more sense. Thoughts?
[15:28:59 CST(-0600)] <jwennmacher> This is a bit out of context so let me know what doesn't make sense.
[15:29:02 CST(-0600)] <EricDalquist> hrm
[15:29:04 CST(-0600)] <EricDalquist> ok
[15:29:08 CST(-0600)] <EricDalquist> looking at this now ...
[15:30:09 CST(-0600)] <jwennmacher> But basically yes, we need to pass multiple tab names into baseAggregationDao.getAggregations. The key as is (a single tabname-group-interval) doesn't work for multiple tabnames (or portlet names or whatever)
[15:30:46 CST(-0600)] <EricDalquist> ah right
[15:30:57 CST(-0600)] <EricDalquist> and BaseStatisticsReportController has native support for multi-group queries
[15:30:59 CST(-0600)] <jwennmacher> JpaBaseAggregationDao.getAggretations has an invocation to bindAggregationSpecificKeyParameters(query, key) which is how that 'extra' dimension would be handled
[15:31:21 CST(-0600)] <jwennmacher> yes, but not multiple whatever-extra-thing queries
[15:31:24 CST(-0600)] <EricDalquist> yeah
[15:31:31 CST(-0600)] <EricDalquist> give me 5 minutes to poke around the API
[15:31:39 CST(-0600)] <jwennmacher> OK
[15:35:22 CST(-0600)] <EricDalquist> ok ... so my thoughts I think line up with yours
[15:35:35 CST(-0600)] <EricDalquist> change createAggregationsQueryKey to return a Set<K>
[15:35:53 CST(-0600)] <EricDalquist> and change BaseAggregationDao
[15:35:54 CST(-0600)] <EricDalquist> getAggregations(DateTime start, DateTime end, K key, AggregatedGroupMapping... aggregatedGroupMappings)
[15:35:57 CST(-0600)] <EricDalquist> to
[15:36:02 CST(-0600)] <EricDalquist> getAggregations(DateTime start, DateTime end, Set<K> key, AggregatedGroupMapping... aggregatedGroupMappings)
[15:36:03 CST(-0600)] <jwennmacher> Adjustment to my suggested solution. Have BaseStatsticsReportController method createAggregationsQueryKey return a Set<K>, and pass that Set<K> into a JpaBaseAggregationDao method that accepts a set<K> instead
[15:36:09 CST(-0600)] <jwennmacher> Yep!!!
[15:36:16 CST(-0600)] <EricDalquist> perhaps add that as an overloaded of the existing getAggregations
[15:36:41 CST(-0600)] <EricDalquist> then you'll have to tweak the various impls of BaseAggregationDao to collate that Set<K> into a single query
[15:36:44 CST(-0600)] <EricDalquist> which shouldn't be too bad
[15:37:21 CST(-0600)] <jwennmacher> Nope. Cool.
[15:37:32 CST(-0600)] <jwennmacher> Thanks for the collaboration.
[15:37:36 CST(-0600)] <EricDalquist> no problem
[15:37:41 CST(-0600)] <EricDalquist> always happy to talk about that stuff
[15:43:46 CST(-0600)] <jwennmacher> BTW one other question. I haven't thought through this all the way but I wanted to get your initial thoughts on aspectj vs. aop proxies. What brought this to mind is I had to make a minor change to JpaAggregatedTabLookupDao. The public method getMappedTabForLayoutId invoked by a controller did not have the @OpenEntityManager annotation but the internal public method it invoked did. The entity manager wasn't created because
[15:44:32 CST(-0600)] <jwennmacher> If using compile-time or run-time weaving, this type of issue of course wouldn't occur.
[15:45:11 CST(-0600)] <EricDalquist> not sure I follow
[15:45:25 CST(-0600)] <EricDalquist> JpaAggregatedTabLookupDao.getTabMapping has the @OpenEntityManager annotation
[15:45:40 CST(-0600)] <EricDalquist> ah right
[15:45:41 CST(-0600)] <EricDalquist> I see
[15:45:47 CST(-0600)] <EricDalquist> ignore my previous statement
[15:46:08 CST(-0600)] <EricDalquist> yeah, I have nothing against compile-time weaving if someone (hint) wants to look into it
[15:46:14 CST(-0600)] <jwennmacher> Yes. The controller was invoking getMappedTabForLayoutId (incorreclty mind you ... I thought the layoutNodeId was the id, but I found out it wasn't and had to implement a get by id method).
[15:46:23 CST(-0600)] <EricDalquist> load-time worries me a little more since then we're adding more complexity to the deployers
[15:46:27 CST(-0600)] <jwennmacher> I like that hint.
[15:46:59 CST(-0600)] <jwennmacher> Yeah I like compile-time, especially on big projects. You don't need anything else happening at startup.
[15:47:10 CST(-0600)] <jwennmacher> Slows things down even more