uPortal IRC Logs-2012-12-13

Owned by Apereo Infrastructure
Last updated: Dec 13, 2012
8 min read

[13:10:37 CST(-0600)] <jwennmacher> EricDalquist: Good afternoon. From your feedback I started the first report (total tab renders). The form lists all the standard fields (start/end date, interval, groups) and I added tabs.

[13:10:53 CST(-0600)] <EricDalquist> great

[13:11:05 CST(-0600)] <jwennmacher> What should the tab names display? Currently in the selection I have 'tab name' / 'fragment owner'

[13:11:45 CST(-0600)] <jwennmacher> In the graph legend I have 'group/fragment name/tab name'.

[13:11:51 CST(-0600)] <jwennmacher> (I'll need to make them consistent)

[13:12:00 CST(-0600)] <EricDalquist> hrm

[13:12:29 CST(-0600)] <EricDalquist> I wish we could just do "tab name"

[13:12:32 CST(-0600)] <jwennmacher> I'm not sure what format would be most logical to the admins; do they need to know the fragment name?

[13:12:36 CST(-0600)] <EricDalquist> but lots of places are going to have duplicate tab names

[13:12:40 CST(-0600)] <EricDalquist> with different fragment owners

[13:12:46 CST(-0600)] <jwennmacher> I was thinking the same

[13:13:09 CST(-0600)] <EricDalquist> if space wasn't an issue we could do "Tab Name (fragmentowner)"

[13:13:17 CST(-0600)] <EricDalquist> but I'd worry that would take up too much ui space

[13:13:23 CST(-0600)] <EricDalquist> so maybe just fragmentowner for now?

[13:13:28 CST(-0600)] <EricDalquist> hrm

[13:13:36 CST(-0600)] <EricDalquist> but that breaks for fragments with multiple tabs

[13:13:37 CST(-0600)] <EricDalquist> (tongue)

[13:13:44 CST(-0600)] <EricDalquist> "Tab Name (fragmentowner)" I guess?

[13:13:52 CST(-0600)] <jwennmacher> Sounds good

[13:15:22 CST(-0600)] <jwennmacher> You have two very long CATCH_ALL fragment owners that show up in the selection box. CATCH_ALL_MISSING_TAB_OWNER and CATCH_ALL_PERSONAL_TAB_OWNER. I was thinking they could change to 'Missing owner' and 'Personal tab' or something like that

[13:15:32 CST(-0600)] <EricDalquist> yeah

[13:15:49 CST(-0600)] <EricDalquist> perhaps just stick some strings in Messages.properties for those?

[13:16:43 CST(-0600)] <jwennmacher> Or should the db be changed? They are currently coming straight from the DB.

[13:17:02 CST(-0600)] <EricDalquist> well I had those because I wanted something with very little chance of having a conflict

[13:19:21 CST(-0600)] <jwennmacher> OK I can look for those strings and replace with message property values. At least it aids internationalization

[13:19:29 CST(-0600)] <EricDalquist> yup

[13:19:41 CST(-0600)] <EricDalquist> figure that the deployer will have locale specific real tab names

[13:20:19 CST(-0600)] <jwennmacher> So for the graph legend do we want tabname (fragname) - groupname?

[13:21:45 CST(-0600)] <jwennmacher> If you are capturing the graph for to insert as a visual I could see having the groupname present. Its a bit wordy though. I was wondering if we needed the group name on the legend

[13:27:06 CST(-0600)] <EricDalquist> yeah, I'd assume you'd want it

[13:27:10 CST(-0600)] <EricDalquist> though perhaps you add some logic

[13:27:18 CST(-0600)] <EricDalquist> and if they only select one group we don';t include it

[13:27:24 CST(-0600)] <EricDalquist> or maybe include the group in the graph title instead

[13:28:35 CST(-0600)] <jwennmacher> I assume you'd want to be able to select multiple tab names.

[13:28:38 CST(-0600)] <EricDalquist> yes

[13:28:42 CST(-0600)] <EricDalquist> just like you do with groups right now

[13:29:00 CST(-0600)] <jwennmacher> With groups I only see 'Everyone'. I was going to look into why I don't see others.

[13:29:10 CST(-0600)] <EricDalquist> that is the only data you have aggregations for

[13:29:31 CST(-0600)] <EricDalquist> take a look at uportal-war/src/main/data/default_entities/event-aggregation/default.event-aggregation.xml

[13:29:42 CST(-0600)] <EricDalquist> you can tweak that file to aggregate more data

[13:29:52 CST(-0600)] <EricDalquist> and then use the import/export portlet to import the udpated file

[13:30:24 CST(-0600)] <EricDalquist> that would actually be another great task after the reporting portlets are done

[13:30:28 CST(-0600)] <EricDalquist> is an admin portlet

[13:30:32 CST(-0600)] <EricDalquist> to manage the data configured via that file

[13:51:14 CST(-0600)] <jwennmacher> OK got multiple groups. MUCH better. Thanks (smile)

[13:52:22 CST(-0600)] <EricDalquist> nice

[13:52:42 CST(-0600)] <EricDalquist> so in theory the reporting portlets should only ever show you options where there is actually data

[13:53:02 CST(-0600)] <jwennmacher> That's pretty cool.

[13:57:01 CST(-0600)] <jwennmacher> I'll finish up this report and check it in UP-3625 for you to take a look at. I should have a few days before my next project to help out a bit more. What do you recommend I target next?

[13:57:35 CST(-0600)] <EricDalquist> awesome

[13:57:52 CST(-0600)] <EricDalquist> well a portlet execution report in the same model would be good

[13:58:09 CST(-0600)] <EricDalquist> and then perhaps the more specific "slow portlet" stuff

[13:58:19 CST(-0600)] <EricDalquist> we've talked about more of a "portal status" dashboard portlet

[13:58:46 CST(-0600)] <EricDalquist> you could have a portlet that shows concurrent users, top 5 slowest portlets

[13:58:51 CST(-0600)] <EricDalquist> tab render times

[13:59:02 CST(-0600)] <EricDalquist> load that data via a resource request & ajax with a timer

[13:59:10 CST(-0600)] <EricDalquist> and you would have a nice portlet to watch and monitor your portal

[13:59:14 CST(-0600)] <drewwills> sounds like some great progress james & eric... i'll be excited to see it

[13:59:19 CST(-0600)] <EricDalquist> (smile)

[13:59:20 CST(-0600)] <EricDalquist> me too

[13:59:50 CST(-0600)] <drewwills> basically all the schools I'm working with were very happy about stats hitting

[14:01:32 CST(-0600)] <jwennmacher> Those are some great ideas. I'll finish up and start working on the portlet execution report as it is more bite-sized.

[14:01:38 CST(-0600)] <EricDalquist> yup

[14:01:48 CST(-0600)] <EricDalquist> well and for the dashboard we can figure out a way to start small there

[14:02:14 CST(-0600)] <EricDalquist> I should also be able to pull this stuff into UWs portal fairly quickly

[14:02:19 CST(-0600)] <EricDalquist> so we can test with some big data sets

[14:02:48 CST(-0600)] <jwennmacher> sounds good

[14:03:15 CST(-0600)] <EricDalquist> like we have 5.8 million rows in UP_TAB_RENDER_AGGR

[14:04:09 CST(-0600)] <EricDalquist> and 34.4 million in UP_PORTLET_EXEC_AGGR

[14:04:12 CST(-0600)] <EricDalquist> (tongue)

[14:04:34 CST(-0600)] <EricDalquist> I wish we had the time to really figure out if there was a NoSQL solution that would be easy for uPortal deployers to setup and work with

[14:04:56 CST(-0600)] <EricDalquist> but I have a feeling that introducing something like that, while great for data processing, would be bad for ease of deployment

[14:08:08 CST(-0600)] <jwennmacher> I'm not very familiar with the noSQL benefits. How would you use it? In its simplest form I understand it's basically a key-value store. How would you use it for this application?

[14:08:21 CST(-0600)] <EricDalquist> so the stats in this case are not very relational

[14:08:26 CST(-0600)] <EricDalquist> the raw stats especially

[14:08:31 CST(-0600)] <EricDalquist> it is more a big log file

[14:08:41 CST(-0600)] <EricDalquist> we just use a DB because we know deployers will have a central db

[14:08:46 CST(-0600)] <EricDalquist> that multiple portal instances can talk to

[14:09:11 CST(-0600)] <EricDalquist> by not having the true relational constraints we could in theory get much better performance and data storage profiles

[14:09:42 CST(-0600)] <EricDalquist> the aggregate data would be a little different, right now that is kind of relational

[14:09:53 CST(-0600)] <EricDalquist> but it wouldn't be hard to turn into a key-value setup either

[14:10:09 CST(-0600)] <EricDalquist> and again have potentially much better performance in both processing and searching

[14:11:18 CST(-0600)] <EricDalquist> who knows ... hibernate 5 should have decent nosql support

[14:11:28 CST(-0600)] <EricDalquist> so maybe at that point we can look at is an optional config thing

[14:11:47 CST(-0600)] <EricDalquist> like places with big data (like uw) could switch to a nosql provider

[14:11:59 CST(-0600)] <EricDalquist> and smaller deployers can just do SQL as they don't have the data volumes

[14:14:44 CST(-0600)] <jwennmacher> makes sense. Thanks

[14:15:00 CST(-0600)] <EricDalquist> it would give us a good buzzword though!

[14:15:13 CST(-0600)] <jwennmacher> (smile)

[15:23:51 CST(-0600)] <jwennmacher> EricDalquist: I'd like to get your feedback on something to make sure I'm going in the right direction.

[15:23:56 CST(-0600)] <EricDalquist> ok

[15:24:27 CST(-0600)] <jwennmacher> I'm making the modifications to allow for multiple tabs to be selected in the render tab report form.

[15:25:38 CST(-0600)] <jwennmacher> I think the current design doesn't allow for multiple tabs to be selected and included in the query.

[15:26:06 CST(-0600)] <EricDalquist> ah the search api?

[15:27:58 CST(-0600)] <jwennmacher> Yes. What I'm thinking is I need to modify the BaseStatsticsReportController method createAggregationsQueryKey to return a Set<K> instead of just a <K> and iterate through the sets, or I could modify TabRenderAggregationKey to have a Set<AggreatedTabMapping> instead of just AggregatedTabMapping.

[15:28:13 CST(-0600)] <jwennmacher> I think the former solution makes more sense. Thoughts?

[15:28:59 CST(-0600)] <jwennmacher> This is a bit out of context so let me know what doesn't make sense.

[15:29:02 CST(-0600)] <EricDalquist> hrm

[15:29:04 CST(-0600)] <EricDalquist> ok

[15:29:08 CST(-0600)] <EricDalquist> looking at this now ...

[15:30:09 CST(-0600)] <jwennmacher> But basically yes, we need to pass multiple tab names into baseAggregationDao.getAggregations. The key as is (a single tabname-group-interval) doesn't work for multiple tabnames (or portlet names or whatever)

[15:30:46 CST(-0600)] <EricDalquist> ah right

[15:30:57 CST(-0600)] <EricDalquist> and BaseStatisticsReportController has native support for multi-group queries

[15:30:59 CST(-0600)] <jwennmacher> JpaBaseAggregationDao.getAggretations has an invocation to bindAggregationSpecificKeyParameters(query, key) which is how that 'extra' dimension would be handled

[15:31:21 CST(-0600)] <jwennmacher> yes, but not multiple whatever-extra-thing queries

[15:31:24 CST(-0600)] <EricDalquist> yeah

[15:31:31 CST(-0600)] <EricDalquist> give me 5 minutes to poke around the API

[15:31:39 CST(-0600)] <jwennmacher> OK

[15:35:22 CST(-0600)] <EricDalquist> ok ... so my thoughts I think line up with yours

[15:35:35 CST(-0600)] <EricDalquist> change createAggregationsQueryKey to return a Set<K>

[15:35:53 CST(-0600)] <EricDalquist> and change BaseAggregationDao

[15:35:54 CST(-0600)] <EricDalquist> getAggregations(DateTime start, DateTime end, K key, AggregatedGroupMapping... aggregatedGroupMappings)

[15:35:57 CST(-0600)] <EricDalquist> to

[15:36:02 CST(-0600)] <EricDalquist> getAggregations(DateTime start, DateTime end, Set<K> key, AggregatedGroupMapping... aggregatedGroupMappings)

[15:36:03 CST(-0600)] <jwennmacher> Adjustment to my suggested solution. Have BaseStatsticsReportController method createAggregationsQueryKey return a Set<K>, and pass that Set<K> into a JpaBaseAggregationDao method that accepts a set<K> instead

[15:36:09 CST(-0600)] <jwennmacher> Yep!!!

[15:36:16 CST(-0600)] <EricDalquist> perhaps add that as an overloaded of the existing getAggregations

[15:36:41 CST(-0600)] <EricDalquist> then you'll have to tweak the various impls of BaseAggregationDao to collate that Set<K> into a single query

[15:36:44 CST(-0600)] <EricDalquist> which shouldn't be too bad

[15:37:21 CST(-0600)] <jwennmacher> Nope. Cool.

[15:37:32 CST(-0600)] <jwennmacher> Thanks for the collaboration.

[15:37:36 CST(-0600)] <EricDalquist> no problem

[15:37:41 CST(-0600)] <EricDalquist> always happy to talk about that stuff (smile)

[15:43:46 CST(-0600)] <jwennmacher> BTW one other question. I haven't thought through this all the way but I wanted to get your initial thoughts on aspectj vs. aop proxies. What brought this to mind is I had to make a minor change to JpaAggregatedTabLookupDao. The public method getMappedTabForLayoutId invoked by a controller did not have the @OpenEntityManager annotation but the internal public method it invoked did. The entity manager wasn't created because

[15:44:32 CST(-0600)] <jwennmacher> If using compile-time or run-time weaving, this type of issue of course wouldn't occur.

[15:45:11 CST(-0600)] <EricDalquist> not sure I follow

[15:45:25 CST(-0600)] <EricDalquist> JpaAggregatedTabLookupDao.getTabMapping has the @OpenEntityManager annotation

[15:45:40 CST(-0600)] <EricDalquist> ah right

[15:45:41 CST(-0600)] <EricDalquist> I see

[15:45:47 CST(-0600)] <EricDalquist> ignore my previous statement

[15:46:08 CST(-0600)] <EricDalquist> yeah, I have nothing against compile-time weaving if someone (hint) wants to look into it

[15:46:14 CST(-0600)] <jwennmacher> Yes. The controller was invoking getMappedTabForLayoutId (incorreclty mind you ... I thought the layoutNodeId was the id, but I found out it wasn't and had to implement a get by id method).

[15:46:23 CST(-0600)] <EricDalquist> load-time worries me a little more since then we're adding more complexity to the deployers

[15:46:27 CST(-0600)] <jwennmacher> I like that hint.

[15:46:59 CST(-0600)] <jwennmacher> Yeah I like compile-time, especially on big projects. You don't need anything else happening at startup.

[15:47:10 CST(-0600)] <jwennmacher> Slows things down even more

[15:48:07 CST(-0600)] <jwennmacher> I didn't know if there was discussion about that before or not. I've used it before and I generally find it better than aop proxies because you don't have to think about these types of things.

[15:48:46 CST(-0600)] <EricDalquist> no, its never really been brought up before

[15:48:53 CST(-0600)] <EricDalquist> I've thought abuot it

[15:49:01 CST(-0600)] <EricDalquist> but just never had the motivation to look into it

[15:49:08 CST(-0600)] <EricDalquist> I'd love to see something like that figured out though

[15:49:24 CST(-0600)] <EricDalquist> I suppose in theory runtime performance would be better with compile-time weaving too

[15:49:30 CST(-0600)] <EricDalquist> fewer wrapper layers

[15:49:44 CST(-0600)] <jwennmacher> If there are any downsides you can think of, I'm interested in hearing that. Otherwise, I'll add that as a todo item. I suspect it might be considered low on the priority list though (which is fine)

[15:50:02 CST(-0600)] <EricDalquist> yeah, I don't know enough about it to really know of down sides

[15:50:07 CST(-0600)] <EricDalquist> but I can't think of any right now

[15:50:16 CST(-0600)] <EricDalquist> just ping here when/if you work on it

[15:50:21 CST(-0600)] <EricDalquist> I'd love to see the possibilities

[15:50:31 CST(-0600)] <jwennmacher> Sounds good. Always love to get others insights.