uPortal IRC Logs-2012-01-11

Owned by Apereo Infrastructure
Last updated: Jan 11, 2012Version comment
10 min read

[10:08:31 CST(-0600)] <EricDalquist> Arvids: you there?

[10:16:39 CST(-0600)] <dd> hey @EricDalquist

[10:16:44 CST(-0600)] <EricDalquist> hi'

[10:17:08 CST(-0600)] <dd> i'm trying to remove the default welcome portlet (the one with tabs) from the welcome fragment

[10:17:15 CST(-0600)] <dd> you answered here: http://jasig.275507.n4.nabble.com/Delete-default-welcome-channel-td1017064.html

[10:17:30 CST(-0600)] <dd> but i don't get a delete option when i am using the welcome fragment user

[10:18:09 CST(-0600)] <EricDalquist> hrm

[10:18:18 CST(-0600)] <EricDalquist> what version of uPortal are you on?

[10:18:23 CST(-0600)] <dd> 3.2.4

[10:18:32 CST(-0600)] <EricDalquist> so I think there is a bug in the UI

[10:18:41 CST(-0600)] <EricDalquist> you first need to edit the settings for that portlet

[10:18:45 CST(-0600)] <EricDalquist> and make it deletable

[10:18:48 CST(-0600)] <EricDalquist> then you can delete it

[10:19:07 CST(-0600)] <EricDalquist> I don't have a running instance of 3.2 handy to give you a more detailed description than that

[10:19:35 CST(-0600)] <dd> hmm, do you know what option needs to be enabled to make it deleteable?

[10:19:47 CST(-0600)] <dd> i see edit

[10:19:54 CST(-0600)] <dd> but that doesn't seem to do anything

[10:19:56 CST(-0600)] <EricDalquist> athena: are you there?

[10:21:16 CST(-0600)] <EricDalquist> I'm trying to remember where it is ...

[10:21:29 CST(-0600)] <EricDalquist> is there any sort of "edit layout" button anywhere?

[10:23:06 CST(-0600)] <dd> there is page layout

[10:23:13 CST(-0600)] <dd> but that is for columns

[10:23:25 CST(-0600)] <EricDalquist> that might be it

[10:23:26 CST(-0600)] <EricDalquist> go to that

[10:23:38 CST(-0600)] <EricDalquist> and for each portlet do you see a little pencil/paper icon?

[10:23:47 CST(-0600)] <EricDalquist> or just try clicking on the portlet title

[10:25:11 CST(-0600)] <dd> ah, ok. it was delete channel to make the delete icon show up

[10:25:18 CST(-0600)] <EricDalquist> yweah

[10:29:59 CST(-0600)] <dd> on Bookmarks Portlet 1.0.10, is there any way to set default bookmarks for all users?

[10:30:08 CST(-0600)] <dd> or users in certain groups

[10:30:13 CST(-0600)] <EricDalquist> I can't remember

[10:30:20 CST(-0600)] <EricDalquist> that would be a good question for the email list

[10:38:29 CST(-0600)] <athena> in 3.2 i don't think there's a way other than the evil legacy customization channel to set fragment permissions

[10:38:35 CST(-0600)] <athena> adding that to the UI was something we did for 4

[10:43:59 CST(-0600)] <Arvids> Hi, EricDalquist

[10:45:44 CST(-0600)] <Arvids> Today I noticed that there something must be done for selects like: http://pastebin.com/HgMawLPT

[10:46:12 CST(-0600)] <Arvids> Is there a way to set fetching strategies using JPA?

[10:46:45 CST(-0600)] <EricDalquist> ?

[10:46:54 CST(-0600)] <EricDalquist> so the sheer number of queries?

[10:47:01 CST(-0600)] <Arvids> yes

[10:47:20 CST(-0600)] <Arvids> I´d like to fetch those entities using join´s, where possible

[10:47:25 CST(-0600)] <EricDalquist> yeah, so the problem there is portal architecture

[10:47:39 CST(-0600)] <EricDalquist> there is nothing to join from

[10:47:48 CST(-0600)] <EricDalquist> it would be the portal layout node

[10:47:54 CST(-0600)] <EricDalquist> but that isn't using JPA yet

[10:48:03 CST(-0600)] <EricDalquist> that is manual SQL in RDBMDistributedLayoutStore

[10:48:15 CST(-0600)] <Arvids> ok

[10:48:32 CST(-0600)] <EricDalquist> the other thing that could help performance would be batching the load

[10:48:47 CST(-0600)] <EricDalquist> but that isn't how the portal it architected

[10:48:58 CST(-0600)] <EricDalquist> it is very "on demand" for loading this stuff

[10:49:07 CST(-0600)] <Arvids> Today i managed to cut down some more miliseconds upon login, but those batches of queries concern me

[10:49:09 CST(-0600)] <EricDalquist> "I need the entity or user X and layout node Y"

[10:49:41 CST(-0600)] <EricDalquist> there are some thinks I'm working on with the hibernate folks that would be a huge help to our performance

[10:49:49 CST(-0600)] <EricDalquist> but that won't be avaialble in hibernate until 4.1

[10:50:10 CST(-0600)] <EricDalquist> https://hibernate.onjira.com/browse/HHH-2879

[10:50:17 CST(-0600)] <Arvids> btw, switching from 3.x to 4.x (hibernate) is painful?

[10:50:36 CST(-0600)] <EricDalquist> I don't think it will be without work

[10:50:46 CST(-0600)] <EricDalquist> but I'd be surprised if it is more than 1-2 days of work

[10:50:52 CST(-0600)] <EricDalquist> can't do that until uP 4.1 though

[10:51:18 CST(-0600)] <EricDalquist> if HHH-2879 (which I've partially completed: https://github.com/edalquist/hibernate-core/compare/master...HHH-2879-nocaching) gets done

[10:51:31 CST(-0600)] <EricDalquist> we could do a single "load all portlet entities for user X" on user login

[10:51:46 CST(-0600)] <Arvids> that would be really nice

[10:51:54 CST(-0600)] <EricDalquist> all the entities would have the naturalId cache populated with (userId, nodeId)->entityId

[10:52:13 CST(-0600)] <EricDalquist> and then all the queries by userId and nodeId wouldn't need to hit the DB at all

[10:52:49 CST(-0600)] <EricDalquist> yes

[10:52:56 CST(-0600)] <Arvids> besides those query batches, i see no real big places where to improve performance

[10:53:17 CST(-0600)] <EricDalquist> I'm sure there are others

[10:53:28 CST(-0600)] <EricDalquist> but they are likely burried in the really old user/layout/auth code

[10:53:39 CST(-0600)] <EricDalquist> there is some poor use of synchronization in there

[10:54:01 CST(-0600)] <Arvids> i´m not so sure about that

[10:54:18 CST(-0600)] <EricDalquist> well ... for creating new users there

[10:54:18 CST(-0600)] <Arvids> ... at least in terms of performance there´s no much places where to look

[10:54:19 CST(-0600)] <EricDalquist> is

[10:54:30 CST(-0600)] <EricDalquist> so the first time user X ever logs in

[10:54:35 CST(-0600)] <Arvids> ahh... that´s out of the scope of my research

[10:54:41 CST(-0600)] <EricDalquist> there is a bunch of stuff done that is sub-optimal

[10:55:02 CST(-0600)] <EricDalquist> the portlet entity lookups do suck though

[10:55:02 CST(-0600)] <Arvids> well... that´s the first time and it is bearable

[10:58:50 CST(-0600)] <EricDalquist> another thing we talked about at the last conference is a need to re-write person directory

[10:58:55 CST(-0600)] <EricDalquist> build a real query API

[10:58:59 CST(-0600)] <EricDalquist> simplify config

[10:59:12 CST(-0600)] <EricDalquist> and most importantly add the ability to query attribute sources concurrently

[11:00:03 CST(-0600)] <Arvids> That could cut only something like 20-60ms

[11:00:15 CST(-0600)] <EricDalquist> well ... that depends on your complexity

[11:00:33 CST(-0600)] <EricDalquist> we query between 4 and 10 attribute sources at login

[11:00:39 CST(-0600)] <EricDalquist> depending on who you are

[11:00:50 CST(-0600)] <Arvids> yeah, my config is really simple - only LDAP and local DB

[11:01:35 CST(-0600)] <EricDalquist> if the portlet entity lookup is REALLY the biggest issue you could fake the naturalid lookup logic locally

[11:01:52 CST(-0600)] <EricDalquist> inject a new ehcache instance into portlet entity registry

[11:02:01 CST(-0600)] <Arvids> One place, where some improvements can be done, is when XML documents are being merged into final layout

[11:02:40 CST(-0600)] <EricDalquist> on user login run a query to load all entities for user in a single query

[11:02:54 CST(-0600)] <Arvids> hrmm... sounds like a good idea

[11:02:57 CST(-0600)] <EricDalquist> store (userId, layoutId, nodeId) -> entityId in the cache

[11:03:23 CST(-0600)] <EricDalquist> then in the entity registry modify the method that queries by (userId, layoutId, nodeId) to look in that cache for the entityId first

[11:03:35 CST(-0600)] <EricDalquist> if that exists load the entity by id (that goes directly against the 2nd level cache)

[11:03:55 CST(-0600)] <EricDalquist> if not fall back to loading by user/layout/node

[11:04:03 CST(-0600)] <EricDalquist> you could inject 2 caches

[11:04:19 CST(-0600)] <EricDalquist> and have the 2nd cache misses or some such

[11:04:51 CST(-0600)] <Arvids> i believe, the same should be done for retrieving stylesheet user preferences

[11:05:19 CST(-0600)] <EricDalquist> yeah

[11:05:30 CST(-0600)] <EricDalquist> again, once we get the layouts into a JPA data model

[11:05:49 CST(-0600)] <EricDalquist> both the node specific stylesheet preferences and portlet entities can have real relations

[11:06:09 CST(-0600)] <EricDalquist> having the user managed by jpa will do the same

[11:06:21 CST(-0600)] <EricDalquist> we can have the user object have bag references to the other types

[11:06:25 CST(-0600)] <EricDalquist> and load them all when the user logs in

[11:06:35 CST(-0600)] <Arvids> ... and one more hotspot - XPath evaluation

[11:07:06 CST(-0600)] <Arvids> I inserted some hand-written code in order to evaluate XPath expression evaluation performance

[11:07:35 CST(-0600)] <Arvids> for example: Evaluated expression '/layout/folder/folder[@ID=$nodeId or descendant::node()[@ID=$nodeId]]/@ID' in 42ms

[11:08:21 CST(-0600)] <EricDalquist> wow

[11:08:24 CST(-0600)] <EricDalquist> just evaluation?

[11:08:26 CST(-0600)] <Arvids> could it be because of lazy loading some entities when visiting nodes... or evaluator is slow?

[11:08:47 CST(-0600)] <EricDalquist> I believe that is running against a fully loaded XML document

[11:09:04 CST(-0600)] <EricDalquist> that's crazy

[11:09:06 CST(-0600)] <Arvids> hmm... then exclude lazy-loading as a cause

[11:09:10 CST(-0600)] <EricDalquist> these aren't huge documents

[11:09:15 CST(-0600)] <Arvids> indeed

[11:09:37 CST(-0600)] <EricDalquist> I mean it does potentially have to do a depth-first search of the tree

[11:09:43 CST(-0600)] <Arvids> but that one expression takes between 1 and 60 ms

[11:09:46 CST(-0600)] <EricDalquist> but with < 100 nodes that should be fast

[11:09:58 CST(-0600)] <EricDalquist> I wonder if there is just something about that xpath that is bad

[11:10:28 CST(-0600)] <Arvids> that´s one of the surprising things - here´s the same expression, but very different evaluation time: Evaluated expression '/layout/folder/folder[@ID=$nodeId or descendant::node()[@ID=$nodeId]]/@ID' in 1ms

[11:11:20 CST(-0600)] <EricDalquist> and that is only timing the call to evaluate()?

[11:11:28 CST(-0600)] <Arvids> yes

[11:12:00 CST(-0600)] <Arvids> i modified a utility class

[11:12:16 CST(-0600)] <Arvids> (don´t remember the exact name)

[11:12:39 CST(-0600)] <Arvids> lol... it´s right in front of me: XPathPoolImpl

[11:12:45 CST(-0600)] <EricDalquist> right

[11:12:58 CST(-0600)] <EricDalquist> the goal of that thing was to try and make xpath faster (tongue)

[11:13:11 CST(-0600)] <EricDalquist> since the JDK doesn't allow thread-safe reuse of compiled expressions

[11:13:15 CST(-0600)] <EricDalquist> which is dumb

[11:13:27 CST(-0600)] <EricDalquist> and unfortunatly none of the 3rd party java XML apis have kept up

[11:13:41 CST(-0600)] <EricDalquist> so they all cause problems if you use jaxp like we do (tongue)

[11:14:14 CST(-0600)] <Arvids> yeah, such extensive use of XML requires really high quality libraries

[11:14:59 CST(-0600)] <Arvids> i was wondering - whether handwritten method that mimics the XPath expression could be faster

[11:15:04 CST(-0600)] <EricDalquist> it could be

[11:15:23 CST(-0600)] <EricDalquist> perhaps try using xpath to find the node with @ID=$nodeId

[11:15:37 CST(-0600)] <EricDalquist> then look up from there to find the tab folder

[11:15:56 CST(-0600)] <EricDalquist> in theory we could have @ID treated by the XML APIs as the elementID

[11:16:06 CST(-0600)] <EricDalquist> but I think we have to associate a XSD with the layout XML to do that

[11:17:31 CST(-0600)] <Arvids> can you explain in some word what "descendant::node()[@ID=$nodeId]" expression means?

[11:18:04 CST(-0600)] <Arvids> because elarier today i was on my way to implement the hand-written search, but couldn´t figure out the last part of that expression

[11:18:21 CST(-0600)] <EricDalquist> find the decended of the current node with attribute ID equals to the $nodeId variable

[11:18:48 CST(-0600)] <EricDalquist> the goal of that method is that the portal knows what the @ID of the layout node targeted by the request is

[11:18:59 CST(-0600)] <EricDalquist> and we need to determine which tab is the parent of that node

[11:19:34 CST(-0600)] <Arvids> so basically "folder[@ID=$nodeId or descendant::node()[@ID=$nodeId]]" means find a node or its subnode whose ID is with value $nodeId

[11:19:44 CST(-0600)] <EricDalquist> yes

[11:19:49 CST(-0600)] <Arvids> got it

[11:20:26 CST(-0600)] <EricDalquist> and it starts with /layout/folder/folder since for the tab/column layout the 2nd level of folder elements are the tabs

[11:20:48 CST(-0600)] <Arvids> yeah, that part was clear for me

[11:21:22 CST(-0600)] <Arvids> another example: Evaluated expression '//channel[@fname=$fname]' in 8ms

[11:21:42 CST(-0600)] <EricDalquist> so // means any channel element in the document

[11:21:51 CST(-0600)] <Arvids> 8ms isn´t too much, but it seems that sums up

[11:22:03 CST(-0600)] <Arvids> ... and makes around 30ms

[11:22:28 CST(-0600)] <EricDalquist> yeah that might be easy enough to fix with a local cache

[11:22:45 CST(-0600)] <EricDalquist> since it appears to be run from DistributedLayoutManager.getSubscribeId

[11:23:22 CST(-0600)] <EricDalquist> like move that logic down into DistributedUserLayout

[11:23:32 CST(-0600)] <EricDalquist> and cache the fname->id in a Map

[11:23:40 CST(-0600)] <EricDalquist> which is invalidated when the layout changes

[11:24:27 CST(-0600)] <Arvids> ok, that will be sufficient for today

[11:24:38 CST(-0600)] <Arvids> thre´ll be a lot of work tomorrow (smile)

[11:25:43 CST(-0600)] <EricDalquist> (smile)

[11:25:48 CST(-0600)] <Arvids> still... now I understand why performance engineers are so highly appraised

[11:26:03 CST(-0600)] <EricDalquist> (smile)

[11:26:10 CST(-0600)] <EricDalquist> well have fun with yourkit as well

[11:26:19 CST(-0600)] <Arvids> yeah, have to get used to it

[11:26:26 CST(-0600)] <EricDalquist> it should give you a lot more insight into what is going on

[11:26:52 CST(-0600)] <Arvids> ok, have to go now

[11:38:49 CST(-0600)] <EricDalquist> blarg

[11:39:08 CST(-0600)] <EricDalquist> I forgot about having to setup a group mapping table for event aggregation

[11:39:16 CST(-0600)] <EricDalquist> and we'll have to do the same for portlet definitions

[11:43:43 CST(-0600)] <athena> yeah

[11:44:21 CST(-0600)] <EricDalquist> so my plan is that during aggregation I'll take a group key (local.32) lookup the group name

[11:44:50 CST(-0600)] <EricDalquist> and the aggregate events db will have a table which maps groupstore+groupname to a synthetic id

[11:44:55 CST(-0600)] <EricDalquist> which will be used when aggregating

[11:45:10 CST(-0600)] <EricDalquist> so that the aggregates are not dependent on the actual group-keys over time

[11:49:48 CST(-0600)] <EricDalquist> athena: would it be the end of the world to have a flat-file for configuring quarters and terms for now?

[11:50:06 CST(-0600)] <EricDalquist> I'm trying to get this stats stuff buttoned up asap

[11:50:19 CST(-0600)] <EricDalquist> and doing those in the db is going to add an extra day or two

[11:51:55 CST(-0600)] <athena> i think that is not hte end of the world

[11:52:02 CST(-0600)] <athena> having db-based seems better in the long run

[11:52:08 CST(-0600)] <athena> but like, this is way better than nothing (smile)

[11:52:21 CST(-0600)] <athena> and most schools know their semesters for years in advance, really

[11:52:40 CST(-0600)] <EricDalquist> yeah

[12:03:54 CST(-0600)] <holdorph> speaking from experience with sakai

[12:04:05 CST(-0600)] <holdorph> it's a bad idea to do that either way

[12:04:18 CST(-0600)] <holdorph> it's always something gets forgotten until it's broken

[12:04:31 CST(-0600)] <holdorph> and while broken certainly reminds you something has to get done

[12:04:36 CST(-0600)] <EricDalquist> to do what either way?

[12:04:39 CST(-0600)] <holdorph> it doesn't change the fact that broken production sucks

[12:04:54 CST(-0600)] <holdorph> requiring semesters to have to be configured by schools

[12:05:07 CST(-0600)] <EricDalquist> ah, it doesn't require it

[12:05:15 CST(-0600)] <EricDalquist> you just don't get term based stats unless you configure it

[12:05:18 CST(-0600)] <holdorph> it's a requirement in sakai or the sakai course site type doesn't work

[12:05:19 CST(-0600)] <EricDalquist> everything else works

[12:06:01 CST(-0600)] <holdorph> ok, but the point is, they won't ever get around to configuring the file until they decide, hmmm.... term based stats aren't working.

[12:06:13 CST(-0600)] <EricDalquist> right

[12:06:19 CST(-0600)] <holdorph> at which point a file is, eh, ok, but if they're looking at the stats in the UI

[12:06:44 CST(-0600)] <holdorph> and they decide they'd like term based stats.... configuring them in the same UI would be better.

[12:07:04 CST(-0600)] <EricDalquist> yeah

[12:07:06 CST(-0600)] <holdorph> but if there's absolutely no UI for the configuration at all, ever... then there's not much difference

[12:07:24 CST(-0600)] <holdorph> but rest assured, it's something that will be a source of 'support' on the mailing lists

[12:07:37 CST(-0600)] <holdorph> it comes up in sakai on a very very very regular basis

[12:08:02 CST(-0600)] <EricDalquist> (smile)

[12:08:11 CST(-0600)] <EricDalquist> well and the fun answer with the stats

[12:08:31 CST(-0600)] <EricDalquist> is that you will only be able to get term based stats data from when you configure terms going forward

[12:08:46 CST(-0600)] <EricDalquist> there isn't really the ability to re-run old data

[12:08:56 CST(-0600)] <EricDalquist> since the raw data gets purged

[12:10:23 CST(-0600)] <EricDalquist> ok, lunch time

[12:23:23 CST(-0600)] <athena> holdorph: the original question was whether we could live with file-based for now so that we can get everything into the trunk and start collaborating on user interfaces

[12:23:35 CST(-0600)] <athena> the long-term intention is still to have database configuration

[12:33:42 CST(-0600)] <holdorph> i understand, i was trying to speak up on the topic of academic term configuration

[12:33:53 CST(-0600)] <holdorph> while it was still in the beginnings of development

[12:34:12 CST(-0600)] <holdorph> to relate a real problem in sakai and hope that we don't recreate the same kind of support problem

[15:20:10 CST(-0600)] <pandisaurus> Is there a best practice to update an 'iFrame' channel for uPortal 4?