uPortal IRC Logs-2012-01-13

[09:52:07 CST(-0600)] <EricDalquist> looks like we might be getting some more QA resources for uP4

[10:16:01 CST(-0600)] <EricDalquist> athena: you there?

[10:26:17 CST(-0600)] <Arvids> EricDalquist, glad you´re here

[10:26:24 CST(-0600)] <EricDalquist> yeah, was out sick yesterday

[10:27:15 CST(-0600)] <Arvids> I rewrote that tab ID lookup - XPath did lookups in 1-50ms, but manual traversing does the same thing in 0.05-0.15ms

[10:27:37 CST(-0600)] <EricDalquist> wow nice

[10:27:44 CST(-0600)] <EricDalquist> +1 for that change

[10:27:45 CST(-0600)] <Arvids> that speeds up response times by 30-40%

[10:28:08 CST(-0600)] * EricDalquist cannot wait to have time to just re-write the layout manager/store layer

[10:28:25 CST(-0600)] <EricDalquist> xslt is find for the rendering pipeline but it would be nice to not treat the actual layouts as DOMs

[10:28:26 CST(-0600)] <Arvids> it turns out that those lookups are made at each URL building request

[10:28:38 CST(-0600)] <EricDalquist> ah ... yeah they would be

[10:28:56 CST(-0600)] <EricDalquist> one thing I've always wanted to write but never had time would be a @RequestCache annotation

[10:29:00 CST(-0600)] <EricDalquist> and supporting code

[10:29:07 CST(-0600)] <Arvids> btw, i can give you a patch, but I won´t have time to commit it today myself

[10:29:14 CST(-0600)] <EricDalquist> that will work

[10:29:28 CST(-0600)] <EricDalquist> being able to annotated a method with @RequestCache and have the method result be cached in a request attribute

[10:30:44 CST(-0600)] <Arvids> https://gist.github.com/1607350

[10:31:42 CST(-0600)] <EricDalquist> so have you tested this on mobile?

[10:31:54 CST(-0600)] <Arvids> no

[10:32:10 CST(-0600)] <EricDalquist> the way things work the UrlNodeSyntaxHelper impl is associated with the structure transform

[10:32:18 CST(-0600)] <EricDalquist> so the logic to find the correct tab needs to be in there

[10:32:57 CST(-0600)] <EricDalquist> since a different structure, like mobile, doesn't have the same url structure as the tab/column structure

[10:34:16 CST(-0600)] <Arvids> ok, i hope i will have some time to inspect that next week

[10:34:31 CST(-0600)] <EricDalquist> well at least we know that it is a problem

[10:35:03 CST(-0600)] <Arvids> yeah, i also did some search on google.. and it gave very little results, but there were some hints

[10:35:12 CST(-0600)] <Arvids> one is: https://issues.apache.org/jira/browse/XALANJ-2540

[10:36:13 CST(-0600)] <Arvids> http://stackoverflow.com/questions/6340802/java-xpath-apache-jaxp-implementation-performance

[10:36:43 CST(-0600)] <Arvids> the one question from stackoverflow is supporting my conclusions - XPath is slow

[10:36:47 CST(-0600)] <EricDalquist> uhg

[10:36:49 CST(-0600)] <EricDalquist> that sucks

[10:37:01 CST(-0600)] <Arvids> yes, it does

[10:37:39 CST(-0600)] <Arvids> it is very nice to write some delarative code that looks up the stuff you want... but the performance penalty is too great

[10:37:50 CST(-0600)] <Arvids> declarative*

[10:37:54 CST(-0600)] <EricDalquist> yeah

[10:38:06 CST(-0600)] <EricDalquist> the sad thing is it shouldn't be that expensive

[10:38:17 CST(-0600)] <EricDalquist> and there are alternative xpath impls for java that I'm betting are fast

[10:38:30 CST(-0600)] <EricDalquist> but last I looked jdom and dom4j are both essentially dead

[10:39:43 CST(-0600)] <Arvids> I also saw some comments saying that VTD-XML is fast, but last release was almost year ago

[10:39:50 CST(-0600)] <EricDalquist> yeah

[10:39:58 CST(-0600)] <EricDalquist> jdom looks like it just had a patch release

[10:40:19 CST(-0600)] <EricDalquist> and dom4j had its last release in april of 2010

[10:41:47 CST(-0600)] <EricDalquist> though due to cernunnous we have jdom 1.0 and dom4j 1.6.1 on the classpath

[10:42:20 CST(-0600)] * EricDalquist gets frustraited by having so many libraries that duplicate functionality in the classpath :/

[10:43:50 CST(-0600)] <Arvids> btw, do you have some ideas regarding new layout architecture?

[10:44:02 CST(-0600)] <EricDalquist> I do have some

[10:44:11 CST(-0600)] <Arvids> will it be XML-based or POJOs?

[10:44:20 CST(-0600)] <EricDalquist> pojo at the data level

[10:44:31 CST(-0600)] <EricDalquist> so create a pojo tree that stores the layout stucture

[10:44:46 CST(-0600)] <EricDalquist> layout nodes can then have true references to IPortletDefinition objects

[10:44:57 CST(-0600)] <EricDalquist> and the IPortletEntity object can be merged into the layout node

[10:45:51 CST(-0600)] <EricDalquist> the pojo structure will need to capture the different types of nodes that can exist in a DLM style

[10:46:17 CST(-0600)] <EricDalquist> folder, portlet, dlm:move, dlm:add, dlm:delete, dlm:edit

[10:46:28 CST(-0600)] <Arvids> btw, I also did some tinkering on JpaPortletEntityDao

[10:46:31 CST(-0600)] <Arvids> here´s diff: https://gist.github.com/1607439

[10:46:58 CST(-0600)] <EricDalquist> part of the goal with moving this to a pojo structure is to be able to more reasonably generate a "state key" for each layout

[10:47:00 CST(-0600)] <Arvids> it turns out that manual loop through user entities fastens the whole thing

[10:47:42 CST(-0600)] <EricDalquist> then the DLM merging logic would need to be re-written from XML based to work with the pojo setup

[10:48:22 CST(-0600)] <EricDalquist> though with the way the rendering pipeline works the DLM merging logic would hopefully become just a rendering pipeline component

[10:48:28 CST(-0600)] <Arvids> ok, i have to go now (it´s friday evening after all)

[10:48:47 CST(-0600)] <EricDalquist> heh

[10:49:05 CST(-0600)] <EricDalquist> I'll see if I can document some of these layout manager re-write ideas

[10:49:13 CST(-0600)] <Arvids> that would be nice

[10:49:30 CST(-0600)] <Arvids> we should somehow address XPath problem, too

[10:49:40 CST(-0600)] <Arvids> manual traversing is painful

[10:49:58 CST(-0600)] <EricDalquist> yeah

[10:50:53 CST(-0600)] <Arvids> ok, bye now... hope you´ll get some pull requests on monday

[10:57:24 CST(-0600)] <athena> hey EricDalquist, what's up?

[10:57:42 CST(-0600)] <EricDalquist> just a few questions about the spring-sec stuff you're working on

[10:57:49 CST(-0600)] <EricDalquist> in relation to some events stuff I'm working on

[10:59:03 CST(-0600)] <EricDalquist> with spring-sec will users "authenticated" on every new session?

[10:59:07 CST(-0600)] <EricDalquist> including guest users?

[11:00:40 CST(-0600)] <athena> yeah but i think they wind up w/ an unauthenticated session

[11:00:44 CST(-0600)] <athena> using the anonymous auth stuff

[11:01:01 CST(-0600)] <EricDalquist> right

[11:01:14 CST(-0600)] <EricDalquist> so in the aggregation we need to track "event sessions"

[11:01:24 CST(-0600)] <EricDalquist> which contains the groups that the user the event was fired for is in

[11:01:45 CST(-0600)] <EricDalquist> what I would like is that a LoginEvent (might need to rename that) is fired for ALL users unauthed and authed

[11:02:02 CST(-0600)] <EricDalquist> just wondering if that was going to fit with your spring-sec work

[11:03:36 CST(-0600)] <athena> yeah i think that sounds doable

[11:03:45 CST(-0600)] <athena> i think there might already be some eventing in there we can hook inot

[11:04:21 CST(-0600)] <EricDalquist> great

[11:19:02 CST(-0600)] <athena> here we go: http://static.springsource.org/spring-security/site/docs/3.0.x/apidocs/index.html

[11:19:13 CST(-0600)] <athena> so actually i think i got everything fixed up enough yesterday to create a branch

[11:19:31 CST(-0600)] <EricDalquist> oh neat

[11:19:51 CST(-0600)] <athena> yeah

[11:19:59 CST(-0600)] <athena> have CAS login working

[11:20:09 CST(-0600)] <athena> and local login, though that's only with spring's default login page

[11:20:14 CST(-0600)] <athena> so outstanding stuff includes

[11:20:23 CST(-0600)] <athena> 1. login portlet that can work w/ the local login stuff

[11:20:31 CST(-0600)] <athena> 2. recreating all the impersonation stuff

[11:20:37 CST(-0600)] <athena> 3. CAS clearpass

[11:20:48 CST(-0600)] <athena> 4. Proxy CAS and making sure CAS tickets get into the userinfomap

[11:20:53 CST(-0600)] <athena> 5. getting passwords into the userinfo map

[11:20:58 CST(-0600)] <athena> and probably some other stuff

[11:21:07 CST(-0600)] <athena> so certainly not done, but maybe working enough to be interesting

[11:22:30 CST(-0600)] <EricDalquist> very interesting

[11:22:37 CST(-0600)] <athena> yeah

[11:22:40 CST(-0600)] <athena> i mean, it does work

[11:22:43 CST(-0600)] <athena> reasonably clean

[11:22:52 CST(-0600)] <athena> not thrilled about the state of their documentation or their autowiring business

[11:23:01 CST(-0600)] <EricDalquist> :/

[11:23:05 CST(-0600)] <athena> and not thrilled that it seems the portal just silently fails to start up when things are misconfigured

[11:23:09 CST(-0600)] <athena> don't know what's going on there

[11:24:37 CST(-0600)] <EricDalquist> :/

[11:24:39 CST(-0600)] <EricDalquist> that is no fun

[11:24:41 CST(-0600)] <athena> yeah

[11:25:13 CST(-0600)] <athena> seems like what crashes it is usually a mismatch between the expected bean properties and what's misconfigured in the spring file

[11:25:22 CST(-0600)] <athena> but don't understand why that doesn't seem to produce more stack traces

[11:25:37 CST(-0600)] <EricDalquist> yeah

[12:00:10 CST(-0600)] <EricDalquist> so just discovered this: http://eggsylife.co.uk/2010/02/03/spring-annotation-based-aop-and-intercepting-the-ball/

[12:00:18 CST(-0600)] <EricDalquist> how to write a custom annotation

[12:00:25 CST(-0600)] <EricDalquist> and associate an interceptor with it

[12:00:30 CST(-0600)] <EricDalquist> in ~ 30 minutes

[12:57:17 CST(-0600)] <athena> ooo

[12:57:25 CST(-0600)] <athena> that sounds aawesome

[12:57:38 CST(-0600)] <EricDalquist> just finished uit

[12:57:51 CST(-0600)] <EricDalquist> there will be a @RequestCache annotation

[12:58:03 CST(-0600)] <EricDalquist> methods annotated with it will cache their result in the attributes of the current request

[12:58:16 CST(-0600)] <EricDalquist> and automatically re-use the cached result for the duration of the request

[12:58:35 CST(-0600)] <EricDalquist> Arvids had noticed that thigns like looking up the default tab nodeid ran for every url generation'

[12:58:44 CST(-0600)] <EricDalquist> even though that is the same for every urkl in the reuqest

[14:03:58 CST(-0600)] <EricDalquist> https://github.com/Jasig/uPortal/commit/68051fb9c9aa683f32d00a0fc8ac97fd900e8faa

[14:04:01 CST(-0600)] <EricDalquist> there we go

[14:05:33 CST(-0600)] <athena> oh nice

[14:05:41 CST(-0600)] <athena> is that the performance stuff he wanted to talk about before?

[14:05:54 CST(-0600)] <EricDalquist> well it is part of addressing it

[14:06:03 CST(-0600)] <EricDalquist> apparently xpath in the core JDK apis is terribly slow

[14:06:22 CST(-0600)] <athena> ah, interesting

[14:06:29 CST(-0600)] <EricDalquist> like 5-60ms to get the tabid for a random layout node id

[14:06:38 CST(-0600)] <EricDalquist> and that would run every time a url was generated

[14:06:41 CST(-0600)] <athena> oh wow

[14:06:47 CST(-0600)] <athena> yeah i guess that'd make a difference

[14:07:19 CST(-0600)] <EricDalquist> he got a huge boost by just navigating the DOM manually

[14:07:25 CST(-0600)] <EricDalquist> his patch just did it in the wrong place

[14:07:43 CST(-0600)] <EricDalquist> the other patch that helped a lot was not looking up portlet entities directly by user+layout+node

[14:07:51 CST(-0600)] <EricDalquist> instead getting all entities for user+layout

[14:07:59 CST(-0600)] <EricDalquist> then iterate on them comparing the nodes

[14:08:04 CST(-0600)] <athena> sounds like really good stuff

[14:08:06 CST(-0600)] <EricDalquist> yeah

[14:08:16 CST(-0600)] <athena> really glad he's involved

[14:08:21 CST(-0600)] <EricDalquist> a lot of this stuff is a result of only being part way to hibernate/jpa

[14:08:30 CST(-0600)] <athena> ah, yeah

[14:08:39 CST(-0600)] <EricDalquist> if everything was actually part of our JPA model we could essentially load it all when the user logs in

[14:08:49 CST(-0600)] <EricDalquist> or load it in reasonable chunks

[14:09:01 CST(-0600)] <EricDalquist> getting users + layouts moved over will be huge for reducing DB io

[14:09:05 CST(-0600)] <athena> yeah

[14:09:09 CST(-0600)] <EricDalquist> since then we can really have one object graph per user

[14:09:11 CST(-0600)] <athena> would love to get that work done

[14:09:18 CST(-0600)] <athena> maybe after stats?

[14:09:20 CST(-0600)] <EricDalquist> and load the whole graph at login with a minimal number of calls

[14:09:21 CST(-0600)] <EricDalquist> yeah

[14:09:24 CST(-0600)] <EricDalquist> after stats

[14:09:28 CST(-0600)] <athena> how's the stats stuff going, by the way?

[14:09:32 CST(-0600)] <EricDalquist> and after I get 4.0 deployed here

[14:09:33 CST(-0600)] <EricDalquist> good

[14:09:40 CST(-0600)] <EricDalquist> the branch is making good progress

[14:09:48 CST(-0600)] <EricDalquist> I just keep realizing more gotchas

[14:09:50 CST(-0600)] <athena> lol

[14:09:55 CST(-0600)] <EricDalquist> I had forgotten how complex that aggregator was

[14:10:01 CST(-0600)] <EricDalquist> its translating over just fine

[14:10:11 CST(-0600)] <EricDalquist> it is just A LOT of daos/entities

[14:10:23 CST(-0600)] <athena> yeahi 'm sure

[14:12:21 CST(-0600)] <EricDalquist> here is the work so far: https://github.com/edalquist/uPortal/compare/master...event-aggr

[14:14:16 CST(-0600)] <EricDalquist> most important stuff that you'll be interested in is:

[14:14:16 CST(-0600)] <EricDalquist> https://github.com/edalquist/uPortal/compare/master...event-aggr#diff-44

[14:14:16 CST(-0600)] <EricDalquist> https://github.com/edalquist/uPortal/compare/master...event-aggr#diff-42

[14:20:46 CST(-0600)] <athena> awesome

[14:21:05 CST(-0600)] <athena> i'll take a look

[14:21:57 CST(-0600)] <EricDalquist> the api is still going to change a little bit

[14:22:03 CST(-0600)] <EricDalquist> that is what I just realized

[14:22:07 CST(-0600)] <EricDalquist> but its getting closer

[14:22:39 CST(-0600)] <EricDalquist> the other thing I need to do (in addition to the term/quarter config) is allow configuration of what groups and intervals each aggregator sees

[14:22:57 CST(-0600)] <EricDalquist> I have a feeling I'm just going to have to write an even aggregation admin flow :/

[14:25:46 CST(-0600)] <athena> ahhh

[14:25:47 CST(-0600)] <athena> hmm.

[14:25:52 CST(-0600)] <athena> yeah