uPortal IRC Logs-2012-01-04

Owned by Apereo Infrastructure
Last updated: Jan 04, 2012Version comment
6 min read

[11:42:08 CST(-0600)] <athena> morning EricDalquist

[11:42:13 CST(-0600)] <EricDalquist> morning athena

[11:42:33 CST(-0600)] <athena> so i'm thinking about creating an experimental branch for spring-security w/ uportal

[11:42:39 CST(-0600)] <athena> specifically converting the authentication stuff over

[11:42:39 CST(-0600)] <EricDalquist> sounds great

[11:43:28 CST(-0600)] <athena> cool

[11:43:45 CST(-0600)] <athena> i'm guessing we'd need to wait until 4.1 to include that in a release?

[11:43:47 CST(-0600)] <EricDalquist> I have the date and time dimension population code done

[11:43:48 CST(-0600)] <EricDalquist> yeah

[11:43:58 CST(-0600)] <athena> wouldn't really affect any portlets, but would require changes to the auth configuration for some people

[11:43:59 CST(-0600)] <EricDalquist> though we could cut a 4.1 as soon as it is ready

[11:44:05 CST(-0600)] <athena> hmm that's a thought

[11:44:05 CST(-0600)] <EricDalquist> if we wanted to

[11:44:21 CST(-0600)] <athena> well, i'll play with it and we can see how it goes

[11:44:27 CST(-0600)] <EricDalquist> that would be great

[11:44:31 CST(-0600)] <athena> cool (smile)

[11:45:12 CST(-0600)] <athena> one nice thing would be it'll be easier to set up oauth if we want

[11:45:21 CST(-0600)] <EricDalquist> yes!

[11:45:24 CST(-0600)] <EricDalquist> that would be wonderful

[11:45:35 CST(-0600)] <EricDalquist> can we do uPortal managed oauth + some external auth like cas?

[11:48:15 CST(-0600)] <athena> i think we can support both oauth and cas, yes

[11:48:20 CST(-0600)] <athena> though cas itself doesn't support oauth

[11:49:33 CST(-0600)] <EricDalquist> right

[11:49:37 CST(-0600)] <EricDalquist> but if uPortal can manage it locally

[11:49:42 CST(-0600)] <EricDalquist> that is better than nothing

[11:49:50 CST(-0600)] <EricDalquist> and more reason to write a java plugin for shib (tongue)

[11:50:51 CST(-0600)] <Arvids> Hi, everyone

[11:51:08 CST(-0600)] <EricDalquist> hello Arvids

[11:51:09 CST(-0600)] <Arvids> haven't had a chance to talk to you lately

[11:51:14 CST(-0600)] <EricDalquist> yeah

[11:51:48 CST(-0600)] <Arvids> The interesting thing is that we've almost done with uP 4.0 setup in production environment

[11:52:07 CST(-0600)] <Arvids> but somehow I feel that it runs a bit slower than it should

[11:52:37 CST(-0600)] <Arvids> each response takes about 1-2seconds to process (measured on server side)

[11:52:55 CST(-0600)] <EricDalquist> hrm

[11:53:07 CST(-0600)] <EricDalquist> is this single user or under load?

[11:53:12 CST(-0600)] <Arvids> single user

[11:53:22 CST(-0600)] <Arvids> we're using MSSQL as DBMS

[11:53:29 CST(-0600)] <Arvids> and JTDS as driver

[11:54:02 CST(-0600)] <Arvids> my observations are that the slow par is in portlet registry when individual portlet definition is retrieved from protlet definition dao

[11:55:00 CST(-0600)] <Arvids> I hope that I'm not missing something

[11:55:06 CST(-0600)] <EricDalquist> how many portlet definitions do you have?

[11:55:22 CST(-0600)] <Arvids> around 100

[11:55:34 CST(-0600)] <EricDalquist> and one thing to look at would be the cache statistics, there is a portlet to view them and they can be seen via JMX

[11:55:58 CST(-0600)] <EricDalquist> do you have a specific code path that you suspect with the portlet registry?

[11:56:01 CST(-0600)] <Arvids> but still... around 300ms for portlet definition retrieval is too much

[11:56:07 CST(-0600)] <EricDalquist> right

[11:56:11 CST(-0600)] <EricDalquist> it should be VERY fast

[11:56:24 CST(-0600)] <EricDalquist> those portlet definitions should also only really ever be retrieved once

[11:56:26 CST(-0600)] <Arvids> that was my assumption, too

[11:56:27 CST(-0600)] <EricDalquist> and then stored in cache

[11:57:46 CST(-0600)] <EricDalquist> I'm wondering if there is some issue around MSSQL

[11:58:04 CST(-0600)] <EricDalquist> we had some other folks bring up an issue that I'm not sure ever got sorted out: http://permalink.gmane.org/gmane.comp.java.jasig.uportal/13830

[11:58:19 CST(-0600)] <EricDalquist> my guess is it is around how hibernate is dealing with CLOBs in MSSQL

[11:59:43 CST(-0600)] <Arvids> btw, I was using http://www.jvmmonitor.org/ on my local machine when I noticed that

[12:00:42 CST(-0600)] <EricDalquist> does that show method level detail?

[12:00:50 CST(-0600)] <Arvids> and it seems that it chokes each time when autoCommit is being set to false (at least i assume that it's setting false)

[12:00:57 CST(-0600)] <EricDalquist> like can you give me a list of methods that show bad response times?

[12:01:30 CST(-0600)] <Arvids> i will be able to do that tomorrow

[12:01:30 CST(-0600)] <EricDalquist> so one thought here ...

[12:01:47 CST(-0600)] <EricDalquist> PostgreSQL requires that CLOBs be read inside of a TX

[12:02:09 CST(-0600)] <EricDalquist> so all of the read methods on JpaPortletDefinitionDao and JpaPortletEntityDao are annotated with @Transactional(readOnly=true)

[12:02:15 CST(-0600)] <EricDalquist> you could try removing that annotation

[12:02:18 CST(-0600)] <EricDalquist> and see what happens

[12:02:28 CST(-0600)] <EricDalquist> it could be that the read-only TX is simply too expensive on MSSQL

[12:02:42 CST(-0600)] <EricDalquist> which means we would have to do a little work to only have that TX functionality for postgres

[12:02:44 CST(-0600)] <EricDalquist> which is doable

[12:02:46 CST(-0600)] <EricDalquist> just annoying (tongue)

[12:03:04 CST(-0600)] <Arvids> but the strange thing is that I also ran MSSQL profiler and each commit executes in 0-1ms

[12:03:37 CST(-0600)] <athena> the issue someone at unicon ran into yesterday w/ MSSQL was that the initial import for 4.0.2 failed with exceptions about connection already closed - think there was some stuff about autocommit in there, too

[12:03:57 CST(-0600)] <EricDalquist> yeah I would be very interested in what you get back from the Monitoring CPU usage feature of that tool

[12:04:17 CST(-0600)] <EricDalquist> I always enjoy (being serious here) looking at method call hotspot listings

[12:04:24 CST(-0600)] <Arvids> :O

[12:04:48 CST(-0600)] <athena> nothing like a nice glass of wine by the fire and a good thread dump . . .

[12:05:13 CST(-0600)] <EricDalquist> lol

[12:05:20 CST(-0600)] <Arvids> indeed (tongue)

[12:05:25 CST(-0600)] <EricDalquist> I do realize there is something wrong with me (tongue)

[12:06:16 CST(-0600)] <athena> lol

[12:06:24 CST(-0600)] <athena> we all have our weak spots (tongue)

[12:06:38 CST(-0600)] <EricDalquist> but it benefits uPortal, so don't look at me too funny now

[12:08:16 CST(-0600)] <athena> it totally does

[12:08:25 CST(-0600)] <athena> and we do all have our exceptionally nerdy spots (tongue)

[12:08:32 CST(-0600)] <Arvids> btw, have you had any chance to look at https://issues.jasig.org/browse/UP-3253

[12:08:50 CST(-0600)] <Arvids> this was the reason why we switched to MSSQL

[12:08:53 CST(-0600)] <EricDalquist> yes

[12:09:01 CST(-0600)] <EricDalquist> that will be fixed in 4.0.2

[12:09:01 CST(-0600)] <Arvids> (... and the presence of MSSQL admin)

[12:09:03 CST(-0600)] <EricDalquist> er 4.0.3

[12:09:10 CST(-0600)] <EricDalquist> it is fixed in master right now

[12:09:15 CST(-0600)] <Arvids> ah, nice

[12:09:21 CST(-0600)] <EricDalquist> I had to completely re-write how we do cluster based locking

[12:09:23 CST(-0600)] <EricDalquist> (tongue)

[12:09:35 CST(-0600)] <Arvids> sounds painful

[12:09:40 CST(-0600)] <EricDalquist> just annoying

[12:09:47 CST(-0600)] <EricDalquist> since the original solution took 2-3 days

[12:09:55 CST(-0600)] <EricDalquist> and then the new solution was another 2-3 days

[12:10:03 CST(-0600)] <EricDalquist> wish I had easy access to all the different DBs for testing

[12:10:10 CST(-0600)] <EricDalquist> that is what we need someone to do

[12:10:23 CST(-0600)] <EricDalquist> setup an EC2 instance with MySQL, Postgres, MSSQL and Oracle on it

[12:10:41 CST(-0600)] <EricDalquist> then we can spin that up 1hr/day and run the uPortal build/init on it

[12:11:21 CST(-0600)] <Arvids> well... I can't do that each day, but it seems that i'll provide you the details regarding MSSQL issues

[12:11:46 CST(-0600)] <EricDalquist> yes

[12:11:47 CST(-0600)] <athena> EricDalquist: is the stats stuff in master?

[12:11:50 CST(-0600)] <EricDalquist> that is nearly as helpful

[12:11:51 CST(-0600)] <EricDalquist> not yet

[12:11:51 CST(-0600)] <Arvids> now that reminds me that we had to remove one uniqueness index

[12:11:55 CST(-0600)] <EricDalquist> I should have it in master tomorrow

[12:12:00 CST(-0600)] <athena> ah ok, cool (wink)

[12:12:01 CST(-0600)] <EricDalquist> with at least one basic aggregation

[12:12:05 CST(-0600)] <athena> perfect

[12:12:06 CST(-0600)] <EricDalquist> https://github.com/edalquist/uPortal/commits/event-aggr

[12:12:10 CST(-0600)] <EricDalquist> that is where I'm working on it

[12:12:15 CST(-0600)] <athena> ah (smile)

[12:12:26 CST(-0600)] <EricDalquist> also ... I love having my own repo with my own branch

[12:12:34 CST(-0600)] <athena> yeah, git is nice!

[12:12:35 CST(-0600)] <EricDalquist> several of those commits are completely non-functional (tongue)

[12:12:51 CST(-0600)] <athena> lol

[12:13:29 CST(-0600)] <Arvids> next week i'll try to spend more time on uPortal platform

[12:13:44 CST(-0600)] <EricDalquist> that's great Arvids

[12:13:51 CST(-0600)] <Arvids> i've implemented a patch for https://issues.jasig.org/browse/UP-3270 but haven't had time to commit it back to our repository (sad)

[12:13:54 CST(-0600)] <EricDalquist> your involvement and interest have been really helpful

[12:13:57 CST(-0600)] <EricDalquist> we need more of you (tongue)

[12:13:57 CST(-0600)] <athena> +1!

[12:14:08 CST(-0600)] <EricDalquist> oh nice

[12:14:36 CST(-0600)] <Arvids> ... and that reminded that we have to be very careful when implementing new layout infrastructure

[12:15:00 CST(-0600)] <EricDalquist> yes

[12:15:09 CST(-0600)] <EricDalquist> we're going to have to do a lot of talking about requirements

[12:15:12 CST(-0600)] <EricDalquist> especially around i18n

[12:15:29 CST(-0600)] <EricDalquist> though simply getting layouts to use the same caching architecture as the rendering pipeline will help A LOT

[12:15:30 CST(-0600)] <Arvids> indeed - i'm also only learning about common issues in this area

[12:15:47 CST(-0600)] <EricDalquist> so that we are actually build the layout cache key to include ALL relavent bits of data

[12:15:55 CST(-0600)] <EricDalquist> so if any of them change we get a cache miss

[12:16:42 CST(-0600)] <Arvids> locale is certainly the one to remember

[12:16:46 CST(-0600)] <EricDalquist> yup

[12:17:36 CST(-0600)] <Arvids> ok, now time to relax (smile)

[12:17:52 CST(-0600)] <Arvids> thank you for your time - it was very valueable

[12:18:09 CST(-0600)] <EricDalquist> no problem

[12:18:15 CST(-0600)] <Arvids> i'll soon report my success regarding performance problems

[12:18:20 CST(-0600)] <EricDalquist> (smile)(]