uPortal IRC Logs-2010-03-22

Owned by Apereo Infrastructure
Last updated: Mar 22, 2010Version comment
8 min read

[08:51:58 EDT(-0400)] * athena (~athena@adsl-99-89-93-113.dsl.wlfrct.sbcglobal.net) has joined ##uportal
[09:05:21 EDT(-0400)] * jessm (~Jess@c-71-232-3-151.hsd1.ma.comcast.net) has joined ##uportal
[09:53:36 EDT(-0400)] * michelled (~michelled@142.150.154.101) has joined ##uportal
[09:56:06 EDT(-0400)] * michelled_ (~michelled@142.150.154.101) has joined ##uportal
[10:25:29 EDT(-0400)] * michelled (~michelled@142.150.154.101) has joined ##uportal
[10:40:30 EDT(-0400)] * EricDalquist (~apollo@76.210.65.83) has joined ##uportal
[10:57:59 EDT(-0400)] * lfuller (~sparhk@wsip-72-215-204-133.ph.ph.cox.net) has joined ##uportal
[11:03:35 EDT(-0400)] * holdorph (~holdorph@wsip-72-215-204-133.ph.ph.cox.net) has joined ##uportal
[11:07:31 EDT(-0400)] * michelled (~michelled@142.150.154.101) has joined ##uportal
[11:33:48 EDT(-0400)] * michelled (~michelled@142.150.154.101) has joined ##uportal
[12:00:00 EDT(-0400)] * awills (~awills@wsip-72-215-204-133.ph.ph.cox.net) has joined ##uportal
[12:45:35 EDT(-0400)] * Sememmon (~Sememmon@unaffiliated/sememmon) has joined ##uportal
[13:11:18 EDT(-0400)] * holdorph (~holdorph@wsip-72-215-204-133.ph.ph.cox.net) has joined ##uportal
[14:12:56 EDT(-0400)] <EricDalquist> any hibernate experts around?
[14:29:38 EDT(-0400)] * athena doesn't really volunteer
[14:29:58 EDT(-0400)] <athena> do you know if GroupsManagerXML gets used by anything other than the groups manager?
[14:30:02 EDT(-0400)] <athena> kind of scary
[14:31:30 EDT(-0400)] <EricDalquist> I don't know
[14:31:48 EDT(-0400)] <EricDalquist> are you working in trunk?
[14:32:15 EDT(-0400)] <athena> yeah
[14:32:36 EDT(-0400)] <athena> sounds like you and nick have made a lot of progress on the caching stuff?
[14:33:38 EDT(-0400)] <EricDalquist> well since channels are going away … "references in workspace" should be fine to figure out where that is used
[14:33:45 EDT(-0400)] <EricDalquist> yeah, we just need to find a host for the lib now
[14:34:02 EDT(-0400)] <EricDalquist> we have fully functional @Cacheable and @TriggersRemove annotations
[14:43:09 EDT(-0400)] <athena> that's awesome!
[14:43:20 EDT(-0400)] <athena> i've been really heads-down since the conference, but i saw the comments on twitter (smile)
[14:43:37 EDT(-0400)] <athena> i'd think that'd be pretty useful outside of uportal as well
[14:43:50 EDT(-0400)] <EricDalquist> yeah, I emailed the spring-extensions address
[14:43:53 EDT(-0400)] <EricDalquist> no idea if that is still alive
[14:44:00 EDT(-0400)] * michelled (~michelled@142.150.154.101) has joined ##uportal
[14:44:02 EDT(-0400)] <EricDalquist> we may just stick it on google code
[14:44:15 EDT(-0400)] <EricDalquist> it works very much like @Transactional for configuring it
[14:44:27 EDT(-0400)] <EricDalquist> also adds self-population and exception caching support
[14:44:44 EDT(-0400)] <EricDalquist> though right now I'm having fun dealing with over-zealous cache replication issues
[14:44:59 EDT(-0400)] <EricDalquist> apparently every time a portlet entity is persisted in uportal the parent definition has that cascaded to it
[14:45:14 EDT(-0400)] <EricDalquist> and even though hibernate realizes nothing changed and doesn't store anything in the db
[14:45:17 EDT(-0400)] <EricDalquist> the cache gets updated
[14:45:23 EDT(-0400)] <EricDalquist> which causes an invalidation message to go out
[14:45:37 EDT(-0400)] <EricDalquist> so its like we have no caching of portlet defintions
[15:07:54 EDT(-0400)] * michelled (~michelled@142.150.154.101) has joined ##uportal
[15:09:59 EDT(-0400)] * b-rock (~80876350@gateway/web/freenode/x-nddkbznyiaocxnji) has joined ##uportal
[15:11:43 EDT(-0400)] <b-rock> Greetings uPortal Devs: were wrapping up some paper work to submitt the grouper group service code to the Jasig Project for uPortal. Does anyone here know where that goes? or who to send it too?
[15:12:38 EDT(-0400)] * awills (~awills@wsip-72-215-204-133.ph.ph.cox.net) has left ##uportal
[15:12:55 EDT(-0400)] <holdorph> does anyone involved have uportal commit access?
[15:13:12 EDT(-0400)] <b-rock> there is probably a link on the jasig site for where to send I guess?...
[15:13:37 EDT(-0400)] <b-rock> no. no one I'm aware of here has commit access.
[15:13:58 EDT(-0400)] <holdorph> if no one has commit privilegles then attach the code as either an svn diff, or as a .zip, to a uportal jira issue.
[15:14:16 EDT(-0400)] <holdorph> then email the uportal list describing the code that was contributed.
[15:14:56 EDT(-0400)] <holdorph> you could also do a confluence page to document it as well, if that works better. but jira is typically where code is contributed by non-committers.
[15:15:09 EDT(-0400)] <b-rock> ok. I'll do that with the code and ping the users list but I have some actual paper work with my signature I need to turn in.
[15:15:26 EDT(-0400)] <b-rock> thanks holdorph
[15:15:28 EDT(-0400)] <holdorph> ah, the ICLA and CCLA ?
[15:15:34 EDT(-0400)] <b-rock> right.
[15:15:49 EDT(-0400)] <holdorph> yup, those need to get fax/sent to the Jasig organization.
[15:15:51 EDT(-0400)] <b-rock> there are some higher ups here that want to check off on that
[15:15:56 EDT(-0400)] <holdorph> nod
[15:16:37 EDT(-0400)] <b-rock> thanks though. it is clear to me now how to get the code into the right hands to get it integrated at some point.
[15:33:30 EDT(-0400)] <athena> b-rock: signed licensing agreements can be emailed to licensing@jasig.org or faxed to +1-303-495-3812
[15:33:38 EDT(-0400)] <athena> the information is here: http://www.ja-sig.org/wiki/display/LIC/Jasig+Licensing+Policy
[15:33:55 EDT(-0400)] <athena> glad to hear that's moving forward!
[15:36:16 EDT(-0400)] <b-rock> thanks athena. I did print copies of the coporate and individual contributors from there but I guess some more folks here need to see the documents befor I have the go ahead to submit it as a jira. but I think/hope it is going to be easy to include / integrate for a committer .
[15:36:48 EDT(-0400)] <athena> hope everything goes well!
[15:37:06 EDT(-0400)] <athena> i'm definitely looking forward to having some open-source grouper integration (smile)
[16:11:24 EDT(-0400)] * michelled (~michelled@142.150.154.101) has joined ##uportal
[17:26:31 EDT(-0400)] * lfuller (~sparhk@ip68-98-56-21.ph.ph.cox.net) has joined ##uportal
[18:59:30 EDT(-0400)] * EricDalquist (~apollo@76.210.65.83) has joined ##uportal
[18:59:52 EDT(-0400)] <EricDalquist> lfuller: you there?
[19:00:14 EDT(-0400)] <lfuller> no
[19:00:22 EDT(-0400)] <lfuller> what's up?
[19:00:27 EDT(-0400)] <EricDalquist> hibernate question
[19:00:40 EDT(-0400)] * lfuller tries to find his hibernate hat and goggles
[19:00:48 EDT(-0400)] <EricDalquist> we recently setup invalidation based clustering in ehcache
[19:00:56 EDT(-0400)] <lfuller> k
[19:01:30 EDT(-0400)] <lfuller> are you using terracotta + ehcache or the ehcache distributed features?
[19:01:36 EDT(-0400)] <EricDalquist> ehcache + jgroups
[19:01:44 EDT(-0400)] <EricDalquist> problem is it appears that some hibernate operations are causing some cascading invalidation
[19:02:18 EDT(-0400)] <EricDalquist> specifically creating a new PortletEntityImpl, which references a PortletDefinitionImpl, invalidates the PortletDefinitionImpl and its associated PortletPreferences
[19:02:26 EDT(-0400)] <EricDalquist> even though the PortletDefinitionImpl never changed
[19:02:47 EDT(-0400)] <EricDalquist> we ended up having to disable the clustering today
[19:03:00 EDT(-0400)] <lfuller> how have you set the ehcache relationships up?
[19:03:11 EDT(-0400)] <EricDalquist> we had a spike in load and realized that this was resulting in the portal never really caching the portlet definition level data
[19:03:19 EDT(-0400)] <EricDalquist> and was really putting the hurt on the DB
[19:03:38 EDT(-0400)] <EricDalquist> what do you mean by ehcache relationships?
[19:04:04 EDT(-0400)] <lfuller> typically in ehcache you would define a cache for PortletEntityImpl, a cache for the list of PortletDefinitionImpl associated with it, and a cache for PortletDefinitionImpl
[19:04:29 EDT(-0400)] <lfuller> make sense?
[19:04:31 EDT(-0400)] <EricDalquist> ah, yeah
[19:04:37 EDT(-0400)] <lfuller> you have that set up?
[19:04:37 EDT(-0400)] <EricDalquist> so in the object model
[19:04:48 EDT(-0400)] <EricDalquist> there is a PortletDefinition
[19:04:54 EDT(-0400)] <EricDalquist> then multiple PortletEntities based on that definition
[19:05:04 EDT(-0400)] <EricDalquist> each PortletEntity has a reference to its parent PortletDefinition
[19:05:12 EDT(-0400)] <EricDalquist> and there is a cache for both the definition and entity
[19:05:34 EDT(-0400)] <EricDalquist> the one thing I'm trying is that entity -> def reference had a bunch of cascade options enabled
[19:05:50 EDT(-0400)] <EricDalquist> even though it is really just a hidden reference there so that hibernate can enforce the referential integrity
[19:06:07 EDT(-0400)] <lfuller> was going to ask that next...
[19:06:10 EDT(-0400)] <EricDalquist> I'm wondering if the cascades could be causing this
[19:06:35 EDT(-0400)] <lfuller> I figured you had already checked on that... so saved that for an 'if all else fails' question.
[19:06:51 EDT(-0400)] <lfuller> can you email me your ehcache config file?
[19:06:59 EDT(-0400)] <EricDalquist> sure
[19:07:09 EDT(-0400)] <lfuller> I still think there is a better than fair chance you may want to make some changes there.
[19:07:46 EDT(-0400)] <lfuller> but yeah... those cascades could very well be at the heart of your problem there.
[19:08:09 EDT(-0400)] <lfuller> I typically on set up cascades for children that do not exist outside the context of their parent.
[19:08:14 EDT(-0400)] <lfuller> er only set up
[19:08:26 EDT(-0400)] <lfuller> lfuller@unicon.net
[19:08:38 EDT(-0400)] <EricDalquist> you can see the persistent objects here: https://www.ja-sig.org/svn/uPortal/tags/rel-3-1-2-GA/uportal-impl/src/main/java/org/jasig/portal/portlet/dao/jpa/
[19:08:40 EDT(-0400)] <EricDalquist> all the impls
[19:09:50 EDT(-0400)] <lfuller> did you email the ehcache config?
[19:09:53 EDT(-0400)] <EricDalquist> just sent
[19:10:00 EDT(-0400)] <lfuller> got it
[19:10:25 EDT(-0400)] <EricDalquist> so when an end user modifies portlet preferences or views a portlet for the first time a PortletEntityImpl is updated or created
[19:11:01 EDT(-0400)] <EricDalquist> as far as I can tell when this happens it triggers the invalidation of the parent PortletDefinitionImpl and the definition's portlet preferences
[19:11:12 EDT(-0400)] <EricDalquist> it doesn't cause a problem on a non-clustered env
[19:11:23 EDT(-0400)] <EricDalquist> since hibernate seems to realize the objects have not changed and doesn't touch the DB
[19:11:44 EDT(-0400)] <EricDalquist> but it does do a Ehcache.put() on the definition and preferences
[19:11:54 EDT(-0400)] <EricDalquist> which results in the invalidation message getting broadcast
[19:13:38 EDT(-0400)] <EricDalquist> I'm working on getting a version of PortletEntityImpl without the cascades to PortletDefinitionImpl on it into our clustered env for testing
[19:13:49 EDT(-0400)] <lfuller> not familiar with this:
[19:13:53 EDT(-0400)] <lfuller> <cache name="org.jasig.portal.portlet.dao.jpa.PortletDefinitionImpl.query.FIND_PORTLET_DEF_BY_CHAN_DEF".....
[19:14:09 EDT(-0400)] <lfuller> never seen that before, cool if it works though.
[19:15:01 EDT(-0400)] <EricDalquist> yeah
[19:15:09 EDT(-0400)] <EricDalquist> you can setup query caches per-query
[19:15:21 EDT(-0400)] <EricDalquist> query.setHint("org.hibernate.cacheRegion", PortletDefinitionImpl.class.getName() + ".query.FIND_PORTLET_DEF_BY_CHAN_DEF");
[19:15:25 EDT(-0400)] <lfuller> nice
[19:15:31 EDT(-0400)] <EricDalquist> I think that is a pending local patch to get back into uportal
[19:15:40 EDT(-0400)] <EricDalquist> it helps a bit with cache tuning
[19:15:46 EDT(-0400)] <lfuller> yeah, would imagine
[19:18:45 EDT(-0400)] <EricDalquist> I'm really hoping removing the cascades will fix it
[19:19:44 EDT(-0400)] <lfuller> fair chance it will help
[19:20:26 EDT(-0400)] <lfuller> am still trying to find the getters and setters that ehcache would use if you were to attempt defining a cache for the associations
[19:21:19 EDT(-0400)] <lfuller> sure... if you want the id for a particular entity fine, but that does not really help out much. Although I guess it could.
[19:22:54 EDT(-0400)] <lfuller> typically... you have class Foo and it has a relationship with Bar classes, represented by a method like List<Bar> Foo.getBars()
[19:23:09 EDT(-0400)] <EricDalquist> right
[19:23:25 EDT(-0400)] <EricDalquist> so PortletDefinitionImpl has a List of PortletEntityImpl
[19:23:27 EDT(-0400)] <lfuller> in which case you set up an echache for whatever.the.path.Foo.bars is
[19:23:28 EDT(-0400)] <EricDalquist> that is lazy init
[19:23:34 EDT(-0400)] <EricDalquist> and never referenced
[19:23:47 EDT(-0400)] <EricDalquist> because a portlet definition will have 1 entity per subscribed user
[19:24:22 EDT(-0400)] <lfuller> could be you don't need it... is just that usually that is how the caching is set up. One each for the entities and one per relationship that you feel you should cache.
[19:26:47 EDT(-0400)] <EricDalquist> yeah
[19:26:55 EDT(-0400)] <EricDalquist> that relationship is never accessed
[19:27:00 EDT(-0400)] <EricDalquist> so there isn't a cache
[19:27:06 EDT(-0400)] <EricDalquist> the portlet object model is a tree
[19:27:14 EDT(-0400)] <EricDalquist> lots of 1-N relationships
[19:27:19 EDT(-0400)] <lfuller> k
[19:27:22 EDT(-0400)] <EricDalquist> but it is only ever walked from the leaves to the root
[19:27:41 EDT(-0400)] <lfuller> then the lists aren't worth your time then... that makes sense
[19:27:42 EDT(-0400)] <EricDalquist> the only time it goes the other way is when non-leaf object is deleted
[19:27:52 EDT(-0400)] <EricDalquist> which is the purpose of that hidden lazy init list
[19:27:59 EDT(-0400)] <EricDalquist> to force the cascading delete
[19:28:00 EDT(-0400)] <EricDalquist> brb
[19:30:01 EDT(-0400)] <lfuller> on your distributed config.... do you have something against using multicast and letting ehcache find/manage its own nodes?
[19:30:12 EDT(-0400)] <lfuller> heck of a lot easier to do it that way...
[19:31:11 EDT(-0400)] * holdorph feels his "clusering made easier" talk was for nothing and silently cries to himself.
[19:33:21 EDT(-0400)] <EricDalquist> some of us were not around to see that talk holdorph
[19:33:40 EDT(-0400)] <EricDalquist> lfuller: that is what it is doin
[19:33:46 EDT(-0400)] <EricDalquist> all we have coded is the multicast address
[19:33:48 EDT(-0400)] <holdorph> the slides are available, i posted them only minutes after i gave the talk (smile)
[19:33:58 EDT(-0400)] <EricDalquist> guess I need to read them then (smile)
[19:34:04 EDT(-0400)] <lfuller> yeah... much, much better to go with terracota+ehcache
[19:34:11 EDT(-0400)] <EricDalquist> brb again
[19:34:27 EDT(-0400)] <holdorph> ehcache + distributed-multicast or jgroups is too chatty
[19:34:40 EDT(-0400)] <holdorph> your 'benefits' start getting erased by other problems pretty quickly
[19:34:41 EDT(-0400)] <lfuller> we use ehcache's distributed configs, but it is not as flexible or as scalable as the terracota + ehcache route
[19:34:59 EDT(-0400)] <lfuller> can definitely attest to that eric
[19:35:27 EDT(-0400)] <lfuller> we just have not been given the go ahead to slide terracota in yet at pearson... however, we know that our 40
[19:35:55 EDT(-0400)] <lfuller> er 40% a year growth will soon have us up against a point where the chattiness outweighs the gains.
[19:36:44 EDT(-0400)] <EricDalquist> well we first tried ehcachermi
[19:36:51 EDT(-0400)] <EricDalquist> and ruled that out as way too inefficient
[19:36:58 EDT(-0400)] <EricDalquist> the jgroups stuff is very low-bandwidth
[19:37:01 EDT(-0400)] <EricDalquist> and much more robust
[19:37:09 EDT(-0400)] <EricDalquist> except for this whole bug
[19:37:12 EDT(-0400)] <EricDalquist> but I have to go grab dinner
[19:37:18 EDT(-0400)] <EricDalquist> thanks for the help
[19:37:24 EDT(-0400)] <EricDalquist> and I'll read your slides tomorrow holdorph
[19:37:25 EDT(-0400)] <EricDalquist> \night
[19:37:31 EDT(-0400)] <holdorph> g'night
[20:03:00 EDT(-0400)] <EricDalquist> hey, you're both still here
[20:03:08 EDT(-0400)] <EricDalquist> holdorph: you have a link to those slides handy>
[20:03:28 EDT(-0400)] <EricDalquist> and I really would have liked to be at that presentation, we'll see what questions I have when I'm done reading
[20:09:42 EDT(-0400)] <holdorph> http://www.ja-sig.org/wiki/display/JCON/Spring+2010+Conference+Presentation+Materials
[20:10:23 EDT(-0400)] <EricDalquist> thanks … found the slideshare
[20:15:49 EDT(-0400)] <EricDalquist> hrm, little annoyed that I have to have an account to read their docs :/
[20:17:41 EDT(-0400)] <holdorph> some links you can get by without the account, like the download, but yeah, some you can't, it's annoying.
[20:29:26 EDT(-0400)] * EricDalquist (~apollo@76.210.65.83) has left ##uportal
[22:04:06 EDT(-0400)] * Sememmon (~Sememmon@unaffiliated/sememmon) has joined ##uportal