uPortal IRC Logs-2013-04-11

Owned by Apereo Infrastructure
Last updated: Apr 11, 2013
3 min read

[12:44:42 CDT(-0500)] <EricDalquist> I think we found a nice infinate loop/memory leak bug

[12:45:04 CDT(-0500)] <EricDalquist> we appear to have threads randomly getting stuck in the DLM merge manager

[12:45:41 CDT(-0500)] <EricDalquist> with some sort of invalid xml document where calling note = node.getNextSibling(); runs forever

[12:45:49 CDT(-0500)] <EricDalquist> and builds an array list of data while it is at it

[12:54:00 CDT(-0500)] <jwennmacher> great find!

[12:54:46 CDT(-0500)] <EricDalquist> https://github.com/Jasig/uPortal/blob/master/uportal-war/src/main/java/org/jasig/portal/layout/dlm/PositionManager.java#L482

[12:54:53 CDT(-0500)] <EricDalquist> so that is the offending method as best as we can tell

[12:55:04 CDT(-0500)] <EricDalquist> that "available" ArrayList grows forever

[12:55:15 CDT(-0500)] <EricDalquist> since we can't actually reproduce this

[12:55:24 CDT(-0500)] <EricDalquist> our plan is to add a HashSet of node ids

[12:55:29 CDT(-0500)] <EricDalquist> and if the same node gets visited twice

[12:55:32 CDT(-0500)] <EricDalquist> just fail with an exception

[12:55:41 CDT(-0500)] <EricDalquist> and at least have something for an initial fix

[12:57:12 CDT(-0500)] <jwennmacher> I assume you found this on your prod server ... yuck.

[12:57:18 CDT(-0500)] <EricDalquist> oh yeah

[12:57:27 CDT(-0500)] <EricDalquist> after 2 jvms died with heap errors in a week (smile)

[12:57:34 CDT(-0500)] <EricDalquist> also glad for a beefy desktop

[12:57:49 CDT(-0500)] <EricDalquist> it took 20 minutes to load the 5.7GB heap dump into yourkit (tongue)

[12:58:42 CDT(-0500)] <jwennmacher> Did you just upgrade your prod environment recently? Couldn't find any recent change that made it appear?

[12:59:06 CDT(-0500)] <EricDalquist> we just updated to 4.0.11 last week

[12:59:13 CDT(-0500)] <EricDalquist> but I can't think of what this would happen from that

[12:59:28 CDT(-0500)] <EricDalquist> its all back to the DLM code

[12:59:34 CDT(-0500)] <EricDalquist> its so hard to figure out wtf is actually going on

[12:59:48 CDT(-0500)] <jwennmacher> Hmm... upgraded from 4.0.what to 4.0.11?

[12:59:59 CDT(-0500)] <EricDalquist> .10

[13:00:33 CDT(-0500)] <jwennmacher> At least it was a small version increase. Hopefully that will help narrow the eventual list of candidates.

[13:00:54 CDT(-0500)] <EricDalquist> yeah

[13:01:01 CDT(-0500)] <EricDalquist> well it could just be dlm layout corruption

[13:01:18 CDT(-0500)] <EricDalquist> due to the direct use of DOM classes there are many places where things can go wrong

[13:01:26 CDT(-0500)] <jwennmacher> I have to head out to a meeting. thanks for notifying us of the issue.

[13:45:36 CDT(-0500)] * Topic is 'http://uportal.pastebin.com/ - https://wiki.jasig.org/display/UPC/uportal+IRC+Logs' set by EricDalquist!~apollo@76.210.64.142 on 17:44:12 CDT(-0500)

[13:45:56 CDT(-0500)] <EricDalquist> testing the bot

[13:46:27 CDT(-0500)] <drewwills> That's quite a find EricDalquist... kinda makes you want to re-do layout persistance I bet (wink)

[13:46:34 CDT(-0500)] <EricDalquist> uhg

[13:46:41 CDT(-0500)] <EricDalquist> I've been wanting to do that for years (tongue)

[14:10:26 CDT(-0500)] * Topic is 'http://uportal.pastebin.com/ - https://wiki.jasig.org/display/UPC/uportal+IRC+Logs' set by EricDalquist!~apollo@76.210.64.142 on 17:44:12 CDT(-0500)

[14:14:00 CDT(-0500)] * Topic is 'http://uportal.pastebin.com/ - https://wiki.jasig.org/display/UPC/uportal+IRC+Logs' set by EricDalquist!~apollo@76.210.64.142 on 17:44:12 CDT(-0500)

[14:20:59 CDT(-0500)] <cwaymire> drewwills: I see you closing those calendar portlet jira tickets. I do not have the necessary perms to do that.

[14:21:07 CDT(-0500)] <EricDalquist> https://github.com/Jasig/uPortal/commit/0c65ad0a6fca43159edb748225ec630c91eb94a2

[14:21:12 CDT(-0500)] <EricDalquist> fix for that issue (I hope)

[14:36:17 CDT(-0500)] <cwaymire> there's a jira (https://issues.jasig.org/browse/ANNPLT-38) to "correct" the fact that the Announcements portlet tables are not prefixed. Table names such as Topic, Audience, Authors. Easy enough fix but not sure if thats something that is considered important enough to go changing the db schema. any thoughts?

[14:59:00 CDT(-0500)] <drewwills> yeah cwaymire I think prefixing them is a good idea, but I'd like to do it (1) with at least a minor release (e.g. 2.1.0) and (2) after we have import/export capabilities

[14:59:34 CDT(-0500)] <drewwills> for announcements

[14:59:58 CDT(-0500)] <cwaymire> yeah. i'd hate to go and change the table names and cause trouble for those that are actively using the portlet.

[15:00:09 CDT(-0500)] <cwaymire> without some clear plan on how to "migrate"