uPortal IRC Logs-2012-03-19

Owned by Apereo Infrastructure
Last updated: Mar 19, 2012
8 min read

[12:50:01 CDT(-0500)] <b-sure> Hello uPortal devs. I"m testing the data imports in the 4.0.4 release with data from the 3.2.x build. It seems to work well on a majority of the records. looking at the target/data-import-reports data, there are a number of individual user files that containg traces like this http://pastebin.com/raw.php?i=vcGqDqbG . Do you know if there is something I can adjust in their files to allow their layout to be imported?

[12:50:50 CDT(-0500)] <EricDalquist> hi b-sure, I'm busy right now but I can take a look at the stacks in about 40 minutes

[12:51:10 CDT(-0500)] <b-sure> np. thanks EricDalquist. I';ll try to stay online her till 4CST today.

[13:17:02 CDT(-0500)] <EricDalquist> b-sure: do you have a corresponding layout file for that?

[13:22:22 CDT(-0500)] <b-sure> i'll dig it up now...

[13:22:34 CDT(-0500)] <EricDalquist> I think I know what is going on though ..

[13:22:39 CDT(-0500)] <EricDalquist> or at least have an idea

[13:23:11 CDT(-0500)] <EricDalquist> do you know if there is an existing jira issue related to this?

[13:25:51 CDT(-0500)] <b-sure> http://pastebin.com/raw.php?i=FHyBmmk2

[13:26:10 CDT(-0500)] <b-sure> I'm not sure if there is a jira fo this. I can submit one if there isn't.

[13:26:20 CDT(-0500)] <EricDalquist> could you do a quick search

[13:26:25 CDT(-0500)] <b-sure> sure...

[13:26:27 CDT(-0500)] <EricDalquist> and then submit one if you don't find one

[13:30:40 CDT(-0500)] <b-sure> looks like this may be similar https://issues.jasig.org/browse/UP-3251

[13:30:49 CDT(-0500)] <b-sure> same oracle error

[13:31:14 CDT(-0500)] <EricDalquist> yup

[13:31:20 CDT(-0500)] <EricDalquist> that looks like the same issue

[13:31:29 CDT(-0500)] <EricDalquist> I'll push a fix out with that issue id

[13:32:23 CDT(-0500)] <b-sure> ok. thanks for looking at this EricDalquist.

[13:51:46 CDT(-0500)] <drewwills1> hi folks

[13:52:08 CDT(-0500)] <EricDalquist> hi drewwills1, when you have a second I'm confused by some layout import logic

[13:53:30 CDT(-0500)] <EricDalquist> it looke like (this is probably my fault) the following code doesn't handle DLM node references:

[13:53:30 CDT(-0500)] <EricDalquist> https://github.com/Jasig/uPortal/blob/master/uportal-war/src/main/java/org/jasig/portal/layout/dlm/RDBMDistributedLayoutStore.java#L929

[13:53:33 CDT(-0500)] <drewwills1> ok... can you post a link? (and i'll look in just a few minutes)

[13:53:34 CDT(-0500)] <EricDalquist> http://pastebin.com/raw.php?i=FHyBmmk2

[13:53:43 CDT(-0500)] <EricDalquist> so the pastebin link is an example layout

[13:54:08 CDT(-0500)] <EricDalquist> the problem is in the code the @ID attribute (line 954) returns null

[13:54:30 CDT(-0500)] <EricDalquist> which looking at the layout makes sense since the folder doesn't have an @ID it has a @dlm:origin

[13:56:53 CDT(-0500)] <b-sure> hello uPortal devs: I've got another issue turning on a 4.0.4 build. I'm not sure where int the configuration I'm messing things up. http://pastebin.com/raw.php?i=qHM2ajLM I can see the the web context fully loads, then something (I assume with filter intialization? ) messes up the deployment and then the portal shuts down.

[13:57:40 CDT(-0500)] <EricDalquist> b-sure: is there anything in your catalina.out log?

[13:57:45 CDT(-0500)] <b-sure> maybe I can make some select packages use DEBUG level to see if there is more infor.

[13:58:17 CDT(-0500)] <b-sure> I'll post some of catalina.out. there is some stuff for the other portlets but nothing I've seen for uPortal itself...

[13:59:03 CDT(-0500)] <EricDalquist> the context should have some sort of mention in there

[14:06:51 CDT(-0500)] <b-sure> ok EricDalquist. I see a bunch of this type of error in catalina.out http://pastebin.com/raw.php?i=ftZ8fKZ5 but nothing I can see yet about uPortal (deployed as ROOT.war).

[14:07:21 CDT(-0500)] <b-sure> should I be looking for some type of exception or maybe something in a particular package.

[14:07:35 CDT(-0500)] <drewwills1> EricDalquest my first reaction is that the import process should be putting ID attributes on the <folder> elements before that method gets called

[14:07:43 CDT(-0500)] <drewwills1> but i'm shooting from the hip

[14:08:31 CDT(-0500)] <drewwills1> another thing – i wouldn't expect an NPE from the valueOf method... should just be an empty string

[14:09:03 CDT(-0500)] <EricDalquist> the NPE isn't there

[14:09:13 CDT(-0500)] <EricDalquist> its later where the null nodeId gets used

[14:09:52 CDT(-0500)] <drewwills1> ah sure... but still should return empty string if it happened that way

[14:10:06 CDT(-0500)] <EricDalquist> right

[14:10:08 CDT(-0500)] <drewwills1> you're seeing this happen I presume?

[14:10:13 CDT(-0500)] <EricDalquist> so it isn't exactly a NPE

[14:10:32 CDT(-0500)] <EricDalquist> it results in a not-null error on oracle DBs

[14:10:39 CDT(-0500)] <EricDalquist> because oracle things "" == null

[14:11:38 CDT(-0500)] <drewwills1> ah yes

[14:11:41 CDT(-0500)] <EricDalquist> so you're thinking that https://github.com/Jasig/uPortal/blob/master/uportal-war/src/main/java/org/jasig/portal/layout/dlm/RDBMDistributedLayoutStore.java#L747

[14:11:49 CDT(-0500)] <EricDalquist> should be resolving those @ID attributes

[14:13:24 CDT(-0500)] <drewwills1> look at line 737

[14:14:10 CDT(-0500)] <drewwills1> i would think an ID attribute would be added to any <folder> element

[14:14:35 CDT(-0500)] <EricDalquist> hrm

[14:14:52 CDT(-0500)] <EricDalquist> take a look at 756

[14:15:08 CDT(-0500)] <EricDalquist> if the pathRef fails to resolve the logic here simply changes the dlm:origin attribute value

[14:15:18 CDT(-0500)] <EricDalquist> the problem is code later on doesn't know that the pathRef was bad

[14:15:26 CDT(-0500)] <EricDalquist> and barfs on the missing @ID

[14:15:43 CDT(-0500)] <EricDalquist> so if we can assume the dlm:origin logic is good

[14:15:58 CDT(-0500)] <EricDalquist> then we can change the logic in the later method to just ignore nodes without an @ID

[14:17:30 CDT(-0500)] <EricDalquist> b-sure: do you have warnings in the output of data-import that look like:

[14:18:07 CDT(-0500)] <EricDalquist> Unable to resolve pathref '....

[14:18:10 CDT(-0500)] <drewwills1> yes i think you're onto it

[14:18:36 CDT(-0500)] <drewwills1> perhaps we could merge that bit with step (6) above though... for simplicity

[14:19:13 CDT(-0500)] <drewwills1> not sure why most nodes get ID in step 6, then some in step (4) – lol at ordering

[14:19:39 CDT(-0500)] <b-sure> I think so. I'd need to run it again. to verify.

[14:19:48 CDT(-0500)] <EricDalquist> well for now I can tweak loadStylesheetUserPreferencesAttributes

[14:20:03 CDT(-0500)] <EricDalquist> and just ignore *-attribute elements where the parent element is missing an @ID

[14:20:17 CDT(-0500)] <drewwills1> yes i think so

[14:20:18 CDT(-0500)] <b-sure> would you like me to run the import again?

[14:20:32 CDT(-0500)] <b-sure> and maybe pipe the output somewhere.

[14:20:44 CDT(-0500)] <EricDalquist> just a minute

[14:20:48 CDT(-0500)] <EricDalquist> I'm going to commit what I think is a fix

[14:20:49 CDT(-0500)] <drewwills1> i have to go to another meeting... bb in the afternoon sometime

[14:21:01 CDT(-0500)] <EricDalquist> you should be able to cherry-pick it into your code base and try again

[14:21:07 CDT(-0500)] <b-sure> ok

[14:25:11 CDT(-0500)] <EricDalquist> just doing one last test here b-sure

[14:25:21 CDT(-0500)] <EricDalquist> I'll have it pushed to Jasig/uPortal in about 5 minutes

[14:25:38 CDT(-0500)] <b-sure> ok.

[14:26:17 CDT(-0500)] <b-sure> by cherry-pick do you mean just copy and paste one of the files into my 4.0.4 release?

[14:28:20 CDT(-0500)] <EricDalquist> sure (smile)

[14:28:28 CDT(-0500)] <EricDalquist> there are about 6 files I think

[14:28:33 CDT(-0500)] <b-sure> oh ok. np.

[14:28:37 CDT(-0500)] <EricDalquist> how are you doing version control?

[14:29:02 CDT(-0500)] <b-sure> well I do have git available on the server where I"m doing the import.

[14:29:41 CDT(-0500)] <b-sure> I was just thinking I should add the changes to the raw release copy I downloaded from jasig. thats what I've been using to do the imports.

[14:30:03 CDT(-0500)] <EricDalquist> ah

[14:30:15 CDT(-0500)] <EricDalquist> then you can just grab the changed files

[14:30:20 CDT(-0500)] <b-sure> ok.

[14:30:26 CDT(-0500)] <EricDalquist> I was saying cherry-pick in the git sense

[14:30:35 CDT(-0500)] <EricDalquist> like we're using git for managing our vendor branch

[14:30:44 CDT(-0500)] <b-sure> thats what I thought. I just havent tried it out yet

[14:30:46 CDT(-0500)] <EricDalquist> and that allows me to do: "git cherry-pick XXXXXX"

[14:30:57 CDT(-0500)] <EricDalquist> while on the myuw branch

[14:31:09 CDT(-0500)] <EricDalquist> that grabs the commit with ID XXXXX and applies it to my branch

[14:31:37 CDT(-0500)] <b-sure> ok. then is your myuw branch the one you use to build and deploy locally and do imports ?

[14:32:08 CDT(-0500)] <EricDalquist> yes

[14:32:16 CDT(-0500)] <EricDalquist> that is what we use for anything myuw related

[14:32:48 CDT(-0500)] <b-sure> ok. I think id like to be able to swicth to the 4.0.4 release branch in git and then cherry pick the id that applies to the branch.

[14:33:15 CDT(-0500)] <EricDalquist> so how are you managing your customizations to uPortal?

[14:33:25 CDT(-0500)] <EricDalquist> are you using git?

[14:34:12 CDT(-0500)] <b-sure> so I'm pulling the 4.0.4 release from maven central and using it as a war file overlay. then any customizations are in the overlay project. they are mostly just configuration xml files at this point.

[14:34:23 CDT(-0500)] <EricDalquist> ah

[14:34:40 CDT(-0500)] <EricDalquist> so your only local version controlled data is the overlay

[14:34:55 CDT(-0500)] <b-sure> yes.

[14:35:02 CDT(-0500)] <EricDalquist> you don't have a vendor branch of uPortal in the traditional sense

[14:35:33 CDT(-0500)] <b-sure> no. although we keep copies of in on the development server to do imports/ exports between versions

[14:35:38 CDT(-0500)] <EricDalquist> then your best bet would be to grab the 9 files that were modified in https://github.com/Jasig/uPortal/commit/f4265f90ae5095d099b417f662c65c6e19854b92

[14:35:52 CDT(-0500)] <EricDalquist> and stick them into your overlay

[14:35:53 CDT(-0500)] <b-sure> and I recently got git installed on that dev server

[14:36:18 CDT(-0500)] <b-sure> ok I'll do that for now and work on the git way once I get this deployed. and import working.

[14:37:38 CDT(-0500)] <b-sure> EricDalquist. This patch is to address the import errors I' assume? not the deploy issue I also poster here?

[14:37:46 CDT(-0500)] <EricDalquist> right

[14:37:55 CDT(-0500)] <b-sure> ok thanks. I'll retry the import again.

[14:37:55 CDT(-0500)] <EricDalquist> I think you need to look earlier in catalina.out

[14:38:01 CDT(-0500)] <EricDalquist> or just rm it

[14:38:02 CDT(-0500)] <EricDalquist> thens tart

[14:38:08 CDT(-0500)] <EricDalquist> and start from the beginning of the log file

[14:38:29 CDT(-0500)] <b-sure> ok i'm gonna remove it and start from scratch since its too big a file now to open w/ nano

[14:49:14 CDT(-0500)] <EricDalquist> b-sure: I missed a file with that first commit: https://github.com/Jasig/uPortal/commit/5490af0b1c46cc4841ee95178e90fe5bc86329fa

[14:53:20 CDT(-0500)] <b-sure> ok. np. I'll restart the import with the updated file.

[15:04:27 CDT(-0500)] <b-sure> EricDalquist: so I am seed these -> WARN No layout node found for pathref: everyday-lo:/layout/folder/f older[3]/folder[1] but not failing of the import yet.

[15:04:39 CDT(-0500)] <EricDalquist> ij'

[15:04:41 CDT(-0500)] <EricDalquist> ok

[15:05:32 CDT(-0500)] <b-sure> EricDalquist: I also see these " [java] ERROR Failed: importing file [/home/billbrown/uPortal-4.0.4/../uPort al-3-2-patches/build/export-2011-10-12/layout/rwilliams12.layout]" but these aren't causing it to bomb either so far.

[15:05:43 CDT(-0500)] <EricDalquist> yeah

[15:05:54 CDT(-0500)] <EricDalquist> 4.0.4 (maybe it was 4.0.3?) changed the import

[15:05:58 CDT(-0500)] <EricDalquist> so it doesn't fail-fast

[15:06:03 CDT(-0500)] <EricDalquist> it imports as much as possib;e

[15:06:10 CDT(-0500)] <EricDalquist> and then logs status

[15:06:12 CDT(-0500)] <b-sure> yeah. I needed to toggle that property in portal.properties

[15:06:37 CDT(-0500)] <b-sure> I think it is set to fail fast by default and you need to tell it not to do that

[15:07:28 CDT(-0500)] <b-sure> so for the failed import lines in the output, do I need to adjust these users import files manually? I'm trying to prevent them from getting an error when they try to access the portal.

[15:08:06 CDT(-0500)] <b-sure> I think users that don't have import data in the db but do have a user record get a broken page when they try to log in.

[15:09:00 CDT(-0500)] <EricDalquist> hrm

[15:09:01 CDT(-0500)] <EricDalquist> I'm not sure

[15:09:07 CDT(-0500)] <b-sure> I'll retest with identity swap.

[15:09:08 CDT(-0500)] <EricDalquist> depends on what the errors are I guess

[15:21:02 CDT(-0500)] <b-sure> hello EricDalquist: the imported errored out but I think it ran to completion and maybe the error is not actually an error http://pastebin.com/raw.php?i=pi9mfWgD.

[15:21:57 CDT(-0500)] <EricDalquist> you'd have to look at the actual per-entity error dumps in /home/billbrown/uPortal-4.0.4/target/data-import-reports

[15:21:59 CDT(-0500)] <EricDalquist> but I have to run

[15:22:05 CDT(-0500)] <EricDalquist> be back tomorrow morning

[15:22:11 CDT(-0500)] <b-sure> oh ok. thanks for your help today