uPortal IRC Logs-2010-09-15

Owned by Apereo Infrastructure
Last updated: Sept 15, 2010Version comment
20 min read

[09:36:52 CDT(-0500)] <athena> so, looks like you found a way around the stax issue?
[09:46:46 CDT(-0500)] <EricDalquist> yes
[09:46:48 CDT(-0500)] <EricDalquist> I think so
[09:48:48 CDT(-0500)] <EricDalquist> there are two JDK bugs that had to be worked around :/
[09:48:56 CDT(-0500)] <EricDalquist> but it appears to all be working
[09:49:04 CDT(-0500)] <EricDalquist> so now I can do what I was hoping to do 1.5 weeks ago
[09:49:07 CDT(-0500)] <EricDalquist> and get stuff cleaned up
[09:49:12 CDT(-0500)] <EricDalquist> do testing
[09:49:17 CDT(-0500)] <EricDalquist> and get it committed at the end of the week
[09:50:06 CDT(-0500)] <EricDalquist> I am working on the channel list performance stuff today
[09:50:09 CDT(-0500)] <EricDalquist> and had a question for you
[09:50:35 CDT(-0500)] <EricDalquist> the controller does: List<IChannelDefinition> allChannels = channelRegistryStore.getChannelDefinitions();
[09:51:12 CDT(-0500)] <EricDalquist> but then gets all the channels per-category again when building the XML
[09:52:27 CDT(-0500)] <EricDalquist> the registry store has an API: public List<IChannelDefinition> getChannelDefinitions(IPerson person)
[09:52:36 CDT(-0500)] <EricDalquist> I'm wondering if there was a reason that wasn't used
[09:52:44 CDT(-0500)] <EricDalquist> since it should get you a pre-filtered list of channels
[09:53:00 CDT(-0500)] <EricDalquist> then those can be added to the XML directly without the extra introspection
[10:03:10 CDT(-0500)] <athena> sorry, wandered off for a minute
[10:03:16 CDT(-0500)] <EricDalquist> no problem
[10:03:19 CDT(-0500)] <EricDalquist> I do that a lot too
[10:03:45 CDT(-0500)] <athena> hm, i don't know of a reason the controller doesn't use the person-specific API method
[10:04:43 CDT(-0500)] <athena> i think dmazurek originally wrote a lot of that code
[10:04:57 CDT(-0500)] <EricDalquist> it looks like they both do the same canSubscribe test
[10:05:09 CDT(-0500)] <athena> oh
[10:05:09 CDT(-0500)] <athena> that'd be why then
[10:05:21 CDT(-0500)] <athena> depending on the request, we need to filter either by subscribe permissions or manage permissions
[10:05:27 CDT(-0500)] <EricDalquist> ah
[10:05:39 CDT(-0500)] <EricDalquist> well I may just create a new registry method
[10:05:44 CDT(-0500)] <athena> so if the registry method only does subscribe, it would only apply to the "add channels" use case, rather than the portlet admin portlet as well
[10:05:50 CDT(-0500)] <athena> makes sense
[10:06:10 CDT(-0500)] <EricDalquist> I think the big problem right now is the current logic gets a List of all channels from JPA then for each category does a JPA.getChannel for each member
[10:06:36 CDT(-0500)] <EricDalquist> even with caching you have to go through the entity manager more than 2*N times since lots of channels end up in multiple categories
[10:06:42 CDT(-0500)] <athena> yes
[10:07:22 CDT(-0500)] <EricDalquist> public List<IChannelDefinition> getChannelDefinitions(IPerson person); isn't currently used by anthing
[10:07:26 CDT(-0500)] <athena> so we could probably prevent all the getChannel calls by pre-fetching them all and then pulling them out of a map or whatever
[10:07:33 CDT(-0500)] <EricDalquist> so I may just modify it and add a 'boolean manage' flag
[10:07:47 CDT(-0500)] <athena> and i still think it might be an improvement to construct the master list only once, then filter through it by permission
[10:07:52 CDT(-0500)] <EricDalquist> yeah
[10:07:53 CDT(-0500)] <athena> yeah that sounds reasonable
[10:07:58 CDT(-0500)] <EricDalquist> that's what this change would do
[10:08:20 CDT(-0500)] <EricDalquist> every time it is called it would do getAllChannels and then filter it for the user
[10:08:33 CDT(-0500)] <EricDalquist> and use that filtered list to build the XML directly
[10:08:49 CDT(-0500)] <athena> would that just return the channel list or the structured category hierarchy?
[10:09:03 CDT(-0500)] <EricDalquist> I'd return the same structure we have now
[10:09:17 CDT(-0500)] <athena> yeah
[10:09:18 CDT(-0500)] <EricDalquist> we should be able to reverse engineer that from the channel list
[10:09:45 CDT(-0500)] <athena> i think all the logic for assembling that structure is in ChannelListController, so you shouldn't have to reverse engineer it
[10:09:54 CDT(-0500)] <athena> most of it's in the addChildren method
[10:11:03 CDT(-0500)] <EricDalquist> yup
[10:17:07 CDT(-0500)] <b-rock> Greetings uPortal Devs: do you know if there is a way to tell the crn-import routine to contine a db import even if it fails on some of the channels it's importing? I'm running it on a full database and it is bombing an some of the channel data.
[10:17:53 CDT(-0500)] <EricDalquist> I don't think there is
[10:18:05 CDT(-0500)] <holdorph> b-rock, yesterday I think you had a problem, but dropped off before we
[10:18:13 CDT(-0500)] <holdorph> were able to tell you, that you can't use ant 1.8
[10:18:22 CDT(-0500)] <holdorph> did you go back to ant 1.7?
[10:19:02 CDT(-0500)] <b-rock> hi holdorph. I fixed the issue from yesterday by adding the uportal-impl jars specifically to the classpath in several targets in the db sections of the build.xml file
[10:19:23 CDT(-0500)] <holdorph> but this could still be related.
[10:19:38 CDT(-0500)] <holdorph> you can't use ant 1.8, you must be using ant 1.7.1
[10:19:42 CDT(-0500)] <b-rock> ok. maybe that is what the classpath errors are about.
[10:19:49 CDT(-0500)] <holdorph> probably
[10:19:51 CDT(-0500)] <EricDalquist> it is a bug that ant 1.8 works at all
[10:19:57 CDT(-0500)] <EricDalquist> the build should be setup to require ant 1.7.1
[10:20:22 CDT(-0500)] <b-rock> ok. I will ask the admins to roll back to 1.7.1 and then try again.
[10:20:23 CDT(-0500)] <EricDalquist> ant very (un)graciously made some really big changes between 1.7 and 1.8 with how macros work (sad)
[10:20:40 CDT(-0500)] <b-rock> it did actually build ok.
[10:20:41 CDT(-0500)] <b-rock> with 1.8.1
[10:20:59 CDT(-0500)] <EricDalquist> right the build stuff works because that is simply calling maven under the hood
[10:21:04 CDT(-0500)] <EricDalquist> none of the command line utilities will work though
[10:21:46 CDT(-0500)] <b-rock> ah ok. do you think that could case the cernunnos importer from completing.
[10:22:11 CDT(-0500)] <b-rock> the error I get there is actually a database error for an invalid field name.
[10:22:24 CDT(-0500)] <b-rock> looks like a temporary field name
[10:35:55 CDT(-0500)] <EricDalquist> uhg this isn't going to work like I hoped athena
[10:36:05 CDT(-0500)] <EricDalquist> I forgot that categories aren't fields on channel defs
[10:36:12 CDT(-0500)] <EricDalquist> you have to go to the group store for that data
[10:36:13 CDT(-0500)] <EricDalquist> (sad)
[10:36:33 CDT(-0500)] <athena> right
[10:36:44 CDT(-0500)] <athena> that's why i was saying we still have to deal w/ all the category membership calls
[10:36:55 CDT(-0500)] <EricDalquist> yeah
[10:37:16 CDT(-0500)] <athena> so the way the old channel registry did it was to assemble the custom category/channel hierarchy structure once for the entire portal, ignoring permissions, then cache it
[10:37:27 CDT(-0500)] <EricDalquist> right
[10:37:44 CDT(-0500)] <athena> then every time a user accessed it, it would take that master XML document and filter through it, producing a custom structure that might match the master or be a subset of it
[10:37:44 CDT(-0500)] <EricDalquist> the problem there is you have to keep it up to date anytime anyone changes anything about a channel
[10:37:46 CDT(-0500)] <EricDalquist> which sucks
[10:37:51 CDT(-0500)] <athena> yep
[10:37:54 CDT(-0500)] <EricDalquist> arg
[10:38:02 CDT(-0500)] <athena> the old version just scrapped the whole registry every time a channel was updated
[10:38:49 CDT(-0500)] <athena> i agree that many args are applicable (smile)
[10:45:31 CDT(-0500)] <EricDalquist> well if all else fails pull out the profiler
[10:45:45 CDT(-0500)] <EricDalquist> doing a 'while true wget channelList' right now with yourkit watching
[10:45:57 CDT(-0500)] <EricDalquist> see if there is anything that can easily be tweaked
[10:46:03 CDT(-0500)] <athena> that'd be great
[10:52:39 CDT(-0500)] <EricDalquist> hrm
[10:52:47 CDT(-0500)] <EricDalquist> yourkit makes it looks like it his hitting the DB every time
[10:52:53 CDT(-0500)] <EricDalquist> I wonder if there is a problem in a cache config
[10:54:02 CDT(-0500)] <athena> hm, interesting
[10:54:13 CDT(-0500)] <athena> wouldn't think it should hit the db for the same user
[10:54:25 CDT(-0500)] <EricDalquist> it shouldn't hit it at all after a single pass
[10:54:34 CDT(-0500)] <EricDalquist> all the channel definitions should be in the 2nd level cache
[10:56:19 CDT(-0500)] <athena> hm, yeah
[10:56:31 CDT(-0500)] <athena> and the other stuff is all groups/permissions
[10:56:43 CDT(-0500)] <EricDalquist> yweah
[10:56:54 CDT(-0500)] <EricDalquist> and that is actually really fast
[10:58:37 CDT(-0500)] <EricDalquist> over 9 calls to channelList
[10:58:47 CDT(-0500)] <EricDalquist> 98% of the time was spent in getChannelDefinitions()
[10:58:58 CDT(-0500)] <EricDalquist> 2% was in addChildren which does the permissions checks
[11:06:48 CDT(-0500)] <athena> interesting
[11:06:54 CDT(-0500)] <athena> why so much time in getChannelDefinitions?
[11:07:02 CDT(-0500)] <athena> shouldn't that be cached as well?
[11:07:04 CDT(-0500)] <EricDalquist> all that does is a jpa call
[11:07:05 CDT(-0500)] <EricDalquist> yeah
[11:07:20 CDT(-0500)] <EricDalquist> so I think something is wrong with the JPA query or cache config there
[11:07:27 CDT(-0500)] <athena> well, at least that might be an easy fix
[11:13:28 CDT(-0500)] <EricDalquist> lfuller1: jpa/hibernate question for you
[11:13:45 CDT(-0500)] <EricDalquist> when loading all entities of a specific type with a query: from ChannelDefinitionImpl channel
[11:13:55 CDT(-0500)] <EricDalquist> it appears that the 2nd level cache is ignored
[11:20:42 CDT(-0500)] <lfuller1> ok... did you turn caching on for the query you are making?
[11:20:59 CDT(-0500)] <lfuller1> cause unless you explicitly set caching to true for that query, it won't work.
[11:21:06 CDT(-0500)] <EricDalquist> it is
[11:21:13 CDT(-0500)] <EricDalquist> I'm debugging through hibernate right now
[11:21:18 CDT(-0500)] <EricDalquist> we're hitting the query cache
[11:21:37 CDT(-0500)] <EricDalquist> but the load of each object from that query cache is very flow
[11:21:41 CDT(-0500)] <EricDalquist> slow*
[11:22:45 CDT(-0500)] <EricDalquist> hrm ...
[11:23:02 CDT(-0500)] <EricDalquist> do you have to do anything special to get one-to-one relationships to be cached?
[11:23:42 CDT(-0500)] <lfuller1> are you using ehcache?
[11:23:51 CDT(-0500)] <EricDalquist> yes
[11:24:10 CDT(-0500)] <lfuller1> would you mind emailing me your ehcache.xml file?
[11:24:24 CDT(-0500)] <lfuller1> or otherwise giving me access to it?
[11:24:27 CDT(-0500)] <EricDalquist> its the default 3.2.2 ehcache.xml
[11:25:02 CDT(-0500)] * lfuller1 has a call he needs to jump on in 4 minutes.
[11:25:03 CDT(-0500)] <EricDalquist> https://source.jasig.org/uPortal/tags/rel-3-2-2/uportal-impl/src/main/resources/properties/ehcache.xml
[11:25:06 CDT(-0500)] <EricDalquist> no problem
[11:25:11 CDT(-0500)] <lfuller1> would just make sure you have defined the cache for it properly.
[11:25:33 CDT(-0500)] <lfuller1> sometimes that just means you let the parent object take care of it... sometimes you don't though.
[11:25:37 CDT(-0500)] <lfuller1> depends on how it is used.
[11:26:09 CDT(-0500)] <EricDalquist> there is a cache defined for it
[11:26:48 CDT(-0500)] <lfuller1> what is the parent object?
[11:26:57 CDT(-0500)] <EricDalquist> so loading all ChannelDefinitionImpls
[11:27:08 CDT(-0500)] <EricDalquist> which have a one to one to PortletDefinitionImpl
[11:27:30 CDT(-0500)] <lfuller1> ok... and what field are the channels stored under?
[11:27:32 CDT(-0500)] <EricDalquist> and it appears that every time a ChannelDefinitionImpl is read from cache the related PortletDefinitionImpl is read from the DB
[11:28:12 CDT(-0500)] <lfuller1> org.jasig.portal.portlet.dao.jpa.PortletDefinitionImpl.whateverFieldChannelDefinitionImplIsIn
[11:28:24 CDT(-0500)] <lfuller1> set up a cache for that
[11:28:38 CDT(-0500)] <EricDalquist> ok
[11:28:54 CDT(-0500)] <EricDalquist> in ChannelDefinitionImpl there is:
[11:28:54 CDT(-0500)]

<EricDalquist> @OneToOne(targetEntity = PortletDefinitionImpl.class, mappedBy = "channelDefinition", fetch = FetchType.EAGER, cascade =

Unknown macro: { CascadeType.ALL }

)


[11:28:54 CDT(-0500)]

<EricDalquist> @Cascade(

Unknown macro: { org.hibernate.annotations.CascadeType.DELETE_ORPHAN, org.hibernate.annotations.CascadeType.ALL }

)


[11:28:54 CDT(-0500)] <EricDalquist> private final IPortletDefinition portletDefinition;
[11:29:17 CDT(-0500)] <EricDalquist> and in PortletDefinitionImpl:
[11:29:17 CDT(-0500)] <EricDalquist> @OneToOne(targetEntity = ChannelDefinitionImpl.class)
[11:29:17 CDT(-0500)] <EricDalquist> @JoinColumn(name = "CHANNEL_DEF_ID", nullable = false, updatable = false, unique = true)
[11:29:17 CDT(-0500)] <EricDalquist> private final IChannelDefinition channelDefinition;
[11:29:36 CDT(-0500)] <lfuller1> org.jasig.portal.portlet.dao.jpa.PortletDefinitionImpl.channelDefinition
[11:29:46 CDT(-0500)] <EricDalquist> got it
[11:30:00 CDT(-0500)] <lfuller1> I don't know off the top of my head how they are used. that is important to how you set up the caching.
[11:32:25 CDT(-0500)] <EricDalquist> ok
[11:32:28 CDT(-0500)] <EricDalquist> I'll give this a try
[12:01:37 CDT(-0500)] <lfuller1> how goes it?
[12:01:41 CDT(-0500)] <EricDalquist> uhg
[12:01:44 CDT(-0500)] <lfuller1> am about to jump into another meeting(sad)
[12:01:46 CDT(-0500)] <EricDalquist> so I found a post on the hibernate forums
[12:02:02 CDT(-0500)] <lfuller1> a credible post?
[12:02:11 CDT(-0500)] <EricDalquist> https://forum.hibernate.org/viewtopic.php?p=2399484
[12:02:22 CDT(-0500)] <EricDalquist> so this is a bidirectional 1-1 relationship
[12:02:31 CDT(-0500)] * lfuller1 clicks on link and tries really hard to act like he is paying attention in a meeting.
[12:02:51 CDT(-0500)] <EricDalquist> the portlet_def table has a FK reference to the channel_def table
[12:03:15 CDT(-0500)] <EricDalquist> according to that post trying to load from the channel_def side of the relationship will always result in a query
[12:03:24 CDT(-0500)] <EricDalquist> since it doesn't know the PK of the portlet_def
[12:04:56 CDT(-0500)] <lfuller1> do we take advantage of that bidirectional relationship?
[12:05:05 CDT(-0500)] <EricDalquist> yes it looks like we do
[12:05:10 CDT(-0500)] <lfuller1> i.e. is it necessary?
[12:05:15 CDT(-0500)] <EricDalquist> I'm wondering if that can be fixed though
[12:32:21 CDT(-0500)] <athena> EricDalquist: did that .channel import issue w/ HTML in portlet preferences get fixed yet?
[13:29:27 CDT(-0500)] <EricDalquist> not yet
[13:29:45 CDT(-0500)] <EricDalquist> I have a number of local mods that I need to create patches for :/
[13:53:48 CDT(-0500)] <EricDalquist> athena: CChannelManager is still in 3.2 but it isn't actually used right?
[13:58:48 CDT(-0500)] <athena> i think there's a method or two that's still calledf
[13:58:53 CDT(-0500)] <athena> think there was some reason i couldn't delete it
[13:59:05 CDT(-0500)] <athena> maybe from the ajax layout preferences target?
[14:00:10 CDT(-0500)] <EricDalquist> hrm, ok
[14:00:35 CDT(-0500)] <athena> maybe that was the channel regsistry manager though
[14:00:49 CDT(-0500)] <EricDalquist> yeah I'm going to try nuking it
[14:24:50 CDT(-0500)] <b-rock> hello again portal devs. I've downgraded ant to 1.7.1 and have got most of our data imported to an empty db. We are getting a failure during the fragment layout import http://uportal.pastebin.com/Rp6VP7kR . Does this trace seem familiar enough to anyone here?
[14:25:40 CDT(-0500)] <awills> userId=0 is the system user
[14:26:08 CDT(-0500)] <b-rock> oh. I think the import created system with userid 1.
[14:26:57 CDT(-0500)] <awills> and this is your data? not data that comes with uP by default?
[14:27:34 CDT(-0500)] <b-rock> yes. this is a full set of data from our production server from a couple weeks back on an crn-export of a 3.1.1 db.
[14:28:26 CDT(-0500)] <awills> and what version are you importing to? (sorry if you've covered this before)
[14:28:39 CDT(-0500)] <b-rock> I found I needed to remove a bunch of dup channel definitions to get those to import
[14:28:52 CDT(-0500)] <b-rock> np. I'm trying to import to 3.2.2
[14:29:15 CDT(-0500)] <b-rock> I'm running this command "ant initdb -Dmaven.test.skip=true -Ddroptables=-D""
[14:29:59 CDT(-0500)] <awills> have you set entities.location?
[14:30:03 CDT(-0500)] <b-rock> I think the only things left to import are the fragment-layouts and layouts
[14:30:09 CDT(-0500)] <awills> in build.properties
[14:30:22 CDT(-0500)] <b-rock> yes I set it to the exported db data from the 3.1.1 build.
[14:30:48 CDT(-0500)] <b-rock> that part seems to be working in that I see it and in the db all of the user and channel definitions.
[14:30:56 CDT(-0500)] <awills> can you peek in the DB and verify USER_ID for the system user?
[14:31:00 CDT(-0500)] <awills> is it 1?
[14:31:56 CDT(-0500)] <b-rock> system user_id = 1 and defaultTemplateUser user_id = 6
[14:32:12 CDT(-0500)] <b-rock> these were both created during the initdb run.
[14:33:02 CDT(-0500)] <b-rock> the user_dflt_us field is null for both of these users and appears to be 6 for the other users.
[14:33:23 CDT(-0500)] <b-rock> and 7 for the fragment owners
[14:36:29 CDT(-0500)] <awills> my mistake... system USER_ID=1 must be ok... it's what I have too
[14:36:39 CDT(-0500)] <awills> used to be always 0
[14:37:41 CDT(-0500)] <awills> i have exactly what you describe too, on UP_USER
[14:38:23 CDT(-0500)] <b-rock> yeah. I haven't tried removing the fragment-layout folder from the rest of the db and then trying again to see if the "layouts" folder will import. I don't mind redoing the fragment layouts if the users individual layouts will import.
[14:39:00 CDT(-0500)] <b-rock> I'm guessing the user layouts probably need the fragment layouts to be there though.
[14:40:16 CDT(-0500)] <awills> what sorts of users are you importing? admin/demo/student/etc, or real users?
[14:41:12 CDT(-0500)] <b-rock> I'm attempting to do real users because we will need to run this on the production database when it comes time to upgrade there.
[14:42:05 CDT(-0500)] <awills> yeah i should think you need the fragment layouts first
[14:45:18 CDT(-0500)] <b-rock> do you know where the cernunnos code is in the portal. I can try digging around there to see where the error "unable to find User Profile for userId 0 and profile default" gets thrown and look for clues there.
[14:46:08 CDT(-0500)] <b-rock> or do you know how to turn off the fail-fast ? maybe that will allow the run to not fail on an error.
[14:46:40 CDT(-0500)] <awills> that error message is RDBMUserLayoutStore.getUserProfileByFname @1873
[14:46:43 CDT(-0500)] <awills> looking at it now
[14:47:22 CDT(-0500)] <EricDalquist> b-rock: fail-fast is there for a reason
[14:47:27 CDT(-0500)] <EricDalquist> you don't want a corrupt database now do you?
[14:47:34 CDT(-0500)] <awills> can you send me a .user + .fragment-layout file for one that's failing?
[14:47:50 CDT(-0500)] <awills> send to up-dev i guess
[14:48:33 CDT(-0500)] <awills> there have been some recent enhancements to profiles... i'm thinking there may be a loose end to tie down
[14:49:23 CDT(-0500)] <b-rock> ok. I can send the one thats failing to the uportal mailing list?
[14:49:44 CDT(-0500)] <b-rock> I'm on the users list there.
[14:49:56 CDT(-0500)] <awills> ok by me... or you could make a JIRA and attach it
[14:51:22 CDT(-0500)] <b-rock> It may be easier to just create a Jira in my opinion. I don't see a lot of traces being passed along through the uportal users list.
[15:01:37 CDT(-0500)] <EricDalquist> WOOHOO
[15:01:44 CDT(-0500)] <EricDalquist> athena: I think I have the load time issues fixed
[15:01:49 CDT(-0500)] <EricDalquist> there were a bunch of little things wrong
[15:01:55 CDT(-0500)] <athena> what happened?
[15:02:02 CDT(-0500)] <athena> cache config?
[15:02:07 CDT(-0500)] <athena> and that's awesome, by the way (smile)
[15:02:07 CDT(-0500)] <EricDalquist> now loading 175 channel definitions in ~ 220ms
[15:02:16 CDT(-0500)] <athena> nice!
[15:03:08 CDT(-0500)] <EricDalquist> missing caches for channel parameters and localizations
[15:03:09 CDT(-0500)] <EricDalquist> and a true bi directional relationship between channel_def and portlet_def
[15:03:12 CDT(-0500)] <EricDalquist> the first one was easy
[15:03:18 CDT(-0500)] <EricDalquist> the second one is tricky
[15:03:27 CDT(-0500)] <athena> ah
[15:03:40 CDT(-0500)] <athena> sounds like good fixes
[15:03:46 CDT(-0500)] <EricDalquist> so apaprently hibernate can only take advantage of the 2nd level cache going from the mappedBy class in a bidirectional relationship
[15:04:00 CDT(-0500)] <athena> interesting
[15:04:03 CDT(-0500)] <EricDalquist> so what I had to do was remove the getPortletDefinition method from IChannelDefinition
[15:04:17 CDT(-0500)] <EricDalquist> there is still a lazy references from chanDef -> portDef to handle cascading deletes
[15:04:25 CDT(-0500)] <EricDalquist> the same thing that portletDef -> portletEnt have setup
[15:04:27 CDT(-0500)] <athena> maybe some of this will get simpler when we get rid of channels?
[15:04:31 CDT(-0500)] <EricDalquist> YES
[15:04:34 CDT(-0500)] <EricDalquist> very much so
[15:04:46 CDT(-0500)] <EricDalquist> this is all a consequence of having IPortletDefinition and IChannelDefinition
[15:04:53 CDT(-0500)] <EricDalquist> when we merge them in trunk it will go away
[15:05:04 CDT(-0500)] <athena> sweet (smile)
[15:05:08 CDT(-0500)] <EricDalquist> I'll see what I can get put together for a patch on this
[15:05:11 CDT(-0500)] <EricDalquist> and our other fixez
[15:05:14 CDT(-0500)] <athena> fantastic!
[15:05:18 CDT(-0500)] <EricDalquist> maybe we can get a 3.2.3 out in a few weeks
[15:05:32 CDT(-0500)] <athena> yeah - that bug where you can't change the skin is actually pretty critical :/
[15:05:44 CDT(-0500)] <EricDalquist> yeah
[15:05:54 CDT(-0500)] <athena> i'm really interested in patches that'll make it easier to use the content portlet with .channel defs too
[15:06:01 CDT(-0500)] <athena> that'd be a huge win
[15:06:06 CDT(-0500)] <EricDalquist> so if you have another bugs you're aware of make sure they are at least in jira
[15:06:07 CDT(-0500)] <EricDalquist> yup
[15:06:14 CDT(-0500)] <EricDalquist> I'll get as many of our local mods in as possible
[15:06:17 CDT(-0500)] <athena> i don't know of any other ones
[15:06:42 CDT(-0500)] <EricDalquist> the missing caches for channel params are likely the source of some of the perf issues brought up about 3.2
[15:07:00 CDT(-0500)] <EricDalquist> since right now every time you ask for a chanDef it does 2 queries even if it was cached
[15:07:06 CDT(-0500)] <awills> b-rock: are there any rows on UP_USER with USER_DFLT_USR_ID=0?
[15:07:07 CDT(-0500)] <EricDalquist> because those collections are not cached
[15:07:31 CDT(-0500)] <EricDalquist> on a nice warm JVM this should perform quite well
[15:07:42 CDT(-0500)] <b-rock> I'll check.
[15:09:32 CDT(-0500)] <b-rock> no USER_DFLT_USR_ID set to 0 during the import.
[15:12:08 CDT(-0500)] <b-rock> I tried deploying the uPortal.war on what the import was able to deposit to the database, and when logging in I get the same error as the import http://uportal.pastebin.com/tN7yHF7Q regarding the
[15:12:20 CDT(-0500)] <b-rock> "Unable to find User Profile for userId 0 and profile default"
[15:13:29 CDT(-0500)] <awills> yeah your stack trace tells me that method recursed 3 times trying to find the right profile to copy
[15:14:30 CDT(-0500)] <awills> "do I have one already, no so look at my parent... does he have one, no so look at his parent... that person appenars to be USER_ID=0, so exception"
[15:15:33 CDT(-0500)] <b-rock> ah ok.
[15:16:17 CDT(-0500)] <awills> did you post the file yet?
[15:16:45 CDT(-0500)] <b-rock> no. I'll do that now. you just want the fragment-layout and the .user file for the owner of the fragment?
[15:17:42 CDT(-0500)] <awills> yep... unless there is a separate .profile file for that account
[15:17:57 CDT(-0500)] <b-rock> I don't think so.
[15:18:15 CDT(-0500)] <awills> i wouldn;'t either
[15:26:18 CDT(-0500)] <b-rock> ok awills, I created the issue here with the .fragment-layout and .user files attached https://issues.jasig.org/browse/UP-2815
[15:30:05 CDT(-0500)] <EricDalquist> so lfuller1 haven't you been doing some load testing with 3.2 recently?
[15:30:44 CDT(-0500)] <lfuller1> no, 3.1.1 or something in that area.
[15:30:49 CDT(-0500)] <EricDalquist> ah ok
[15:30:59 CDT(-0500)] <lfuller1> Alex did 3.2, but claimed he found some data corruption... and punted.
[15:31:06 CDT(-0500)] <lfuller1> he did not give me any details though(sad)
[15:31:34 CDT(-0500)] <lfuller1> so we are on 3.1.x until such time as we get another iteration to upgrade/update the uportal version we are using.
[15:32:47 CDT(-0500)] <awills> b-rock i have them, they don't look off in any way
[15:32:56 CDT(-0500)] <awills> trying to import them now
[15:33:02 CDT(-0500)] <b-rock> ok
[15:33:12 CDT(-0500)] <b-rock> on a 3.2.2 build?
[15:33:19 CDT(-0500)] <b-rock> thats where I'm doing this.
[15:33:40 CDT(-0500)] <awills> but i think the issue might be "upstream" – like with fragmentTemplate
[15:33:44 CDT(-0500)] <awills> yeah on 3.2.2
[15:33:49 CDT(-0500)] <b-rock> ok
[15:40:38 CDT(-0500)] <awills> i think i see part of the issue b-rock
[15:41:00 CDT(-0500)] <b-rock> swell. How can I help you fix it?
[15:41:01 CDT(-0500)] <awills> can you change your -lo users to <default-user>defaultTemplateUser</default-user>
[15:41:12 CDT(-0500)] <awills> to see if it works?
[15:41:26 CDT(-0500)] <awills> from <default-user>fragmentTemplate</default-user>
[15:41:28 CDT(-0500)] <b-rock> ok. I'll do that. I just grepped and there are about 8 or 9 fragment useres with that setting
[15:41:39 CDT(-0500)] <b-rock> I'll change all of them.
[15:42:01 CDT(-0500)] <awills> cool, if it makes a difference I'll update the JIRA with the details
[15:42:28 CDT(-0500)] <b-rock> ok. I'll do the mod and then rerun the intidb
[15:42:34 CDT(-0500)] <EricDalquist> yay the patch applies cleanly!
[15:42:55 CDT(-0500)] <awills> grats eric
[15:43:22 CDT(-0500)] <EricDalquist> athena: do you know why the JS UIs for add portlets & manage portlets use XML instead of JSON?
[15:43:34 CDT(-0500)] <EricDalquist> just because that's what the JS already did?
[15:44:39 CDT(-0500)] <athena> hang on - presenting on a call
[15:45:51 CDT(-0500)] <EricDalquist> np
[15:54:52 CDT(-0500)] <awills> b-rock i got the same exception as you when i did a re-init including your file... 1 gave you 1 suggestion and tried the other approach myself... i changed the files extension for fragmentTemplate.user to .template-user, and it works... did it export as .user (not .template-user) from your 3.1.x?
[15:55:43 CDT(-0500)] <b-rock> oh ok good. the change from fragmentTemplate to defaultTemplateUser did not work for me.
[15:56:12 CDT(-0500)] <awills> hmmm... i thought either approach would work
[15:56:24 CDT(-0500)] <b-rock> I'll try the other approach now.
[15:57:14 CDT(-0500)] <awills> did it export as .user?
[15:57:40 CDT(-0500)] <b-rock> actually it is already called fragmentTemplate.template-user in the users folder.
[15:58:00 CDT(-0500)] <awills> that would be good, i think
[16:00:04 CDT(-0500)] <awills> can you pastebin the fragmentTemplate.layout & fragmentTemplate.user files?
[16:00:14 CDT(-0500)] <b-rock> sure.
[16:03:27 CDT(-0500)] <b-rock> awills here are the 2 files http://uportal.pastebin.com/fxjnkBw5
[16:05:59 CDT(-0500)] <b-rock> awills I've got to break from this chat now. can I try debugging this some more with you tomorrow?
[16:06:34 CDT(-0500)] <athena> EricDalquist: yes you're right that the JS uses XML just because that's what it did back when we used the channel registry directly
[16:06:45 CDT(-0500)] <athena> that's all changed in trunk - it's been refactored to use JSON
[16:06:47 CDT(-0500)] <EricDalquist> great
[16:06:55 CDT(-0500)] <EricDalquist> since that appears to be ~35% smaller
[16:07:03 CDT(-0500)] <athena> ya (smile)
[16:07:09 CDT(-0500)] <awills> i won't be on tomorrow actually
[16:07:13 CDT(-0500)] <athena> i was pretty happy about that part
[16:07:15 CDT(-0500)] <EricDalquist> so now the authz checks are the slowest part
[16:07:22 CDT(-0500)] <athena> and it's just sorta better now
[16:07:23 CDT(-0500)] <EricDalquist> but not much that can be easily done there
[16:07:29 CDT(-0500)] <athena> are they problematically slow?
[16:07:31 CDT(-0500)] <EricDalquist> no
[16:07:38 CDT(-0500)] <athena> ok, well that's good
[16:07:38 CDT(-0500)] <awills> but 1 more suggestion: try changing fragmentTemplate.layout --> fragmentTemplate.fragment-layout
[16:07:45 CDT(-0500)] <EricDalquist> average response times are now easily sub-second for 175 channels
[16:07:53 CDT(-0500)] <athena> good
[16:07:58 CDT(-0500)] <EricDalquist> portlet manager loaded that in ~ 750ms
[16:07:59 CDT(-0500)] <athena> definitely bad to have them be higher
[16:08:07 CDT(-0500)] <EricDalquist> used to take nearly 10 seconds
[16:08:07 CDT(-0500)] <b-rock> ok. I'll try that in the morning and report to the jira about what happens. thanks awills for your help today.
[16:08:15 CDT(-0500)] <EricDalquist> already applied the patch in 3.2
[16:08:16 CDT(-0500)] <awills> you bet, gl
[16:08:17 CDT(-0500)] <athena> i imagine that fix might improve some other things too?
[16:08:21 CDT(-0500)] <EricDalquist> likely yes
[16:08:24 CDT(-0500)] <athena> we do load up portlets a lot i'd think
[16:08:32 CDT(-0500)] <EricDalquist> I'm guessing every time a channel definition was loaded it would be affected by this
[16:08:36 CDT(-0500)] <athena> yeah
[16:08:40 CDT(-0500)] <EricDalquist> not even loaded
[16:08:48 CDT(-0500)] <EricDalquist> just accessed via the dao
[16:08:50 CDT(-0500)] <EricDalquist> so yeah
[16:08:59 CDT(-0500)] <athena> wonder if that explains some of the 3.1 -> 3.2 performance diffs lfuller1 and others saw
[16:09:00 CDT(-0500)] <EricDalquist> this will be a big fix
[16:09:02 CDT(-0500)] <EricDalquist> yup
[16:09:04 CDT(-0500)] <athena> awesome (smile)
[16:09:11 CDT(-0500)] <EricDalquist> I'm guessing it is the primary cause of that
[16:09:26 CDT(-0500)] <athena> should be great to get a patch release w/ that out
[16:09:34 CDT(-0500)] <EricDalquist> yeah
[16:09:45 CDT(-0500)] <EricDalquist> that plus lfuller1's change to not create new portlet entity objects for everyone
[16:10:34 CDT(-0500)] <athena> yeah
[16:10:39 CDT(-0500)] <athena> very cool (smile)
[16:12:43 CDT(-0500)] <EricDalquist> oh athena any new stuff you'd like in resource-server?
[16:12:52 CDT(-0500)] <EricDalquist> I think the ehcache bug has been fixed
[16:13:03 CDT(-0500)] <athena> yeah actually awills was asking today for a new release w/ the fluid patch releases
[16:13:04 CDT(-0500)] <athena> i
[16:13:35 CDT(-0500)] <athena> i owe him instructions today on actually generating the fluid file to get in there
[16:13:49 CDT(-0500)] <athena> and it would be great to have a release w/ that ehcache issue fixed
[16:14:13 CDT(-0500)] <EricDalquist> yeah, think you have time to do the fluid files today?
[16:14:37 CDT(-0500)] <athena> or at least send awills the instructions for doing it
[16:14:59 CDT(-0500)] <EricDalquist> think you could stick those on the resoure-server wiki page?
[16:15:09 CDT(-0500)] <athena> definitely
[16:15:14 CDT(-0500)] <EricDalquist> great
[16:15:17 CDT(-0500)] <athena> was discussing the need to publicly document that
[16:16:04 CDT(-0500)] <awills> i do think i can get on that today eric & athena
[16:16:10 CDT(-0500)] <awills> if I have instructions
[16:18:26 CDT(-0500)] * lfuller1 notes he has yet to check his changes in... Will do that first thing tomorrow I guess. has been crazy lately(sad)
[16:18:58 CDT(-0500)] <lfuller1> The other improvement you were talking about Eric would not impact 3.1, right?
[16:19:19 CDT(-0500)] <EricDalquist> no, the stuff I just fixed is 3.2 only
[16:20:55 CDT(-0500)] <lfuller1> k
[17:08:33 CDT(-0500)] <athena> ok awills
[17:08:52 CDT(-0500)] <athena> so for the custom fluid build we need to exclude jQuery, jQueryUICore, jQueryUIWidgets
[17:09:23 CDT(-0500)] <athena> in the past we also excluded bgiframe because we added that into jqueryui by hand, but it looks like that never happened for 1.8, so i guess we can just include it
[17:09:44 CDT(-0500)] <athena> the other thing i'd recommend is that we add the unminified version of fluid, not the minified one, maybe?
[17:10:02 CDT(-0500)] <athena> we used to do that, though it looks like the minified version got added for 1.2
[17:10:04 CDT(-0500)] <awills> yeah unminnified
[17:10:15 CDT(-0500)] <athena> not sure what happened w/ 1.2, but whatever
[17:10:17 CDT(-0500)] <athena> does that make sense?
[17:10:21 CDT(-0500)] * athena is really fried
[17:10:22 CDT(-0500)] <awills> we use the compressor to handle minnification at build time
[17:10:29 CDT(-0500)] <awills> mostly makes sense
[17:10:36 CDT(-0500)] <athena> yes, we do
[17:10:54 CDT(-0500)] <awills> it's extremely handly to be able to remove '.min' from a JSP somewhere and debug the JS
[17:11:01 CDT(-0500)] <athena> so like -Dexcludything="jQuery,jQueryUICore,jQueryUIWidgets", i think
[17:11:04 CDT(-0500)] <athena> yes, i totally agree
[17:11:08 CDT(-0500)] <athena> giant pain when we can't
[17:11:13 CDT(-0500)] <awills> yes
[17:11:13 CDT(-0500)] <athena> useless errors
[17:11:17 CDT(-0500)] <athena> no breakpoints
[17:11:18 CDT(-0500)] <athena> rrarrrr
[17:11:24 CDT(-0500)] <awills> lol
[17:11:49 CDT(-0500)] <awills> ok, i can get on that today i think
[17:11:51 CDT(-0500)] <athena> so adding the 1.1.3 and 1.2.1 tags would be super-awesome
[17:11:53 CDT(-0500)] <athena> you rock
[17:12:18 CDT(-0500)] <athena> hmmm
[17:12:26 CDT(-0500)] <athena> looks like we still need to update ehcache
[17:12:46 CDT(-0500)] <athena> there's a crappy bug that prevents the resources from being cached properly, which sorta defeats the point
[17:13:30 CDT(-0500)] <awills> yeah sounds like it