[10:08:31 CST(-0600)] <EricDalquist> Arvids: you there?
[10:16:39 CST(-0600)] <dd> hey @EricDalquist
[10:16:44 CST(-0600)] <EricDalquist> hi'
[10:17:08 CST(-0600)] <dd> i'm trying to remove the default welcome portlet (the one with tabs) from the welcome fragment
[10:17:15 CST(-0600)] <dd> you answered here: http://jasig.275507.n4.nabble.com/Delete-default-welcome-channel-td1017064.html
[10:17:30 CST(-0600)] <dd> but i don't get a delete option when i am using the welcome fragment user
[10:18:09 CST(-0600)] <EricDalquist> hrm
[10:18:18 CST(-0600)] <EricDalquist> what version of uPortal are you on?
[10:18:23 CST(-0600)] <dd> 3.2.4
[10:18:32 CST(-0600)] <EricDalquist> so I think there is a bug in the UI
[10:18:41 CST(-0600)] <EricDalquist> you first need to edit the settings for that portlet
[10:18:45 CST(-0600)] <EricDalquist> and make it deletable
[10:18:48 CST(-0600)] <EricDalquist> then you can delete it
[10:19:07 CST(-0600)] <EricDalquist> I don't have a running instance of 3.2 handy to give you a more detailed description than that
[10:19:35 CST(-0600)] <dd> hmm, do you know what option needs to be enabled to make it deleteable?
[10:19:47 CST(-0600)] <dd> i see edit
[10:19:54 CST(-0600)] <dd> but that doesn't seem to do anything
[10:19:56 CST(-0600)] <EricDalquist> athena: are you there?
[10:21:16 CST(-0600)] <EricDalquist> I'm trying to remember where it is ...
[10:21:29 CST(-0600)] <EricDalquist> is there any sort of "edit layout" button anywhere?
[10:23:06 CST(-0600)] <dd> there is page layout
[10:23:13 CST(-0600)] <dd> but that is for columns
[10:23:25 CST(-0600)] <EricDalquist> that might be it
[10:23:26 CST(-0600)] <EricDalquist> go to that
[10:23:38 CST(-0600)] <EricDalquist> and for each portlet do you see a little pencil/paper icon?
[10:23:47 CST(-0600)] <EricDalquist> or just try clicking on the portlet title
[10:25:11 CST(-0600)] <dd> ah, ok. it was delete channel to make the delete icon show up
[10:25:18 CST(-0600)] <EricDalquist> yweah
[10:29:59 CST(-0600)] <dd> on Bookmarks Portlet 1.0.10, is there any way to set default bookmarks for all users?
[10:30:08 CST(-0600)] <dd> or users in certain groups
[10:30:13 CST(-0600)] <EricDalquist> I can't remember
[10:30:20 CST(-0600)] <EricDalquist> that would be a good question for the email list
[10:38:29 CST(-0600)] <athena> in 3.2 i don't think there's a way other than the evil legacy customization channel to set fragment permissions
[10:38:35 CST(-0600)] <athena> adding that to the UI was something we did for 4
[10:43:59 CST(-0600)] <Arvids> Hi, EricDalquist
[10:45:44 CST(-0600)] <Arvids> Today I noticed that there something must be done for selects like: http://pastebin.com/HgMawLPT
[10:46:12 CST(-0600)] <Arvids> Is there a way to set fetching strategies using JPA?
[10:46:45 CST(-0600)] <EricDalquist> ?
[10:46:54 CST(-0600)] <EricDalquist> so the sheer number of queries?
[10:47:01 CST(-0600)] <Arvids> yes
[10:47:20 CST(-0600)] <Arvids> I´d like to fetch those entities using join´s, where possible
[10:47:25 CST(-0600)] <EricDalquist> yeah, so the problem there is portal architecture
[10:47:39 CST(-0600)] <EricDalquist> there is nothing to join from
[10:47:48 CST(-0600)] <EricDalquist> it would be the portal layout node
[10:47:54 CST(-0600)] <EricDalquist> but that isn't using JPA yet
[10:48:03 CST(-0600)] <EricDalquist> that is manual SQL in RDBMDistributedLayoutStore
[10:48:15 CST(-0600)] <Arvids> ok
[10:48:32 CST(-0600)] <EricDalquist> the other thing that could help performance would be batching the load
[10:48:47 CST(-0600)] <EricDalquist> but that isn't how the portal it architected
[10:48:58 CST(-0600)] <EricDalquist> it is very "on demand" for loading this stuff
[10:49:07 CST(-0600)] <Arvids> Today i managed to cut down some more miliseconds upon login, but those batches of queries concern me
[10:49:09 CST(-0600)] <EricDalquist> "I need the entity or user X and layout node Y"
[10:49:41 CST(-0600)] <EricDalquist> there are some thinks I'm working on with the hibernate folks that would be a huge help to our performance
[10:49:49 CST(-0600)] <EricDalquist> but that won't be avaialble in hibernate until 4.1
[10:50:10 CST(-0600)] <EricDalquist> https://hibernate.onjira.com/browse/HHH-2879
[10:50:17 CST(-0600)] <Arvids> btw, switching from 3.x to 4.x (hibernate) is painful?
[10:50:36 CST(-0600)] <EricDalquist> I don't think it will be without work
[10:50:46 CST(-0600)] <EricDalquist> but I'd be surprised if it is more than 1-2 days of work
[10:50:52 CST(-0600)] <EricDalquist> can't do that until uP 4.1 though
[10:51:18 CST(-0600)] <EricDalquist> if HHH-2879 (which I've partially completed: https://github.com/edalquist/hibernate-core/compare/master...HHH-2879-nocaching) gets done
[10:51:31 CST(-0600)] <EricDalquist> we could do a single "load all portlet entities for user X" on user login
[10:51:46 CST(-0600)] <Arvids> that would be really nice
[10:51:54 CST(-0600)] <EricDalquist> all the entities would have the naturalId cache populated with (userId, nodeId)->entityId
[10:52:13 CST(-0600)] <EricDalquist> and then all the queries by userId and nodeId wouldn't need to hit the DB at all
[10:52:49 CST(-0600)] <EricDalquist> yes
[10:52:56 CST(-0600)] <Arvids> besides those query batches, i see no real big places where to improve performance
[10:53:17 CST(-0600)] <EricDalquist> I'm sure there are others
[10:53:28 CST(-0600)] <EricDalquist> but they are likely burried in the really old user/layout/auth code
[10:53:39 CST(-0600)] <EricDalquist> there is some poor use of synchronization in there
[10:54:01 CST(-0600)] <Arvids> i´m not so sure about that
[10:54:18 CST(-0600)] <EricDalquist> well ... for creating new users there
[10:54:18 CST(-0600)] <Arvids> ... at least in terms of performance there´s no much places where to look
[10:54:19 CST(-0600)] <EricDalquist> is
[10:54:30 CST(-0600)] <EricDalquist> so the first time user X ever logs in
[10:54:35 CST(-0600)] <Arvids> ahh... that´s out of the scope of my research
[10:54:41 CST(-0600)] <EricDalquist> there is a bunch of stuff done that is sub-optimal
[10:55:02 CST(-0600)] <EricDalquist> the portlet entity lookups do suck though
[10:55:02 CST(-0600)] <Arvids> well... that´s the first time and it is bearable
[10:58:50 CST(-0600)] <EricDalquist> another thing we talked about at the last conference is a need to re-write person directory
[10:58:55 CST(-0600)] <EricDalquist> build a real query API
[10:58:59 CST(-0600)] <EricDalquist> simplify config
[10:59:12 CST(-0600)] <EricDalquist> and most importantly add the ability to query attribute sources concurrently
[11:00:03 CST(-0600)] <Arvids> That could cut only something like 20-60ms
[11:00:15 CST(-0600)] <EricDalquist> well ... that depends on your complexity
[11:00:33 CST(-0600)] <EricDalquist> we query between 4 and 10 attribute sources at login
[11:00:39 CST(-0600)] <EricDalquist> depending on who you are
[11:00:50 CST(-0600)] <Arvids> yeah, my config is really simple - only LDAP and local DB
[11:01:35 CST(-0600)] <EricDalquist> if the portlet entity lookup is REALLY the biggest issue you could fake the naturalid lookup logic locally
[11:01:52 CST(-0600)] <EricDalquist> inject a new ehcache instance into portlet entity registry
[11:02:01 CST(-0600)] <Arvids> One place, where some improvements can be done, is when XML documents are being merged into final layout
[11:02:40 CST(-0600)] <EricDalquist> on user login run a query to load all entities for user in a single query
[11:02:54 CST(-0600)] <Arvids> hrmm... sounds like a good idea
[11:02:57 CST(-0600)] <EricDalquist> store (userId, layoutId, nodeId) -> entityId in the cache
[11:03:23 CST(-0600)] <EricDalquist> then in the entity registry modify the method that queries by (userId, layoutId, nodeId) to look in that cache for the entityId first
[11:03:35 CST(-0600)] <EricDalquist> if that exists load the entity by id (that goes directly against the 2nd level cache)
[11:03:55 CST(-0600)] <EricDalquist> if not fall back to loading by user/layout/node
[11:04:03 CST(-0600)] <EricDalquist> you could inject 2 caches
[11:04:19 CST(-0600)] <EricDalquist> and have the 2nd cache misses or some such
[11:04:51 CST(-0600)] <Arvids> i believe, the same should be done for retrieving stylesheet user preferences
[11:05:19 CST(-0600)] <EricDalquist> yeah
[11:05:30 CST(-0600)] <EricDalquist> again, once we get the layouts into a JPA data model
[11:05:49 CST(-0600)] <EricDalquist> both the node specific stylesheet preferences and portlet entities can have real relations
[11:06:09 CST(-0600)] <EricDalquist> having the user managed by jpa will do the same
[11:06:21 CST(-0600)] <EricDalquist> we can have the user object have bag references to the other types
[11:06:25 CST(-0600)] <EricDalquist> and load them all when the user logs in
[11:06:35 CST(-0600)] <Arvids> ... and one more hotspot - XPath evaluation
[11:07:06 CST(-0600)] <Arvids> I inserted some hand-written code in order to evaluate XPath expression evaluation performance
[11:07:35 CST(-0600)] <Arvids> for example: Evaluated expression '/layout/folder/folder[@ID=$nodeId or descendant::node()[@ID=$nodeId]]/@ID' in 42ms
[11:08:21 CST(-0600)] <EricDalquist> wow
[11:08:24 CST(-0600)] <EricDalquist> just evaluation?
[11:08:26 CST(-0600)] <Arvids> could it be because of lazy loading some entities when visiting nodes... or evaluator is slow?
[11:08:47 CST(-0600)] <EricDalquist> I believe that is running against a fully loaded XML document
[11:09:04 CST(-0600)] <EricDalquist> that's crazy
[11:09:06 CST(-0600)] <Arvids> hmm... then exclude lazy-loading as a cause
[11:09:10 CST(-0600)] <EricDalquist> these aren't huge documents
[11:09:15 CST(-0600)] <Arvids> indeed
[11:09:37 CST(-0600)] <EricDalquist> I mean it does potentially have to do a depth-first search of the tree
[11:09:43 CST(-0600)] <Arvids> but that one expression takes between 1 and 60 ms
[11:09:46 CST(-0600)] <EricDalquist> but with < 100 nodes that should be fast
[11:09:58 CST(-0600)] <EricDalquist> I wonder if there is just something about that xpath that is bad
[11:10:28 CST(-0600)] <Arvids> that´s one of the surprising things - here´s the same expression, but very different evaluation time: Evaluated expression '/layout/folder/folder[@ID=$nodeId or descendant::node()[@ID=$nodeId]]/@ID' in 1ms
[11:11:20 CST(-0600)] <EricDalquist> and that is only timing the call to evaluate()?
[11:11:28 CST(-0600)] <Arvids> yes
[11:12:00 CST(-0600)] <Arvids> i modified a utility class
[11:12:16 CST(-0600)] <Arvids> (don´t remember the exact name)
[11:12:39 CST(-0600)] <Arvids> lol... it´s right in front of me: XPathPoolImpl
[11:12:45 CST(-0600)] <EricDalquist> right
[11:12:58 CST(-0600)] <EricDalquist> the goal of that thing was to try and make xpath faster 
[11:13:11 CST(-0600)] <EricDalquist> since the JDK doesn't allow thread-safe reuse of compiled expressions
[11:13:15 CST(-0600)] <EricDalquist> which is dumb
[11:13:27 CST(-0600)] <EricDalquist> and unfortunatly none of the 3rd party java XML apis have kept up
[11:13:41 CST(-0600)] <EricDalquist> so they all cause problems if you use jaxp like we do
[11:14:14 CST(-0600)] <Arvids> yeah, such extensive use of XML requires really high quality libraries
[11:14:59 CST(-0600)] <Arvids> i was wondering - whether handwritten method that mimics the XPath expression could be faster
[11:15:04 CST(-0600)] <EricDalquist> it could be
[11:15:23 CST(-0600)] <EricDalquist> perhaps try using xpath to find the node with @ID=$nodeId
[11:15:37 CST(-0600)] <EricDalquist> then look up from there to find the tab folder
[11:15:56 CST(-0600)] <EricDalquist> in theory we could have @ID treated by the XML APIs as the elementID
[11:16:06 CST(-0600)] <EricDalquist> but I think we have to associate a XSD with the layout XML to do that
[11:17:31 CST(-0600)] <Arvids> can you explain in some word what "descendant::node()[@ID=$nodeId]" expression means?
[11:18:04 CST(-0600)] <Arvids> because elarier today i was on my way to implement the hand-written search, but couldn´t figure out the last part of that expression
[11:18:21 CST(-0600)] <EricDalquist> find the decended of the current node with attribute ID equals to the $nodeId variable
[11:18:48 CST(-0600)] <EricDalquist> the goal of that method is that the portal knows what the @ID of the layout node targeted by the request is
[11:18:59 CST(-0600)] <EricDalquist> and we need to determine which tab is the parent of that node
[11:19:34 CST(-0600)] <Arvids> so basically "folder[@ID=$nodeId or descendant::node()[@ID=$nodeId]]" means find a node or its subnode whose ID is with value $nodeId
[11:19:44 CST(-0600)] <EricDalquist> yes
[11:19:49 CST(-0600)] <Arvids> got it
[11:20:26 CST(-0600)] <EricDalquist> and it starts with /layout/folder/folder since for the tab/column layout the 2nd level of folder elements are the tabs
[11:20:48 CST(-0600)] <Arvids> yeah, that part was clear for me
[11:21:22 CST(-0600)] <Arvids> another example: Evaluated expression '//channel[@fname=$fname]' in 8ms
[11:21:42 CST(-0600)] <EricDalquist> so // means any channel element in the document
[11:21:51 CST(-0600)] <Arvids> 8ms isn´t too much, but it seems that sums up
[11:22:03 CST(-0600)] <Arvids> ... and makes around 30ms
[11:22:28 CST(-0600)] <EricDalquist> yeah that might be easy enough to fix with a local cache
[11:22:45 CST(-0600)] <EricDalquist> since it appears to be run from DistributedLayoutManager.getSubscribeId
[11:23:22 CST(-0600)] <EricDalquist> like move that logic down into DistributedUserLayout
[11:23:32 CST(-0600)] <EricDalquist> and cache the fname->id in a Map
[11:23:40 CST(-0600)] <EricDalquist> which is invalidated when the layout changes
[11:24:27 CST(-0600)] <Arvids> ok, that will be sufficient for today
[11:24:38 CST(-0600)] <Arvids> thre´ll be a lot of work tomorrow 
[11:25:43 CST(-0600)] <EricDalquist>
[11:25:48 CST(-0600)] <Arvids> still... now I understand why performance engineers are so highly appraised
[11:26:03 CST(-0600)] <EricDalquist>
[11:26:10 CST(-0600)] <EricDalquist> well have fun with yourkit as well
[11:26:19 CST(-0600)] <Arvids> yeah, have to get used to it
[11:26:26 CST(-0600)] <EricDalquist> it should give you a lot more insight into what is going on
[11:26:52 CST(-0600)] <Arvids> ok, have to go now
[11:38:49 CST(-0600)] <EricDalquist> blarg
[11:39:08 CST(-0600)] <EricDalquist> I forgot about having to setup a group mapping table for event aggregation
[11:39:16 CST(-0600)] <EricDalquist> and we'll have to do the same for portlet definitions
[11:43:43 CST(-0600)] <athena> yeah
[11:44:21 CST(-0600)] <EricDalquist> so my plan is that during aggregation I'll take a group key (local.32) lookup the group name
[11:44:50 CST(-0600)] <EricDalquist> and the aggregate events db will have a table which maps groupstore+groupname to a synthetic id
[11:44:55 CST(-0600)] <EricDalquist> which will be used when aggregating
[11:45:10 CST(-0600)] <EricDalquist> so that the aggregates are not dependent on the actual group-keys over time
[11:49:48 CST(-0600)] <EricDalquist> athena: would it be the end of the world to have a flat-file for configuring quarters and terms for now?
[11:50:06 CST(-0600)] <EricDalquist> I'm trying to get this stats stuff buttoned up asap
[11:50:19 CST(-0600)] <EricDalquist> and doing those in the db is going to add an extra day or two
[11:51:55 CST(-0600)] <athena> i think that is not hte end of the world
[11:52:02 CST(-0600)] <athena> having db-based seems better in the long run
[11:52:08 CST(-0600)] <athena> but like, this is way better than nothing
[11:52:21 CST(-0600)] <athena> and most schools know their semesters for years in advance, really
[11:52:40 CST(-0600)] <EricDalquist> yeah
[12:03:54 CST(-0600)] <holdorph> speaking from experience with sakai
[12:04:05 CST(-0600)] <holdorph> it's a bad idea to do that either way
[12:04:18 CST(-0600)] <holdorph> it's always something gets forgotten until it's broken
[12:04:31 CST(-0600)] <holdorph> and while broken certainly reminds you something has to get done
[12:04:36 CST(-0600)] <EricDalquist> to do what either way?
[12:04:39 CST(-0600)] <holdorph> it doesn't change the fact that broken production sucks
[12:04:54 CST(-0600)] <holdorph> requiring semesters to have to be configured by schools
[12:05:07 CST(-0600)] <EricDalquist> ah, it doesn't require it
[12:05:15 CST(-0600)] <EricDalquist> you just don't get term based stats unless you configure it
[12:05:18 CST(-0600)] <holdorph> it's a requirement in sakai or the sakai course site type doesn't work
[12:05:19 CST(-0600)] <EricDalquist> everything else works
[12:06:01 CST(-0600)] <holdorph> ok, but the point is, they won't ever get around to configuring the file until they decide, hmmm.... term based stats aren't working.
[12:06:13 CST(-0600)] <EricDalquist> right
[12:06:19 CST(-0600)] <holdorph> at which point a file is, eh, ok, but if they're looking at the stats in the UI
[12:06:44 CST(-0600)] <holdorph> and they decide they'd like term based stats.... configuring them in the same UI would be better.
[12:07:04 CST(-0600)] <EricDalquist> yeah
[12:07:06 CST(-0600)] <holdorph> but if there's absolutely no UI for the configuration at all, ever... then there's not much difference
[12:07:24 CST(-0600)] <holdorph> but rest assured, it's something that will be a source of 'support' on the mailing lists
[12:07:37 CST(-0600)] <holdorph> it comes up in sakai on a very very very regular basis
[12:08:02 CST(-0600)] <EricDalquist>
[12:08:11 CST(-0600)] <EricDalquist> well and the fun answer with the stats
[12:08:31 CST(-0600)] <EricDalquist> is that you will only be able to get term based stats data from when you configure terms going forward
[12:08:46 CST(-0600)] <EricDalquist> there isn't really the ability to re-run old data
[12:08:56 CST(-0600)] <EricDalquist> since the raw data gets purged