[09:23:27 CST(-0600)] <jwennmacher> Hi Eric.
[09:23:45 CST(-0600)] <EricDalquist> hello
[09:24:27 CST(-0600)] <jwennmacher> I was thinking about the event aggregation issue Anthony reported. I think both he and the person at the Apereo conference indicated their raw events table was growing at around 1M rows a day.
[09:25:35 CST(-0600)] <jwennmacher> I'm not sure if this is that there were 1M inserts or the net is a growth of 1M rows. I just did a rough calculation. If the aggregation is running every 20 minutes and it does at most 10,000 aggregations, it will aggregate at most 720,000 rows in a day.
[09:26:02 CST(-0600)] <EricDalquist> well it runs ever 60 seconds
[09:26:19 CST(-0600)] <EricDalquist> and it catch-up mode (like there were more events to aggregate than aggregated) it should run continueously
[09:26:25 CST(-0600)] <EricDalquist> the problem could be aggregation rate
[09:26:40 CST(-0600)] <EricDalquist> one thing I thought about is that I test this all on oracle
[09:26:50 CST(-0600)] <EricDalquist> which has different index behavior than some other DBs
[09:26:59 CST(-0600)] <EricDalquist> like FKs in oracle are indexes
[09:27:07 CST(-0600)] <EricDalquist> but in some other DBs explicit indexes are also needed
[09:27:21 CST(-0600)] <EricDalquist> that is why I'm interested in seeing a chunk from his 4.0.10 log
[09:27:30 CST(-0600)] <EricDalquist> to see if the aggr is running in catch up mode
[09:27:38 CST(-0600)] <EricDalquist> and if it is what the aggregation rate is
[09:28:05 CST(-0600)] <EricDalquist> next steps would be to filter some of the raw events before they persist
[09:28:11 CST(-0600)] <EricDalquist> or just truncate the raw events table
[09:28:20 CST(-0600)] <EricDalquist> and see if it can keep up with a fresh data set
[09:29:04 CST(-0600)] <jwennmacher> Makes sense.
[09:29:31 CST(-0600)] <jwennmacher> In the log he sent it looked like aggregation was running every 20 minutes or am I misinterpreting the log statements??
[09:30:03 CST(-0600)] <EricDalquist> my guess is that it was completing 10k events every 20 minutes
[09:30:13 CST(-0600)] <EricDalquist> it can't run concurrently
[09:30:23 CST(-0600)] <EricDalquist> so if it starts processing
[09:30:51 CST(-0600)] <EricDalquist> and it takes 20 minutes to process the 10k
[09:31:03 CST(-0600)] <EricDalquist> then you only see one aggregation message every 20 minutes
[09:31:50 CST(-0600)] <jwennmacher> How long does it take to process 10K events at your site?
[09:38:30 CST(-0600)] <EricDalquist> let me check ...
[09:39:59 CST(-0600)] <EricDalquist> we're all caught up so I'm trying to find the biggest interval
[09:40:11 CST(-0600)] <EricDalquist> right now its only doing about 1500 events aggregation run
[09:40:15 CST(-0600)] <EricDalquist> since that is all there is to process
[09:40:52 CST(-0600)] <jwennmacher> In the line Aggregated 10000 events created at 15.5280 events/second between 2013-01-10T22:02:11.000Z and 2013-01-10T22:12:55.595Z in 1311626ms - 7.6241 e/s a 0.4910x speedup
[09:41:02 CST(-0600)] <EricDalquist> Aggregated 2249 events created at 18.5868 events/second between 2013-02-19T13:52:37.960-06:00 and 2013-02-19T13:54:38.994-06:00 in 44113ms - 50.9827 e/s a 2.7430x speedup.
[09:41:11 CST(-0600)] <jwennmacher> what is the 15.5280 events/sec calculation mean?
[09:41:26 CST(-0600)] <jwennmacher> and the 7.6241 e/s?
[09:41:57 CST(-0600)] <EricDalquist> so that first number is 10000 / (2013-01-10T22:12:55.595Z - 2013-01-10T22:02:11.000Z)
[09:42:14 CST(-0600)] <EricDalquist> so for the time range the aggr ran events were created at that rate
[09:42:35 CST(-0600)] <EricDalquist> and then the 2nd number is just 10000/1311.626
[09:42:59 CST(-0600)] <EricDalquist> so the 1311626ms it took to process 10k events works out to 7.6241
[09:43:05 CST(-0600)] <EricDalquist> and this is the problem he has
[09:43:35 CST(-0600)] <EricDalquist> Aggregated 2718 events created at 14.9341 events/second between 2013-02-19T14:55:31.684-06:00 and 2013-02-19T14:58:33.908-06:00 in 46410ms - 58.5650 e/s a 3.9216x speedup.
[09:46:43 CST(-0600)] <jwennmacher> I see. Big difference in performance between yours and his.
[09:46:48 CST(-0600)] <EricDalquist> yes
[09:47:06 CST(-0600)] <EricDalquist> I think we might be at the point where he tries truncating the raw events table
[09:47:11 CST(-0600)] <EricDalquist> and seeing if that makes a difference
[09:47:19 CST(-0600)] <EricDalquist> he's on mysql right?
[09:47:29 CST(-0600)] <EricDalquist> I wonder if there are some other performance tweaks needed
[09:47:29 CST(-0600)] <jwennmacher> One interesting thing I see in Anthony's data is that the time reported for aggregation is fairly constant; e.g. 880s - 1311s when processing 10,000 or 2800 events. I could see that possibly being related to db performance.
[09:47:42 CST(-0600)] <EricDalquist> yeah
[09:47:58 CST(-0600)] <jwennmacher> I don't know if he's on mysql
[09:49:03 CST(-0600)] <EricDalquist> I believe he is
[09:49:07 CST(-0600)] <EricDalquist> since we had to fix a db bug
[09:49:11 CST(-0600)] <EricDalquist> for mysql and floats
[10:11:29 CST(-0600)] <mmoayyed> Hi Eric. Are u there? 
[10:11:44 CST(-0600)] <EricDalquist> yes
[10:12:14 CST(-0600)] <mmoayyed> hey there, question for you: could you plz allow me and matt polizzotti to have commit rights to the phonegap repo?
[10:12:23 CST(-0600)] <mmoayyed> https://github.com/Jasig/umobile-app-phonegap
[10:12:23 CST(-0600)] <EricDalquist> sure
[10:12:30 CST(-0600)] <mmoayyed> tnx!
[10:12:39 CST(-0600)] <EricDalquist> github usernames?
[10:13:08 CST(-0600)] <mpolizzotti> mpolizzotti
[10:13:26 CST(-0600)] <mmoayyed> mmoayyed
[10:13:27 CST(-0600)] <mmoayyed> yup
[10:14:15 CST(-0600)] <EricDalquist> done
[10:14:45 CST(-0600)] <mpolizzotti> Thanks.
[10:15:30 CST(-0600)] <mmoayyed> tnx. one more question: do you know where the code lines on github for the android-umobile-plugin project?
[10:15:34 CST(-0600)] <mmoayyed> lives*
[10:15:46 CST(-0600)] <EricDalquist> nope
[10:16:19 CST(-0600)] <mmoayyed> hmmm{color}
[10:16:31 CST(-0600)] <mmoayyed> I think i might have to download the src from sonatype!
[11:49:23 CST(-0600)] <cwaymire> EricDalquist: hey. you around and got a few minutes?
[12:12:02 CST(-0600)] <EricDalquist> back now cwaymire
[12:14:04 CST(-0600)] <cwaymire> EricDalquist: Hey. Working on an "issue" related to session timeout between the portal and a portlet and looking for some input. Basically the situation I am dealing with is on SSP and the fact that the workflow within it is primarily via api calls that do not go through the portal container which ultimately results in the uportal session timing out.
[12:14:54 CST(-0600)] <cwaymire> EricDalquist: The initial "quick fix" was to simply create a filter that maps to the api calls and makes a restful call back to uportal and "touches" the session thus keeping it alive. It's kind of dirty and not very efficient but it seems to work
[12:15:10 CST(-0600)] <cwaymire> just wondering if theres a cleaner/more efficient method to achieve that
[12:16:10 CST(-0600)] <cwaymire> one thought was to enable crossContext in tomcat to expose some sort of handler to be able to update the session timer but havent had time to think through the implications or issues with that.
[12:19:48 CST(-0600)] <EricDalquist> what do these api calls go to?
[12:20:15 CST(-0600)] <cwaymire> make service calls that implement the SSP workflow
[12:20:30 CST(-0600)] <EricDalquist> ok ... so where does that code live?
[12:20:32 CST(-0600)] <EricDalquist> is that a portlet
[12:20:34 CST(-0600)] <EricDalquist> or a servlet
[12:21:05 CST(-0600)] <cwaymire> it's in the SSP portlet, but all of these are just in standard Spring MVC controllers
[12:21:17 CST(-0600)] <cwaymire> the actual portlet just loads up the main display page
[12:21:20 CST(-0600)] <EricDalquist> so it is a portlet/servlet hybrid app
[12:21:23 CST(-0600)] <cwaymire> yeah
[12:21:26 CST(-0600)] <EricDalquist> ah
[12:21:27 CST(-0600)] <EricDalquist> yeah
[12:21:31 CST(-0600)] <EricDalquist> there isn't a great way to fix that
[12:21:33 CST(-0600)] <cwaymire> all of the functioanlity is in the servlet part
[12:21:44 CST(-0600)] <EricDalquist> I mean the best would be to turn that all into portlet resource controllers
[12:21:48 CST(-0600)] <EricDalquist> instead of servlet controllers
[12:21:52 CST(-0600)] <EricDalquist> then you go through the portal
[12:21:54 CST(-0600)] <cwaymire> that was suggested. they are reluctant/resistant to migrating the functionality
[12:22:02 CST(-0600)] <EricDalquist> but other than that ...
[12:22:35 CST(-0600)] <EricDalquist> stick some JS in the portal theme that pings a keep-alive controller in uportal
[12:23:20 CST(-0600)] <cwaymire> The reason I was reluctant to do something like that is it nullifies timeout
[12:23:32 CST(-0600)] <EricDalquist> maybe you can have fancy js
[12:23:39 CST(-0600)] <EricDalquist> that listens for any user interaction
[12:23:50 CST(-0600)] <EricDalquist> and only pings if it has seen interaction int he last X minutes
[12:24:51 CST(-0600)] <cwaymire> Yeah. I also thought of adding some logic like that to what I already have. Right now anytime an SSP api call is made it uses the cross context restful api invoker to make a call back to the /session api method on uportal.
[12:25:29 CST(-0600)] <EricDalquist> yeah
[12:25:32 CST(-0600)] <EricDalquist> that seems reasonable as well
[12:25:36 CST(-0600)] <EricDalquist> I mean it is all hackish
[12:25:41 CST(-0600)] <EricDalquist> short of actually making this stuff portlets
[12:26:12 CST(-0600)] <cwaymire> yeah. ok thanks. I havent been able to come up with a better approach and wanted to see if you had any suggestions before moving forward with this.
[12:26:40 CST(-0600)] <EricDalquist> sorry, nothing that isn't hackish
[12:26:45 CST(-0600)] <cwaymire> heheh
[14:12:09 CST(-0600)] <bnason> I have a multi tiered setup and would like to add the text "Tier #" below the copyright on the bottom left. Could anyone point me in the right direction to do this?
[14:12:28 CST(-0600)] <EricDalquist> what version of uPortal are you using?
[14:14:04 CST(-0600)] <bnason> hrm, im not sure, its technically luminis 4.3.0.81
[14:14:21 CST(-0600)] <EricDalquist> no idea then sorry :/
[14:14:37 CST(-0600)] <EricDalquist> if you can let us know the uPortal version we might be abkle to help
[14:14:43 CST(-0600)] <bnason> let me try to see if I can find a version ={color}
[14:14:44 CST(-0600)] <bnason> k
[14:15:16 CST(-0600)] <bnason> Is there stored anywhere within uPortal code?
[14:15:44 CST(-0600)] <EricDalquist> not really ...
[14:15:56 CST(-0600)] <EricDalquist> I'm guessing it isn't uPortal 4 based though
[14:16:00 CST(-0600)] <EricDalquist> which makes it a lot harder
[14:20:11 CST(-0600)] <bnason> When was 4 released?
[14:20:45 CST(-0600)] <EricDalquist> 4.0.0 was released Aug 31 2011
[14:21:14 CST(-0600)] <bnason> ah, it appears version 2.5
[14:21:40 CST(-0600)] <EricDalquist> yeah I'm not sure how to do it in 2.5 easily
[14:22:06 CST(-0600)] <EricDalquist> off the top of my head ... you'd have to modify the rendering pipeline class and have it add some additional info into the theme XSLT
[14:22:16 CST(-0600)] <EricDalquist> or figure out how to have a different theme XSL on each server
[14:22:36 CST(-0600)] <EricDalquist> guess that 2nd option might be easier depending on how you deploy your code to the servers
[14:23:04 CST(-0600)] <bnason> yea currently not easy with out deployment setup
[14:23:08 CST(-0600)] <bnason> s/out/our/
[14:24:27 CST(-0600)] <bnason> I found some of their official documentation, so Ill take a look at these few hundred pages ha
[14:24:56 CST(-0600)] <EricDalquist> yeah I doubt it is in their
[14:25:03 CST(-0600)] <EricDalquist> that just wasn't an easy thing to do in 2.x
[14:25:06 CST(-0600)] <EricDalquist> or 3.x really
[14:25:13 CST(-0600)] <EricDalquist> it comes as part of uPortal 4 :/