[11:14:08 CDT(-0500)] <EricDalquist> hi there athena
[11:17:41 CDT(-0500)] * athena waves
[11:19:50 CDT(-0500)] <athena> so . . . stats?
[11:29:02 CDT(-0500)] <EricDalquist> yes
[11:29:09 CDT(-0500)] <EricDalquist> just had a chat with our DB
[11:29:10 CDT(-0500)] <EricDalquist> DBA
[11:29:25 CDT(-0500)] <EricDalquist> so my question is "how useful are the relational raw stats tables"
[11:30:14 CDT(-0500)] <EricDalquist> with the though that we could store more event data with less work by having a raw_stats table with just 3 columns: id, timestamp, data_clob
[11:31:07 CDT(-0500)] <athena> i think mostly we'd just want to do reporting on the aggregated tables, right?
[11:31:13 CDT(-0500)] <athena> that seems like the more usual thing to do
[11:31:16 CDT(-0500)] <EricDalquist> right
[11:31:21 CDT(-0500)] <EricDalquist> this is part of the idea that we're building aggregation in with uportal itself
[11:31:27 CDT(-0500)] <athena> yeah
[11:31:36 CDT(-0500)] <EricDalquist> so when the DB event DAO would persist a list of events
[11:31:38 CDT(-0500)] <athena> i don't think we'd want to build reporting tools against the raw stats
[11:31:45 CDT(-0500)] <athena> so we could just maybe consider that table 100% internal?
[11:31:52 CDT(-0500)] <EricDalquist> by serializing them to a string either via JAXB or jackson
[11:32:31 CDT(-0500)] <EricDalquist> then we can easily add new data fields to the aggregate data
[11:32:35 CDT(-0500)] <EricDalquist> yeah
[11:32:41 CDT(-0500)] <EricDalquist> I'm thinking that is a good idea
[11:32:51 CDT(-0500)] <EricDalquist> where that table gets written to by all servers in a cluster
[11:33:12 CDT(-0500)] <EricDalquist> then the aggregation runs on events older than (now - 1 minute)
[11:33:32 CDT(-0500)] <athena> sounds good to me 
[11:33:39 CDT(-0500)] <EricDalquist> and immediately deletes aggregated events
[11:34:18 CDT(-0500)] <EricDalquist> that should greatly simplify some things
[11:34:25 CDT(-0500)] <EricDalquist> and let us store more data in the event objects
[11:34:40 CDT(-0500)] <EricDalquist> I should have the raw events stuff merged in this weekend
[11:34:57 CDT(-0500)] <EricDalquist> and I'll get the aggregator framework in place as well
[11:35:16 CDT(-0500)] <EricDalquist> then it will just be the work of writing aggregator plugins for each report type we want to have
[11:35:52 CDT(-0500)] <athena> oh that's terrific
[11:36:03 CDT(-0500)] <athena> so the reporting tool is on the coop dev list
[11:36:14 CDT(-0500)] <athena> so i should have some availability to help out with that
[11:36:22 CDT(-0500)] <EricDalquist> great
[11:37:29 CDT(-0500)] <athena> there anything i can do to help right now or should i wait for you to get the framework up and running and then touch base?
[11:38:12 CDT(-0500)] <EricDalquist> I think wait until the raw stats stuff is sorted out
[11:38:21 CDT(-0500)] <athena> ok
[11:38:24 CDT(-0500)] <EricDalquist> though maybe cataloging desired reports
[11:38:28 CDT(-0500)] <EricDalquist> and prioritizing them
[11:38:35 CDT(-0500)] <EricDalquist> then we can work through the various aggregation tasks
[11:39:48 CDT(-0500)] <athena> sounds great
[11:39:55 CDT(-0500)] <athena> do we have a list of the available information?
[11:40:58 CDT(-0500)] <EricDalquist> my piddly notes so far: https://source.jasig.org/uPortal/branches/UP-3171/statsDesign.txt , though with the new clob format it will be very easy to add additional info
[11:41:07 CDT(-0500)] <athena> sounds good
[11:41:21 CDT(-0500)] <athena> so probably also info about sessions?
[11:41:33 CDT(-0500)] <EricDalquist> yeah there is a login/logout event
[11:41:40 CDT(-0500)] <athena> yeah figured
[11:41:51 CDT(-0500)] <athena> well i can start capturing some reporting ideas
[11:41:54 CDT(-0500)] <EricDalquist> and a unique/session token is generated and used to tie events together
[11:42:05 CDT(-0500)] <EricDalquist> also events now include the host name of the server that spawned them
[11:42:17 CDT(-0500)] <EricDalquist> so we can do per/server in a cluster reporting if needed
[11:42:20 CDT(-0500)] <athena> oh nice
[11:42:22 CDT(-0500)] <athena> that's pretty cool
[11:42:39 CDT(-0500)] <EricDalquist> login event captures the user's groups & attributes
[11:43:01 CDT(-0500)] <EricDalquist> I think with the clob store I'm going to add portlet request parameters to the portlet execution events as well
[11:43:16 CDT(-0500)] <EricDalquist> with some limits in place for not storing huge values
[11:45:28 CDT(-0500)] <athena> oh that sounds like we'll be able to get some great info
[11:46:20 CDT(-0500)] <EricDalquist> yeah
[12:02:16 CDT(-0500)] <athena> so i'm wondering if there's also a use case for auditing vs. stats aggregation here
[12:02:44 CDT(-0500)] <athena> things like swapping identity might potentially be something you'd want to save in a more permanent log, but be less interested in aggregating
[12:59:30 CDT(-0500)] <b-rock> Greetings uPortal Devs: does someone know if the ""org.jasig.portlet.USER_INFO_MULTIVALUED"" changed names in up4.0.1 from up3.2?
[12:59:57 CDT(-0500)] <b-rock> Im' seeing a null pointer for it in one of my portlets
[13:01:25 CDT(-0500)] <b-rock> well I still see it in the IPortletRender so it probably didn't change.
[13:10:31 CDT(-0500)] <b-rock> ok. nevermind. it was returning null becuase I have the portlet running as the quest (non auth / authorized) user so there aren't any person attributes for it
[13:10:50 CDT(-0500)] <b-rock> apologies for the confusion.
[13:26:35 CDT(-0500)] <EricDalquist> I think it got left out of 4.0 b-rock
[13:26:38 CDT(-0500)] <EricDalquist> nick just fixed that
[13:26:42 CDT(-0500)] <EricDalquist> and it will be back in 4.0.2
[13:55:16 CDT(-0500)] <b-rock> Hi EricDalquist. I'm not sure if you already have a trace like this for your data import work for layouts http://pastebin.com/k5nXrY8h but this is where I"m currently set to debug. All other data pieces are importing cleanly now.
[13:56:31 CDT(-0500)] <b-rock> based on the patch I submitted for the grouper stuff yesterday.
[14:45:02 CDT(-0500)] <holdorph> EricDalquist ?
[14:45:08 CDT(-0500)] <EricDalquist> hi holdorph
[14:45:52 CDT(-0500)] <holdorph> hey, i'm not sure if you knew or not, but Lennard isn't at Unicon right now, so him not having a github account is ok. It's a bummer for history purposes, but there's no way to make him create one right now
[14:46:05 CDT(-0500)] <holdorph> I'll try to track down pollizzoti and g. thompson though.
[14:46:44 CDT(-0500)] <holdorph> by 'not at unicon' i mean, he quit and no longer works for unicon.
[14:46:48 CDT(-0500)] <EricDalquist> right
[14:47:03 CDT(-0500)] <EricDalquist> so that isn't a huge deal the author mapping falls back on: https://github.com/Jasig/svn2git-migration/blob/master/resolvedAuthors.txt
[14:47:15 CDT(-0500)] <EricDalquist> which uses everyones crowd name/email
[14:47:26 CDT(-0500)] <holdorph> yeah, but we still have high hopes he'll one day return
[14:47:42 CDT(-0500)] <EricDalquist> so if they create a github account later and then map the email from that file to their account it will "just work"
[14:48:09 CDT(-0500)] <EricDalquist> since git doesn't really have a central server there is no "username" per-say, it just uses your email address on the commits
[14:48:32 CDT(-0500)] <EricDalquist> and in github you list the email address(es) that are you for association purposes
[14:48:34 CDT(-0500)] <holdorph> ok. seems reasonable to assume he'd get the same email address back
[14:48:54 CDT(-0500)] <holdorph> i thought it was weird that github didn't authenticate your email addresses
[14:49:03 CDT(-0500)] <EricDalquist> Matthew Polizzotti <mpolizzotti@unicon.net> and Gary Thompson <gthompson@unicon.net> are in the mapping file as well
[14:49:04 CDT(-0500)] <EricDalquist> yes
[14:49:06 CDT(-0500)] <holdorph> you can just put in whatever addresses you want that aren't already used.
[14:49:10 CDT(-0500)] <EricDalquist> I thought it strange as well
[14:49:23 CDT(-0500)] <EricDalquist> I wonder how they deal with conflicts