[11:14:08 CDT(-0500)] <EricDalquist> hi there athena
[11:17:41 CDT(-0500)] * athena waves
[11:19:50 CDT(-0500)] <athena> so . . . stats?
[11:29:02 CDT(-0500)] <EricDalquist> yes
[11:29:09 CDT(-0500)] <EricDalquist> just had a chat with our DB
[11:29:10 CDT(-0500)] <EricDalquist> DBA
[11:29:25 CDT(-0500)] <EricDalquist> so my question is "how useful are the relational raw stats tables"
[11:30:14 CDT(-0500)] <EricDalquist> with the though that we could store more event data with less work by having a raw_stats table with just 3 columns: id, timestamp, data_clob
[11:31:07 CDT(-0500)] <athena> i think mostly we'd just want to do reporting on the aggregated tables, right?
[11:31:13 CDT(-0500)] <athena> that seems like the more usual thing to do
[11:31:16 CDT(-0500)] <EricDalquist> right
[11:31:21 CDT(-0500)] <EricDalquist> this is part of the idea that we're building aggregation in with uportal itself
[11:31:27 CDT(-0500)] <athena> yeah
[11:31:36 CDT(-0500)] <EricDalquist> so when the DB event DAO would persist a list of events
[11:31:38 CDT(-0500)] <athena> i don't think we'd want to build reporting tools against the raw stats
[11:31:45 CDT(-0500)] <athena> so we could just maybe consider that table 100% internal?
[11:31:52 CDT(-0500)] <EricDalquist> by serializing them to a string either via JAXB or jackson
[11:32:31 CDT(-0500)] <EricDalquist> then we can easily add new data fields to the aggregate data
[11:32:35 CDT(-0500)] <EricDalquist> yeah
[11:32:41 CDT(-0500)] <EricDalquist> I'm thinking that is a good idea
[11:32:51 CDT(-0500)] <EricDalquist> where that table gets written to by all servers in a cluster
[11:33:12 CDT(-0500)] <EricDalquist> then the aggregation runs on events older than (now - 1 minute)
[11:33:32 CDT(-0500)] <athena> sounds good to me 
[11:33:39 CDT(-0500)] <EricDalquist> and immediately deletes aggregated events
[11:34:18 CDT(-0500)] <EricDalquist> that should greatly simplify some things
[11:34:25 CDT(-0500)] <EricDalquist> and let us store more data in the event objects
[11:34:40 CDT(-0500)] <EricDalquist> I should have the raw events stuff merged in this weekend
[11:34:57 CDT(-0500)] <EricDalquist> and I'll get the aggregator framework in place as well
[11:35:16 CDT(-0500)] <EricDalquist> then it will just be the work of writing aggregator plugins for each report type we want to have
[11:35:52 CDT(-0500)] <athena> oh that's terrific
[11:36:03 CDT(-0500)] <athena> so the reporting tool is on the coop dev list
[11:36:14 CDT(-0500)] <athena> so i should have some availability to help out with that
[11:36:22 CDT(-0500)] <EricDalquist> great
[11:37:29 CDT(-0500)] <athena> there anything i can do to help right now or should i wait for you to get the framework up and running and then touch base?
[11:38:12 CDT(-0500)] <EricDalquist> I think wait until the raw stats stuff is sorted out
[11:38:21 CDT(-0500)] <athena> ok
[11:38:24 CDT(-0500)] <EricDalquist> though maybe cataloging desired reports
[11:38:28 CDT(-0500)] <EricDalquist> and prioritizing them
[11:38:35 CDT(-0500)] <EricDalquist> then we can work through the various aggregation tasks
[11:39:48 CDT(-0500)] <athena> sounds great
[11:39:55 CDT(-0500)] <athena> do we have a list of the available information?
[11:40:58 CDT(-0500)] <EricDalquist> my piddly notes so far: https://source.jasig.org/uPortal/branches/UP-3171/statsDesign.txt , though with the new clob format it will be very easy to add additional info
[11:41:07 CDT(-0500)] <athena> sounds good
[11:41:21 CDT(-0500)] <athena> so probably also info about sessions?
[11:41:33 CDT(-0500)] <EricDalquist> yeah there is a login/logout event
[11:41:40 CDT(-0500)] <athena> yeah figured
[11:41:51 CDT(-0500)] <athena> well i can start capturing some reporting ideas
[11:41:54 CDT(-0500)] <EricDalquist> and a unique/session token is generated and used to tie events together
[11:42:05 CDT(-0500)] <EricDalquist> also events now include the host name of the server that spawned them
[11:42:17 CDT(-0500)] <EricDalquist> so we can do per/server in a cluster reporting if needed
[11:42:20 CDT(-0500)] <athena> oh nice
[11:42:22 CDT(-0500)] <athena> that's pretty cool
[11:42:39 CDT(-0500)] <EricDalquist> login event captures the user's groups & attributes
[11:43:01 CDT(-0500)] <EricDalquist> I think with the clob store I'm going to add portlet request parameters to the portlet execution events as well
[11:43:16 CDT(-0500)] <EricDalquist> with some limits in place for not storing huge values
[11:45:28 CDT(-0500)] <athena> oh that sounds like we'll be able to get some great info
[11:46:20 CDT(-0500)] <EricDalquist> yeah
[12:02:16 CDT(-0500)] <athena> so i'm wondering if there's also a use case for auditing vs. stats aggregation here
[12:02:44 CDT(-0500)] <athena> things like swapping identity might potentially be something you'd want to save in a more permanent log, but be less interested in aggregating
[12:59:30 CDT(-0500)] <b-rock> Greetings uPortal Devs: does someone know if the ""org.jasig.portlet.USER_INFO_MULTIVALUED"" changed names in up4.0.1 from up3.2?
[12:59:57 CDT(-0500)] <b-rock> Im' seeing a null pointer for it in one of my portlets
[13:01:25 CDT(-0500)] <b-rock> well I still see it in the IPortletRender so it probably didn't change.
[13:10:31 CDT(-0500)] <b-rock> ok. nevermind. it was returning null becuase I have the portlet running as the quest (non auth / authorized) user so there aren't any person attributes for it
[13:10:50 CDT(-0500)] <b-rock> apologies for the confusion.
[13:26:35 CDT(-0500)] <EricDalquist> I think it got left out of 4.0 b-rock
[13:26:38 CDT(-0500)] <EricDalquist> nick just fixed that
[13:26:42 CDT(-0500)] <EricDalquist> and it will be back in 4.0.2
[13:55:16 CDT(-0500)] <b-rock> Hi EricDalquist. I'm not sure if you already have a trace like this for your data import work for layouts http://pastebin.com/k5nXrY8h but this is where I"m currently set to debug. All other data pieces are importing cleanly now.