[13:59:55 CDT(-0500)] <drewwills> question about the new events architecture: is there a way to configure how much of raw events is captured, and how long it's held in either raw or aggregated state?
[14:00:10 CDT(-0500)] <EricDalquist> yes, though not well documented yet
[14:00:23 CDT(-0500)] <EricDalquist> give me ~ 5 mintues
[14:00:28 CDT(-0500)] <EricDalquist> and I can chat more about it
[14:00:46 CDT(-0500)] <drewwills> i don't see it in portal.props or eventpersistencecontext
[14:08:21 CDT(-0500)] <EricDalquist> drewwills: there are a bunch of properties in PortalEventAggregationManagerImpl that I forgot to transcribe into portal.properties
[14:08:41 CDT(-0500)] <EricDalquist> (I'm thinking about writing a script to generate a base .properties file from the @Value annotations)
[14:08:52 CDT(-0500)] <EricDalquist> org.jasig.portal.event.aggr.PortalEventAggregationManager.purgeDelay:P1D
[14:08:59 CDT(-0500)] <EricDalquist> @Value("$ ")
[14:09:02 CDT(-0500)] <EricDalquist> there
[14:09:14 CDT(-0500)] <EricDalquist> so that sets the default raw event purge delay to 1 day
[14:09:30 CDT(-0500)] <EricDalquist> that 1 day is calculated as (mostrecentlyAggregatedEventDate - 1 day)
[14:10:02 CDT(-0500)] <EricDalquist> take a look at uportal-war/src/main/data/default_entities/event-aggregation/default.event-aggregation.xml
[14:10:25 CDT(-0500)] <EricDalquist> that lets you configure the intervals and groups each aggregator generates aggregates for
[14:10:34 CDT(-0500)] <EricDalquist> there is an open jira to write an event admin portlet
[14:10:39 CDT(-0500)] <EricDalquist> to pull this all together
[14:10:42 CDT(-0500)] <drewwills> so there aren't lines in portal.properties for it out-of-the-box, but if I put them in they'll be honored? do i have that right?
[14:10:45 CDT(-0500)] <EricDalquist> but that won't happen until conf time
[14:10:47 CDT(-0500)] <EricDalquist> yes
[14:10:55 CDT(-0500)] <drewwills> roger
[14:11:18 CDT(-0500)] <EricDalquist> they syntax of that @Value says "Use the property value org.jasig.portal.event.aggr.PortalEventAggregationManager.purgeDelay but if that isn't found use the value after the :"
[14:11:31 CDT(-0500)] <EricDalquist> there is no build in mechanism to purge aggregated data
[14:12:01 CDT(-0500)] <drewwills> no worries... I can put a section for that, commented-out, in portal.props if you like?
[14:12:15 CDT(-0500)] <EricDalquist> yes
[14:12:21 CDT(-0500)] <drewwills> cool
[14:12:40 CDT(-0500)] <EricDalquist> like I said I want to write a little shell script we can run to look in .java and .xml for property placeholders
[14:12:49 CDT(-0500)] <EricDalquist> and generate a default .properties file for us
[14:12:58 CDT(-0500)] <EricDalquist> so a purge-aggreagted-data tool could be a fairly easy addition to the IPortalEventAggregator interface
[14:13:05 CDT(-0500)] <drewwills_bbl> sounds good
[16:30:13 CDT(-0500)] <drewwills> EricDalquist? You still there? I'm looking at PortalEventAggregationManagerImpl and default.event-aggregation.xml, but it doesn't jump out at me how to make the system write less to the database... maybe reduce the purge time to 12 hrs... maybe change <exclude>MINUTE</exclude> to something longer than a minute?
[16:30:45 CDT(-0500)] <EricDalquist> that minute value is from an enum AggregationInterval
[16:31:02 CDT(-0500)] <EricDalquist> you could also exclude FIVE_MINUTE, HOUR, etc...
[16:31:08 CDT(-0500)] <drewwills> in the old stats system, there was an xml file where you could list the events you cared about (e.g. login), and exclude the ones you didn't (e.g. portlet renderings)
[16:31:17 CDT(-0500)] <EricDalquist> the other thing that is happening is you are getting one aggregation per event
[16:31:19 CDT(-0500)] <EricDalquist> ah
[16:31:19 CDT(-0500)] <EricDalquist> hrm
[16:31:24 CDT(-0500)] <drewwills> i figured it was from a set of values
[16:32:02 CDT(-0500)] <EricDalquist> PortalEventDaoQueuingEventHandler
[16:32:03 CDT(-0500)] <drewwills> i bet if i told it not to record portlet renderings, it would make less data
[16:32:12 CDT(-0500)] <EricDalquist> yup
[16:32:19 CDT(-0500)] <EricDalquist> do you have a specific table that is a concern?
[16:32:23 CDT(-0500)] <EricDalquist> just raw_events?
[16:32:48 CDT(-0500)] <drewwills> i have this to go on: "it seems like uPortal is sticking a lot of large objects into postgres and the event tracker has around 13 million records. We might want to see if we can purge this data and stop the recording of this raw event data."
[16:34:02 CDT(-0500)] <EricDalquist> well you can disable event persistence completely (though not easily 
) by removing @Service from PortalEventDaoQueuingEventHandler
[16:34:24 CDT(-0500)] <EricDalquist> you can also inject a List<ApplicationEventFilter<E>> into PortalEventDaoQueuingEventHandler.setApplicationEventFilters
[16:34:31 CDT(-0500)] <EricDalquist> to do more specific filtering
[16:34:45 CDT(-0500)] <EricDalquist> there is a ClassFilteringApplicationListener and a UserFilteringPortalEventHandler
[16:34:55 CDT(-0500)] <EricDalquist> though again not much in the way of examples for configuring all that stuff yet
[16:37:46 CDT(-0500)] <EricDalquist> as for trimming down the aggregation side
[16:38:01 CDT(-0500)] <EricDalquist> excluding more intervals would help
[16:38:13 CDT(-0500)] <EricDalquist> as well as either excluding or including specific groups
[16:38:15 CDT(-0500)] <EricDalquist> https://gist.github.com/2175330
[16:38:23 CDT(-0500)] <EricDalquist> that would only aggregate data for the Everyone group
[16:38:34 CDT(-0500)] <EricDalquist> instead of the default which creates an aggregate for every group
[16:42:28 CDT(-0500)] <EricDalquist> creating jiras to document these pain points: https://issues.jasig.org/browse/UP-3397
[16:49:38 CDT(-0500)] <EricDalquist> drewwills: how many users does this deployment have?
[16:49:45 CDT(-0500)] <EricDalquist> just out of curiosity
[17:01:54 CDT(-0500)] <drewwills> sorry got distracted... getting caught up