[09:23:27 CST(-0600)] <jwennmacher> Hi Eric.
[09:23:45 CST(-0600)] <EricDalquist> hello
[09:24:27 CST(-0600)] <jwennmacher> I was thinking about the event aggregation issue Anthony reported. I think both he and the person at the Apereo conference indicated their raw events table was growing at around 1M rows a day.
[09:25:35 CST(-0600)] <jwennmacher> I'm not sure if this is that there were 1M inserts or the net is a growth of 1M rows. I just did a rough calculation. If the aggregation is running every 20 minutes and it does at most 10,000 aggregations, it will aggregate at most 720,000 rows in a day.
[09:26:02 CST(-0600)] <EricDalquist> well it runs ever 60 seconds
[09:26:19 CST(-0600)] <EricDalquist> and it catch-up mode (like there were more events to aggregate than aggregated) it should run continueously
[09:26:25 CST(-0600)] <EricDalquist> the problem could be aggregation rate
[09:26:40 CST(-0600)] <EricDalquist> one thing I thought about is that I test this all on oracle
[09:26:50 CST(-0600)] <EricDalquist> which has different index behavior than some other DBs
[09:26:59 CST(-0600)] <EricDalquist> like FKs in oracle are indexes
[09:27:07 CST(-0600)] <EricDalquist> but in some other DBs explicit indexes are also needed
[09:27:21 CST(-0600)] <EricDalquist> that is why I'm interested in seeing a chunk from his 4.0.10 log
[09:27:30 CST(-0600)] <EricDalquist> to see if the aggr is running in catch up mode
[09:27:38 CST(-0600)] <EricDalquist> and if it is what the aggregation rate is
[09:28:05 CST(-0600)] <EricDalquist> next steps would be to filter some of the raw events before they persist
[09:28:11 CST(-0600)] <EricDalquist> or just truncate the raw events table
[09:28:20 CST(-0600)] <EricDalquist> and see if it can keep up with a fresh data set
[09:29:04 CST(-0600)] <jwennmacher> Makes sense.