uPortal IRC Logs-2012-10-16

Owned by Apereo Infrastructure
Last updated: Oct 16, 2012

[10:56:20 CDT(-0500)] <EricDalquist> this timezone stuff is going to drive me insane!

[10:58:40 CDT(-0500)] <athena> noooooo!

[10:58:47 CDT(-0500)] <athena> pull back from the dark side!

[11:16:14 CDT(-0500)] <athena> so our data model for fragment defs allows more than one audience

[11:16:25 CDT(-0500)] <athena> but i don't think that actually works, does it?

[11:16:39 CDT(-0500)] <athena> i suspect that may be an artifact of shoehorning those into the existing data structures

[11:22:30 CDT(-0500)] <EricDalquist> I'm not sure

[11:22:40 CDT(-0500)] <EricDalquist> I've purposely not really looked at it

[11:22:55 CDT(-0500)] <EricDalquist> as if I do I feel an overwhelming urge to refactor it all into sanity

[11:23:06 CDT(-0500)] <EricDalquist> but yes

[11:23:13 CDT(-0500)] <EricDalquist> I believe you can target multiple audiences

[11:47:29 CDT(-0500)] <EricDalquist> @#$!@#$

[11:47:51 CDT(-0500)] <EricDalquist> The JDBC specification does not define any details with regards to time zone

[11:47:55 CDT(-0500)] <EricDalquist> yay

[11:48:13 CDT(-0500)] <athena> ummmmm

[11:48:14 CDT(-0500)] <athena> wow.

[11:48:25 CDT(-0500)] <athena> hey that tz stuff isn't important, lets ignore it (smile)

[11:48:29 CDT(-0500)] <EricDalquist> seriously

[11:48:33 CDT(-0500)] <athena> crazy

[11:48:35 CDT(-0500)] <EricDalquist> I'm going to start throwing things

[11:48:55 CDT(-0500)] <athena> hands over more objects to throw

[13:14:31 CDT(-0500)] <EricDalquist> hrm

[13:14:37 CDT(-0500)] <EricDalquist> maybe this is just a hsqldb bug

[13:14:40 CDT(-0500)] <EricDalquist> time to test oracle

[13:14:48 CDT(-0500)] <EricDalquist> I've at least isolated the problem

[13:15:13 CDT(-0500)] <EricDalquist> the jodatime mapping coverts the LocalDate of 2012-10-15 into a sql Date object

[13:15:20 CDT(-0500)] <EricDalquist> which ends up looking like "2012-10-14T19:00:00.000-0500"

[13:15:29 CDT(-0500)] <EricDalquist> the problem is hsql seems to store that in the local time zone

[13:15:37 CDT(-0500)] <EricDalquist> so it stores "2012-10-14"

[13:15:52 CDT(-0500)] <EricDalquist> and on the way back out of the DB the sql Date object is set to "2012-10-14T00:00:00.000-0500"

[13:19:08 CDT(-0500)] <EricDalquist> I wonder if this is a hsqldb bug even

[13:19:31 CDT(-0500)] <EricDalquist> since java.sql.Date says it is supposed to track seconds since GMT epoch

[13:19:39 CDT(-0500)] <EricDalquist> which is what the value that is passed in was

[13:54:52 CDT(-0500)] <athena> interesting

[14:01:45 CDT(-0500)] <EricDalquist> angering more like it (tongue)

[14:28:47 CDT(-0500)] <athena> well

[14:28:55 CDT(-0500)] <athena> an hsqldb bugt sounds better than a jdbc bug?

[14:28:59 CDT(-0500)] <EricDalquist> yeah

[14:29:16 CDT(-0500)] <EricDalquist> I think I might have a full scale test involving plan JDBC access to figure this out :/

[14:29:30 CDT(-0500)] <EricDalquist> my fear is that the mapping from the joda LocalDate -> JDBC is wrong

[14:29:36 CDT(-0500)] <athena> ergh :/

[14:29:51 CDT(-0500)] <EricDalquist> and we're going to have to figure out how to modify a bunch of already processed stats data

[14:30:42 CDT(-0500)] <athena> eep

[14:30:45 CDT(-0500)] <EricDalquist> yeah

[15:08:13 CDT(-0500)] <EricDalquist> yup ... its official

[15:08:18 CDT(-0500)] <EricDalquist> the JDBC Date type sucks

[15:08:48 CDT(-0500)] <EricDalquist> and I'm honestly not sure what the fix is

[15:08:56 CDT(-0500)] <EricDalquist> since TZ stuff is so freaking nebulous

[15:09:45 CDT(-0500)] <EricDalquist> https://gist.github.com/3901664

[15:09:47 CDT(-0500)] <EricDalquist> that fails

[15:09:50 CDT(-0500)] <EricDalquist> with both hsql and oracle

[15:10:27 CDT(-0500)] <EricDalquist> in both cases my LocalDate(2012-10-15) gets coverted to a JDBC Date object with an internal value of "2012-10-14T19:00:00.000-0500"

[15:10:37 CDT(-0500)] <EricDalquist> and the DB only stores the y/m/d biy

[15:10:38 CDT(-0500)] <EricDalquist> bit

[15:10:49 CDT(-0500)] <EricDalquist> so it comes back out as ""2012-10-14T00:00:00.000-0500""

[15:26:10 CDT(-0500)] <EricDalquist> UHG

[15:26:15 CDT(-0500)] <EricDalquist> so I think I figured out a fix ...

[15:27:07 CDT(-0500)] <EricDalquist> so it looks like the usertype library should be setting the databaseZone to the current JVM TZ

[15:27:26 CDT(-0500)] <EricDalquist> instead of UTC

[15:27:34 CDT(-0500)] <EricDalquist> the problem is when I make that change

[15:28:03 CDT(-0500)] <EricDalquist> aggregation is going to jump ahead by 1 day if you have a negative utc offset

[15:28:10 CDT(-0500)] <EricDalquist> or behind by 1 day if you have a positive utc offset

[15:28:13 CDT(-0500)] <EricDalquist> which is bad

[15:28:39 CDT(-0500)] <athena> oh (sad)

[15:29:17 CDT(-0500)] <EricDalquist> grrr

[15:29:27 CDT(-0500)] * EricDalquist really hates timezones (tongue)

[15:29:33 CDT(-0500)] <athena> no kidding.

[15:29:55 CDT(-0500)] <EricDalquist> time to figure out a way to fix this ...

[15:30:13 CDT(-0500)] <EricDalquist> the big problem is the people with a positive TZ offset

[15:30:22 CDT(-0500)] <EricDalquist> for the negative people they get a hole in their stats data

[15:30:25 CDT(-0500)] <EricDalquist> which isn't horrible

[15:30:32 CDT(-0500)] <EricDalquist> but for positive TZ offsets

[15:30:33 CDT(-0500)] <athena> yeah (sad)

[15:30:37 CDT(-0500)] <EricDalquist> it will crash the aggregation

[15:30:38 CDT(-0500)] <athena> that's not so good

[15:30:52 CDT(-0500)] <EricDalquist> since data will already exist and be in a "closed" state

[15:31:02 CDT(-0500)] <EricDalquist> so that means to fix this

[15:31:12 CDT(-0500)] <EricDalquist> 1. disable stats aggregation on ALL machines in the environment

[15:31:30 CDT(-0500)] <EricDalquist> 2. run program to "fix" dates (not sure how that will work yet)

[15:31:38 CDT(-0500)] <EricDalquist> 3. roll out new portal code

[15:31:51 CDT(-0500)] <EricDalquist> 3 would enable aggregation again

[15:31:53 CDT(-0500)] <EricDalquist> grr

[15:32:13 CDT(-0500)] <athena> yeah :/

[15:37:29 CDT(-0500)] <EricDalquist> I have a feature I realized I forgot in the works

[15:37:48 CDT(-0500)] <EricDalquist> that essentially stops aggreattion from running where the code version is less than the db version

[15:38:07 CDT(-0500)] <EricDalquist> so if you upgrade one server of a cluster from 4.0.7 to 4.0.8

[15:38:11 CDT(-0500)] <EricDalquist> and that includes running db-update

[15:38:24 CDT(-0500)] <EricDalquist> only server(s) running 4.0.8 will do even aggreation

[15:38:37 CDT(-0500)] <athena> yeah

[15:38:38 CDT(-0500)] <EricDalquist> if that was in place now I think I could mostly automate this process

[15:38:43 CDT(-0500)] <EricDalquist> but it isn't

[15:38:47 CDT(-0500)] <athena> oh (sad)

[15:38:53 CDT(-0500)] <EricDalquist> the db versioning is there

[15:38:59 CDT(-0500)] <EricDalquist> but I forgot the check in the aggregator

[15:39:41 CDT(-0500)] <athena> ah

[15:39:54 CDT(-0500)] <EricDalquist> well ... I'll do a bunch of testing here

[15:40:00 CDT(-0500)] <EricDalquist> and see what I can figure out for a fix

[15:40:02 CDT(-0500)] <EricDalquist> (sad)

[16:30:51 CDT(-0500)] <EricDalquist> yup ... fixed the bug

[16:31:04 CDT(-0500)] <EricDalquist> and now all my event related dates jumped back in time by 5 hours

[16:31:06 CDT(-0500)] <EricDalquist> (sad)

[16:42:41 CDT(-0500)] <EricDalquist> uhg ... 8 columns across 5 tables affected by this TZ issue

[16:42:47 CDT(-0500)] <EricDalquist> this is not good at all

[16:43:20 CDT(-0500)] <EricDalquist> hrm

[16:43:33 CDT(-0500)] <EricDalquist> though setting the TZ to "jvm" isn't great either

[16:43:46 CDT(-0500)] <EricDalquist> I need to test what happens when the server crosses a DST boundary

[16:47:21 CDT(-0500)] <EricDalquist> so really it seems like storing stuff in UTC would be the best bet

[16:47:28 CDT(-0500)] <EricDalquist> the problem is this LocalDate ....

[16:47:39 CDT(-0500)] <EricDalquist> and its interaction with Date

[16:50:36 CDT(-0500)] <athena> yikes

[16:50:44 CDT(-0500)] <athena> that all sounds kind of concerning

[16:50:50 CDT(-0500)] <EricDalquist> yeah

[16:50:51 CDT(-0500)] <EricDalquist> it is

[16:50:54 CDT(-0500)] <EricDalquist> oh

[16:50:55 CDT(-0500)] <EricDalquist> ok

[16:50:56 CDT(-0500)] <EricDalquist> just tested

[16:51:02 CDT(-0500)] <EricDalquist> it is handling DST correctly

[16:51:08 CDT(-0500)] <EricDalquist> when using the jvm TZ

[16:51:10 CDT(-0500)] <EricDalquist> I believe

[16:51:11 CDT(-0500)] <EricDalquist> still

[16:51:16 CDT(-0500)] <EricDalquist> more testing is going to be needed

[16:51:25 CDT(-0500)] <EricDalquist> and probably a fairly complex java app will be needed to "fix" this

[16:51:28 CDT(-0500)] <EricDalquist> (sad)

[16:51:43 CDT(-0500)] <EricDalquist> oh well ... bus time