jasig-ssp IRC Logs-2013-08-22

Owned by Apereo Infrastructure
Last updated: Aug 22, 2013
9 min read

[12:25:12 CDT(-0500)] <TonyUnicon> external_student_transcript_course

[12:25:16 CDT(-0500)] <TonyUnicon> Jason you mentioned

[12:25:23 CDT(-0500)] <TonyUnicon> that with music classes and such

[12:25:26 CDT(-0500)] <TonyUnicon> a student could take

[12:25:36 CDT(-0500)] <TonyUnicon> the same course in the same term

[12:25:48 CDT(-0500)] <TonyUnicon> section is also nullable on this table

[12:25:58 CDT(-0500)] <TonyUnicon> section_number

[12:26:43 CDT(-0500)] <JasonElwood> yep

[12:26:46 CDT(-0500)] <TonyUnicon> so I think we'll also have to handle this

[12:26:52 CDT(-0500)] <TonyUnicon> (sad)

[12:27:56 CDT(-0500)] <JasonElwood> so one idea is to make section_number not nullable now in faculty_course, faculty_course_roster and transcript_course IF we are going to write liquibase in 2.1 to construct the section_code

[12:28:54 CDT(-0500)] <TonyUnicon> nullable but not unique?

[12:28:59 CDT(-0500)] <TonyUnicon> or

[12:29:09 CDT(-0500)] <TonyUnicon> not nullable now but not unique

[12:29:23 CDT(-0500)] <JasonElwood> if it doesn't break code, they should be unique, not nullable

[12:29:43 CDT(-0500)] <JasonElwood> well...

[12:30:04 CDT(-0500)] <JasonElwood> it depends on whether or not we use section_code in transcript_course as the key in the future

[12:30:25 CDT(-0500)] <JasonElwood> if we use section_code across the board, maybe nullable is ok

[12:30:33 CDT(-0500)] <TonyUnicon> this is the same conversation though, we'd have to comb through the code

[12:30:45 CDT(-0500)] <TonyUnicon> and change everything to make sure it mentions section

[12:31:34 CDT(-0500)] <JasonElwood> do we know if the app blows up now for dupes?

[12:31:48 CDT(-0500)] <TonyUnicon> i dont know, im still working on the liquibase

[12:31:59 CDT(-0500)] <TonyUnicon> i can check

[12:32:01 CDT(-0500)] <TonyUnicon> hold on

[12:34:53 CDT(-0500)] <TonyUnicon> does not look like the java code will because we never try to fetch a single transcript course

[12:35:02 CDT(-0500)] <TonyUnicon> but hard to tell what the UI will do

[12:35:30 CDT(-0500)] <JasonElwood> so you propose to not add keys to transcript_course?

[12:35:33 CDT(-0500)] <TonyUnicon> its just an unknown what the impact will be is we decide to go full monty with a natural key

[12:36:06 CDT(-0500)] <TonyUnicon> i agree with what we spoke about on the phone if we're serious about wrapping up tomorrow

[12:36:34 CDT(-0500)] <TonyUnicon> but its a big change that will need soak, and we've increased the scope since this morning so we're looking at atleast 4 days dev time

[12:36:41 CDT(-0500)] <TonyUnicon> its just a big change

[12:37:29 CDT(-0500)] <TonyUnicon> i agree with ideally doing it right is what I want but I think there is a decent amount of risk

[12:38:07 CDT(-0500)] <TonyUnicon> and there are still other tickets

[12:38:10 CDT(-0500)] <TonyUnicon> I have to get to

[12:38:20 CDT(-0500)] <JasonElwood> ok

[12:39:20 CDT(-0500)] <TonyUnicon> we can do it first thing next iteration so it will have a chance to soak

[12:39:31 CDT(-0500)] <TonyUnicon> and we'll know its stable

[12:39:40 CDT(-0500)] <TonyUnicon> especially if there is a work around locally

[12:39:43 CDT(-0500)] <JasonElwood> maybe a silly question, but won't all these PK changes have the possibility of blowing the app

[12:39:59 CDT(-0500)] <TonyUnicon> no because most if it is below what the app cares about

[12:40:11 CDT(-0500)] <TonyUnicon> it cares that the views have ids

[12:40:19 CDT(-0500)] <TonyUnicon> and all it cares about is it's unique

[12:40:23 CDT(-0500)] <dmccallum54> EA is a special case b/c it happens to have assumptions about what the correct natural key is

[12:40:30 CDT(-0500)] <dmccallum54> which are in fact incorrect

[12:40:33 CDT(-0500)] <JasonElwood> are there others besides transcript then?

[12:40:54 CDT(-0500)] <TonyUnicon> where a natural key column was not nullable before?

[12:40:55 CDT(-0500)] <TonyUnicon> hold on

[12:41:22 CDT(-0500)] <TonyUnicon> i did add code/to external_course

[12:41:24 CDT(-0500)] <TonyUnicon> that was missing

[12:41:32 CDT(-0500)] <TonyUnicon> but that wont break the code

[12:41:38 CDT(-0500)] <TonyUnicon> it was just an oversight

[12:41:53 CDT(-0500)] <JasonElwood> cool

[12:42:09 CDT(-0500)] <TonyUnicon> there are other examples

[12:42:16 CDT(-0500)] <TonyUnicon> but I know they are fine

[12:42:20 CDT(-0500)] <TonyUnicon> because they were ones i worked on

[12:42:24 CDT(-0500)] <TonyUnicon> I just didn't add them

[12:42:34 CDT(-0500)] <dmccallum54> what is the proposed key for external_student_transcript_course?

[12:42:51 CDT(-0500)] <TonyUnicon> it has the same problem

[12:42:57 CDT(-0500)] <TonyUnicon> student can be in same course and same term

[12:43:06 CDT(-0500)] <TonyUnicon> but section info is nullable

[12:43:07 CDT(-0500)] <JasonElwood> right

[12:43:22 CDT(-0500)] <dmccallum54> so we were thinking it would have been 'school_id, formatted_course, term_code' except for that wrinkle?

[12:43:32 CDT(-0500)] <TonyUnicon> yeah

[12:43:41 CDT(-0500)] <TonyUnicon> and Jason had mentioned some valid examples at sinclair

[12:43:44 CDT(-0500)] <TonyUnicon> where that is the case

[12:43:48 CDT(-0500)] <dmccallum54> k. sending vicky a request to see if she can check on the actual state of the data in NC databases

[12:44:42 CDT(-0500)] <JasonElwood> Tony- was external_student_test.sub_test_code changes to be not null in 1224?

[12:44:56 CDT(-0500)] <TonyUnicon> yes it was

[12:45:22 CDT(-0500)] <TonyUnicon> but I know the code would break as well

[12:45:25 CDT(-0500)] <TonyUnicon> if that was not unique

[12:46:12 CDT(-0500)] <JasonElwood> problem is that there isn't always a sub_test_code

[12:46:39 CDT(-0500)] <TonyUnicon> sweet!

[12:46:53 CDT(-0500)] <JasonElwood> none of Sinclair's data has sub_test_code

[12:47:04 CDT(-0500)] <TonyUnicon> do they have dup test codes?

[12:47:21 CDT(-0500)] <TonyUnicon> or rather

[12:47:35 CDT(-0500)] <JasonElwood> yeah

[12:47:50 CDT(-0500)] <TonyUnicon> dup school_id/test_code combinations

[12:47:58 CDT(-0500)] <TonyUnicon> does the test stuff render ok?

[12:48:37 CDT(-0500)] <JasonElwood> haven't tried it

[12:48:43 CDT(-0500)] <JasonElwood> just running the queries

[12:48:52 CDT(-0500)] <JasonElwood> test_date should be in there for sure

[12:49:02 CDT(-0500)] <JasonElwood> with that only 8 dupe combos

[12:51:32 CDT(-0500)] <JasonElwood> they could always put in bogus sub_test_code data I guess

[12:52:31 CDT(-0500)] <TonyUnicon> its displayed in the UI though, let me test for sure that it breaks

[12:52:37 CDT(-0500)] <TonyUnicon> I remember having to add it when I had to work on that screen

[12:52:43 CDT(-0500)] <TonyUnicon> but maybe I'm wrong

[12:53:58 CDT(-0500)] <TonyUnicon> going to grab lunch quick, afk for about 20-30 minutes

[13:25:36 CDT(-0500)] <TonyUnicon> so for the tables in which we are passing on pk's… what should we do as far as ID generation?

[13:25:52 CDT(-0500)] <dmccallum54> i think we just have to leave them alone

[13:26:20 CDT(-0500)] <dmccallum54> i.e. we have no reliable natural keys, so it's back to row_number

[13:26:32 CDT(-0500)] <TonyUnicon> right, ok

[13:30:52 CDT(-0500)] <JasonElwood> Did we come to a conclusion on external_student_test?

[13:31:32 CDT(-0500)] <TonyUnicon> no not yet, havent gotten to test it

[13:31:36 CDT(-0500)] <TonyUnicon> let me get this script in

[13:31:42 CDT(-0500)] <JasonElwood> sure

[13:31:44 CDT(-0500)] <JasonElwood> np

[13:31:44 CDT(-0500)] <TonyUnicon> so i can move on

[13:32:03 CDT(-0500)] <TonyUnicon> its just a question if it blows up with dup sub test codes

[13:32:16 CDT(-0500)] <TonyUnicon> if it doesn't, ill change the key to be the two columns

[13:32:55 CDT(-0500)] <JasonElwood> hold off on that please. it may need to be more complex

[13:33:12 CDT(-0500)] <TonyUnicon> ok

[13:47:12 CDT(-0500)] <JasonElwood> the scenario for external_student test is that conceivably a student could take 2 or more tests on a single day represented by school_id, test_code, sub_test_code and test_date

[13:51:56 CDT(-0500)] <JasonElwood> Testing in Sinclair's data, there are 8 pairs that are dupes using test_code & test_date or test_code & test_date & sub_test_code. for them adding sub_test_code makes not difference, but test_date reduces the dupe list from 1131 dupe rows

[15:03:31 CDT(-0500)] <dmccallum54> i swear to god. SSP is a database's worst nightmare

[15:03:56 CDT(-0500)] <dmccallum54> makes a rails app look downright angelic

[15:04:47 CDT(-0500)] <TonyUnicon> new problem>

[15:06:47 CDT(-0500)] <dmccallum54> ?

[15:07:07 CDT(-0500)] <TonyUnicon> did you come across a new problem?

[15:07:15 CDT(-0500)] <TonyUnicon> or just generally fed up with the old ones?

[15:07:35 CDT(-0500)] <JasonElwood> my guess is both

[15:07:36 CDT(-0500)] <dmccallum54> oh. debugging this first-time login problem. which boils down to hib sessions being closed unexpectedly

[15:07:44 CDT(-0500)] <dmccallum54> so turned on more spring+hib logging

[15:08:07 CDT(-0500)] <dmccallum54> now witnessing us pound the ever living crap out of the db just to launch the UI

[15:08:16 CDT(-0500)] <TonyUnicon> yeah

[15:08:20 CDT(-0500)] <dmccallum54> tons and tons of transaction commits on read-only ops

[15:08:37 CDT(-0500)] <TonyUnicon> pretty much a DOS attack(smile)

[15:09:00 CDT(-0500)] <dmccallum54> that's eactly what GET /api/1/person/ is (smile)

[15:09:13 CDT(-0500)] <TonyUnicon> hehe

[15:09:28 CDT(-0500)] <dmccallum54> no joke

[15:09:35 CDT(-0500)] <dmccallum54> try it in linux ci

[15:09:36 CDT(-0500)] <dmccallum54> will bring it down

[15:09:46 CDT(-0500)] <dmccallum54> so, er, don't try it, rather

[15:10:35 CDT(-0500)] <TonyUnicon> oh i don't doubt it

[18:42:40 CDT(-0500)] <JasonElwood> Tony just put in a change to the PK external_student_test. That along with other changes today before you build any code, you should drop the keys on external_faculty_course, external_faculty_course_roster and external_student_test. The latest build will replace the external_student_test but the other two will not have keys until 2.1

[18:55:50 CDT(-0500)] <TonyUnicon> this is a checksum error jason

[18:56:45 CDT(-0500)] <JasonElwood> sweet

[18:57:54 CDT(-0500)] <TonyUnicon> can we drop the keys manually on that env?

[18:58:01 CDT(-0500)] <JasonElwood> I did

[18:58:08 CDT(-0500)] <TonyUnicon> ok

[19:00:38 CDT(-0500)] <JasonElwood> was there another change to configuration.js?

[19:00:46 CDT(-0500)] <JasonElwood> my build is being held up on that again

[19:02:21 CDT(-0500)] <TonyUnicon> not I

[19:04:02 CDT(-0500)] <dmccallum54> pspaude made a change…. 16m ago

[19:04:31 CDT(-0500)] <pspaude> Yeah sorry, overrode Tony's deletion for the coachSet

[19:04:37 CDT(-0500)] <pspaude> Added it back in.

[19:05:37 CDT(-0500)] <dmccallum54> what's the error jason?

[19:06:11 CDT(-0500)] <JasonElwood> I can fix it. np

[19:29:08 CDT(-0500)] <JasonElwood> Paul- do you have a build with the POM you just checked in?

[19:33:29 CDT(-0500)] <pspaude> Yes

[19:34:30 CDT(-0500)] <pspaude> Is there an issue? I've built on it a couple of times and I think on that one on Win and Linux.

[19:34:58 CDT(-0500)] <JasonElwood> it failed on win ci a minute ago

[19:35:08 CDT(-0500)] <JasonElwood> re-running it now

[19:38:14 CDT(-0500)] <pspaude> Ok, building it now on my win box to see too

[19:39:36 CDT(-0500)] <JasonElwood> it ran the second time. who knows

[19:40:47 CDT(-0500)] <pspaude> Yep mine is good too. It's just one of those days

[19:41:08 CDT(-0500)] <TonyUnicon> ^^ what he said(smile)

[19:42:04 CDT(-0500)] <JasonElwood> so that build is working, but I don't have the external_student_test PK

[19:44:54 CDT(-0500)] <JasonElwood> I'm building on linux ci which hasn't been touched much today except to remove the 3 keys.

[19:45:15 CDT(-0500)] <JasonElwood> but win ci and Sinclair are behaving the same

[19:46:05 CDT(-0500)] <JasonElwood> Paul- if you build lately, what PK do you have on external_student_test?

[19:52:38 CDT(-0500)] <pspaude> primary key is none

[19:53:52 CDT(-0500)] <JasonElwood> Thanks man

[19:54:09 CDT(-0500)] <TonyUnicon> ...

[19:54:20 CDT(-0500)] <TonyUnicon> was your flag turned off?

[19:54:28 CDT(-0500)] <JasonElwood> It didn't add on win or linux ci or Sinclair either

[19:58:15 CDT(-0500)] <TonyUnicon> run this query

[19:58:45 CDT(-0500)] <TonyUnicon> select count(star) from databasechangelog where id = 'Add natural key to v_external_student_test'

[19:58:50 CDT(-0500)] <TonyUnicon> if its >0

[19:58:53 CDT(-0500)] <TonyUnicon> delete it

[19:58:54 CDT(-0500)] <TonyUnicon> and rerun

[20:00:28 CDT(-0500)] <JasonElwood> checksum also?

[20:00:51 CDT(-0500)] <TonyUnicon> coulkdnt hurt

[20:16:13 CDT(-0500)] <JasonElwood> worked on postgres. still waiting on the sql envs

[20:21:35 CDT(-0500)] <JasonElwood> sql failed

[20:21:39 CDT(-0500)] <JasonElwood> I'll email the log

[20:22:52 CDT(-0500)] <TonyUnicon> you send it?

[20:22:57 CDT(-0500)] <JasonElwood> yeah

[20:23:33 CDT(-0500)] <TonyUnicon> fudge

[20:23:49 CDT(-0500)] <TonyUnicon> cant use the date as part of the id string

[20:24:05 CDT(-0500)] <TonyUnicon> did I mention I think sql server is the best ever?

[20:24:08 CDT(-0500)] <TonyUnicon> (smile)

[20:24:35 CDT(-0500)] <JasonElwood> you didn't have to say it, the SQL Server Rocks t-shirt you are wearing speaks loudly.

[20:25:18 CDT(-0500)] <TonyUnicon> there is a guy head banging on it

[20:25:24 CDT(-0500)] <TonyUnicon> its a nice t-shirt

[20:25:44 CDT(-0500)] <TonyUnicon> maybe there is a function I can use, let me use the googles

[20:27:09 CDT(-0500)] <JasonElwood> the google machine will work

[20:33:01 CDT(-0500)] <TonyUnicon> this is what the googles tells me

[20:33:08 CDT(-0500)] <TonyUnicon> does this run on sqlserver

[20:33:09 CDT(-0500)] <TonyUnicon> select school_id||test_code||sub_test_code||cast(test_date as nvarchar(120)) as id,* from external_student_test

[20:34:42 CDT(-0500)] <TonyUnicon> select school_id+test_code+sub_test_code+cast(test_date as nvarchar(120)) as id,* from external_student_test

[20:34:43 CDT(-0500)] <TonyUnicon> rather

[20:35:07 CDT(-0500)] <JasonElwood> yes

[20:35:14 CDT(-0500)] <TonyUnicon> hooray

[20:35:28 CDT(-0500)] <JasonElwood> definitely

[20:37:29 CDT(-0500)] <TonyUnicon> ok

[20:38:00 CDT(-0500)] <TonyUnicon> its in

[20:40:20 CDT(-0500)] <JasonElwood> just run it or checksum, delete row too

[20:42:26 CDT(-0500)] <TonyUnicon> checksum but no delete row

[20:42:34 CDT(-0500)] <TonyUnicon> dont think it inserts till it completes the job

[20:43:25 CDT(-0500)] <JasonElwood> thanks. both are building and they are pretty slow

[20:43:48 CDT(-0500)] <TonyUnicon> i knew this was going to be a nightmare

[20:43:51 CDT(-0500)] <JasonElwood> is the change only for sql server

[20:43:53 CDT(-0500)] <TonyUnicon> i groaned when I saw the ticket(smile)

[20:44:06 CDT(-0500)] <TonyUnicon> it will work with postgres as well

[20:44:10 CDT(-0500)] <JasonElwood> and we avoided the biggest part of the nightmare

[20:44:12 CDT(-0500)] <TonyUnicon> you will have to clear the check sum

[20:46:44 CDT(-0500)] <TonyUnicon> in the postgres instances

[20:57:52 CDT(-0500)] <JasonElwood> postgresq and one sql came up. we can call that victory I think

[20:57:58 CDT(-0500)] <JasonElwood> at 10p I'll take anything

[21:01:37 CDT(-0500)] <TonyUnicon> ok

[21:01:49 CDT(-0500)] <TonyUnicon> chat with you in the mornin

[21:01:55 CDT(-0500)] <TonyUnicon> or not?

[21:01:57 CDT(-0500)] <TonyUnicon> your off

[21:02:08 CDT(-0500)] <JasonElwood> I'm off

[21:02:15 CDT(-0500)] <JasonElwood> unless my wife divorces me

[21:02:16 CDT(-0500)] <TonyUnicon> have a good weekend then

[21:02:21 CDT(-0500)] <JasonElwood> then I'll have plenty time to work

[21:02:21 CDT(-0500)] <TonyUnicon> hah

[21:02:24 CDT(-0500)] <TonyUnicon> take her out to dinner

[21:03:04 CDT(-0500)] <JasonElwood> gonna take more than that. today is our anniversary, I'm here chatting with you.

[21:03:19 CDT(-0500)] <TonyUnicon> oh geez

[21:03:49 CDT(-0500)] <TonyUnicon> i wont keep you then, good luck!

[21:03:53 CDT(-0500)] <JasonElwood> take off man. thanks for staying on late. I'll be able to get Sinclair going.

[21:04:16 CDT(-0500)] <TonyUnicon> alright, np, have a good weekend