...
[11:45:29 CST(-0600)] <dmccallum54> raw files -> per-row type, width, nullity validations -> filtered/validated files -> batched inserts to stage tables -> upserts to live tables
[11:46:52 CST(-0600)] <TonyUnicon> but lets say we have a dup row in the file
[11:46:56 CST(-0600)] <TonyUnicon> in that workflow
[11:47:04 CST(-0600)] <TonyUnicon> it would fail on the stage inserts
[11:47:11 CST(-0600)] <TonyUnicon> which I thought you wanted to avoid
[11:47:24 CST(-0600)] <dmccallum54> that particular error would not be caught until [batched inserts to stage tables] and the resulting error would be as inspecific as the batch size is large
[11:47:42 CST(-0600)] <dmccallum54> inspecific. wtf.
[11:47:44 CST(-0600)] <dmccallum54> non-specific