ssp-data-importer

ssp-data-importer is a JASIG SSP project for importing csv data for the external tables into the SSP database. This program inserts data from csv files into a database. Csv files have some latitude in parsing including whitespace, separator, and quote. Currently, the program supports only updates and inserts. It is possible to upload partial data updating specific columns. When used as a method to update already existing data, the only requirement is that the columns containing key values be supplied. Only supplied column values will be overwritten.

High Level Workflow

The ssp-data-importer is expected to be used as part of a cronjob or other periodic method for running the program, e.g. Windows Task Scheduler. The job should be set up to run with a period that is appropriate for how often the external tables are to be updated. It is important that the upload to the input folder be timed to the running of the cronjob.

1. A folder that will contain the .csv files (input folder) to import is monitored at runtime.
2. If the folder contains .csv files and the files have had a sufficient soak time since modification, the files are transfered to a folder (process folder).
3. Files in the process folder are validated. Validation is on a column basis. All errors in a given row are documented for the final job-level report. Validation is based on database metadata constraints.
4. Validated rows are written to a second folder (upsert folder).
5. Files in the upsert folder are inserted to staging tables in the database.
6. Additional validation steps are taken on the complete data set evaluating for inconsistencies and any potential duplications.
7. The validated staging tables are then used to update/insert (upsert) data into the correspondingexternal_* tables.
8. A report is generated and emailed giving pertinent information including any validation errors, total lines per table processed etc.
9. Finally, staging tables are truncated, processing and upset folders are removed and the processed files are archived.

For a full description of ssp-data-importer including requirements, how to run the program, file formats etc. Please see the Jasig SSP project readme.md at GitHub.