SSP Source Code Upgrade Process

This page still in progress...

Overview

All real-world SSP deployments fork SSP-Platform source code. Most end up forking SSP source code as well. Thus upgrades typically involve some amount of source code merging, at a minimum for SSP-Platform.

By definition, these instructions cannot provide comprehensive advice for all the situations you may encounter when upgrading source code forks, and they should be understand as a best-effort attempt at a generic recipe for working through any given upgrade, with pointers on known issues and "gotchas" mixed in.

In many cases, a systems administrator with relatively limited SSP-specific experience can successfully execute an upgrade. But again, because upgrading a source code fork almost always involves making decisions about what does and doesn't constitute a valid local change that should be carried forward into the upgraded source code, it may be necessary to involve a developer or technical staff already familiar with SSP internals.

The SSP team strongly recommends a multi-stage process for upgrading institutional SSP installations. Start by attempting the upgrade locally, even in the absence of a complete copy of UAT or production databases. Work out the obvious problems there, then move to your shared QA/Test/UAT/Dev environment and attempt a deployment. You'll usually want to refresh the database in that environment from a recent production backup, if possible, before attempting the deployment. Once everything has been tested successfully in that environment, you should have a very good idea of the exact steps you'll need to reproduce to execute the deployment in production. 

SSP Source Code Upgrade

If your deployment is not derived from a fork of the SSP source code project, skip this section and go to SSP-Platform Source Code Upgrade below.

Deployments typically upgrade/merge SSP source code before upgrading/merging SSP-Platform code because SSP-Platform has a build-time dependency on SSP.

In general, the SSP source code upgrade process steps through:

  1. Inventory local customizations
  2. Identify customizations to be preserved
  3. Merge latest upstream code into local development branch
  4. Resolve conflicts
  5. Test locally
  6. Commit, Push
  7. [Upgrade SSP-Platform]
  8. Test in shared QA
  9. Merge development branch into prod branch
  10. Deploy to PROD

SSP Source Code Upgrade - Example

The following is a schematic example of the high-level source code merge process.

# By convention, the 'origin' remote typically points to your fork in your Git hosting service,
# and upstream points to the canonical Apereo SSP repo in Github. E.g.:
#    $ git remote -vvv
#    origin	git@github.com:edu.edu/SSP.git (fetch)
#    origin	git@github.com:edu.edu/SSP.git (push)
#    upstream	git@github.com:Jasig/SSP.git (fetch)
#    upstream	git@github.com:Jasig/SSP.git (push)
# 
# Get the latest remote code
$ git fetch origin
$ git fetch upstream
 
# Make sure you're on your dev branch. The convention is <institution>-ssp-dev
$ git checkout edu-ssp-dev
 
# If that errors out, with an error like:
#    error: pathspec 'edu-ssp-dev' did not match any file(s) known to git.
# Then you probably need to create the branch and link it the remote copy that should already exist in 'origin':
$ git checkout -b edu-ssp-dev origin/edu-ssp-dev
 
# Create a branch to hold your place:
$ git checkout -b edu-ssp-dev-pre-upgrade
 
# Push it to your hosted repo:
$ git push origin edu-ssp-dev-pre-upgrade
 
# Switch back to your actual dev branch:
$ git checkout edu-ssp-dev
 
# Capture local changes. This will help with two things:
#
#   1. Deciding whether to merge or reset-and-cherry-pick, and
#   2. Sanity checking the result of either approach
#
# This can be a bit complicated (http://stackoverflow.com/questions/1527234/finding-a-branch-point-with-git)
# but usually it's sufficient to get the diff against the current state of the upstream
# branch on which your dev branch is based:
#
#  $ git diff upstream/<most-recently-merged-branch>..HEAD
# 
# E.g. for a local dev branch based on rel-2-0-patches:
$ git diff upstream/rel-2-0-patches..HEAD > /my/upgrade/notes/pre-upgrade-local-diff
 
# Or, get a list of all the commits in your dev branch that aren't in the most
# recently merged upstream branch. Again, for a 2.0-based dev branch:
$ git rev-list upstream/rel-2-0-patches..HEAD
 
# Or:
$ git log upstream/rel-2-0-patches..HEAD
 
# It may also be useful to use a tool like Atlassian SourceTree to help visualize how your dev branch has diverged
# and inventory the patches to be preserved.
 
# If local changes are minimal, such that a merge is likely to create more problems
# than it solves because of unrelated conflicts in upstream code, it's often easiest
# to just reset your dev branch to point to the target upstream branch and cherry-pick
# local customizations into the result.
#
# Start by forcing the local dev branch to point at upgrade target. E.g. for a 2.5 upgrade:
$ git reset --hard origin/rel-2-5-patches
# Re-apply local changes via cherry-pick. E.g. for each commit in
# 'git rev-list upstream/rel-2-0-patches..HEAD' from above:
$ git cherry-pick <commit>
# If git reports conflicts on a cherry-pick command, you'll need to:
#  - Get the list of conflicted files. Thesea re all the paths listed below "Unmerged paths"
#    in the output from 'git status':
$ git status
#  - Open each conflicted file, correct the conflicts and
#    'git add' the corrected files (http://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging):
$ git add <corrected-file>
#  - Complete the cherry-pick once all conflicts are corrected. You should
#    be prompted to edit the commit message.
$ git cherry-pick --continue
 
# Or, if local changes are extensive, such that history is important and isolating each
# relevant commit for cherry-pick is not feasible, use a standard 'git merge'. Again
# assuming an upgrade to 2.5.x:
$ git merge origin/rel-2-5-patches
# Deal with conflicts in the same way as described above for cherry-picks:
#  - Get the list of conflicted files. Thesea re all the paths listed below "Unmerged paths"
#    in the output from 'git status':
$ git status
#  - Open each conflicted file, correct the conflicts and
#    'git add' the corrected files (http://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging):
$ git add <corrected-file>
#  - Commit the result once all conflicts are corrected
$ git commit
 
# See below for more information on typical merge conflicts.
 
# Try building SSP:
$ mvn -Dmaven.test.skip=true clean install
 
# If there are problems, the errors will usually come from the Java compiler and will list the problematic
# source files. For each such problematic file, open it for edit, fix the problem, and 'git add' the change:
$ git add <fixed-file>
# Repeat the fix/add/build process until the build completes successfully. Then commit the change.
# You should be prompted to supply a commit message:
$ git commit
 
# Sanity check that the differences relative to the upgrade target reflect the local customizations
# you decided should be preserved. Again for a 2.5.x upgrade:
$ git diff origin/rel-2-5-patches HEAD
 
# If anything seems to be missing either fix it right away or take a note to address it once
# you get to a more stable point in the overall upgrade.
 
# At this point you'll probably want to back up your work to your hosted repo.
# Push your dev branch to a temporary remote branch:
$ git push origin edu-ssp-dev:edu-ssp-dev-upgrade
 
# Now switch to the SSP-Platform source code upgrade process described below.
# Once that is complete and tested, you'll want to push your SSP dev branch
# to your hosted repo using its canonical name.
#
# If you used the 'reset' approach above, you'll need to specify the '-f' argument:
$ git push -f origin edu-ssp-dev
# Else if you used the 'merge' approach, leave the '-f' off:
$ git push origin edu-ssp-dev
 
# You may want to leave the placeholder 'edu-ssp-dev-pre-upgrade' branch as a
# reference point until the upgrade is complete. Once it's outlived its usefulness,
# you can delete it with:
$ git push origin :edu-ssp-dev-pre-upgrade

TODO: prod merge, tagging instructions

TODO: notes on what to do when switching from a customized branch to non-customized GA

TODO: notes on what to do when the post-merge diff shows spurious deltas (just checkout the files in question (see CPCC notes))

SSP Source Code Upgrade - Common Conflicts

SSP Source Code Upgrade - Common Conflicts - Through 2.2.x

FileNote
pom.xmlConflict almost always trivial and caused by the way the SSP release process bumps artifact versions in this file. Can almost always just accept the upstream version wholesale (the 'checkout --theirs' described below).

src/main/java/org/jasig/ssp/service/reference/impl/MessageTemplateServiceImpl.java

 
src/main/java/org/jasig/ssp/transferobject/TemplateOutputTO.java 
src/main/resources/org/jasig/ssp/database/masterChangeLog.xml 
src/main/webapp/app/controller/admin/crg/EditReferralViewController.js 
src/main/webapp/app/controller/tool/map/LoadTemplateViewController.js 
src/main/webapp/app/controller/tool/map/SemesterPanelContainerViewController.js 
src/main/webapp/app/controller/tool/map/SemesterPanelViewController.js 
src/main/webapp/app/service/MapPlanService.js 
src/main/webapp/app/store/external/Terms.js 

If you're seeing conflicts on a small subset of those files and you know you don't need to preserve any local customizations, it's easiest to just accept the upstream state of those files in their entirety. For each such file:

$ git checkout --theirs <conflicted-path>

Or, if you're seeing conflicts on all a large set of files either in or not in this list, and you know you don't need to preserve any local customizations for those files, it's easiest to just revert the merge and start over specifying that the upstream version should be preferred in the case of conflict:

$ git merge --abort
$ git merge -s recursive -Xtheirs upstream/<upgrade-target-branch>

Otherwise familiarize yourself with the way Git represents conflicts within files, open each conflicted file for edit, fix the problem, and git add the file to indicate that the conflict has been resolved.

SSP Source Code Upgrade - Common Conflicts - Through 2.3.x

FileNotes
pom.xmlConflict almost always trivial and caused by the way the SSP release process bumps artifact versions in this file. Can almost always just accept the upstream version wholesale (the 'checkout --theirs' described below).
src/main/java/org/jasig/ssp/web/api/TemplateController.java 
src/main/java/org/jasig/ssp/web/api/reports/PersonHistoryReportController.java 

src/main/webapp/app/controller/tool/map/SaveTemplateViewController.js

 

src/main/webapp/app/view/tools/map/StudentTranscriptViewer.js

 

The 2.2.x recommendations above also apply here.

SSP Source Code Upgrade - Common Conflicts - Through 2.4.x

FileNotes
pom.xmlConflict almost always trivial and caused by the way the SSP release process bumps artifact versions in this file. Can almost always just accept the upstream version wholesale (the 'checkout --theirs' described below).

SSP Source Code Upgrade - Common Conflicts - Through 2.5.x

FileNotes
pom.xmlConflict almost always trivial and caused by the way the SSP release process bumps artifact versions in this file. Can almost always just accept the upstream version wholesale (the 'checkout --theirs' described below). 

SSP-Platform Source Code Upgrade

The process for upgrading SSP-Platform source code is effectively identical to that described above for SSP, except that:

  1. At this writing all known real-world SSP deployments do maintain SSP-Platform source code forks, so dealing with source code upgrades is effectively non-optional in a way that it's not for SSP itself.
  2. The local branch naming convention is <institution>-ssp-platform-[dev|prod]. E.g. for a dev branch: edu-ssp-platform-dev.
  3. SSP-Platform patches tend to deal mostly with complex local integration configurations, especially with directory services. As such, it is much more common to be carrying truly local customizations, as opposed to SSP, which tends to carry select upstream patches for bug fixes and feature adds. Thus the git reset --hard option is typically not a realistic option when upgrading SSP-Platform. Most upgrades tend to use a traditional git merge
  4. If local changes to SSP are modeled as "overlay" files in ./uportal-portlets-overlay/ssp, Git cannot help detect conflicts nor merge in the latest upstream changes. Thus that approach for managing local SSP customizations is not recommended since it can become very easy to "forget" to upgrade individual files, which leads to difficult to debug problems during upgrade QA.

TODO: notes on where the SSP dep is set

TODO: notes on what to do with the Platform artifact version, even when originally based on a GA

TODO: Special notes on dealing with filters file... actually, that should go in the release notes b/c filters/ssp-platform-config.properties really shoudlnt be in SCM (mainly thinking about doco for the one-liner for figuring out which property is referenced where)