This Proposal

This proposal describes an overhaul of uPortal as far as the way project sources are organized, and as far as how the project is built & deployed. The high-level aims of this proposal are to make the project more "modern" and support IT professionals – developers, designers, sysadmins, etc. – who wish to interact with the project in ways the current setup doesn't lend itself to. These changes are proposed for uPortal 5, which (we hope) would see a release before the Open Apereo 2017 conference.

High-Level Goals

Shift the customary deployment paradigm away from building on each portal server

Many uPortal adopters build & deploy in other ways, but the only deployment paradigm uPortal fully supports out-of-the-box requires the following steps (beyond those required for running uPortal) on each webserver:

Install Ant, Maven, & Git
```
$git clone the project repo
```

$cp build.properties.sample build.properties and configure locally

Configure filters/{env}.properties locally

```
$ant -Denv={env} deploy-ear
```

Updates require additional $git commands and additional builds.

We would like to make uPortal more agreeable to building once, deploying everywhere.

Allow multiple adopters to deploy the same, Apereo-provided binaries

Apereo uPortal is Free & Open Source Software (FOSS). It will always be possible to clone the uPortal repo and build your own copy. Many adopters will aways choose to build uPortal themselves owing to local customizations or enhancements. Adopters who do not have local customizations or enhancements to the core uPortal technology, however, should not be obligated to build it.

It should be able to apply your own...

Configuration
Data
Modules
Skin

To uPortal binaries provided by Apereo.

Move uPortal toward "cloud native"

The artifacts produced by a uPortal build should be ready to deploy to the cloud. They should be capable of taking advantage of what the cloud has to offer. The uPortal service should be decomposed into smaller, coordinated Microservices.

To see plans in progress for Docker, see the Docker Build Artifacts page.

Parker's List

Reduce size of Core uPortal and potentially containerize (Docker)
Break out all (or a subset) of portlets that are distributed with uPortal into containerized soffits (at least the ones that are not widely used)
Consider use of Distributed Caching (Elasticache/memcached/redis) instead of shared uPortal database for clustering
User a cloud notification service (AWS SNS) to push notifications between portlets/soffits and even mobile devices
Move event logging out of the uPortal DB into a NOSQL DB i.e. Dynamo db using eventual consistency, or provide a logging event interface to allow developers to write their own event logging backends
Consider new auth technologies i.e. OpenIdConnect that would allow for authentication context/state to be maintained on the client.
Provide REST APIs for as much of the uPortal admin functionality as possible

The Cloud Native Computing Foundation defines Cloud Native in its charter:

Container packaged. Running applications and processes in software containers as an isolated unit of application deployment, and as a mechanism to achieve high levels of resource isolation. Improves overall developer experience, fosters code and component reuse and simplify operations for cloud native applications.
Dynamically managed. Actively scheduled and actively managed by a central orchestrating process. Radically improve machine efficiency and resource utilization while reducing the cost associated with maintenance and operations.
Micro-services oriented. Loosely coupled with dependencies explicitly described (e.g. through service endpoints). Significantly increase the overall agility and maintainability of applications. The foundation will shape the evolution of the technology to advance the state of the art for application management, and to make the technology ubiquitous and easily available through reliable interfaces.

Further Reading

This blog post from Andrew Petro makes a lot of points that line up nicely with this proposal.

Make uPortal easier to work with for developers, especially new developers

There are several things we'd like to see in this area.

Fewer prerequisites
More clarity about which things to configure or customize (as an adopter), and which to leave alone
Faster builds
Fewer required restarts
Reduced "learning curve"

Remove sources and/or features that are @Deprecated, unused, or unreachable (within reason)

uPortal is a project with over a decade and a half of history. The framework – and web development generally – have seen many changes in this period. There are many sources and/or features in uPortal that don't contribute to future success. We should "prune" as much of this stuff as we reasonably can for two important reasons:

To reduce the maintenance burdon
To make the platform more nimble

We need to steer uPortal tolwards a successful future, but the sheer size of the vessel makes maneuvering more difficult.

There is an epic in JIRA for this effort.

Tentative list of items to remove

Items on this list are subject to review & discussion. All removals should be handled through a plull request process on GitHub. Discussions about removals on the dev/user lists are also encouraged. If you see something on this list that you feel should not be removed, please speak up! If you have a suggestion for something that should be on the list, please say so!

Legacy (XML file configuration) DLM fragments (Pull Request: https://github.com/Jasig/uPortal/pull/843)
Legacy (XML file configuration) PAGS groups
Tab groups
Opt-in fragments
A few items in LESS and JS specific to old (unsupported) versions of IE
- https://github.com/Jasig/uPortal/blob/c5d425de9738e73432bf4c2d066b67782495b001/uportal-war/src/main/webapp/media/skins/common/javascript/uportal/flyout-nav.js#L96
- https://github.com/Jasig/uPortal/blob/0a084305db85da9adcc71429e57e86c6835f60df/uportal-war/src/main/webapp/media/skins/respondr/common/less/tags.less#L67

Embrace the Classpath

Remove WEB-INF altogether as much and as many times as possible. Everything should preferably be under src/main/resources within the appropriate module. (Many modules will contain resources.) Make JARs, not WARs.

Challenges

Some of the goals outlined above are at odds with the way in which uPortal source code is organized and the build system operates. As one of the outcomes of this effort, we should look at reorganizing uPortal project sources and build processes to be "more friendly" to these goals.

Essence of a uPortal Implementation

The following items are what make your portal your portal. Customarily they are maintained in the same Git repo as – and mixed with – upstream sources from Apereo in a variety of locations all over the repo. If we get to a point where many adopters can use the same binaries, they will nevertheless need to maintain custom versions of these items.

Configuration

Most configuration settings are in uportal-war/src/main/resources/properties, but not all of them. Configuration comes primarily in two forms: properties files and XML files. Properties are much simpler and can be externally overridden (outside the uPortal webapp and Tomcat) using a properties overrides file.

Data

Your mix of database objects is also unique to your portal implementation. uPortal has a system for serializing & deserializing data into XML files called entities or entity files. These XML files are normally maintained for the life of the project and kept in uportal-war/src/main/data.

Skin

The appearance and brand identity of your portal is unique to your implementation. Visual customizations are implemented as one or more skins, and are maintained in the Git repository under uportal-war/src/main/webapp/media/skins/respondr.

Shell-Based Administrative Tools

uPortal includes several administrative tools that are available from the command line within the root directory of a uPortal Git clone. These tools are indispensable, and not all of them have a web interface equivalent. It would be difficult to operate a uPortal service without these tools. These tools are built on the project sources, local configuration files (not the copies in Tomcat), and other resources within the Git clone; they are integrated with – and offered as a part of – the uPortal build system.

Partial List

data-export
data-import
deployPortletApp
md5passwd
hsql
initdb

Shared Classloader Resources

Java portlet support requires that the Portlet API jar, the Pluto runtime, and a few other jar files get copied to a shared.loader location within Tomcat. The uPortal build system takes care of these items. If we switch to a build system on a separate machine, we will need another way to handle this requirement.

Outcomes

We should reorganize uPortal source files using an approach that is more contemporary and more friendly to the goals described above. This change will make the builds faster, and it will allow us to "reinvent" what implementing uPortal looks like. We will be able to have a build system that takes binaries built independently – quite possibly by Apereo itself – and combine them with locally-managed configuration and skin resources into a runnable service. These changes are desired by members of the uPortal community for the next major version (uPortal 5), and the time to work on the next major version is at hand. This outcome includes a migration from the current Ant/Maven based build system to Gradle – a newer, more contemporary Java build system.

uPortal sources decomposed into +/- 12 modern components with Gradle-based build scripts
Shell-based administrative tools migrated from Ant
Current Ant/Maven build scripts removed, together with any uPortal components that exist solely to support them
uPortal 5 released

Plan of Action

Update the version of the master branch to 5.0.0-SNAPSHOT; the kinds of changes we will be making should appear only in a new major version
Add a Gradle Wrapper based on the latest version of Gradle to uPortal and a build.gradle to the root directory; this step means you won't have to install Gradle to build uPortal
Teach the existing build.xml (Ant file) to invoke the Gradle build before the Maven build; this step means we can have a functioning uPortal with a working build during the early transition process
Starting at the root directory, lay out a starting collection of hierarchical sub-modules using names that begin with up[ortal]-; as they move, Re-package all Java classes & classpath resources starting with org.apereo... (now is a good time to make this change); examples...
- up-core
- up-authn
- up-user-attributes
- up-groups
  - up-pags
- up-layouts
- etc.
Include a build.gradle for each level of this module hierarchy
Update the the root build.gradle to invoke its children; update each of those to invoke it's own children, etc.
Start porting code & classpath resources from uportal-war into this structure appropriately
Add dependencies to each build.gradle as you go; make sure we have a working build after each move; commit after each move
Several resposibilities need to be moved out of Maven entirely...
- Compilation of all Java sources
- Packaging of all classpath resources
- All the little functions that the Maven plugins provide – at least all the functions we want o retain
Once all these things are ported out of Maven, remove Maven from the equasion
At this point we can teach Gradle to invoke Ant; this tactic will get us through the phase where we no longer need Maven to build, but we rely on Ant for shell-based admin functions
Reproduce necessary shell tools without Ant
Eliminate Ant, together with any uPortal components that exist solely to support the Ant/Maven build
Move the uPortal Git repo to the uPortal-Project Organization in Github

Work in Progress

Branch in Github

There is a branch in Github where this work is already underway. Please look over the commits to get a better sense of these changes.

Further Reading