uP2 XML Import Export with Cernunnos

Proposed presenter

Drew Wills via dial in. Drew has generously agreed to update this page today with more info on his availability and interest in doing this and should be contacted for more info as needed.

Clock time required

TBD.

Slot concept / outline / content

I understand Drew Wills has got some really neat XML export-import code using his Cernunnos framework for uP2 domain entities including layouts, channels, users, groups, etc. I wonder if he could be prevailed upon to say a few words about this, via a dial-in if necessary, to twig people to the availability of this code.

Discussion of whether and when this should be part of uPortal itself.

XML Import/Export for uP v2.5 Using Cernunnos

Thanks to Andrew P. for taking an interest and allowing me to abuse his patience with these ideas.

Motivation

Every uPortal deployment needs a collection of basic data – groups, channels, permissions, fragments, etc. – in order to do it's job delivering content to users. This data needs to reflect the goals of the institution sponsoring the portal, and these needs differ significantly from one institution to the next.

I spend a lot of time planning, doing, and thinking about migrating uPortal-based portals to 2.5 from earlier versions. I performed this upgrade for Cal Poly and University of Colorado System myself, and I advised the team that prepared Chico State's migration. I've also met with UNICON PMs on several occasions to develop our knowledge base surrounding the tasks and challenges associated with upgrading uPortal.

A very large portion of the effort involved in migrating to uPortal 2.5 is spent managing portal data. The challenge is that you want some stuff from the existing portal database, some stuff from the base data set contained in the new version, and some stuff needs to be newly developed to meet institution-specific needs that come into focus with the migration.

Each time I have invested a lot of effort analyzing the delta between the institution's existing data and the base data set for the current uPortal version, as well as the delta between the base data set for the current version compared with the new version. This process gives me the union of that which is desirable from the current portal and that which is desirable from the new version. Thereafter, more data comes to light periodically as the team goes through the process of rethinking groups, layouts, and fragments.

These changes are very difficult to manage in data.xml. Sometimes changes are made by hand, sometimes changes are made in a running portal instance and exported using the 'dbunload' target. Either way it's cryptic, tedious, time-consuming, and error-prone.

Why do we go to all this trouble? Why don't we just deploy the new code base and connect to the existing database? Well, normally institutions upgrade uPortal in order to take advantage of new features (especially DLM). Often these new features represent a paradigm shift from the way things worked previously. Using DLM effectively means radical changes to your template users and adding fragment owners and their layouts. It may also mean reevaluating the geography of your groups and permissions. Upgrade teams may also take advantage of this opportunity to do some housecleaning: pruning user accounts that are out of use, etc.

Why deal with data.xml? Why not make all these changes using the portal UI and simply go forward with the resulting database? Well, typically the upgrade team will need to spin up several portal environments before the new portal is ready to go into production. Each developer may use a local deployment to make his or her changes, and the new portal may use separate, server-hardware environments for Development, QA, and (ultimately) Production. Each of these needs its own database.

It's natural to use the 'initportal' Ant target to propagate the data set to these environments, as well as to use source code control for safeguarding the "official" data set and tracking the evolution of changes to it. And when you're ready to deploy the new portal to your production hardware, you want to have confidence that you can configure your groups, permissions, channels, and fragments quickly and dependably.

Version migrations are not the only scenarios where these sorts of considerations come into play. Every time we develop new content for the portal we may need to push more data – channel definitions, fragments, permissions – through several environments on the way to production. And when we're merely adding content, moreover, it's especially unattractive to use data.xml because calling 'initportal' resets the entire portal database, causing users to loose their preferences settings, etc.

Proposed Solution

I believe it would be better to move away from using data.xml for the initial portal data set and instead use a collection of domain-specific XML documents. This approach would make this data more readable (and manageable) by human beings. It would allow institutions to "develop" and maintain data for things like channels, groups, permissions, and fragments in source code control. Modified documents could be re-imported individually, and new documents could be imported freely within existing portals.

These XML documents should not contain any database IDs so that new entities may be "authored" in XML and freely imported into existing databases. Entities that reference other entities (e.g. foreign keys) must do so by another unique attribute – e.g. username, fname, group name (or path), etc.

I believe the Cernunnos Project is a good match for implementing these features quickly and reliably. It allows for each of the required import or export operations to be expressed as a succinct, XML-based script, and it avoids the need for custom Java code to perform behaviors like XSL transformations, RDBMS upsert, DOM manipulation, evaluating XPath expressions, etc. Cernunnos scripts (and the resources they use) can be invoked from the classpath, which means they will be available both from the console and within the running webapp.

About Cernunnos

Cernunnos is an open source project hosted by Google Code. Andrew Wills (Deactivated) is the founder and (as yet) the sole committer. Cernunnos aims to make Java developers more effective by providing solid, reusable implementations for common behaviors that are easy to instrument through scripts and even integrate with custom or existing code.

Example Cernunnos Script

The following script imports an XML structure that represents a permissions entry. Scroll down the page for an example of a permissions entry.

<sql-upsert>
    <update-statement>UPDATE up_permission SET owner = ?, permission_type = ? WHERE principal_type = (SELECT entity_type_id FROM up_entity_type WHERE entity_type_name = ?) AND principal_key = ? AND activity = ? AND target = ?</update-statement>
    <insert-statement>INSERT INTO up_permission(owner, permission_type, principal_type, principal_key, activity, target) VALUES(?, ?, (SELECT entity_type_id FROM up_entity_type WHERE entity_type_name = ?), ?, ?, ?)</insert-statement>
    <parameter value="${valueOf(owner)}"/>
    <parameter value="${valueOf(permission-type)}"/>
    <parameter value="${valueOf(principal-type)}"/>
    <parameter value="${org.jasig.portal.security.GroupKeyOrLiteralPhrase(${singleNode(principal/*)})}"/>
    <parameter value="${valueOf(activity)}"/>
    <parameter value="${org.jasig.portal.security.GroupKeyOrLiteralPhrase(${singleNode(target/*)})}"/>
</sql-upsert>

Overview of Portal Entities

Here is a 10,000' view of data in uPortal. I prepared this list while working on the upgrade for Cal Poly and it's come in handy on several occasions.

From this list you can see that later entities have dependencies (foreign keys) on earlier items. Put another way, these entities must be imported into the portal database in the order shown.

Status

I've been prototyping this approach at odd moments for a couple weeks.

I did it because:

• I believe it would be tremendously useful, especially to me; and
• These features seemed like a good match for the Cernunnos technology I've been working on

I started with Layouts on the list above and worked "backwards" through Groups. In other words, the working prototype (attached to this page and described below) can import and export the following types of entities:

• Groups
• Channels
• Group Memberships
• Permissions
• Layouts

Example XML Documents

Note that each of the following XML structures contains a script attribute on the root element. This attribute tells the import service what script to use to import it. Using this approach, we will be able simply to drop in new scripts to support new types of entities or new formats for existing entities. If a future version of uPortal adds more members to one of these entities, we could create a new XML format to support these members and a new script to import it. The "old" script could then be modified to transform the "old" XML to the new format (using intelligent defaults) and subsequently invoke the new script.

Groups

<group script="classpath://org/jasig/portal/groups/import-group.crn">
  <name>Developers</name>
  <entity-type>org.jasig.portal.security.IPerson</entity-type>
  <description>All IT developers</description>
</group>

Memberships

<membership script="classpath://org/jasig/portal/groups/import-membership.crn">
  <parent>Everyone</parent>
  <child> 
    <group>Staff</group> 
  </child>
</membership>

Channels

Note that the channel example uses nearly the same format as the 'pubchan' Ant target.

The only differences are:

• Removal of the DOCTYPE declaration
• Addition of the script attribute

The script for importing channels actually calls the same Java class that is invoked by the 'pubchan' target.

<channel-definition script="classpath://org/jasig/portal/channels/import-channel.crn">
  <title>Daily Business Cartoon</title>
  <name>Daily Business Cartoon</name>
  <fname>daily-business-cartoon</fname>
  <desc>Daily Business Cartoon by Ted Goff, www.tedgoff.com</desc>
  <type>Image</type>
  <class>org.jasig.portal.channels.CImage</class>
  <timeout>30000</timeout>
  <hasedit>N</hasedit>
  <hashelp>N</hashelp>
  <hasabout>N</hasabout>
  <secure>N</secure>
  <locale>en_US</locale>
  <categories>
    <category>CImage</category>
    <category>Entertainment</category>
  </categories>
  <groups>
    <group>Everyone</group>
  </groups>
  <parameters>
    <parameter> 
      <name>img-border</name>  
      <value>0</value>  
      <description/>  
      <ovrd>N</ovrd> 
    </parameter>
    <parameter> 
      <name>img-height</name>  
      <value>300</value>  
      <description/>  
      <ovrd>N</ovrd> 
    </parameter>
    <parameter> 
      <name>img-link</name>  
      <value>http://www.tedgoff.com</value>  
      <description/>  
      <ovrd>N</ovrd> 
    </parameter>
    <parameter> 
      <name>img-uri</name>  
      <value>http://www.tedgoff.com/mb/images/today.gif</value>  
      <description/>  
      <ovrd>N</ovrd> 
    </parameter>
    <parameter> 
      <name>img-width</name>  
      <value>300</value>  
      <description/>  
      <ovrd>N</ovrd> 
    </parameter>
  </parameters>
</channel-definition>

Permissions

<permission script="classpath://org/jasig/portal/security/import-permission.crn">
  <owner>org.jasig.portal.channels.groupsmanager.CGroupsManager</owner>
  <principal-type>org.jasig.portal.groups.IEntityGroup</principal-type>
  <principal> 
    <group>Developers</group> 
  </principal>
  <activity>SELECT</activity>
  <target> 
    <group>Faculty</group> 
  </target>
  <permission-type>GRANT</permission-type>
</permission>

Layouts

<layout username="admin" script="classpath://org/jasig/portal/layout/simple/import-layout.crn">
  <root name="Root folder" immutable="N" unremovable="Y">
    <header name="Header folder" immutable="Y" unremovable="Y">
      <channel fname="header"/>
      <channel fname="portal/login/general"/>
      <channel fname="session-locales-selector"/>
    </header>
    <tab name="TRAINING" immutable="N" unremovable="N">
      <column name="Column" immutable="N" unremovable="N">
        <channel fname="books"/>
        <channel fname="SpringGuessNumberPortlet"/>
        <channel fname="BookmarkPortlet"/>
        <channel fname="GuessNumberPortlet"/>
        <channel fname="HelloWorldPortlet"/>
      </column>
    </tab>
    <tab name="TEST" immutable="N" unremovable="N">
      <column name="Column" immutable="N" unremovable="N">
        <channel fname="GatewayPortlet"/>
        <channel fname="PermissionsPortlet"/>
        <channel fname="MessagingPortlet"/>
        <channel fname="BriefcasePortlet"/>
      </column>
    </tab>
    <tab name="Main" immutable="N" unremovable="N">
      <column name="Column 1" immutable="N" unremovable="N">
        <channel fname="word-of-the-day"/>
        <channel fname="number-guessing-game">
          <param>
            <name>minNum</name>
            <value>1</value>
          </param>
          <param>
            <name>maxNum</name>
            <value>300</value>
          </param>
        </channel>
      </column>
      <column name="Column 2" immutable="N" unremovable="N">
        <channel fname="daily-business-cartoon"/>
      </column>
    </tab>
    <tab name="Admin Tools" immutable="N" unremovable="N">
      <column name="Column" immutable="N" unremovable="N">
        <channel fname="groupsmanager"/>
      </column>
      <column name="Column" immutable="N" unremovable="N">
        <channel fname="permissionsmanager"/>
      </column>
    </tab>
    <tab name="Development" immutable="N" unremovable="N">
      <column name="Column 1" immutable="N" unremovable="N">
        <channel fname="person-attributes"/>
        <channel fname="css-viewer"/>
      </column>
      <column name="Column 2" immutable="N" unremovable="N">
        <channel fname="uportal-development-books"/>
        <channel fname="uportal-developers-reference"/>
        <channel fname="snooper"/>
        <channel fname="uportal-data-dictionary"/>
      </column>
    </tab>
    <tab name="CWebProxy Examples" immutable="N" unremovable="N">
      <column name="Column 1" immutable="N" unremovable="N">
        <channel fname="tomcat-number-guess"/>
        <channel fname="tomcat-number-guess"/>
      </column>
      <column name="Column 2" immutable="N" unremovable="N">
        <channel fname="tomcat-servlet-examples"/>
      </column>
    </tab>
    <footer name="Footer folder" immutable="N" unremovable="N">
      <channel fname="footer"/>
    </footer>
  </root>
</layout>

Working Prototype

I've attached my working prototype to this page. To install it, just dump the zip into a uPortal 2.5.3 (or later) distribution. I only added new files, no existing uPortal files were changed to build this prototype. There are 28 files in total.

uPortal_2-5-3_GA [uPortal ROOT]
  import-export.xml
  |
  + lib/
    cernunnos.jar
    commons-jexl-1.1.jar
    dom4j.jar
    jaxen.jar
  |
  + source/
    |
    + org/
      |
      + jasig/
        |
        + portal/
          export.crn
          import.crn
          |
          + channels/
            export-channel.crn
            import-channel.crn
            PortletPreferenceOverridePhrase.java
          |
          + groups/
            export-group.crn
            export-memberships.crn
            GetMemberIsGroupPhrase.java
            GetMemberKeyPhrase.java
            GetMemberServicePhrase.java
            import-group.crn
            import-membership.crn
          |
          + layout/
            |
            + simple/
              export-layout.crn
              export-layout.xsl
              import-layout.crn
              import-layout.xsl
          |
          + security/
            DefaultUsernamePhrase.java
            export-permissions.crn
            export-user.crn
            GroupKeyOrLiteralPhrase.java
            import-permission.crn
            import-user.crn
          |
          + tools/
            |
            + chanpub/
              UrlChannelPublisher.java

Prototype Files: Detailed Notes

import-export.xml

This is an Ant script that allows you to invoke the import-export features from the command line. It contains two targets: export and import. If this technology is adopted by the community, these targets could be added to build.xml, allowing this file to be removed.

JAR Files

The four JAR files listed above contain the Cernunnos technology and the libraries required by it.

Cernunnos Scripts (*.crn)

These files are where the import-export features are implemented. See above for an example script.

Custom Phrase Implementations

Phrase is a Java interface within Cernunnos. This prototype contains four custom Phrase implementations: GetMemberIsGroupPhrase, GetMemberKeyPhrase, GetMemberServicePhrase, and GroupKeyOrLiteralPhrase. Each of these implements a short series of operations that would be cumbersome and inelegant to write in Cernunnos.

XSL Transformations

This prototype contains two XSL stylesheets: export-layout.xsl and import-layout.crn. These implement operations that are more easily expressed in XSL than either Cernunnos or Java.

UrlChannelPublisher.java

This prototype uses uPortal's 'pubchan' functionality to import channels. I had to extend the existing Java implementation to allow pubchan to read documents that are not located in the properties/chanpub/ directory within the classpath.

Using the Prototype

To use the prototype, extract the contents into the root of a uPortal 2.5 (or later) distribution. You can then invoke it using Ant as described below.

• ant -f import-export.xml export -Ddir=dir -Dtype=type (-Dsysid=sysid)
• ant -f import-export.xml import (-Ddir=dir) (-Dpattern=pattern)