March 2005 uPortal Developers Meeting Minutes

Revision of April 24, 2005, 23:45 EDT

uPortal 2.4 performance and memory

by Scott Battaglia, Rutgers (on behalf of Bill Thompson)

(These notes are based in part on the Slides, which has more details.)

we have 10,000 unique users per day, with over 30,000 total logins. We load balance per session to 4 machines.
we are seeing a bad memory leak.
eventually uportal stops working properly and hangs
we didn't see this in QA
seen in production since November 2004

Yale, U LA Lafayette, U CA Urvine, Cornell also see this we bounce portal when available memory goes below 5%
– may be too aggressive
– people lose their session
– if we don't catch it soon enough Apache also goes down this just makes it barely livable

fixes they've done so far:
– removed caching of IPerson

– CError and CSecureInfo now pass events to wrapped channels (they did not before; this meant the channels would hang on to objects)
– restrict access to ChannelFactory's channel cache, and sync instantiateChannel method
– guest sessions created on time out
– AbstractMultithreadedChannels were not cleaning out their channel state maps

3 months later –
-- leaks still exist
– the search continues

now looking for more leaks

– retooled load tests
– production snapshots
– incremental updates
– reaffirm that loadtest system matches production system retooling: attempt to mimic more closely what a user does in prod
– use more custom layouts in test environment
– fewer people logging out (only 25 percent formally do that)
– hitting more popular channels more aggressively try to match the production throughput

– imitate average user session length
– determine rate at which users access system bought test system with same specs as prod system
– ensure database optimizations are the same
– make sure configuration is the same

take production snapshots
– JVM heap size initially 2 GB
lowered JVM heap size to 128 MB on machine, allows us to compare snapshots (on our developer machines which have only 1G. larger snapshots won't compare)

when memory reaches 10% take that one production server out of load balancing rotation
garbage collect
capture snapshot
wait past session timeout (15 minutes)
take another snapshot
compare

– what objects are still in memory
– how much memory they are using

– how much memory things they reference are using
– they are using YourKit to take the snapshots and compare them

– YourKit reports incoming and outgoing references
– totals for objects of each type
– how much memory they consume
– allows us to compare snapshots, showing the details of each object type

understanding the snapshots

– name
– objects
– shallow size (object itself)
– retained size (plus referenced objects)

can trace the path to the root of the garbage collector

another practice to get a handle on the problem: incremental updates:

instead of doing massive patches (either Rutgers' local fixes, or from JA-SIG), we do one little fix at a time, first on the loadtester, then in production
(Scott gave a couple examples of the minimal fixes)

there is a flurry of discussion on JASIG-DEV about memory issues one thing being discussed is to backport the concurrent threadpool
– there are two other issues in the discussion as potential causes

The Concurrent Thread Library, because the threadpool has some problems.

Aaron Hamid at Cornell wrote a patch to replace the thread library

Rutgers manually applied the patch to 2.4.1 into production
This was during spring break.
It seems to have improved performance rather than fixing the leaks.

AuthorizationImpl
– retains references to principals
– no explicit removal of principal from cache
– whenever a new principal is created it copies the map

– Rutgers has a patch for this that they are going to loadtest then place in production

– then they will put it in the JA-SIG CVS HEAD

– introduced a CacheFactory (generic), an interface. they are trying out WhirlyCache
– -- allows for declaring cache settings and policy in XML
– -- allows for fine-grained caching strategies for each part of uPortal.

General principles

– implementing a finalizer should be a last resort
– consider using an open source solution

– be aware of proper caching (where is needed vs. not, weak and soft references, etc.)
– avoid circular references whenever possible

Adam: (heard this at last year's Java One) Now that Sun has 3 different heaps in JVM, caching is not as important as it used to be for temporary objects.

Eric: Rutgers is contributing things to HEAD. Looks good. The rest of us should be careful not to make the same mistakes.

Peter K.: One problem is you can't replicate this well? Q&A does not find it? Do you do snapshots in Q&A? You see ChannelRender instances are being held?

Scott: Yes, by the finalizer. We are probably going to change so the thread pool uses explicit calls rather than finalizers.

Peter K.: How many requests are necessary to saturate the server? Find that out. Use 'last render time' per user to get an equivalent to 'logout'.

Scott: We have four machines in the production pool.

Peter K: Be sure to track how often channels timeout. When you have a huge memory most of that is objects that can be GC'd. But if you have only 128M for your experiment, there will be very little. Most of Peter's comments here relate to trying to figure out why the QA machine does not experience the problem as clearly, (because if it did the problem would be easier to solve).

Eric and Adam questioned whether 128M is enough to realistically run the portal. Make the heap bigger and see if you can fit it on your developer system. Scott replied that 128M is big enough to help diagnose the memory leaks, which is their purpose in the runs in question.

Adam: You should try visualgc

free from Sun (it is no longer available for 1.4, only for 1.5 any more).

Scott: (Adam asked what JVM and parms on it they are using) Solaris, don't know if we're running in 64bit. They are using the default GC algorithm, and using Apache. Someone asked how many AJP threads, Scott didn't know. (are you overriding the default stack size per thread?) Scott didn't know. (-Xss:threadStackSize=)

Nick from U. Wisconsin – been looking into storing sessions into database (for another application), but Andrew says the uportal session is not fully serializable AND we need more stuff than just the session (static caches etc.), so this isn't doable.

Nick gave a few recommendations

uPortal 2.4.x, Future Release, Branch maintenance

by Andrew Petro, Yale

(This session regards how the uPortal project uses its CVS archive)

(The notes are taken in part from the Slides.)

This will cover
– what we currently do
– our deployer community
– what we could do

what we've done

– patches for only the current release
– in up2: One active patches branch, one Head.

before 2.4 was released, there was:

– the current release (2.3.4)
– the 2-3 patches branch (would become 2.3.5)
– the HEAD – would become uportal 2.4
– the SANDBOX – uportal 3 pre-M1

the 2.4 release

– head was tagged as 2.4

– head continues and becomes uPortal 2.5
– created branch for 2-4-patches
– stopped adding to 2-3-patches, and stopped releasing further 2-3
releases.

right now:

– the current release (uportal 2.4.2)
– the 2-4-patches branch
– the HEAD
– the SANDBOX

Andrew made the point that a lot of schools are deployed on older releases.

– we need timely new releases adding functionality we discover we need and can achieve
– high quality of releases
– patches and support for older versions
– -- there are still questions, problems in uP 2.1
– -- back-porting of fixes.

Andrew wants to keep multiple active patches branches.

Problem – too many places where progress can be made, the small developer community becomes diluted.

Andrew says – the idea would be, the older patches branches remain open, but commits and releases will be only in response to contributions, rather than official.

Patches branches would dry up naturally when the contributor community has moved on.

Adam: Closing older branches was an inducement to encourage people to keep up to date.

Jim: Many sites are highly modified, such that it's a big chore to move up. The original decision was a pragmatic one; it was all that he (Ken) could manage to do.

Peter K: The way we did this encourages development and fixes on newer releases. If we keep the older branches open how would we encourage people to commit to later ones? (especially without people riding herd on it like Ken did). If someone commits a patch to an old branch, the committer should very clearly state that in a particular place. (so that others can consider the patch for later branches) Open a Jira issue on the head and latest patches?

John F: The problem would be, how would they test the newer release if they have no instance of it? It may actually already be fixed in a later release, and such a person might not know.

Andrew: The most appropriate way to track patches against older releases is to put it in CVS.

Spring and uPortal 2.x

by Andrew Petro, Yale

(These notes are based in part on the Slides which has more details)

The problem that the Spring framework effort addresses is dependencies.

We developers are addicts to making dependencies – how do we find all the other things we need for a particular class to do its work.

Traditionally we use ("lookup"): JNDI, or servlet context, or a static object, or use a factory.

The more recently introduced way is "Dependency Injection"

– your object knows what it needs
– exposes constructor arguments/ setter methods to fulfill these dependencies
– (some other package goes around and resolves them)
– (therefore,) the object assumes that dependencies have been fulfilled before it tries to do anything
– bonus points: fail gracefully if they have not

Spring as a dependency injector.
You have an XML configuration of the dependencies.
Our IChannel instances, for example, are not Spring-configured.

In JSR-168
Spring PortletMVC can be used for portlets.

Our traditional solution is to use portal.properties.

PersonDirectory reimplemented for uPortal 2.5: it uses Spring so it can gather up the 'way it works' from various pieces.

"Using Spring for the sake of using Spring" is not what this is about. Instead, this is about producing loosely coupled configurable objects.

A Yale use case: At Yale, there's an XML file available on the web that defines
e.g. student support people; How can Yale bring that into the IPerson?
By wiring in more code, another source. So instead of having a local hack, the Spring thing allows you to paste in a wide variety of implementations.

PersonDirectory becomes a façade in front of the Spring-configured person attribute Data Access Object (DAO) that allows the old code to call PersonDirectory in the traditional way.

If there were any other Spring-configured objects, they would not need the façade or the static services, they could have Spring inject their dependencies. Non-Spring-configured objects can use the static way.

a proposed way forward:

more services being Spring-configured when they need to be significantly touched

Packaging:

Peter K recommended that this could be spun out as a micro-project producing a jar
This might ease maintenance:
– multiple codebases can use this code.
Dan E: Good idea. For example, that would make migration of PAGS (into 3.0) easier.

Distributed Layout Manager integration with base code

by Mark Boyd of SunGard SCT

SCT has their own unique layout manager; although it has another name internally, when discussed for JA-SIG purposes, it's called by its old name, the Distributed Layout Manager, called here DLM.

This will be provided in the base code as an alternative to JA-SIG's Aggregated Layout Manager (ALM).

This project has two phases.

First, the DLM will be added to the base code with the capabilities it has at present in Luminis' code. This has only pushed fragments, configured with an XML file rather than a GUI. Each fragment is associated with a userid set aside for the purpose. To configure the contents of the fragment, the fragment owner logs in to that userid and sets up the tabs, columns, and so on.

– this much will be in 2.5. (his part will be done in about 2 weeks from this meeting)

Second, a user interface for creating and editing fragments, will be created for the DLM, based on the code for the present ALM. Pulled fragments will also be implemented.

In the XML, the 'audience declaration' describes the authorization (What is the audience for this fragment?). Keith Stacks of SunGard SCT is working on the authorization part (see another session elsewhere in these minutes).

Mark demonstrated in Luminis (SCT's portal offering) how their DLM layout works. It has finer grained control for what is locked down in a fragment, than does the ALM. So a user can add columns or channels to a pushed fragment if the owner permits it. Preferences can be pushed along with the fragment.

It indicates locked down channels, columns, or tabs by graying out delete buttons and/or leaving out moving buttons.

for 2.6: they will have their part done in 2 or 3 months
both pushed and pulled fragments
define fragments via UI
new permissions enhancements for defining audiences, differentiating
publishing capabilities.

Nick (U. Wisconsin) – they want to roll out with 2.4 but wish they could use DLM anyhow.

2.5 will be out in two or three weeks
2.6 will be out about 6 months later.

uPortal 2.5 release issues

facilitated by Adam Rybicki
(release schedule or release content)

what's new?

– portlet adapter updates
the adapter was portlet API compliant but there were some bugs. e.g. in ProcessAction, one can now programmatically change the window state of the portlet.
redirects from inside a portlet now work.
we now have expiration caching. (optional feature)
enhancements to Exclusive Window State implementation (uPortal enhancement)
e.g. to support downloads via a portlet.

– RDBMServices refactoring
the ant tasks do the db stuff; JNDI can be used only by the portal itself, and the rdbm.properties are used by the ant tasks.
we need pooled connections – Eric wrote RDBMServices to create its own pooled resource when it can't get to JNDI. therefore able to check out spooled connections outside of Tomcat. It will now give you metadata about the nature of the JDBC connection.
– Person Directory Spring re-write had a whole session on that (see above)
– moved to JAXP XML APIs
Sun has backported these API's from 1.5 to 1.4 so the API can look the same.
This does not conflict with the old 1.4 API's.
– JDK1.4 is now a minimum requirement
don't use 1.3 any more, period
– all exceptions are chained
all through the code.

release topics

release management responsibility
– Rutgers did 2.4.2, can we count on them again for 2.5? Scott from Rutgers says Bill (who was not able to attend) was kind of expecting this.
what remains to be completed
– rewrite of the thread pool – Doug Lee's actual package
– about 55 issues in Jira that are targeted to 2.5. are there critical issues that block the release?
– memory leaks (is it a critical enough issue?)

release date – tentatively april 18th

there was a discussion about whether 2.5 should be held up for the memory leak fixes because Rutgers and others really want that in the release, but some people don't want to wait for it to come out because if it delays too long they couldn't commit it to a fall rollout.

Dan: What is the real issue here? Are we worried about selling this to our respective management, or is it a developer concern, we need to do this by such a date etc.

Mark Boyd: Why don't the people who have strong feelings about this work out the issue?

Cris H (Unicon): From our point of view, it's best to have uPortal work well on Java 1.5, because what if the memory leak problem has been fixed in 1.5? So, I recommend we go forward.

LUNCH BREAK

uPortal Clearinghouse

with Patty Gertz and Paul Lynn from Princeton (via speakerphone)

Clearinghouse has two meanings:

The JA-SIG Clearinghouse application (which has much material related to uPortal, and some other things)

and the Clearinghouse machine that the application runs on, which was acquired by a grant from Sun.
Most of Patty's questions today are about the machine.

It's a Sunfire 280R, 64G disk, 8G ram, 2 processors
Princeton charges an annual fee for hosting a server – includes nightly backups, 24 hour monitoring, OS patching. The first year (well, 18 months) of service was donated by Princeton, but it is not clear as yet whether this will be an ongoing contribution.

This is the home of Jira and Confluence. If someone wants to put up some other app and take full responsibility for maintaining it, that's okay.

There are 3 chunks of filesystem space. a production clearinghouse, a dev clearinghouse, and postgres. Jira and Confluence are both on the development disk. Right now there's only one Tomcat. She wants there to
be two Tomcats, and move Jira and Confluence to the production partition. The second Tomcat (dev) is NOT brought up automatically if it dies or the host reboots.

Who will do the work? Jason at Rutgers probably.

Domain names and certs for this server. What do we want to do in terms of standardizing the JA-SIG domain names? Until this week, they didn't have incorporation papers.

Patty prefers
www.ja-sig.org/something
Someone else assumed something.ja-sig.org
Patty points out that the latter would make it more difficult to get security certificates (for SSL and site identification).
Group agreed with Patty to do it first way.

Paul at Princeton asked if we should add more partitions. Our group had no strong feeling about this.

Patty: If new tools want to be added, be sure not to put up tools that require maintenance or licensing fees.
Each tool needs an owner.
Please notify the JA-SIG board (e.g. Patty) that you want to put it on the machine.
Provide the license agreement and if it's open source, put the source itself into the clearinghouse, so that if the tool owner moves on to other things people will still know why it is there.
Andrew pointed out that the CAS (Central Authorization Service, from Yale) project might choose to use the machine as their home.

Patty thought Eric Dalquist was wanting to move other uportal stuff onto the clearing house machine (but Eric is still not present at this meeting.)
Delaware which hosts the CVS archive (people at first surmised that was Eric's concern) doesn't look like they will pull out.
Adam found Eric's notes, he was suggesting the CVS monitor (provides statistics about the use of a CVS instance) should be moved over to the Clearinghouse; it is currently hosted at Unicon, but it's on a low
powered machine with no provision for regular updating of the database. JA-SIG has a license for FishEye

but that would have to run on the same machine as the CVS.
The group approved of the idea in principle.

What is the legal entity for JA-SIG? Patty says it is now a non-profit corporation in New Jersey.

"JA-SIG, Inc." is the official name.

uPortal Documentation

facilitated by John Fereira

Most of the documentation is now being created on the Wiki (Confluence), so do we keep putting things on the main website or mostly on the Wiki?

Recently there has been less opportunity for people to write the formal documentation. Easier to write briefer pieces – how to solve a specific problem.

John thought that the things on the Wiki would be early drafts of things that would later be turned into formal docs. But that process is not yet happening.

Some months ago John demonstrated running a tool called Anakia on the CVS material to produce more nicely formatted and organized docs. He suggests that this sort of thing should be done.

At present, the entire www.uportal.org IS NOTHING BUT a live view on the docs subdirectory of the CVS project.

Doug G.:
(a) CVS has a way to run a script on check-in that could e.g. generate Anakia documentation if needed.
(b) Wouldn't it be better to have a way to 'digest' the Confluence material, which is inherently in a state of flux, to make more static documentation for releases?

John: has a module in CVS called uportal_documentation that contains a snapshot of all the documentation in xml format. The Anakia ant task transforms the xml documentation into html pages, creating "web" and "print" versions.

uPortal-Sakai integration

Andrew Petro did most of this presentation, with Charles Severance present (he arrived during lunchtime)

(This presentation is in part from the Slides

Sakai had navigation requirements. Originally David Haines at Sakai modified up2.4 to make possible what Sakai wanted, but this proved too drastic.

The new plan is to introduce IFrames and WSRP to Sakai, and look at Sakai through uPortal that way. That's phase I. The second phase will be to make Sakai tools available as JSR-168 portlets that can run in uPortal.

Sakai is a Learning Management System, not a portal as such. A portal should be customizable, many things available at the same time. An LMS is not usually highly customizable by its end users, and users want to work with only one thing at a time.

integration goals:

– Sakai TPP Tools will run as JSR-168 portlets.
– An entire Sakai site can be included at some point in an enterprise portal
IFrames - separate sign on
WSRP – shared sign on.
– Sakai sites, tools, or pages, can be aggregated to produce a personal federated view for an individual – moves toward a personal learning and research environment.

these are NOT goals:
– uPortal administration replaces Sakai administration. No, Sakai has its own.
– uPortal navigation supports Sakai's every Learning and Collaboration environment need. No.

Integration directions

Summer 2004 – decided that Sakai will be at first portal-agnostic

Outline
1.0 – integral aggregator with IFrames 10/04
1.5 – IFrames 02/05 just came out
2.0 – minimal WSRP (05/05)
Post 2.0 – Sakai tools available as JSR-168 portlets that run in uPortal

Phase I

Sakai and uPortal deployed separately based on IFrames in uPortal pointing at Sakai or WSRP pointing at Sakai

This integration will work between Sakai and any portal product which supports IFrames
Management and andministration will be done separately for the two.

Phase II

– Get uPortal and Sakai in the same JVM with uPortal handling navigation and layout for Sakai portlets.
– Sakai portlets will be JSR-168 adapted.
– In phase II Sakai will continue to interoperate with other portals using WSRP and IFrames, with uPortal being WSRP producer.
– There are many technical challenges.

Charles did part of the presentation:
Sakai integration phase I through Sakai 2.0
– parts can appear as IFrames within uportal
– adding a WSRP producer to Sakai so portals can show things thereby.
(Charles showed what this looks like)
Charles wants to use a portal to federate among many different instances of Sakai.

Phase 1 tasks:
WSRP Consumer in uportal

IFrame producer in Sakai (1.5)
WSRP producer in Sakai (2.0)
Working on this: Beth Kirshner (UM), Vishal Prashant (SunGard)
May start effort to build a better WSRP consumer for uPortal if Sakai's producer becomes "better".

Vishal Goenka: I've been comparing WSRP and JSR-168 APIs to see what commonality there is. Whether it is easier to go through the JSR-168 adapter to do a particular function or to write it natively.

Phase II

Sakai tools will appear in uPortal as JSR-168 channels

Administered by uPortal admin tools; that is, with respect to publishing, subscribing, positioning in layout.
Run inside same JVM
uPortal will render all navigation (as now)

Sakai tools use Velocity and JSF )Java Server Faces) view technologies.

Sakai's internals are being rearchitected to have less in its "kernel"
– the environment-specific part. This will be the only part that will have to be modified to work with other view technologies such as JSR-168.

JSR-168 portlets other than Sakai Tools may wish to use the Sakai Applications Framework (SAF) Kernel.

The Phase II work will be led by Yale. This is going to be Andrew Petro's primary responsibility for a while.

Make JSR-168 and JSF work in uportal
Make SAF kernel work in uportal
Make JSR-168 Velocity gateway work
Build plugins for Sakai Common Services which talk to uPortal users and groups

Broader opportunities:

Solve the ability to deliver asynchronous events to the browser.
(Sakai's 'courier')

Dan: give an example of uPortal 'managing' a Sakai service, as you said in one of your slides?

Andrew: the portlets will be Sakai because they use Sakai resources out the back end. But uPortal will manage the portlet parts of their nature.

It was never to be intended to be delivered before December 2005 anyway. Charles thought that development was going to be starting in June, but Yale and Andrew offered to do some work starting in April. The phase II effort will not be as aggressive a schedule as phase I has been.

Charles: I have a 20-slide presentation that I will send around instead of presenting here. We have figured out a way to write Sakai services in a flexible, Spring-like way. the way we do this may be helpful to the uPortal 3.0 effort.

uPortal 2.6:

by Andrew Petro

(These notes are taken in part from the Slides which has more detail.)

We usually put out dot releases every six months or so.

The Jira tool allows us to track issues which could be proposed new features for a release, allowing us to contruct a "Roadmap".

Andrew showed (statically) what Jira can keep track of, including its integration with CVS, so you can see e.g what commits mentioned this issue I.D.

he suggests things to consider for 2.6:
– stability
plugging memory leaks
fixing bugs
making our portal robust

– skinning
Jason Shao's ideas and efforts
more use of CSS's and nested divs instead of explicit markup.
– more layout work
DLM
DLM / ALM marriage
programmatic creation of fragments

– nested tabs
make more screen real estate
– WSRP consumption
improve it
– Sakai integration work
improving uPortal by taking advantage of resources made available by Sakai.

There are some suggested new features left over from Bugzilla (the bug tracking software that uPortal used before Jira), some good ideas that we haven't yet implemented and they would be worth reviewing.

An example of a leftover from Bugzilla is persisting minimizedness of channels as part of layout preferences.
(Peter K. suggested a very simple way to do this)

undoubtedly some 2.5 planned stuff did not get done.

Documentation
Administrator tools (duplicating ant tasks)

Groups and Permissions

– extract into standalone API and/or a jar

– Keith Stacks' permissions ideas/work.

What enhancements made locally? What enhancements do we need?

Nick: Some management types have asked for session replication.

Duffy: I am trying to work on 'programmatically creatable fragments' and 'internal navigation'. Is that something we want for 2.6 or wait until 3.0?

Peter K: It will be a lot easier for someone to do that in 3.0.

(Since the meeting, a path forward has been agreed upon: discover uP 2.6 goals via list discussion, and in our Confluence Wiki, then make these into a roadmap of scheduled issues in Jira.)

Permissions Framework

– Keith Stacks

SunGard SCT refers to the principal associated with a channel as the "audience" (who can get
at that channel). Their permissions model has a much more flexible way of granting permissions, using complex boolean logic expressions.
SunGard SCT is contributing this model to the JA-SIG code base.

They have an easy way to edit these complex expressions with a series of selections from drop down lists rather than having to type expressions by hand. He showed in slides how it works. It will be available both from the Channel Manager when setting channel permissions, and from the Permissions Manager.

The user attributes from the PAGS can be used directly; other groups are from the regular groups service.

There are several new classes, including IEntityFilterGroup (which holds the new style of permission expressions) and IOperator (which represents the individual AND/OR/NOT operators, and which can be expanded with new operators).

Further modifications needed to the existing infrastructure consist of:

Need to add a column to the UP_PERMISSION table IPermission updates

IPermissibles API to indicate whether an activity supports a restriction
A new User Attributes servant
Runtime updatable PAGS store
PAGS store will support named and unnamed groups (unnamed are the AND/OR things)

Dan: Keith spoke of fine-grained permissions – refers to the GUI and it could be done a variety of ways. Fine-grained restrictions is the part that changes the underlying design. Dan has a concern that this design requires adding PAGS in order to work. Keith says that the "User Attributes" part of their permissions is a pluggable configuration, which is currently by default configured to use PAGS.

Charles asked whether this was usable from JSR-168, which cannot get directly at Java Interfaces.

NOTE: On Monday evening we met for dinner at a nearby restaurant in honor of Ken Weiner who served as project lead of uPortal for several years, but who recently left for another position. Ken joined us for the dinner.

Eric Dalquist
– we want to put the presentations online

uPortal 3

facilitated by Peter Kharchenko

(These notes are taken in part from the Slides, which has more detail.)

Strategic goals:
– configuration flexibility
rendering mechanisms
HTTP parameter generating/processing mechanisms
data layer implementations
– long-term maintenance
isolated, individually configured groups of components

transparent, sparse coupling

Tactical goals
– JSR-168 support – in uportal 2 it's more obviously an afterthought; make it more integral
– Sakai integration
– uPortal2 support
– Complete WSRP integration

Sakai beta – June 2005
implements minimum Sakai phase II requirements

3.0 Beta: non-2.x users – August 2005
completes Sakai integration requirements
completes core features for new uportal adopters

3.0 Final: Upgrades – September/October 2005
migration tools for existing 2.x users

IChannel adapter, DB migration
Requirements not finalized at this time

up3 dependencies

uPortal
make sure that available 3.0 resources are actually being applied to 3.0 (resources as in people's time; this means don't get sidetracked on other things)

Support from 2.x resources
Define upgrade/migration tools

Sakai
Refining Sakai requirements past June release
Development
WSRP consumer, deployment testing

alternative (to up2)
skin/structure
OSID Authentication/Authorization

Testing
identify resources/institutions
QA and Performance Test

Quality Assurance and Testing

facilitated by Eric Dalquist

Standardization on what testing processes are done before a release is cut
There should be scripts etc. to do the cutting of a release
There needs to be documentation on how to test. e.g. Simple Layouts were not being tested.

Curt: Is it considered too difficult to set up regression tests (a test suite)?

Eric: That would be great. Would someone be interested in writing more unit tests for up 2.x? There are not many as yet.

Andrew: in 2.5 there is an approach to creating a fake data source, that can be used for unit testing. Perhaps an evolving test structure will be happening in 2.x, rather than 'test everything ASAP', such that as people touch various parts of code, they can add JUnit tests.

Scott: Faizan has user acceptance tests. We're setting up CruiseControl (automated build tool) to run on Confluence on the Clearinghouse machine. Maybe we should run it on uPortal also. And we have a license for Clover (unit testing evaluator).

Eric: The people who do the next few releases should be writing up what they do, which will be the beginning of the documentation.

Dan: Some people at various institutions would test things relevant to their individual site. We should gather all these tests.

Duffy: Mentioned 'HttpUnit' which can do live scripted testing.

Eric: Whoever is in charge of making the release, has the responsibility of finding others to test it.

Scott: We need a documented process of testing, in case the person responsible has an emergency of some kind.

Peter K: We are being very careful in writing unit tests for 3.0. But we don't have a functional test. What experience do people have with automated functional testing?

Peter K: For some things it is more realistic to automate testing, than for other things. For example publishing a portlet, subscribing to it, seeing if it actually renders.

Nick: You said there would be a nightly build process; how about a nightly deploy process? Maybe we could deploy it on the clearinghouse machine and people could try it out.

Nick: Regression testing tools are very expensive and none of them beats the human eye.

Peter K: I think Cruise Control will help a good bit.

Scott: I agree we need a deployed instance available somewhere.

Mark: (a) Documentation of unit testing (b) Add a unit test to test anything newly coded (c) A place to run these tests (d) a deployed instance.

Cris: Then you can talk about automated performance testing. Which would find memory leaks right when they are added.

There was a discussion of whether we can expect a 'dot zero' release to have no known unresolved bugs. Some thought that either this should be required, or we should only call it a "release candidate". Others thought that even commercial software does not achieve this; therefore we should inform the user community to proceed with caution.

Another issue discussed was, can we identify more people's time to apply to do additional pre-release testing? Charles Severance said regarding that: Sakai has a volunteer QA effort. The people who are responsible for making Sakai work at various sites, some are willing to try out the new code to see if it is good. We have gotten volunteer QA to work. We tell them to use Jira to report problems, then the developers monitor that. There needs to be leadership for QA, but it probably should not be a developer.

Rendering Contexts and components for uportal 3

by Peter Kharchenko

(These notes are taken in part from the Slides which has more detail.)

(The concept is, that the entire way that uPortal determines a user's layout, prepares it for display, and responds to inputs, can be different for different users in the same instance of uPortal. Actually, a user might even switch between contexts within the scope of the same session.)

What is a uportal 3 context?
defines all aspects of a traditional uportal instance
design, navigation, profiles
parameter syntax
portlet selections, session scopes
multiple concurrent contexts can be used

example contexts
static frame showing a single portlet
simple navigation with human-readable URLs
uportal 2 clone

(a graphic) showing a context resolver at the top that routes requests to appropriate context, and one of the contexts under it is the imitation of uportal2.

context resolvers:

Servlet Path context resolver

switch contexts by going to a specific servlet path
default, ServletPathContextResolver implementation

Atlernatively contexts can be resolved using
user information
authentication state
user affiliation (groups), attributes (permissions)
user agent information (browser)
internal states, session (e.g. down for maintenance)

Mark: could you make a context that does not create a session? e.g. for Guest user.
Peter K: Yes.

Context responsibilities:
User interaction
Presentation

IContext
service (HTTPRequest, HttpResponse)

CompoundContext implements IContext
separates parameter processing and rendering
IRequestParameterController

processparameters(HttpRequest,HttpResponse)
IPortalRenderer
render(HttpRequest, HttpResponse)

Rendering: drilling down

TreeRenderer implements IPortalRenderer
Assumes a rooted structure of components
Responsible for rendering cycle initiation/events

IRenderingComponentManager
Defines the topology of rendering components

Responsible for finding optimal cache entry points

PipelineComponentManager
defines linear pipeline rendering structures

Rendering: SAX pipelines

uportal2-like pipeline in which components pass SAX events to each other

ISaxConsumer and ISaxRenderer

components
abstract filters and buffers
Serializers
XSL transformers

SAX Cache entry points
know how to replay cache
returned by cache managers
implementing IPipelineCacheManager

Rendering: String pipelines

Pipeline that operates by transforming Strings
quick, non-restricted

IStringRenderer and IStringFilter

Components
RegexStringFilter
PortletRenderingInitiationFilter
StringResponseWriter
String cache filters, managers, entry points

Rendering: Hybrid pipelines

Pipelines that can be configured with multiple types of components
org.jasig.portal.rendering.hybrid
component manager implementation
connector structure
cache manager implementation
multiple cache types
prioritized

Sould be primary choice for configuring context rendering

up2 context example

(He showed an XML example of a uPortal components configuration)

Rendering: XML Layout sources

Starting point for the Sax pipelines
XmlFileLayoutSource

org.jasig.portal.layout.uportal2 package
LegacyLayoutSource
wraps IUserLayoutManager
SimpleUserLayoutManager

lightly modified uportal2 version
with simplified DAO Impl

Rendering: XSLT TransformationFilter

A flexible XSLT SAX Rendering component
configured with ITransformerPreferences
insert arbitrary element attributes and params

Transformer preferences
contain IStylesheetDescriptoin
location, mime type, etc.
IParamDescription and
IElementAttributeDescription
default
values, persistence scope, target elements
responsible for keeping track of current values of stylesheet params, attributes
RuntimeTransformerPreferences (in memory)
TransformerPreferencesImpl (persistent)
ITransformerHelper - stylesheet handling

(He showed an XML example of attributes configuration)

now he went back to an earlier slide to talk about the

IRequestParameterController

CompoundContext parameter processing

Before the rendering starts, need to make sense of the request parameters

RequestParameterProcessorListController
based on a prioritized list of parameter processors
each parameter processor talks to some component
ordered list of IRequestParameterProcessors
boolean processParameters(req, res)
allows multiple rounds of processing for variable evaluation
set of IUrlConstructorProviders

Example: uportal2 transformation legacy syntax

Context patterns: parameter-aware component

Goal
isolation and targeted injection of components Parameter-aware context component
directly involved in the rendering
assisting other rendering components
assisting other parameter processors

Pattern:
implement URL constructor (ie. how to write)
implement ParameterProcessor (i.e. how to parse)
Connect processor, component itself and other pieces using dependency injection
Test URL constructor and processor against each other

Example 1: Legacy up2 Layout param syntax

Example 2: persistence manager
job: fire off persistence events
request param command to initiate persistence
no rendering, just param processing ...

The fate of the URL constructors

Controller configuration is passed to Pluto's PortletURLFactoryImpl

PortletURL delivered to the portlets
implements IPortletURL
adds ability to obtain uPortal-specific URL constructors
getUrlConstructor (name)
IPortletUrlConstructor
implements portlet URL syntax
also configurable

Pluto integration, WSRP producer and consumer

by Michael Ivanov

JSR-168 Pluto container portlet rendering engine implements JSR-168 provides a runtime environment for portlets and calls Portlet invoker to execute them contains interfaces that ...

(He showed a diagram of the parts of Pluto)

container methods

init()/shutdown() lifecycle methods called only once
portletLoad() called only once before aggregation
renderPortlet() – caled for all portlets once per refresh
processPortletAction() – called for only one portlet

Container Services

Mandatory Services
Information Provider Service
Factory Manager Service
Log Service
Optional services
Property manager service

Additional portal-specific services

Information Provider Service
callback mechanism for the container into the portal
Dynamic and static information provider

Portal Context Provider

Provides Pluto with the portal context information

Pluto's Object Model
Represents portlet information on different levels
('definition' has parameters of a 'class' nature; 'entity' is of an 'object' nature.)

WebApplicationDefinition
PortletApplicationDefinition
PortletApplicationEntity

ServletDefinition
PortletDefinition
PortletEntity
PortletWindow

for uportal3:
implement Pluto's object model interfaces

implement uportal-specific portlet URL
implement dynamic and static information providers
create factories for pluto's components that need to be replaced
implement configurable factory management service
provide factoryManagerService that allows to replace major container
factories within the portal
provide LogService

in uportal2 we have a 'published' and 'subscribed' channel
'deployments' are the static configuration info. analogous to 'channel
types' – they have readonly information from the portlet.xml files.

'definitions' – publish information for a particular portlet. (channel params)
'entities' – subscribed portlet (user params)
'windows' – appearing somewhere in a window (as yet, no distinct prefs for uportal)

Portlet URL implementation

PortletURL Constructor

Constructs Portlet URLs

Factory Manager Service

Vikram: Are you leveraging the sample portal code that comes with Pluto? Peter K: No, we are specifically trying to avoid using that. We are writing our side from scratch. uPortal is too different from the Pluto portal skeleton code. We did use some things from the container implementation, but we copied it rather than extended it.

WSRP Producer in uPortal 3

(refer to Michael's slides)

the org.jasig.portal.wsrp.producer.WsrpEngine is the main class it requires Axis

org.jasig.portal.wsrp.producer.provider.Provider

an access point for WSRP engine to uportal object model.

The WSRP component of the uPortal 3 development is at first using JDO and JPOX for WSRP Persistence

Why JDO (he explains this)

Vikram: WSRP4J uses a different db implementation, doesn't it?
Michael: Yes, but we modified it to use JDO.

Duffy: What about Hibernate vs. JDO?

Peter K: JDO is easier because in this case we don't know what we are going to be persisting. That would be hard to do with DAO, wouldn't it?
Michael: It would not be hard to change the implementation itself, or the ones that call it. But we're using JDO to get the WSRP feature working. Then we will switch over to the standard DAO things in uportal. By then the latter will be more mature.

(He gave an example of JDO meta-data)

What has been done?
What is in progress?
Open issues

Do we need OpenSAML or WS-Security? Is it possible to implement them in WSRP 1.0?
Charles: The real question is, do we need to do more than just trust the consumer.
Vikram: We can use transport mechanisms to do that – e.g. SSL.
Charles: First, deal with the problem of trusting our own consumers.
Secondly, deal with the problem of trusting others' consumers.
Adam: We can have transport security, and as several have suggested, CAS Proxy for authentication. What about signing?

Vikram: Message-level security may not buy us much. (later) WSRP has a concept of "user category" which is like an authorization "role". (there was much discussion of WSRP security)

Is it possible for WSRP Producer to share one Pluto container's instance with uPortal 3 by changing dynamic providers on-the-fly? i.e. if regular portal requests and WSRP requests come into the same instance of the portal servlet can we handle both of them? There was some discussion of this suggesting how to do it.

LUNCH BREAK

uPortal 3 – Portlet Development

Eric Dalquist

One problem is, providing access to portal services from a portlet
(portlets have their own class loader so you can't
get at uPortal classes in the normal way)
Special access-layer classes (shared library location in the servlet container)
if the portal API changes, then the interfaces in this layer have to change as well as their implementations.
this increases maintenance overhead and debugging effort.
Moving services to shared location

Peter K: the way Sakai does the first one: Constructed cross-context class loaders with a binding to Spring. From the user application you are using Spring injection without being aware of the problem.
Adam: Is this servlet-container independent?
Peter K: If the servlet context listener works in a certain way, that would be the only relevant dependency.
Eric: Casting classes is very difficult between class loaders.

The other way is to move all the services into a jar in the container shared library. e.g. the Groups and Permissions.
Adam: Be sure it will work in any J2EE container. Some have had difficulty with specific containers.
Eric: Well if this would not work, then Pluto would not work anyway because it depends on the same ability.
Adam: There are already industry-standard ways to do this sort of thing, maybe this is the time to consider them.
Eric: Pluto works with WebLogic, Orion, Tomcat. Therefore, this would probably also work.
Adam: We should not require one particular servlet container.

Eric: If you put the Groups jar in the shared library, then your portlet has more dependencies and it does not purely meet the standard.
Cris: Running a portable portlet is not a very exciting thing right now. There is little you can do with them. The meaty type portlets are going to require dependencies to the particular portal you want to use.
Is there an effort to enhance JSR-168?
Eric: I haven't seen one.
Vikram: There is some sense that once the WSRP revision is done, JSR-168 will then be revised.

Vikram: Where does the Group service fit into a portlet? Is it used via 'IsUserInRole' portlet method, which is static access control, whereas groups are dynamic?
(there was an extensive discussion of this question)
Some people supposed that we would write only fully standard portlets, others supposed that we would also write things with portlet APIs but that also used uportal services ("portlet-like things"). The debate can be summarized like this: 'But that's not a portlet'. 'Not completely, but we don't mind.'

Cris: Here's a tip. If one group (such as JA-SIG) is building a variety of portlets, you should build your portlets into one web application, so that you won't get like fifty web applications all running at once. That's a big performance problem.

(NOTE: Bill Thompson was on speakerphone from Rutgers for part of the afternoon. From sometime in the uPortal3 portlets session, through the 'leadership' session after it)

Administrative Portlets
Consistent and flexible UI
Inclusion in the portal web application

(The rest of this session consisted of Cris H. on his investigation into what's available for the "consistent and flexible UI" part)

Spring PortletMVC
Bill Thompson – support: it's been in there about a year. The code seems to be stable enough for production. But maybe it doesn't have the elegance of the rest of the Spring framework and it has some rough edges around it. There is some talk to roll it out as a separate Spring subproject, so that it can be released on its own release cycle. It just needs some leadership and attention.

Struts
Beyond what we know and love as Struts today. First of all, there is a part of the Struts project being spun off called 'Portals Bridges'.
it's part of Jetspeed 2 and they are trying to split it off from Jetspeed.

Then there are two proposals for Struts 2 one is called Shale, and the other is called Jericho.
The fact that there are two proposals, hints how far away from Struts 2 they are.
Both of them mention portlet support.

Another thing called MVCPortlet (not Spring) that is being open sourced by a company. This is the only one of these that says it's an official release (and they sell commercial support for it.)

JSF (Java Server Faces)

Sakai makes heavy use of JSF. JSF kind of takes development back from web applications to more the way stand alone apps were written. With for example, events and listeners. Supposedly there exists shim code to make this work within a portlet.

uPortal Leadership

discussion facilitated by Eric Dalquist

Ken has left. What do we do now?
– Long Term Release Management
– Development Issue Decision Making
– Concrete Leadership Role

Development Issue Decision Making
there was a benevolent dictator kind of position that he had.
how will we handle such issues now?

Andrew: Should we have a "Ken version 2" Project Manager?
Doug: Should we split it up into different projects with different people?
Adam: It would be easier to ask someone to volunteer to lead a project (instead of being specifically paid to do so) if it is smaller than 'forever'.

Ken was funded 80 percent development and 20 percent project management. Does the board want to continue funding someone? Bill Thompson (on speakerphone) is on the JA-SIG board. The board is looking to the community for the immediate future. Bill thinks that people who have the thing in production would pitch in for it. Rutgers would be willing to help with 2.4.3 and may need some help with working on 2.5. Andrew (Yale) is very involved with 2.5 so could give advice to Rutgers about that.

Jim: Peter K is the chief architect for 3.0.
Peter K: As yet 3.0 is in its infancy so most decisions are still architectural. As it matures there will be more managerial type decisions, with which I have much less prior experience. So there should be more community participation than just me. Or perhaps Eric could help with that.

Andrew: Should we say: Please Unicon you're doing the development, could you also do project management? Or should we say you're doing enough so someone else should do it?

Jim: Until December 30, 2005, Carl Jacobsen is responsible for the Mellon funding and making decisions, so he should make the call on the 3.0 effort. After December 30 the funding ends and it would be approached differently.

Doug: from Unicon's perspective, within the company, Eric would be the best choice to do 3.0 project management.

Bill T: I'm not sure we need a meta manager for 2.x as long as there are volunteers for each dot release.

Eric: One thing that isn't covered is when Ken used to make sure that a patch on the HEAD also goes into the patches branch or vice versa.

Bill T: Regarding the memory leak issues you discussed. Our intent with Scott was to share what our struggles have been so we can head off things before they bite us in production. We released with expectations of quality (that were not met). I think we need to adopt the concept of quality and have a QA process. So that people will know what to expect.

BREAK

Miscellaneous uPortal 3 Questions

by Peter Kharchenko
(he just brought up these things that need to be decided eventually; there was no attempt to discuss them at length or make decisions)

(These notes are based in part on the Powerpoint presentation, which has more detail.)

layout managers

Choices:
simple user layout manager
ALM
DLM

Integration? (we want to eventually have one code base for those two things, merge the features; but the effort has been pushed off into the future because there are not resources for it at present)

Sakai Priorities
simple layout manager is sufficient for Phase II

Community evaluation of choices/requirements

uportal3 authentication

uportal2 interfaces

multiple authentication context
opaque credentials

why not use some existing framework to do some of it for us?

Acegi (AY-see-gee)
default db, ldap
CAS

What information do we need from authentication subsystem?

uportal3: navigation, back button

uportal2 structure/theme
Alternative skin (theme) – SAKAI
Alternative structure
trees
nested tabs

tabs/trees combo (U. Wisconsin)

Back button behavior
– GET
– POST

To make the back button work properly we would have to have idempotent (can only have an effect once) URLs.

Eric: The biggest problem is to be able to have two windows open on the same portal and interact with both of them at once. Because the location in the portal is part of your session, not part of the link.

Andrew: Also, bookmarkable links would be great. (going directly to a specific tab or focused portlet).

service integration

Slides
Main services from uPortal2
directory
groups
permissions

how to share?
avoid maintaining two copies (for up2 and up3)
separate deployment of each service

(i.e. pull these pieces out of the code base so they can be the same
for up2 and up3)

community involvement requested for the development of uPortal 3:

Internationalization (XLIFF, etc.)

Administrative Portlets
publish, subscribe (extensions)
compound preferences portlet
profiles, skins, internationalization
runtime administration
cache control (empty them or tune performance characteristics) (2.5 we can dump some caches from the Channel Admin channel)

IChannel adapter to allow old channels to run in uPortal 3
Upgrade tools

NOTE: A few minutes before the very end of our meeting, we were reminded that we were meeting in California, by way of a magnitude 3.4 (i.e minor) earthquake, whose epicenter was 4 miles to the southwest beneath Santa Monica Bay. It lasted about 3 seconds.