The complete minutes from the Spring 2005 developers meeting are currently being reviewed. They will be posted as soon as they are readyRevision of April 24, 2005, 23:45 EDT
| Monday, March 21 |
|
|---|---|---|
09:00am | Introduction~edalquist | |
09:15am | ||
09:55am | ||
10:10am | ||
10:30am | Morning Break |
|
10:45am | ||
11:25am | ||
12:00pm | Lunch Break |
|
01:00pm | ||
01:20pm | ||
01:50pm | ~awp9 Andrew Petro and Chuck Severance | |
02:40pm | Afternoon Break |
|
02:45pm | ||
03:25pm | ||
04:00pm | End of day |
|
|
|
|
| Tuesday, March 22 |
|
09:00am | ||
09:10am | ||
10:00am | Morning Break |
|
10:20am | ||
11:20am | ||
11:45am | ||
12:20pm | Lunch Break |
|
01:20pm | ||
02:20pm | ||
03:10pm | Afternoon Break |
|
03:25pm | ||
03:35pm | ||
03:45pm | ||
03:55pm | ||
04:00pm | End of day |
|
uPortal 2.4
...
performance and
...
memory
| Anchor | ||||
|---|---|---|---|---|
|
...
...
...
...
...
...
...
uPortal Developers' meeting
March 21-22, 2005 Monday and Tuesday
at the offices of Unicon, Inc., El Segundo, CA.
revision of April 24, 2005, 23:45 EDT
Attendees
Duffy Gillman – University ofArizona
Dan Ellentuck – Columbia University
John Fereira – Cornell University
Michael Oltz – Cornell University
Sriman Ravi – University of Georgia
Jim Farmer – instructional media + magic, inc.
Patty Gertz* – Princeton
Paul Lynn* – Princeton
William G. Thompson, Jr.* – Rutgers University
Scott Battaglia – Rutgers University
Chuck Severance – Sakai - University of Michigan
Ann Badertscher – Sigma Systems, Inc.
Mark R. Boyd – SunGard SCT
Vishal Goenka – SunGard SCT
Keith Stacks – SunGard SCT
Mike Zackrison – SunGard SCT
Eric Dalquist – Unicon, Inc.
Michael Ivanov – Unicon, Inc.
Freddy Lopez – Unicon, Inc.
Cris J. Holdorph – Unicon, Inc.
Peter Kharchenko – Unicon, Inc.
Adam Rybicki – Unicon, Inc.
Doug Gouldin – Unicon, Inc.
Nicholas Blair – University of Wisconsin - Madison
Curt Ellmann – University of Wisconsin - Madison
Andrew Petro – Yale University
Useful links of concepts often referred to.
uPortal homepage
uPortal Confluence
uPortal CVS archive
uPortal Jira
JA-SIG Clearinghouse
MONDAY March 21
uPortal 2.4 performance and memory
by Scott Battaglia, Rutgers on behalf of Bill Thompson
[by Scott Battaglia, Rutgers (on behalf of Bill Thompson)
(These notes are based in part on the Powerpoint presentation Slides, which has more details.])
we have 10,000 unique users per day, with over 30,000 total logins. We load balance per session to 4 machines.
we are seeing a bad memory leak.
eventually uportal stops working properly and hangs
we didn't see this in QA
seen in production since November 2004
...
– retooled load tests
– production snapshots
– incremental updates
– reaffirm that loadtest system matches production system retooling: attempt to mimic more closely what a user does in prod
– use more custom layouts in test environment
– fewer people logging out (only 25 percent formally do that)
– hitting more popular channels more aggressively try to match the production throughput
...
– name
– objects
– shallow size (object itself)
– retained size (plus referenced objects)
can trace the path to the root of the garbage collector
...
instead of doing massive patches (either Rutgers' local fixes, or from JA-SIG), we do one little fix at a time, first on the loadtester, then in production
(Scott gave a couple examples of the minimal fixes)
there is a flurry of discussion on JASIG-DEV about memory issues one thing being discussed is to backport the concurrent threadpool
– there are two other issues in the discussion as potential causes
...
– introduced a CacheFactory (generic), an interface. they are trying out WhirlyCache
– -- allows for declaring cache settings and policy in XML
– -- allows for fine-grained caching strategies for each part of uPortal.
...
Scott: (Adam asked what JVM and parms on it they are using) Solaris, don't know if we're running in 64bit. They are using the default GC algorithm, and using Apache. Someone asked how many AJP threads, Scott didn't know. (are you overriding the default stack size per thread?) Scott didn't know. (-Xss:threadStackSize=)
Nick from U. Wisconsin – been looking into storing sessions into database (for another application), but Andrew says the uportal session is not fully serializable AND we need more stuff than just the session (static caches etc.), so this isn't doable.
Nick gave a few recommendations
uPortal 2.4.x, Future Release, Branch maintenance
| Anchor | ||||
|---|---|---|---|---|
|
by Andrew Petro, Yale
[(This session regards how the uPortal project uses its CVS archive])
[(The notes are taken in part from the Powerpoint presentation Slides.])
This will cover
– what we currently do
– our deployer community
– what we could do
...
– the current release (2.3.4)
– the 2-3 patches branch (would become 2.3.5)
– the HEAD – would become uportal 2.4
– the SANDBOX – uportal 3 pre-M1
...
Peter K: The way we did this encourages development and fixes on newer releases. If we keep the older branches open how would we encourage people to commit to later ones? (especially without people riding herd on it like Ken did). If someone commits a patch to an old branch, the committer should very clearly state that in a particular place. (so that others can consider the patch for later branches) Open a Jira issue on the head and latest patches?
...
Andrew: The most appropriate way to track patches against older releases is to put it in CVS.
Spring and uPortal 2.x
| Anchor | ||||
|---|---|---|---|---|
|
by Andrew Petro, Yale
[(These notes are based in part on the Powerpoint presentation Slides which has more details])
The problem that the Spring framework effort addresses is dependencies.
...
– your object knows what it needs
– exposes constructor arguments/ setter methods to fulfill these dependencies
– (some other package goes around and resolves them)
– (therefore,) the object assumes that dependencies have been fulfilled before it tries to do anything
– bonus points: fail gracefully if they have not
...
Peter K recommended that this could be spun out as a micro-project producing a jar
This might ease maintenance:
– multiple codebases can use this code.
Dan E: Good idea. For example, that would make migration of PAGS (into 3.0) easier.
Distributed Layout Manager integration with base code
| Anchor | ||||
|---|---|---|---|---|
|
by Mark Boyd of SunGard SCT
...
In the XML, the 'audience declaration' describes the authorization (What is the audience for this fragment?). Keith Stacks of SunGard SCT is working on the authorization part (see another session elsewhere in these minutes).
Mark demonstrated in Luminis (SCT's portal offering) how their DLM layout works. It has finer grained control for what is locked down in a fragment, than does the ALM. So a user can add columns or channels to a pushed fragment if the owner permits it. Preferences can be pushed along with the fragment.
...
2.5 will be out in two or three weeks
2.6 will be out about 6 months later.
uPortal 2.5 release issues
| Anchor | ||||
|---|---|---|---|---|
|
facilitated by Adam Rybicki
(release schedule or release content)
what's new?
– portlet adapter updates
the adapter was portlet API compliant but there were some bugs. e.g. in ProcessAction, one can now programmatically change the window state of the portlet.
redirects from inside a portlet now work.
we now have expiration caching. (optional feature)
enhancements to Exclusive Window State implementation (uPortal enhancement)
e.g. to support downloads via a portlet.
– RDBMServices refactoring
the ant tasks do the db stuff; JNDI can be used only by the portal itself, and the rdbm.properties are used by the ant tasks.
we need pooled connections – Eric wrote RDBMServices to create its own pooled resource when it can't get to JNDI. therefore able to check out spooled connections outside of Tomcat. It will now give you metadata about the nature of the JDBC connection.
– Person Directory Spring re-write had a whole session on that (see above)
– moved to JAXP XML APIs
Sun has backported these API's from 1.5 to 1.4 so the API can look the same.
This does not conflict with the old 1.4 API's.
– JDK1.4 is now a minimum requirement
don't use 1.3 any more, period
– all exceptions are chained
all through the code.
...
release management responsibility
– Rutgers did 2.4.2, can we count on them again for 2.5? Scott from Rutgers says Bill (who was not able to attend) was kind of expecting this.
what remains to be completed
– rewrite of the thread pool – Doug Lee's actual package
– about 55 issues in Jira that are targeted to 2.5. are there critical issues that block the release?
– memory leaks (is it a critical enough issue?)
release date – tentatively april 18th
...
Cris H (Unicon): From our point of view, it's best to have uPortal work well on Java 1.5, because what if the memory leak problem has been fixed in 1.5? So, I recommend we go forward.
LUNCH BREAK
uPortal Clearinghouse
| Anchor | ||||
|---|---|---|---|---|
|
with Patty Gertz and Paul Lynn from Princeton [(via
speakerphone])
Clearinghouse has two meanings:
The JA-SIG Clearinghouse application (which has much material related to uPortal, and some other things)
and the Clearinghouse machine that the application runs on, which was acquired by a grant from Sun.
Most of Patty's questions today are about the machine.
...
Patty: If new tools want to be added, be sure not to put up tools that require maintenance or licensing fees.
Each tool needs an owner.
Please notify the JA-SIG board (e.g. Patty) that you want to put it on the machine.
Provide the license agreement and if it's open source, put the source itself into the clearinghouse, so that if the tool owner moves on to other things people will still know why it is there.
Andrew pointed out that the CAS (Central Authorization Service, from Yale) project might choose to use the machine as their home.
Patty thought Eric Dalquist was wanting to move other uportal stuff onto the clearing house machine (but Eric is still not present at this meeting.)
Delaware which hosts the CVS archive (people at first surmised that was Eric's concern) doesn't look like they will pull out.
Adam found Eric's notes, he was suggesting the CVS monitor (provides statistics about the use of a CVS instance) should be moved over to the Clearinghouse; it is currently hosted at Unicon, but it's on a low
powered machine with no provision for regular updating of the database. JA-SIG has a license for FishEye
...
"JA-SIG, Inc." is the official name.
...
.
uPortal Documentation
| Anchor | ||||
|---|---|---|---|---|
|
facilitated by John Fereira
...
John: has a module in CVS called uportal_documentation that contains a snapshot of all the documentation in xml format. The Anakia ant task transforms the xml documentation into html pages, creating "web" and "print" versions.
uPortal-Sakai integration
| Anchor | ||||
|---|---|---|---|---|
|
Andrew Petro did most of this presentation, with Charles Severance present (he arrived during lunchtime)
[(This presentation is in part from the Powerpoint presentation which has much more detail.] Slides
Sakai had navigation requirements. Originally David Haines at Sakai modified up2.4 to make possible what Sakai wanted, but this proved too drastic.
...
Charles did part of the presentation:
Sakai integration phase I through Sakai 2.0
– parts can appear as IFrames within uportal
– adding a WSRP producer to Sakai so portals can show things thereby.
(Charles showed what this looks like)
Charles wants to use a portal to federate among many different instances of Sakai.
...
Sakai tools use Velocity and JSF [)Java Server Faces] ) view technologies.
Sakai's internals are being rearchitected to have less in its "kernel"
– the environment-specific part. This will be the only part that will have to be modified to work with other view technologies such as JSR-168.
...
Solve the ability to deliver asynchronous events to the browser.
(Sakai's 'courier')
Dan: give an example of uPortal 'managing' a Sakai service, as you said in one of your slides?
...
Charles: I have a 20-slide presentation that I will send around instead of presenting here. We have figured out a way to write Sakai services in a flexible, Spring-like way. the way we do this may be helpful to the uPortal 3.0 effort.
uPortal 2.6:
| Anchor | ||||
|---|---|---|---|---|
|
by Andrew Petro
[(These notes are taken in part from the Powerpoint presentation Slides which has more detail.])
We usually put out dot releases every six months or so.
...
An example of a leftover from Bugzilla is persisting minimizedness of channels as part of layout preferences.
(Peter K. suggested a very simple way to do this)
undoubtedly some 2.5 planned stuff did not get done.
Documentation
Administrator tools (duplicating ant tasks)
Groups and Permissions
– extract into standalone API and/or a jar
...
Peter K: It will be a lot easier for someone to do that in 3.0.
(Since the meeting, a path forward has been agreed upon: discover uP 2.6 goals via list discussion, and in our Confluence Wiki, then make these into a roadmap of scheduled issues in Jira.)
Permissions Framework
| Anchor | ||||
|---|---|---|---|---|
|
– Keith Stacks
SunGard SCT refers to the principal associated with a channel as the "audience" (who can get
at that channel). Their permissions model has a much more flexible way of granting permissions, using complex boolean logic expressions.
SunGard SCT is contributing this model to the JA-SIG code base.
...
IPermissibles API to indicate whether an activity supports a restriction
A new User Attributes servant
Runtime updatable PAGS store
PAGS store will support named and unnamed groups (unnamed are the AND/OR things)
Dan: Keith spoke of fine-grained permissions – refers to the GUI and it could be done a variety of ways. Fine-grained restrictions is the part that changes the underlying design. Dan has a concern that this design requires adding PAGS in order to work. Keith says that the "User Attributes" part of their permissions is a pluggable configuration, which is currently by default configured to use PAGS.
...
NOTE: On Monday evening we met for dinner at a nearby restaurant in honor of Ken Weiner who served as project lead of uPortal for several years, but who recently left for another position. Ken joined us for the dinner.
TUESDAY March 22
Eric Dalquist
– we want to put the presentations online
uPortal 3
| Anchor | ||||
|---|---|---|---|---|
|
facilitated by Peter Kharchenko
[(These notes are taken in part from the Powerpoint presentation Slides, which has more detail.])
Strategic goals:
– configuration flexibility
rendering mechanisms
HTTP parameter generating/processing mechanisms
data layer implementations
– long-term maintenance
isolated, individually configured groups of components
...
uPortal
make sure that available 3.0 resources are actually being applied to 3.0 (resources as in people's time; this means don't get sidetracked on other things)
Support from 2.x resources
Define upgrade/migration tools
...
Testing
identify resources/institutions
QA and Performance Test
Quality Assurance and Testing
| Anchor | ||||
|---|---|---|---|---|
|
facilitated by Eric Dalquist
...
Curt: Is it considered too difficult to set up regression tests (a test suite)?
Eric: That would be great. Would someone be interested in writing more unit tests for up 2.x? There are not many as yet.
...
Another issue discussed was, can we identify more people's time to apply to do additional pre-release testing? Charles Severance said regarding that: Sakai has a volunteer QA effort. The people who are responsible for making Sakai work at various sites, some are willing to try out the new code to see if it is good. We have gotten volunteer QA to work. We tell them to use Jira to report problems, then the developers monitor that. There needs to be leadership for QA, but it probably should not be a developer.
Rendering Contexts and components for uportal 3
| Anchor | ||||
|---|---|---|---|---|
|
by Peter Kharchenko
[(These notes are taken in part from the Powerpoint presentation Slides which has more detail.])
(The concept is, that the entire way that uPortal determines a user's layout, prepares it for display, and responds to inputs, can be different for different users in the same instance of uPortal. Actually, a user might even switch between contexts within the scope of the same session.)
What is a uportal 3 context?
defines all aspects of a traditional uportal instance
design, navigation, profiles
parameter syntax
portlet selections, session scopes
multiple concurrent contexts can be used
example contexts
static frame showing a single portlet
simple navigation with human-readable URLs
uportal 2 clone
(a graphic) showing a context resolver at the top that routes requests to appropriate context, and one of the contexts under it is the imitation of uportal2.
...
Sould be primary choice for configuring context rendering
up2 context example
(He showed an XML example of a uPortal components configuration)
Rendering: XML Layout sources
...
Transformer preferences
contain IStylesheetDescriptoin
location, mime type, etc.
IParamDescription and
IElementAttributeDescription
default
values, persistence scope, target elements
responsible for keeping track of current values of stylesheet params, attributes
RuntimeTransformerPreferences (in memory)
TransformerPreferencesImpl (persistent)
ITransformerHelper - stylesheet handling
(He showed an XML example of attributes configuration)
now he went back to an earlier slide to talk about the
...
PortletURL delivered to the portlets
implements IPortletURL
adds ability to obtain uPortal-specific URL constructors
getUrlConstructor (name)
IPortletUrlConstructor
implements portlet URL syntax
also configurable
Pluto integration, WSRP producer and consumer
| Anchor | ||||
|---|---|---|---|---|
|
by Michael Ivanov
JSR-168 Pluto container portlet rendering engine implements JSR-168 provides a runtime environment for portlets and calls Portlet invoker to execute them contains interfaces that ...
(He showed a diagram of the parts of Pluto)
container methods
init()/shutdown() lifecycle methods called only once
portletLoad() called only once before aggregation
renderPortlet() – caled for all portlets once per refresh
processPortletAction() – called for only one portlet
...
Pluto's Object Model
Represents portlet information on different levels
('definition' has parameters of a 'class' nature; 'entity' is of an 'object' nature.)
WebApplicationDefinition
PortletApplicationDefinition
PortletApplicationEntity
...
'definitions' – publish information for a particular portlet. (channel params)
'entities' – subscribed portlet (user params)
'windows' – appearing somewhere in a window (as yet, no distinct prefs for uportal)
Portlet URL implementation
...
WSRP Producer in uPortal 3
(refer to Michael's slides)
the org.jasig.portal.wsrp.producer.WsrpEngine is the main class it requires Axis
...
The WSRP component of the uPortal 3 development is at first using JDO and JPOX for WSRP Persistence
Why JDO (he explains this)
Vikram: WSRP4J uses a different db implementation, doesn't it?
Michael: Yes, but we modified it to use JDO.
...
Peter K: JDO is easier because in this case we don't know what we are going to be persisting. That would be hard to do with DAO, wouldn't it?
Michael: It would not be hard to change the implementation itself, or the ones that call it. But we're using JDO to get the WSRP feature working. Then we will switch over to the standard DAO things in uportal. By then the latter will be more mature.
(He gave an example of JDO meta-data)
What has been done?
What is in progress?
Open issues
...
Vikram: Message-level security may not buy us much. (later) WSRP has a concept of "user category" which is like an authorization "role". (there was much discussion of WSRP security)
Is it possible for WSRP Producer to share one Pluto container's instance with uPortal 3 by changing dynamic providers on-the-fly? i.e. if regular portal requests and WSRP requests come into the same instance of the portal servlet can we handle both of them? There was some discussion of this suggesting how to do it.
LUNCH BREAK
uPortal 3 – Portlet Development
| Anchor | ||||
|---|---|---|---|---|
|
Eric Dalquist
One problem is, providing access to portal services from a portlet
[ (portlets have their own class loader so you can't
get at uPortal classes in the normal way])
Special access-layer classes (shared library location in the servlet container)
if the portal API changes, then the interfaces in this layer have to change as well as their implementations.
this increases maintenance overhead and debugging effort.
Moving services to shared location
...
Vikram: Where does the Group service fit into a portlet? Is it used via 'IsUserInRole' portlet method, which is static access control, whereas groups are dynamic?
(there was an extensive discussion of this question)
Some people supposed that we would write only fully standard portlets, others supposed that we would also write things with portlet APIs but that also used uportal services ("portlet-like things"). The debate can be summarized like this: 'But that's not a portlet'. 'Not completely, but we don't mind.'
Cris: Here's a tip. If one group (such as JA-SIG) is building a variety of portlets, you should build your portlets into one web application, so that you won't get like fifty web applications all running at once. That's a big performance problem.
(NOTE: Bill Thompson was on speakerphone from Rutgers for part of the afternoon. From sometime in the uPortal3 portlets session, through the 'leadership' session after it)
Administrative Portlets
Consistent and flexible UI
Inclusion in the portal web application
(The rest of this session consisted of Cris H. on his investigation into what's available for the "consistent and flexible UI" part)
Spring PortletMVC
Bill Thompson – support: it's been in there about a year. The code seems to be stable enough for production. But maybe it doesn't have the elegance of the rest of the Spring framework and it has some rough edges around it. There is some talk to roll it out as a separate Spring subproject, so that it can be released on its own release cycle. It just needs some leadership and attention.
...
Another thing called MVCPortlet (not Spring) that is being open sourced by a company. This is the only one of these that says it's an official release (and they sell commercial support for it.)
JSF [(Java Server Faces])
Sakai makes heavy use of JSF. JSF kind of takes development back from web applications to more the way stand alone apps were written. With for example, events and listeners. Supposedly there exists shim code to make this work within a portlet.
uPortal Leadership
| Anchor | ||||
|---|---|---|---|---|
|
discussion facilitated by Eric Dalquist
...
Ken was funded 80 percent development and 20 percent project management. Does the board want to continue funding someone? Bill Thompson (on speakerphone) is on the JA-SIG board. The board is looking to the community for the immediate future. Bill thinks that people who have the thing in production would pitch in for it. Rutgers would be willing to help with 2.4.3 and may need some help with working on 2.5. Andrew (Yale) is very involved with 2.5 so could give advice to Rutgers about that.
...
Bill T: Regarding the memory leak issues you discussed. Our intent with Scott was to share what our struggles have been so we can head off things before they bite us in production. We released with expectations of quality (that were not met). I think we need to adopt the concept of quality and have a QA process. So that people will know what to expect.
BREAK
Miscellaneous uPortal 3 Questions
| Anchor | ||||
|---|---|---|---|---|
|
by Peter Kharchenko
(he just brought up these things that need to be decided eventually; there was no attempt to discuss them at length or make decisions)
[(These notes are based in part on the Powerpoint presentation, which has more detail.])
layout managers
Choices:
simple user layout manager
ALM
DLM
Integration? (we want to eventually have one code base for those two things, merge the features; but the effort has been pushed off into the future because there are not resources for it at present)
Sakai Priorities
simple layout manager is sufficient for Phase II
...
why not use some existing framework to do some of it for us?
Acegi (AY-see-gee)
default db, ldap
CAS
What information do we need from authentication subsystem?
...
To make the back button work properly we would have to have idempotent (can only have an effect once) URLs.
Eric: The biggest problem is to be able to have two windows open on the same portal and interact with both of them at once. Because the location in the portal is part of your session, not part of the link.
Andrew: Also, bookmarkable links would be great. (going directly to a specific tab or focused portlet).
service integration
| Anchor | ||||
|---|---|---|---|---|
|
Slides
Main services from uPortal2
directory
groups
permissions
how to share?
avoid maintaining two copies (for up2 and up3)
separate deployment of each service
[(i.e. pull these pieces out of the code base so they can be the same
for up2 and up3])
community involvement requested for the development of uPortal 3:
...
Administrative Portlets
publish, subscribe (extensions)
compound preferences portlet
profiles, skins, internationalization
runtime administration
cache control (empty them or tune performance characteristics) (2.5 we can dump some caches from the Channel Admin channel)
IChannel adapter to allow old channels to run in uPortal 3
Upgrade tools
...