August 2004 uPortal Developers Meeting Minutes

uPortal Developers' Meeting
MIT August 30-31, 2004

Minutes released September 24, 2004

Attendees

Al Wold Arizona State University
Dan Ellentuck Columbia University
John Fereira Cornell University
Michael Oltz Cornell University
Jim Thomas Indiana University
Jim Farmer instructional media + magic, inc. Sakai Educational Partners Program (SEPP)
Michael Ivanov instructional media + magic, inc.
Craig Counterman Massachusetts Institute of Technology (MIT)
Michael De Simone the r•smart group, inc.
Dmitriy Kopylenko Rutgers University
Bill Thompson Rutgers University
Charles Severance Sakai Project
Mark Norton Sakai Educational Partners Program (SEPP)
Casey Dunn Stanford University
Daisy Flemming Stanford University
Mark Boyd SunGard SCT, Inc.
Mike Zackrison SunGard SCT, Inc.
Nick Bolton Unicon, Inc.
Eric Dalquist Unicon, Inc.
Peter Kharchenko Unicon, Inc.
Jason Lacy Unicon, Inc.
Ken Weiner Unicon, Inc.
David Haines University of Michigan
Nick Blair University of Wisconsin - Madison
Curt Ellmann University of Wisconsin - Madison
Susan Bramhall Yale University (monday only)
Howard Gilbert Yale University (tuesday only)
Andrew Petro Yale University

MONDAY, August 30

Ken: we will talk about 2.x a little
but this afternoon we will be talking about what has been done so far on 3.0
tomorrow discuss what to do

Sakai/Portal Integration – Charles Severance
How will sakai deal with things that are not uportal? charles was asked this.
From now on there will be more emphasis on interoperability with sakai running on other portals.

The original plan was to use JSR-168 (portlet specification) and modify uPortal to meet navigational needs of Sakai; Sakai would have full functionality only on uPortal. They started down that path and discovered they needed mods to uPortal to fulfill the original vision

– dynamic very large layouts
– layout info from some plug-in sources [a concept similar to composite group services]
– a given instance of a portlet needs to know where it is in the layout hierarchy
– multilevel hierarchies
– change what the url's look like – humanly readable bookmarking of portlets, so you can go straight to the portlet you want.

– there is a large need to be able to integrate non-java apps
– need for a web-services approach rather than a one-JVM approach
– Sakai needs to run in other portals

Because of the above, and
– performance problems
– multiple webapps
– the tools team has UI requirements

It was decided that instead of JSR-168, WSRP ("Web Services for Remote Portlets") needs more emphasis in the Sakai code.

Examples of the URL convention:

/ or /site – the whole page

/site/sp04/eecs/280 the entire page pre-navigated to the EECS280 site

[more examples in slides; I couldn't type fast enough]

Their new navigation model is nested rectangles, rather than tabs and columns
These URL's are either local navigation patterns; or WSRP patterns of what to go get.

The WSRP allows the user to federate many different Sakai sites. (supposing single-signon issues are resolved). The portal does the WSRP consuming and federating; Sakai sites will be producers.

Sakai will use the O.K.I. OSIDs (api's describing the manner of doing persisted-info access, etc.) to talk to enterprise information.

Charles' main workpoints today are:

– Sakai will be concentrating on WSRP development, let's share experiences

– Suggest to uPortal that they use O.K.I. OSIDs for AUTH and [something else, I missed it] particularly. Not at all a requirement. Ken pointed out that if WSRP is going to be used, then OSIDs should not be needed. It's too 'under the hood'; WSRP is supposed to protect uPortal from having to know how to do those things – the OSID kind of stuff should only be used on the producer side; if the portal is only a consumer it doesn't belong there.

One question is how should this be layered in the software?

Sakai tools and apps/'magic fairy dust' interfacing code/Wsrp4j/WSRP-protocol-as-consumer

Sakai/dust/JSR-168/Wsrp4j/WSRP-protocol-as-consumer

Sakai/dust/JSR-168/uPortal/WSRP-protocol-as-producer/...

The first one looks like more work for Sakai but in a better position afterwards

The second one would make Sakai tools and apps themselves somewhat easier to develop but would have to make more mods to other existing code to do it.

The third one is a rather unlikely choice

Dan asked what is the relationship in terms of making design decisions, between Sakai and O.K.I. OSIDs?

Charles compared the old Chef (earlier U Michigan project) model in comparison with OSIDs.

Chef's goal is to make the writing of tools as easy as possible; lots of details are moved into the services.

In the OSID world, the goal is to allow a wide variety of implementations of the services; this is more sensible when each service is comparatively small in power and complexity. [but there are higher level application services as well]
Now, there is the Sakai API which "looks like OSIDs" at one level, but there are many, more powerful and more specific services as well. Eventually Sakai will write OSIDs on top of itself so tools can call certain capacities of Sakai that do OSID-like stuff, so that those parts of Sakai can look like OSIDs from that tool's point of view.

The original plan was to live in the same JVM with uportal and make the UI as flexible and nice as Sakai wants. This proved overly complex to do.

Earlier Charles said that portlets would know about their position in the hierarchy.

Ken asked with reference to that: Will the Sakai parts that are WSRP producers need to know about their 'portlet location' etc. This would influence how the consumer would ask for that producer, and wouldn't the consumer have to know where it is? What Charles essentially explained is that the URL of a producer does look like it 'knows' where it is physically in the hierarchy, but really it's just a long name for the producer, and you just tell the other site what URL to put and it just works. [that is, a WSRP producer really isn't 'in' a layout, so it doesn't have a 'position'. It just has a 'name' that happens to look like a subdirectory path.]

Jim: Are the various tools in Sakai separately addressable via WSRP?

Charles: Does Sakai provide a general purpose chat that the consumer can use as a utility any way they want, with the consumer deciding the purpose of the chat, etc?. Charles says no; the configuring of the chat will happen at the producer side "Here's the chat for U Mich's Chemistry 280 course, do you want to be a consumer of it?" i.e. Sakai provides a collection of instanced portlets.

Charles recognizes that the Sakai funding has influenced the direction of uPortal, but that the primary purpose of the funding was to help uPortal continue, and he feels that this change of direction will make the interaction between the two groups easier "We are supporting uPortal in general"

Charles wants to do 'http rectangles' (the url stuff) for Sakai 1.0 because this makes the iFrames he showed possible. and this will be as it were a reference implementation. there will be instructions of how to make this work.

WS-Security (WSRP security) WSRP4J doesn't say anything about what security model is used; it can just pass along an authentication string but has nothing official to do with where the string comes from.

Mike Zackrison: with the URL hierarchy, how do you deal with cross-listed courses? How does the SAME portlet show from two different URL's?
Craig: we (Sakai) will need to have that somehow but not sure how yet. Daisy: They (Stanford) pick one of the depts as the primary course, and that's how you get to the course on the web.

[morning break was here]

Portlet development challenges – Eric Dalquist, Unicon, Inc.
[bugs and issues encountered while developing portlets in uPortal]

File uploading – uPortal wasn't allowing portal to access uploaded data

File downloading – bug in Pluto and Tomcat made the recommended way of sending binary data not work

Portal preferences – was not allowing prefs on a per portlet instance basis

Class loader issues – each portlet has own class loader

Portlet API access – does not have access to the portal's libraries or api's

Portlet portability – needs to be cleaner

File uploading – (RequestParamWrapper.java) the uPortal was writing the data into a temporary file, so when the portlet looked the input stream was already gone. Code changed so if it is a portlet the portal doesn't touch the input. This fix implemented in 2.3.3

File downloading – JSR-168 does not provide a specific way to provide non-markup output. It does require that a portlet and a servlet executing in the same environment share a session, which would be a sneaky way for the portlet to pass non-markup output to uPortal. But there is a bug in Pluto and Tomcat such that they get different sessions, and it can't work. Several people suggested programming solutions. Then it was pointed out that to sneak around this would violate the security defined by a servlet. The conclusion was, it's in Tomcat's lap for a permanent solution. Temporarily for uPortal: a portlet can extend its window state. So they did that ("exclusive window state"), allowing portlet access to the output so it can set the content type and output stream. This is in the MAIN branch, so it would be in 2.4, not 2-3-patches. Another bug in Pluto was that it hard codes the setContentType to text/html.

Portlet preferences – the preferences are correctly being read from portlet.xml and distributed. The entity preferences were being scoped on a per-published basis. All instances of a portlet would have the same preferences, which is not what the spec says, but it's kind of muddy; is it per-user, or per-user per-instance? Eric made it the latter so that each instance can be different. He also added a layer of prefs that influences every user in every instance "administrative only"; these can override publish parms of a portlet (the ones that come from the WAR file), which properly cannot be overridden by the user. This will also be in 2.4. Entity level preferences can be changed programmatically by the portlet; they could be changed by user when subscribing as well, BUT the latter is not yet implemented at the framework level (no UI changes to support it). There is now a portal preference store class, and a uPortal property that determines what kind of backing store is used.

Class loader issues – Each portlet application has its own class loader. This is also confusing with respect to static initializers (they get run once per class loader). [common/lib and shared/lib are in all class loaders]

Portal API access (from a portlet) – Either put uPortal in one of those two directories or make a lightweight interface that allows a developer to at least compile stuff without having to do the preceding; but then you couldn't run on the compiling machine. It was pointed out that using the uPortal API at all, impedes portability of the portlet. The servlet spec and portlet spec are at odds about this; there needs to be a way around it.

Someone pointed out we should have a closer relationship with the standards committees; the ongoing problem is lack of funding to let someone do so officially. However, ken and others at Unicon are posting on a Wiki which the committees read, to make recommendations about future changes. Ken: there is very little discussion on the Pluto and WSRP mailing lists, except for newbie questions. The portlet spec seems to have been created as a lowest common denominator of what was observed to be in existing portlets, rather than of what a portlet ought to be able to do.

Charles: Sun feels that ESB's solve this problem of portable portal (portlets/channels); IBM feels their 'concrete portlets' are the solution. This disagreement is ongoing and impedes progress on the spec.

U Wisconsin has some 'modules' in their portal that give them access to enterprise info or provide it to remote services.

Eric: There is no way in a portlet to get at groups and permissions, for example.

Ken: what are people doing out there? everything seems to be 'hello world' portlets

Susan: or oracle or ibm portlets
Charles: Various groups are just ad hoc solving the problem separately. Somebody gets together with their "five closest friends" [with whom they need to have interoperability of a particular application] and they decide how they in particular are going to get around it.

Ken: let's do a get-at-portal-capabilities-from-a-portlet API thing in such a way that it could be reimplemented straightforwardly by any other portal.

Dan: is there a relationship between such an idea of a portal API and the services package?

Ken: Yes some of those services are what we'd want in the API.

Michael Ivanov (from instructional media + magic) had something to say in this session as well.
How to use stylesheets from a portlet. You can't use it in a standard way. Because the baseactionURL can't be changed in a stylesheet; this would have to be done in Java.

He rewrote [x]htmlSerializer.java that allows you to add a new element inside a <a href> for example, <param name="" value=""> this is recognized and accepted by the XSLTSerializer and altered later on by XSLT into a real URL.
He also added XSLTPortlet class.

Other approaches to solving this problem would be

1) Create the URL's and pass them as parameters to the stylesheets.
2) You could write an extension to the XSLT engine.
3) Use JSP's or the tagged kind of them.

Ken: For those who are going to write portlets, how many are going to use XSLT and how many not? There were several in each group.

Mark Boyd: He did some timing tests, and JSP was almost twice as fast as XSLT.

Someone else asked; Is anybody outside of Sakai using Java Server Faces? Answer was, really not much among those present.

[Lunch break was here]

WSRP – Ken Weiner and Michael Ivanov
Ken tried to get WSRP consumer and producer working for 2.2 but it was really early after the standard was finalized and he wrote it from scratch without any helper libraries. Now he is trying to integrate WSRP4J (Apache implementation of WSRP). In the MAIN branch (to be 2.4) there is a consumer based on it, with minimal testing. This did not require much effort. There are two different consumers provided with WSRP4J; one is a Swing consumer, the other a 'proxy portlet'. All he had to do was drop that portlet in, and do some configuration. He wanted to let people specify the producer for an instance, when they publish the channel. This is implemented. This consumer thing is now a channel type. The unmodified WAR file is in lib/portlets, and then he has his changes in source/org/jasig/portal/wsrp/consumer java files. These overwrite stuff that comes from the source tree. but if he had built these into the war file then where should the source live? He also had to modify some of the WSRP base code also (submitted as bug fix suggestions). He documented these changes in the build.xml file.

Nobody present seems to have actually tried this in the MAIN branch.

Both for WSRP4J directly, and for the way Ken implemented it in uportal 2.2, you have to configure four different URLs. Some portals use one URL to identify the producer and its services, but WSRP4J doesn't support that. Right now, there is not authentication, user identity, etc.

He was hoping to implement a producer for 2.4 as well, but it looks non trivial. He wanted it to sit on top of the IChannel, so it could serve up content both from channels and portlets. In one sense, doing this in 2.4 would sort of be 'end of life', because 3.0 is down the pike, but 2.x will probably last a year or two after 3.0 comes out. So if anyone wants to go and implement this for 2.x it will not be a waste of time.

Michael Ivanov tried to implement WSRP in 3.0. It now works with PortletEntityRegistry. But that's mostly the only change he made so far. We need to write a WSRP 'engine' and 'composer'. How can we get at the static and dynamic information from portlets/channels? They will get back to work on it after certain other things are done. We need to create information producers and WSRPPortletEntity. Ken asked isn't WSRP based on the container? Michael said it actually needs more information from the portal than from the container. The consumer in 3.0 will probably also be the 'proxy portlet'.

Charles: compared to splicing in the Pluto stuff, is WSRP4J looking harder or easier to add in? Ken and Michael: about the same amount of work.

Nick Blair: Upcoming PeopleSoft will have WSRP producers. Next release will be read only, one after (Tools X) will be read write.

Ken: we need a way to dynamically discover producers, rather than hard coded. We don't have a 'producer registry' which WSRP4J requires. In 2.4 he fudged this, you specify the producer at publish time.

Charles: Are you configuring producers at the producer end or consumer end? Ken says there exists a way to configure a producer from the consumer end but it isn't implemented yet on uportal.

uPortal 3.0 Code Overview – Ken Weiner, Mike DeSimone, Michael Ivanov
Ken's PowerPoint slides (second slide is the domain object architecture diagram)

Ken showed his 'domain object architecture' diagram from February.

There are domain objects that lash together different pieces of the portal code. You get the information in a domain object by querying a 'registry'. (this is a bit like JNDI but it applies to local code, rather than 'wherever it may be' stuff). The registries use 'domain object factories' to produce the domain objects given their related transfer objects. There will be caching of registry values.

There will be DAO (Data Access Object) implementations to get at persistent data, and 'transfer objects' which are called to read or write particular data.

They are using Spring to organize the code. The applicationContext.xml tells how each aspect of the portal is implemented.

Among these are the implementations of the DAO's. each implementation tells what its dependencies are.

As an aside, all of the implementations are by default singletons, which they used to have to do by hand.

Mike DeSimone – shows how the DAO's work as an example of this.

His PowerPoint presentation (some of the notes below are from this)

Mark Boyd: What about database sync?
You specify a transaction manager as one of the dependencies of a DAO.

How did uportal 2 do it?
– straight low-level JDBC
– One service class (RDBMServices)
– 3 or 4 classes roughly divided into functional areas (RDBMUserIdentityStore, RDBMUserLayoutStore, etc.)
– One of those two stores just mentioned, is a rather large chunk of code

issues

– ResultSets were left open
– abandoned connections
– missing keys/inidices in generated DDL (xsl fix)
– performance issues

Separate the business logic from the data access

Alternatives which were looked at:
– Object relationship managers: JDO, Hibernate
– iBatis SqlMaps
– Spring JDBC support

iBatis SqlMaps
– has a somewhat simpler xml configuration but still not trivial
– geared toward persisting objects that have basic types, rather than more complex objects
– stability and longevity were unknowns

Hibernate
– requires many fewer lines of java code
– provides caching of objects and lazy-initialization
– requires extensive and complex XML configuration
– understands many major DBMS sql dialects
– has a hefty learning curve for effective use

Daisy: error messages were difficult, and found it hard to retrofit an existing db but much better when starting from scratch.

Spring JDBC support
– the programming style is pretty close to vanilla JDBC
– requires a lot of Java code to implement persistence, but not as much XML config
– does not have caching support

Factors in choosing a "winner"
– nobody's a "luser"
– easier than JDBC?
– maturity, stability
– community vibrancy/activity
– minimize barriers to future contributors

Spring JDBC support
– a package that allows DB in an object oriented dmanner
– execute queries and get a list of results back

What you need to do to use this:
– extend mappingsqlquery for selects
– extend sqlupdate for insert, update
– implement mapRow to map relational data to Java objects
– inject the data source
– transaction support is transparent to code

Dan: Do you have any idea what the performance penalty is on these things?
Michael DeSimone: I can't quantify it, but i would think not much.
Ken: Spring JDBC doesn't really change what JDBC does, it just organizes how you write the JDBC code.

Ken: Re Spring – I think this is a better match for the existing skill sets of our current user community, you can actually see the exact SQL statements in the code.

[Daisy advocates Hibernate]

Ken: Often the person who is willing to do the work is the one who makes the decision, and Mike DeSimone was willing. But I support the decision based on what i said earlier. If you think hibernate is better, you are welcome to write it and check it in in parallel to the Spring one.

Charles: His view of the difference between Spring and Hibernate is that Hibernate gets a lot of data and does the parsing locally, whereas Spring only requests the columns you want.

Ken: Where is the new stuff in the code? org.jasig.portal.dao.spring has the code Mike De Simone was just discussing. some DAO's just read from a config xml file, so they are read-only. Ken showed an example of a database DAO, and of one that reads from an XML file. As was said before, keep all business logic out of the DAO layer.

Also look at org.jasig.portlet.deployment

[afternoon break happened here]

Relationship between the various levels of portlet classes in the specification – Ken Weiner
Ken's PowerPoint slides (first slide is a list of the levels explained here)

At each level there is a "Something" class and an "ApplicationSomething" class; we didn't make this distinction (about our channels) in the portal before.

A portlet application is possibly more than one portlet sharing a context, generally delivered in a single war file. So for deployment, you could query a PortletApplicationDeployment to see what PortletDeployments it has in it. The other "something" class has to do with exactly one portlet.
– information from portlet.xml file
... has a one-to-many relationship to:

PortletApplicationDefinition
PortletDefinition
– this knows about the Deployment AND has publish time preferences.

PortletApplicationEntity
PortletEntity
– this knows what the Definition knows and has an additional layer of Preferences; this provides something a user can subscribe to.

PortletWindow
– instances of portletEntity painted on the screen
– carries portletwindowstate
[the window states are not persisted]

Then Ken showed from the source what a PortletDefinitionTransferObject looks like.

The registries (see previous session) can communicate with transfer objects in order to create, for example, PortletApplicationDefinitions.

Then Ken showed how the caching works; the caching implementation is loaded by a TransferObject as called by the Registry. Right now they have a rudimentary custom caching implementation that is being written.

Each DomainObject can have a different caching implementation which is specified in the applicationContext.xml. There are some objects that have a read-only version, and also a read-write version (these would have "Control" on end of class name) which also has set methods.

JUnit tests – There is much more emphasis on unit testing in 3.0. All of these are in the tests directory. Ken wants someone to look at the tests as a whole and evaluate consistency and thoroughness. Some setup is required before you can use the tests, e.g. you have to create the database.

The tests are written against the interface, not the impl. Then there's a configuration file that says what implementation you are testing for this particular run. If you write a component of 3.0 you are expected to write a test for it.

Pluto Object Model implementation.

We didn't just want to code up uPortal 3.0 to use Pluto's interfaces. Maybe Pluto will be obsolete some day, something better, etc. So we wanted to be independent somehow.

In order to use Pluto, the implementations we provide must meeet Pluto's specs. But we don't want to be welded to that. So instead we wrote the classes the way we needed to, and made sure that the impl class also implements pluto's interfaces.

How did he make up names for all these new classes just for us? Peter suggested: "If the Pluto name has 'Portlet' in it, take it out, if it doesn't have 'Portlet' in it, put it in."

Dan: what is the scope of the persistence effort? Have you thought yet about the effects on when you are writing e.g. publishing tools? There are locking issues etc. it would be better to think about this now rather than later.

TUESDAY, August 31

More About 3.0 – Peter Kharchenko
His PowerPoint slide (a diagram of the traditional context, with annotations)

In 3.0 there can be more than one code flow (a 'rendering context') for creating the content of a subwindow

e.g traditional one is

– parameter processing; parses request URL syntax; initiates actions on portletes, or rendering pipeline elements
– layout generator; generates initial user layout markup; aggregated, and simple layouts [aggregated layout code is not going to be rewritten at this point, too much else to get done]
– transformer; a chain of transformers and filters compiles presentation markup [See UserInstance.java]
– filter
– transformer
– serializer

URL command syntax can be customized by providing alternate processing and URL generation components.

The 'context resolver' chooses a rendering context for the user. The default one will use something in the URL so you hit a different URL and you could get a different context.

Initially we will implement a context that is more configurable but has the same (above) functionality of 2.x.

Why have these contexts? Because there are various requirements of modifying the rendering pipeline; this provides more flexibility.

For example an error context could be just a more static page generated by a separate context.

One person supposed that the context was a URL parameter, but Peter was intending that it would be the context path itself. Actually the resolver calls everything else, and actually returns the rendered stuff, not just a 'use this context' value.

Peter showed context resolving code and an example configuration with a "dummy" context. To configure contexts with appropriate scoping of necessary support beans, as required by Spring-specific configuration issues, multiple hierarchical Spring BeanFactories are used, and a filesystem-like syntax is used to refer to different beans throughout the bean factory hierarchy. After he showed the simple case, he then showed the 'CompoundContext' that handles more complicated situations.

SAXPipelineRenderer is the main rendering class, with some XSLT steps after it. then Peter showed the configuration file of the rendering, which sets up SAXPipelineRenderer. Enclosed in it is a list of filters that are called in succession.
The complexity of context configuration comes from the fact that there can be multiple components within contexts, which are capable of rendering URLs.

He segued into discussing URL processing. URLInfo class utility class that allows different constructors to work on the same URL. IURLConstructorProvider this allows components to provide objects for custom URL construction.

Peter was asskd what he calls a 'pathElement' and a 'fileElement''

In a URL the part that looks like a subdirectory path constitutes the path elements. The last part of the URL just before the question mark and parameters if any, is a 'file' (Mark Boyd pointed out that's what the Java API docs call it) and the tokens of that file name are 'fileElements'.

Ken: IURLController extends Pluto. Why? Peter: that is a Pluto-specific class, not a 'JA-Sig only, portlet-engine-independent' class.

People were puzzled by the 'URLIinfo' name of that class.

Ken: As yet we don't have any 3.0 classes that represent a layout. Will we? The layoutGenerator does not actually return a layout object, it does a lot of SAX processing. The IPortalRenderer interface renders a request to a response. (during the talk Peter went and looked through the code to demonstrate that as yet they don't really have a 'layout object')

Ken: What's the general status of this?

Peter: The part that is not checked in is any caching in the layout pipeline for 3.0. This should be configurable. there is also a StringRenderer that is not checked in either; it will have be similarly configurable and also cacheable.

Ken: What do we need to do to render a portlet? We need a PortletManager to call the portlets, and a filter to insert the returned stuff into the output stream. These don't exist yet.

Portlet Deployment – Ken
In 2.3 we have a class that deploys a portlet via an Ant target. It assumes that the portlet is in a present war, and it rewrites web.xml with servlet definitions and aliases that are needed to wrap the portlets. But according to the spec, the deployment process is supposed to prompt the administrator for information. In 2.3 these are done from a config file instead. How should we seriously do that part for 3.0? Should it be a web application? A Swing app? Command line interface? Ken says, most of the time you do want an automated process.

Bill from Rutgers: something like a PersonDirs.xml which was one thing Ken mentioned.

Mark Boyd: It's something like a properties editor for a bean.

There wasn't a really strong feeling in any particular direction. Ken asked if anyone was interested in working on it. One of the people from Unicon said they had someone back at the office specializing in deployment and they might be interested in helping. Ken "I just want someone who can concentrate on making this a good process."

morning break

Porting independent parts of the portal to 3.0 – Ken
Some parts that we're trying to think of as more independent parts of uportal; how do they port to 3.0?

Logging – we wanted to use the logging package built into java 1.4 yet have it configurable, so they chose commons logging from Apache instead because it provides an abstraction layer. Should we configure it the old way or use IoC? They decided to use a LogFactory instead. (this will require adding one line of initialization per class). Ken will leave out any logging implementation in the quickstarts, so it will choose the java 1.4 logger by default. but you can put in log4j or whatever.

Pluto has a logging interface you have to implement. We will probably pass the same as for the portal infrastructure.

Mark Boyd: We have a script that goes through source files and tinkers the LogService calls to be commons logging instead. The buy-in is you can have much finer control over what parts of the code generate what level of logging. Mark and friends will do it. (but this will touch just about every source file)

Security – Should we use the same security pckage as in 2.3 and how to port it over? Do we want a separate security project and then use it from uportal? we were looking at Acegi Security[*]. it is similar to yale's security. you can put security around a class and its objects if you want. this allows much more flexible configuration. ken wants yale to participate in this decision becacuse they developed the original code.

[* a security framework that is written to work with Spring. Say 'ah-see-gee'. The name is just every other letter of the alphabet; it doesn't stand for anything.]

Howard – The original design was to provide credentials to various back end processes – to log on multiple times to multiple back end security systems. People don't seem to have used this capability. acegi would be external, better documented, etc. so Yale, for one, would support moving on.

Bill from Rutgers says they have been watching Acegi and think it's well designed and will fit in well with uPortal's new Spring design.

Ken: Andrew is coming to coding session the rest of the week; let's experiment with Acegi.

Eric: a) allowing for any SecurityContext class to be configured. Currently no way to send parms in. He figured out a way to do it in the initializer class. This allows multiple LDAP servers to be used for security without multiple java sources. Andrew from Yale: why not put this in the factory for the particular kind of security? Eric: because how would the factory know which instance of the LDAP security it was creating? The initializer would know this so it should be done from there. Ken said to go ahead and do this.

b) Why should we have to read the security context properties from portal.properties every time someone logs in or out? Let's cache that.

Groups and Permissions: Dan: in both groups and permissions, there is a) a runtime functionality, making queries b) provisioning activities. I think those two sets of activities need to be considered completely separately. Minimal for b) is to wrap groups manager and permissions manager, as they are, for 3.0. Biggest concern for a) is configuring the consolidated groups and permissions. How do we package these two services for runtime use?

Mark Boyd mentioned a requirement: at SCT in AggregatedLayoutManager they have their own configuration that has more complex boolean logic for permissions. Would it be possible to make the fragment and channel permissions pluggable and allow differing policies depending on what you are inquiring about?

Dan pointed out that there is already a 'permissions policy' which is selectable on the properties-file level but could be more dynamically selected.

At what level do we want to make it pluggable?
– a group service itself?
– a group store?
– an authorization (permissions) service?
– an authorization store?
– a composite authorization store has been suggested but never done

Ken: Do we want a groups thing to be a server? a jar? source that's mixed in like in 2.x? People don't want the latter option.
Mark from Sakai: Sakai is discussing this very issue with respect to their own code, and it will probably be based on O.K.I.. He wrote the proposed API yesterday.

Ken: Make a groups module parallel to up3.uportal in the sandbox (e.g. up3.groups). Get Dan to move his code into that, and put a jar into the portal libs to use it. Dan: Agreed, also make one for authorization. Ken: The tricky thing will be to configure database access. Dan suggests still having the groups tables in the portal database. Ken: Yes, but have a separate database config for it.

Jim: The database for authentication and authorization should be completely separate, for sociological reasons.

LUNCH BREAK

Error Handling: Andrew Petro, Yale
His PowerPoint presentation (much of the notes below are from this)

What reporting is desired?
How are we presently doing this?

How might it be done better?
Errors and exceptions in a server are different – you can't just bomb, you need to handle it and recover.

For example, Yale had a "yesterday's weather" situation, involving DocumentCacher and the Weather Channel. If it's pouring rain outside one does not want to be saying that it's partly sunny.

Could we look inside a running portal and see what's been amiss lately? At Yale they have a /problems servlet that shows a page of recent errors and you can click and see the stack trace etc.

How should we do it in portal?

– logging, certainly
– component-level reporting (proposed PropertiesManager e.g.)
– failure-response components e.g. CError channel
– and the dirty trick;

instantiating PortalException – make it have the side effect of REPORTING the exception at that moment to the ProblemsManager. So even if the developer is not conscientious enough to handle or report it properly, you get a record anyway.

Currently (for 2.x in Yale's code):

– create an UndeclaredPortalException

instances have ErrorID which are keyed to the messsage
– Custom Exceptions

As convenience implementations of the necessary calls to UndeclaredPortalException

(Undeclared)PortalExceptions have a logging flag to help ProblemsManager decide what/how to report about them.

Going Forward

ProblemsManager should have been behind an interface, to allow multiple custom implementations.

Revisit the ErrorID's
– key to categorization
– key to parameterized message

1:1 mappings:
– messages to ErrorIDs
– Exceptions to ErrorIDs

Convenience implementation pattern and ErrorIDs.

Alternatively

ErrorID roles can be fulfilled by the fully qualified name of the IPortalException implementing class.
– key to categorization
– key to templated message

Where is the value in doing this?

The value of it is the custom Exception class, with its own documentation.
and in templating and I18ning the Exception messages.

Howard: wanted to point out the key issue. He objected to the concept "a string value shall be a parameter to the Exception class" that is, the message string. This was the only way to differentiate PortalExceptions. He wanted to use an ErrorID value because it's more convenient to write. The Java, stricter object oriented way would be to create a lot of different classes. Retrospectively, this was the chief objection people had to Yale's proposal. So Howard is willing to concede that point and let the design use lots of different classes to differentiate the PortalExceptions, so that the error handling issue can be addressed in the shared code. However, these classes must not be allowed to send in a message, only the parameters to that message. Because the main text of the message needs to be either built into the class or read (I18n) from a properties file.

General comment: Don't forget to allow a Cause parameter if some other exception caused 'this one'.

So, for uportal3 (this is not implemented yet):

Lose the ErrorIDs

There will be a uPortal exception class hierarchy based on PortalException and UndeclaredPortalException

Custom exceptions know how to obtain (from Spring?) their template message and merge in their parameters

Exceptions report themselves on instantiation to an IProblemsManager implementation.

In Java you can't 'catch' an interface, but after you catch 'Exception' or 'Throwable' then you can then look at whether you got an instance of IPortalException.

Bill from Rutgers: A cleaner way is to use an AOP throw advisor, which happens without any mods of the exception classes themselves. [the above link goes to the doc page for the AOP support in Spring]

Andrew: But the PortalException and UndeclaredPortalException constructors are the only places where we need to change the code.

Mark Boyd: but if we use AOP it will advise even non-portal exceptions and you can report them the same way.

Ken: Check it in for 3.0 [well Yale hasn't done it yet]

General comment: what about putting some of this in 2.4?

After some discussion, it was decided to change the instantiation of the two base classes to call the ProblemsManager, in 2.4. and leave alone the specific exception clauses scattered throughout the code base.

Peter: It's important to plan the hierarchy of classes for 3.0, carefully, so people create them properly and it won't have to be corrected later.

Dan: How would I make these things? (for groups and permissions) [Answer: make one (or two, declared and undeclared) derived exception for your component of the code, and then derive a bunch of more specific exceptions from that as needed]

Ken: How to decide which kind of exception to throw? For example, our DAO implementations don't throw declared exceptions but the interfaces do. When do we declare and when don't we?

Howard: You should not throw an undeclared portal exception if you EXPECT the immediate caller to catch and handle it ("we're going to retry this individual thing, okay caller, go to plan B."). The undeclared ones are those which you expect to be tossed up to a GENERIC exception handling clause ("it makes no sense to retry this individual thing, we have to give up the whole operation").

Howard: When you are writing code in Eclipse and it asks you: you didn't handle this exception do you want to catch it here or reethrow it? if it's a declared exception you should catch it.

[there was further discussion of this distinction]

Peter: Whenever we DO declare an exception we should document how we expect the caller to handle it.

Upgrade Tools – Al Wold
upgrade21.xml – upgrades 2.1 to 2.3, will probably work in 2.4

One task is upgradedb. It passes to DbLoader what your current database version is.

In the data.xml and tables.xml, now there are version numbers on the <table> <column> and <row> elements as attributes. The upgrader uses these numbers to decide what to add to the existing tables.

Originally Al decided to switch to pushfragments and he would wipe out the existing templates. Then he decided to use brand-new templates instead. So the "convertprofiles" target wipes people's templates except for bookmarks and other shared data. it changes everybody's theme and structure stylesheets. This destroys people's existing layouts.

For existing rows in a table which has added a new column, there will be null in that column. New rows wiil get a new value in that new row.

Ken: I wrote a class that returns the portal version number, starting with 2.4.

Mark Boyd: I wrote a version manager that can keep track of versions of the portal itself, and of for example channels. It stores these things in a database table.

John Fereira: You said that you copy the database manually first then alter the copy. Since we have those version numbers in the XML, could you do this after dbunload instead, changing the xml file? The reaction was 'this would be an enormous file and take a really long time'

AFTERNOON BREAK

Documentation – John Fereira

Ken: doc is one of our weakest points
John: the current doc page tries to document every version at once. For 3.0 why not start a new page that documents only that version.

I tried to categorize documentation by what kind of user was looking for it. There are now a number of documents that cross these boundaries. Can we continue to do this? Should this be an emphasis on how the new docs are written? In general there should exist documentation on any new features.

Ken: there does exist documentation for some things that we never got on our own pages.

John: I put on pointers to some docs that live elsewhere. Lots of mailing list questions could be answered by sending someone to the right docs.

I created a docs directory in the sandbox.

On the train ride here I experimented with the look of the documentation of the portal. I used the portal icons and stylesheet to make it resemble the portal even though it is static.

In Mozilla there is a separate project for its documentation.

I didn't want to put restrictions on how people wrote docs. doc, pdf, html, whatever.

Ken: I don't want much non-javadoc docs in the source tree.

John: maybe put them in a separate zip for each release.

Ken: Maybe we could use Maven to help us build a better doc site. I don't think it's time yet to use it for any portal source projects.

A question: should we have a Wiki?

Ken: Yeah, why doesn't somebody put one up?

Bill from Rutgers: Maven is good for keeping consistency across multiple projects. But for one project that benefit doesn't apply.

Dave: I build uPortal under Maven.

r.smart: When he downloaded Sakai it downloaded many versions of some of the jars.

Proposal for 2.4 by Mark Boyd
Enhancements to channel archives.

In the channel archive deployment descriptor you can put in a <database> element to do dbloading things, and a <channel-definition> element. he has a <processIf> – for example if the channel is already there but is an earlier version, you can upgrade existing channel definition and change the appropriate database table(s).

Things for 2.4, 3.0, and wrapup – Ken Weiner
What things are going in to 2.4?

Andrew Petro: – exception handling [relatively small change]

Freddy Lopez's (rutgers) bug fixes [as mentioned on the mailing list]

Mark Boyd: changes to channel archives [see his brief talk above]

Mods to security properties to have configurable properties

Don't parse the security context configuration every time, cache it.

could do a new release within the next 2 or 3 weeks.

Yale and Rutgers: both are experiencing some memory growth issues with 2.3.3 in production
Howard: this seems more like a loop than a leak.

Ken: This release is called 2.4 rather than 2.3.5 because it has a few new features; however that doesn't mean it will be difficult to upgrade – there will be very little change in the database.

There will also be a 2.3.5 which will be the last patches for 2.3 and it will come out the same time as 2.4.

3.0 should come out sometime next calendar year.

Someone asked 'what's the earliest date before which 3.0 definitely could not come out.' The point is how can we decide whether to go to 2.4 or wait until adoption.

Ken: We're definitely less than halfway. He wants to have a genuine alpha and beta this time, without the upgrade aspect of it, then do a production-quality release with everything that was promised.
He asked, what do you want developers to keep in mind while 3.0 is developed so they don't become show stoppers later?

Dan: thinking of an enhancement to groups and permissions design, with specific built-in modeling for template permissions. To reduce the number of group and permission objects that have to be created in a medium to large scale site.

Ken: Any comments on how this meeting is run. is this the best way to do it? There was agreement that given the current size it is not yet necessary to split the agenda into more than one track.

At the last JA-Sig conference we had a meeting about the community process and how it's gone about. So that newcomers will know how we operate. Bill Thompson was chosen to write up how it is done.

Ken: Maybe this can be more effectively written by a new person, because they'll know what the questions are. a newcomer present volunteered to do that

Next meeting will be somewhere around February, on the west coast or on the extremely west coast.

Howard: [when choosing a developer meeting location] give some priority to decent coffee.