Clustering JSR-168 Portlet Applications in Tomcat

John A. Lewis
Unicon, Inc.

Introduction

JSR-168 Portlet applications represent a special challenge when it comes to clustering within Tomcat (or any other servlet container, for that matter). In order to effectively cluster web applications, session data must be replicated or shared between the nodes in the cluster. Otherwise, the user experiences a complete loss of context during a node failover. While Tomcat has provided session replication for quite some time, it has not supported replication of session changes resulting from a cross-context call from one webapp to another.

Portlets that are deployed as separate webapps from the portal webapp must be accessed by cross-context calls from the portal. This cross-context access creates a number of issues: servlet filters are not applied, session sharing between servlets and portlets is difficult, and session replication in Tomcat did not work.

The University of Wisconsin-Madison (www.wisc.edu) and Pearson Education (www.pearsoned.com) both recently identified a need for full support of portlet session replication within Tomcat. They engaged with Unicon (www.unicon.net) to work with the Tomcat community to provide this capability. We've worked directly with the Tomcat developers to get Cross-Context Session Replication built into Tomcat 5.5 and we've demonstrated that portlet session data can now be properly replicated. I'd like to give a special thanks to Peter Rossbach from the Tomcat team for working on this with us and for getting this done so quickly. The changes for Tomcat are currently in the latest development codebase and will be included in version 5.5.16 and later.

In this article, I'll discuss how to use Tomcat 5.5, mod_jk 1.2, Apache 2.0, and Pluto 1.0.1 to construct a functioning Tomcat cluster that will properly replicate portlet session information.

Background Information

This article assumes you are already familiar with JSR-168 Portlets and have a general understanding of Tomcat, Apache, and mod_jk. It is also important to understand the concepts in using a Tomcat cluster and then load-balancing connections to that cluster. Review the following documentation for the necessary background.

Tomcat 5.5 - Clustering and Session Replication:
http://tomcat.apache.org/tomcat-5.5-doc/cluster-howto.html

Tomcat 5.5 - Load Balancing:
http://tomcat.apache.org/tomcat-5.5-doc/balancer-howto.html

Step 1: Set up Tomcat

In this example, we are going to set up a Tomcat cluster consisting of two instances of Tomcat running on the same system. There are no real differences in setting it up across multiple systems – only the hostnames, IP addresses, and port numbers would be different.

The support for cross-context session replication is available in Tomcat as of version 5.5.16. This version has not been released yet at the time of the writing of this article, so until it is available, you will need a nightly-build or self-build development version.

Tomcat 5.5 - Building Tomcat:
http://tomcat.apache.org/tomcat-5.5-doc/building.html

We'll start by building up one instance of Tomcat with everything we need and when we are done we will make a copy of it and make some minor changes. We'll call the two instances 'tomcatA' and 'tomcatB', so we'll name the directories this way as well.

Unpack your 5.5.16+ build or development build of Tomcat and rename the directory 'tomcatA'.

Make sure that the CATALINA_HOME and CATALINA_BASE environment variables are not set. If they are set and you want to leave them set, modify the startup.sh/startup.bat files in tomcatA/bin to explicitly set the CATALINA_HOME and/or CATALINA_BASE variables. Without all this, you won't be able to start two instances of Tomcat on that same system.

Replace the entire tomcatA/conf/server.xml file with the following:

<Server port="8006" shutdown="SHUTDOWN">

  <Listener className="org.apache.catalina.core.AprLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.storeconfig.StoreConfigLifecycleListener"/>

  <GlobalNamingResources>

    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
       description="User database that can be updated and saved"
           factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
          pathname="conf/tomcat-users.xml" />

  </GlobalNamingResources>

  <Service name="Catalina">

    <Connector port="8081" maxHttpHeaderSize="8192"
               maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
               enableLookups="false" acceptCount="100"
               connectionTimeout="20000" disableUploadTimeout="true"
               emptySessionPath="true" />

    <Connector port="8010"
               enableLookups="false" redirectPort="8081" protocol="AJP/1.3"
               emptySessionPath="true" />

    <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcatA">

      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>

      <Host name="localhost" appBase="webapps"
       unpackWARs="true" autoDeploy="true"
       xmlValidation="false" xmlNamespaceAware="false">

        <Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"
                 managerClassName="org.apache.catalina.cluster.session.DeltaManager"
                 expireSessionsOnShutdown="false"
                 useDirtyFlag="true"
                 notifyListenersOnReplication="true">

            <Membership
                className="org.apache.catalina.cluster.mcast.McastService"
                mcastAddr="228.0.0.4"
                mcastPort="45564"
                mcastFrequency="500"
                mcastDropTime="3000"/>

            <Receiver
                className="org.apache.catalina.cluster.tcp.ReplicationListener"
                tcpListenAddress="auto"
                tcpListenPort="4002"
                tcpSelectorTimeout="100"
                tcpThreadCount="6"/>

            <Sender
                className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"
                replicationMode="pooled"
                ackTimeout="15000"/>

            <Valve className="org.apache.catalina.cluster.tcp.ReplicationValve"
                   filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.css;.*\.txt;"/>

            <Deployer className="org.apache.catalina.cluster.deploy.FarmWarDeployer"
                      tempDir="/tmp/war-temp/"
                      deployDir="/tmp/war-deploy/"
                      watchDir="/tmp/war-listen/"
                      watchEnabled="false"/>

            <ClusterListener className="org.apache.catalina.cluster.session.ClusterSessionListener"/>

        </Cluster>

      </Host>

    </Engine>

  </Service>

</Server>

The important differences here from the standard server.xml are the following:

The HTTP <Connector> declaration and the AJP/1.3 <Connector> declaration both contain 'emptySessionPath="true"' – this is necessary so that we can share session information between servlets and portlets by having the path on the session cookie be set to '/' instead of the path of the webapp. Without this, the portal webapp and the portlet webapp will get separate session cookies and therefore separate session IDs.
The <Engine> declaration has 'jvmRoute="tomcatA"' set in it. This is needed to identify the engine to enable sticky sessions (i.e. a user stays with the same engine for his entire session unless the engine becomes unavailable).
The entire section for the <Cluster> declaration has been enabled using all the default settings.
The 'port' for the <Server> declaration, the 'port' for both <Connector> declarations, the 'redirectPort' for the AJP <Connector> declaration, and the 'tcpListenPort' for the <Receiver> declaration inside the <Cluster> declaration have all been incremented by one in order to not conflict with any preexisting install of Tomcat.

Step 2: Deploy the Pluto Portal Driver & Test Suite

Pluto is the reference implementation of the JSR-168 specification for a portlet-container. It also contains an application called the "Portal Driver" which can be used as a skeletal portal environment for developing and testing portlet applications without using a real portal. We'll use Pluto and the Pluto Portal Driver in this example. The equivalent steps should work with any JSR-168 portal platform that runs in Tomcat.

Obtain the current pluto-lib-core and pluto-lib-tools downloads from the Pluto website – they are available in both .zip and .tar.gz format, depending on your preference. I'm working with the 1.0.1 release at this point.

Apache Pluto Website:
http://portals.apache.org/pluto/

Deploy the 'pluto-1.0.1.jar' file from the pluto-lib-core archive into the tomcatA/shared/lib directory.

You'll also need the 'portlet.jar' file that provides the formal JSR-168 API. You can download the API by following the links from the main JSR-168 site.

JSR-168 Specification Website:
http://jcp.org/aboutJava/communityprocess/final/jsr168/

Deploy the 'portlet.jar' file into the 'tomcatA/shared/lib' directory as well. You might want to rename the file as 'portlet-api.jar' so you know exactly what it is. You can also get this file out of the full binary Pluto distribution that includes a pre-configured Tomcat instance.

You'll also need an XML Parser library available in your classpath. Download the latest XercesJ2 release and put xercesImpl.jar and xml-apis.jar into the tomcatA/common/endorsed directory.

Apache Xerces Website:
http://xerces.apache.org/

From the pluto-lib-tools archive, retrieve the 'pluto-driver.war' and 'pluto-testsuite.war' files. Rename 'pluto-driver.war' as just 'pluto.war' and rename 'pluto-testsuite.war' as just 'testsuite.war'.Deploy these into the webapps directory of the Tomcat instances. You should probably unzip them by hand because we need to make some changes before we use them. Make sure they end up deployed as 'tomcatA/webapps/pluto' and 'tomcatA/webapps/testsuite'.

Create a context configuration file for the pluto webapp. In tomcatA/conf/Catalina/localhost, create a file called pluto.xml that contains the following:

<Context path="/pluto" crossContext="true" />

The definition of 'crossContext="true"' is critical here – this allows the Pluto Portal Driver to make calls into the portlets running inside other webapps.

The testsuite webapp comes packaged generically and must be deployed into the target JSR-168 portal platform. Unfortunately, the pluto webapp comes configured as if the testsuite was already installed. So instead of running the testsuite through the deployment process, we will hand-modify it to complete the deployment. To do this, modify the tomcatA/webapps/testsuite/WEB-INF/web.xml file.

Add the following <servlet> entries after the existing ones:

<servlet>
    <servlet-name>TestPortlet1</servlet-name>
    <servlet-class>org.apache.pluto.core.PortletServlet</servlet-class>
    <init-param>
        <param-name>portlet-class</param-name>
        <param-value>org.apache.pluto.portalImpl.portlet.TestPortlet</param-value>
    </init-param>
    <init-param>
        <param-name>portlet-guid</param-name>
        <param-value>testsuite.TestPortlet1</param-value>
    </init-param>
    <security-role-ref>
        <role-name>plutoTestRole</role-name>
        <role-link>tomcat</role-link>
    </security-role-ref>
</servlet>

<servlet>
    <servlet-name>TestPortlet2</servlet-name>
    <servlet-class>org.apache.pluto.core.PortletServlet</servlet-class>
    <init-param>
        <param-name>portlet-class</param-name>
        <param-value>org.apache.pluto.portalImpl.portlet.TestPortlet</param-value>
    </init-param>
    <init-param>
        <param-name>portlet-guid</param-name>
        <param-value>testsuite.TestPortlet2</param-value>
    </init-param>
    <security-role-ref>
        <role-name>plutoTestRole</role-name>
        <role-link>tomcat</role-link>
    </security-role-ref>
</servlet>

And then add the following <servlet-mapping> entries after the existing ones:

<servlet-mapping>
    <servlet-name>TestPortlet1</servlet-name>
    <url-pattern>/TestPortlet1/*</url-pattern>
</servlet-mapping>

<servlet-mapping>
    <servlet-name>TestPortlet2</servlet-name>
    <url-pattern>/TestPortlet2/*</url-pattern>
</servlet-mapping>

At this point you should be able to start the tomcatA instance and connect to http://localhost:8081/pluto/portal/ and see the portal and try out the test suite portlets. Do not proceed until this is all working correctly.

Step 3: Deploy the Sample Web Application

For this article, I've developed a small sample web application to test and demonstrate that session replication is working correctly. The webapp is called 'session-test'. Download it using the link below.

Session-Test Sample Application:
session-test.war

Use the deployment portlet inside Pluto to deploy the session-test.war file. Do not just deploy the war file into Tomcat directly – the deployment process makes some necessary modifications to the web.xml file. To do this, complete the following:

In the left-hand navigation of the Pluto Portal Driver, select 'Admin'.
In the 'Deploy War Admin Portlet', click 'Browse...' and select the 'session-test.war' file that you downloaded. Then click 'Submit'.
In the form for the layout information, set the Title as 'Session' and the descriptios as 'Cross-Context Session Replication Test'. Set the number of columns to '2' and leave the number of rows as '1'. Then click 'Submit'.
Leave the entry in both column layouts as 'session-test' and click 'Submit'.

Unfortunately, the deployment tool in the Pluto Portal Driver is not smart enough to realize that we wanted two copies of the same portlet instead of them being two different portlets, and it generates some invalid changes to the web.xml file that we must clean up. So, stop Tomcat and edit tomcatA/webapps/session-test/WEB-INF/web.xml and remove the redundant <servlet> and <servlet-mapping> declarations from the file. The webapp won't start correctly until you do this.

The reason that we added two instances of the session-test portlet to the page is to demonstrate multiple portlets are sharing application-level session data but retaining separate portlet-level session data.

Create a context configuration file for the session-test webapp. In tomcatA/conf/Catalina/localhost, create a file called session-test.xml that contains the following:

<Context path="/session-test" crossContext="true" />

Even though portlet webapps do not need to be declared as cross-context for normal communication with the portal, they must be declared as cross-context for the session replication to work properly.

Now restart Tomcat and test that you are able to see the session-test portlets correctly by visting http://localhost:8081/pluto/portal/ and clicking on 'Session' in the left navigation.

Step 4: Prepare for Clustering

In order for Tomcat to replicate session data, the web application must be declared as "distributable". This means that the web application is built to be deployed into a distributed servlet container, such as a Tomcat cluster. The key requirement for this is that all session attributes must implement java.io.Serializable.

The Pluto Portal Driver does not come declared as distributable. We specifically held off on modifying the pluto webapp for this because the 'Deploy War Admin Portlet' violates the rules – it puts something in the session that is not serializable. Since we are done using that tool now, we can proceed.

Modify tomcatA/webapps/pluto/WEB-INF/web.xml by adding the '<distributable/>' tag. This should appear immediately after the <display-name> and <description> entries (there may not be a description).

In the case of JSR-168, both the portal webapp and the portlet webapp must be declared as distributable for the portlet session data to be replicated. The session-test webapp comes already declared as distributable, so it does not need to be modified.

Step 5: Create 2nd Tomcat Instance

At this point, make a complete copy of the tomcatA directory and rename it as tomcatB. Go into the 'logs' and 'work' directories in tomcatB and delete everything in them.

If you modified the startup.sh/startup.bat files in tomcatA/bin to explicitly set the CATALINA_HOME and/or CATALINA_BASE variables, go modify these files again in tomcatB/bin to reflect the directory name for this instance.

Edit the tomcatB/conf/server.xml file ard make the following changes:

Change the <Engine> declaration to contain 'jvmRoute="tomcatB"' set in it. Again this is needed to identify this engine for sticky sessions.
Increment by one the 'port' in the <Server> declaration, the 'port' in both <Connector> declarations, the 'redirectPort' in the AJP <Connector> declaration, and the 'tcpListenPort' for the <Receiver> in the <Cluster>. This is necessary so they will not conflict with the tomcatA instance.

You should now be able to start both Tomcat instances and see in their logs that they are communicating as a cluster. If you are not seeing happy messages about cluster membership in the log, go back and review the Tomcat documentation and resolve this issue before proceeding.

Step 6: Set up Apache & mod_jk Load-Balancing

Obtain Apache 2 and the mod_jk 1.2 connector for it.

Apache Web Site:
http://httpd.apache.org/

mod_jk Web Site:
http://tomcat.apache.org/connectors-doc/

Follow the normal installation process for Apache and then place the mod_jk.so file into the modules directory of your Apache installation.

To enable mod_jk, include the following directives in your Apache httpd.conf file:

LoadModule jk_module modules/mod_jk.so

JkWorkersFile   conf/workers.properties
JkLogFile       logs/mod_jk.log
JkLogLevel      warn

JkMount /jkstatus/* status

JkMount /pluto/* loadbalancer

JkMount /testsuite/* loadbalancer

JkMount /session-test/* loadbalancer

Then create a file in the conf directory of your Apache installation called 'workers.properties'. This file should contain the following configuration:

worker.list=status,loadbalancer

worker.status.type=status

worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=tomcatA,tomcatB
worker.loadbalancer.sticky_session=1

worker.tomcatA.port=8010
worker.tomcatA.host=localhost
worker.tomcatA.type=ajp13
worker.tomcatA.lbfactor=1

worker.tomcatB.port=8011
worker.tomcatB.host=localhost
worker.tomcatB.type=ajp13
worker.tomcatB.lbfactor=1

With both of your Tomcat instances up and running, restart Apache. Now browse to http://localhost/jkstatus/ and you should see the details about your load balancer and your two Tomcat workers. The tomcatA and tomcatB workers should both have status as 'OK'. If this is not the case, resolve this before proceeding.

This article assumes your Apache is listening on port 80. Modify the URL above and subsequent URL as necessary if a different port number should be used.

Step 7: Test it!

Everything should now be installed and configured properly for cross-context session replication in JSR-168 portlets. We will use the sample application to prove that this is working.

Bring up one browser window pointing to the JK status page at http://localhost/jkstatus/ and verify that both engines are still listed as OK.

Now open a separate browser window and go to http://localhost/pluto/portal/session-test/ – you should see the Pluto environment and the two instances of the session-test portlet. You should also see an entry at the bottom of the portlets that lists the phyiscal location of the JSP file you are viewing. The path in this entry will indicate whether the loadbalancer is sending you to tomcatA or tomcatB.

Portlets have two different session scopes: portlet scope and application scope. Information in the portlet scope session is just for that instance of the portlet and is not shared with other portlets. Information in the application scope session is shared between all the portlets in the same webapp and also with servlets in that webapp (if you have 'emptySessionPath' enabled). The session-test portlet lets you enter a String value into either scope. Play with these two instances of the portlet and note that changes in the portlet scope are properly limited to the specific portlet and that changes in the application scope appear in both portlets.

Once you have initialized the various sessions with data, it is time to cause your connection to fail-over to the other Tomcat instance to see if your session data is preserved. Go back to your window with the JK status page and select the worker that you have been running against. In the 'Edit worker settings' section, check the box for 'Stopped' and press 'Update Worker'. Verify in the status list that the worker is listed as 'Stopped'.

Now go back to your browser window that is displaying the portlets and click one of the 'Refresh' buttons inside one of the portlets. This will cause the portlet to rerender, but since JK is not able to send requests to the engine you were previously using, it will send you to the other one. You should see that the physical path of the displayed JSP changes to the other Tomcat instance. You should also see that your session data values stayed the same. This means that your portlet session data was replicated between the nodes in the cluster and everything is working.

Go ahead and make more changes to the session data and then go back to the JK status window and restart the first Tomcat instance. Now hit one of the 'Refresh' buttons again and you should see that you have switched back to the other instance and that the values are still maintained correctly.

As a further test, there is also a servlet in the sample webapp that shows the value in the application scope. In a new browser window, go to http://localhost/session-test/ and you will see the current value and again see the physical path of the JSP. You can also edit the value here and then go see that the application scope value changed in the portlets when you refresh them. This sharing of session data between portlets and servlets is only possible in Tomcat 5.5.4 and later since the 'emptySessionPath' feature is needed for this to work.

Conclusion

The configuration demonstrated above is genuinely useful on a single system. It allows you to take down one instance of Tomcat and performance maintenance and updates while the other instance continues to provide service. Then you can bring up the updated instance and take down the other instance for the same maintenance and updates. Then you can bring the second instance up and your overall service was never offline while you did maintenance. Realize that this configuration may use a great deal more memory than just a single instance of Tomcat.

We have now installed, configured, and tested a load-balanced Tomcat cluster that properly supports JSR-168 Portlet applications. Now all that remains is to deploy this into a real cluster and deploy a real application into it!