Clustering with Terracotta

Purpose

This tutorial describes how to integrate Terracotta (TC) into AppFuse using the TC Maven Plugin. We hence assume that you have a general understanding of what these applications can do. At the end of this tutorial, you should have a TC enabled application capable of scaling up to several thousands of simultaneous users within a few clicks

Since Terracotta is a library used to cluster data seamlessly amongst many nodes, we'll be sharing parts of the info that are running in every instance of AppFuse amongst n-number of Jetty Servers.

Setting up the environment

For the following, we'll assume you've already setup your AppFuse environment, if not, please refer to AppFuse's QuickStart with Spring MVC web framework. Please notice that Terracotta can also be integrated in any other web framework in AppFuse.

Start by creating a new AppFuse Spring MVC project:

GroupId: com.mycompany
ArtifactId: myproject
Version: 2.1.0-M1
Web Framework: Spring MVC

Create the new project by executing the following generated Maven command:

mvn archetype:generate -B -DarchetypeGroupId=org.appfuse.archetypes \
-DarchetypeArtifactId=appfuse-basic-spring-archetype \
-DarchetypeVersion=2.2.0 -DgroupId=com.mycompany -DartifactId=myproject

Run the following command inside the project's directory:

mvn jetty:run-war

This will run the AppFuse application on http://localhost:8080.

Integrating Terracotta

Terracotta runs as a standalone server on a machine. It accepts 'client' connections from remote applications, namely our AppFuse based applications.
In order to tell AppFuse where to find the central Terracotta server, we use the TC configuration file, named "tc-config.xml" by convention. Inside this file, we usually specify the hostname of the TC server, the data to be shared, where to log the server and client logs, which classes to instrument and which modules to include.

Place the following configurations inside a file named "tc-config.xml" file and save it in the same directory that contains the file pom.xml of the AppFuse project.

<tc:tc-config xmlns:tc="http://www.terracotta.org/config">

<!--Tell DSO where the Terracotta server can be found;

See the Terracotta DSO Guide for additional information.-->

<servers>
    <server host="%i" name="sample">
        <data>target/terracotta/server/data</data>
        <logs>target/terracotta/server/logs</logs>
        <statistics>target/terracotta/server/statistics</statistics>
    </server>
</servers>
<!--Tell DSO where to put the generated client logs
See the Terracotta DSO Guide for additional information.-->
<clients>
   <logs>target/terracotta/clients/logs/%(tc.nodeName)</logs>
   <statistics>target/terracotta/clients/statistics/%(tc.nodeName)</statistics>
   <modules>
        <!-- this module will automatically tell the Jetty container to share and cluster the session map -->
        <module name="tim-jetty-6.1" version="2.1.1" />
   </modules>
</clients>
<application>
<dso>
<!--Our app requires these custom objects/classes to be shared - the following declarations
tells DSO which ones they are. When the app runs under DSO, instances of these classes
will broadcast changes in their state.
A good idiom when writing an app that you intend to cluster via TC DSO, is to group the
classes you wish to share under a single package (although if you follow the MVC pattern
this tends to happen naturally) - this way the list of classes you wish to instrument
can be concise-->
<instrumented-classes>
<!--Include all application classes that are clustered-->
<include>
    <class-expression>com.opensymphony.clickstream.Clickstream</class-expression>
</include>
<include>
    <class-expression>com.opensymphony.clickstream.ClickstreamRequest</class-expression>
</include>
<include>
    <class-expression>org.springframework.security.ui.savedrequest.SavedRequest</class-expression>
</include>
<include>
    <class-expression>org.springframework.security.ui.savedrequest.SavedCookie</class-expression>
</include>
<include>
    <class-expression>org.springframework.security.context.SecurityContextImpl</class-expression>
</include>
<include>
    <class-expression>org.springframework.security.providers.UsernamePasswordAuthenticationToken</class-expression>
</include>
<include>
    <class-expression>org.springframework.security.providers.AbstractAuthenticationToken</class-expression>
</include>
<include>
    <class-expression>org.appfuse.model.User</class-expression>
</include>
<include>
    <class-expression>org.appfuse.model.BaseObject</class-expression>
</include>
<include>
    <class-expression>org.springframework.security.ui.WebAuthenticationDetails</class-expression>
</include>
<include>
    <class-expression>org.appfuse.model.Role</class-expression>
</include>
<include>
    <class-expression>org.appfuse.model.Address</class-expression>
</include>
<include>
    <class-expression>org.hibernate.collection.PersistentSet</class-expression>
</include>
<include>
    <class-expression>org.hibernate.collection.AbstractPersistentCollection</class-expression>
</include>
<include>
    <class-expression>org.springframework.security.BadCredentialsException</class-expression>
</include>
<include>
    <class-expression>org.springframework.security.AuthenticationException</class-expression>
</include>
<include>
    <class-expression>org.springframework.security.SpringSecurityException</class-expression>
</include>
<include>
    <class-expression>org.springframework.core.NestedRuntimeException</class-expression>
</include>
</instrumented-classes>
<additional-boot-jar-classes>
    <include>java.lang.String$CaseInsensitiveComparator</include>
    <include>java.util.Locale</include>
</additional-boot-jar-classes>
<!--Tell DSO which applications in your web container is using DSO-->
<web-applications>
    <web-application>myproject-1.0-SNAPSHOT</web-application>
</web-applications>
</dso>
</application>
</tc:tc-config>

We now need to TC-enable the AppFuse generated project. In order to keep things simple, we'll use the TC Maven plugin which downloads a mock Terracotta server and runs it on the localhost. It will download Terracotta's core libraries and use them to run the servers and nodes. This plugin will NOT use the Terracotta distribution you installed manually. The above distribution ships with an extra Swing GUI called the Dev Console that will allow you to debug the TC memory in real time.

Add the following to the pom.xml file in their corresponding places:

<repository>
    <id>terracotta-repository</id>
    <url>http://www.Terracotta.org/download/reflector/maven2</url>
    <releases>
        <enabled>true</enabled>
    </releases>
    <snapshots>
    <enabled>true</enabled>
    </snapshots>
</repository>

<pluginRepository>
    <id>terracotta-repository</id>
    <url>http://www.terracotta.org/download/reflector/maven2</url>
    <releases>
        <enabled>true</enabled>
    </releases>
    <snapshots>
        <enabled>true</enabled>
    </snapshots>
</pluginRepository>

<!-- this will add the Jetty module library. This library will be used by the container -->
<dependency>
    <artifactId>tim-jetty-6.1</artifactId>
    <version>2.1.1</version>
    <groupId>org.terracotta.modules</groupId>
    <scope>provided</scope>
</dependency>

<!--
This will create a new profile that will be used in order to run the Terracotta server and two Terracotta nodes (application server instances).
This will download the Terracotta core libraries (version 3.2.1) that will be used.
This will download the Jetty container also.
-->
<profile>
    <id>jetty6x</id>
    <build>
        <plugins>
            <plugin>
                <artifactId>tc-maven-plugin</artifactId>
                <version>1.5.1</version>
                <groupId>org.terracotta.maven.plugins</groupId>
                <configuration>
                    <processes>
                        <process nodeName="cargo" count="2">
                            <container>
                                <containerId>jetty6x</containerId>
                                <zipUrlInstaller>
                                    <url>http://dist.codehaus.org/jetty/jetty-6.1.22/jetty-6.1.22.zip</url>
                                    <installDir>installs</installDir>
                                </zipUrlInstaller>
                            </container>
                        </process>
                    </processes>
                </configuration>
            </plugin>
        </plugins>
    </build>
</profile>

Running AppFuse with Terracotta

Now go to your project's directory. Make sure you generated the war file of the project in the target folder. If you don't have the war file, execute the following command to generate it:

mvn jetty:run-war

Then execute the following command to run the project with Terracotta:

mvn -Pjetty6x tc:run

If you run the command for the first time then you'll have to wait after getting this because Maven will be downloading the "installs" folder which contains Jetty (24.3 MB).

[INFO] Resolving modules: [<xml-fragment name="tim-jetty-6.1" version="2.1.1" xmlns:tc="http://www.terracotta.org/config"/>]
[INFO] Resolving bundle: org.terracotta.modules:excludes-config:3.2.1
[INFO] Resolving bundle: org.terracotta.modules:modules-base:1.2.1
[INFO] Resolving bundle: org.terracotta.modules:guimodels-config:3.2.1
[INFO] Resolving bundle: org.terracotta.modules:jdk15-preinst-config:3.2.1
[INFO] Resolving bundle: org.terracotta.modules:standard-config:3.2.1
[INFO] Resolving location: org.terracotta.modules:tim-jetty-6.1:2.1.1
[INFO] Resolving bundle: org.terracotta.modules:tim-session-common:2.1.1
[INFO] Resolving bundle: org.terracotta.modules:tim-distributed-cache:1.3.1
[INFO] Resolving bundle: org.terracotta.modules:tim-concurrent-collections:1.3.1
[INFO] Starting DSO nodes
[INFO] [dso start] 2010-03-28 13:57:37,180 INFO - Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work.

This command will choose the Jetty profile and execute the tc:run using the tc-maven-plugin. This will start one Terracotta server. This will also run two client nodes which will connect automatically to the server. The two client nodes will automatically use the tc-config.xml we created as their configuration files.

Make sure you remove the your old cookie session from your browser.

Congratulations !!!
You're all set. Your myproject application will be run by two client nodes (listening on different ports):

http://localhost:8080/myproject-1.0-SNAPSHOT
http://localhost:8081/myproject-1.0-SNAPSHOT

Follow those steps to make sure that Terracotta is clustering your session:

go to http://localhost:8080/myproject-1.0-SNAPSHOT
login using admin/admin
go to http://localhost:8081/myproject-1.0-SNAPSHOT/admin/users.html (using the other port)
if you are not redirected to the login screen then it worked

You can monitor your Terracotta server and the two clients by running the dev console:

Linux:

$TC_INSTALL_DIR/bin/dev-console.sh

Windows:

$TC_INSTALL_DIR/bin/dev-console.bat

Configure your loadbalancer to dispatch the load on the 2 nodes. At anytime, if any of the nodes fails, your inmemory data is safely clustered on the TC Server (which can be installed in a highly available mode).

From here on

You should be able to setup your AppFuse instances to connect to an actual TC Server in no time. To do this, refer to the 'host' parameter inside the config.xml file and change it to point to the newly installed TC Server.

Helpful sources:

Ayman Abou Hamra
abouhamra@setic.eu
Setic http://www.setic.eu

In this article, we've used

Maven 2.2.1
Java: 1.6.0_10
Terracotta: 3.2.1
AppFuse 2.1.0-M1 with the Spring MVC framework
Jetty 6.1.22