ResponseMX Blog

Load Balancing ResponseMX over clustered Tomcats.

What is load balancing and clustering. Well a lot cleverer people then me have wrote a lot of text on the different type of load balancing algorithms and clustering methodologies, so instead of regurgitating what my learned colleagues have wrote I’ll just briefly sum up.

Load Balancing is a method of distributing requests onto multiple servers seamlessly to the user doesn’t realise.

Clustering is slightly more in-depth. For static web sites load balancing is fine for distributing load, however for statefull applications running on app servers load balancing alone won’t work. This problem is load balanced servers work completely independently, so consecutive requests sent to different application servers will lose their session. Clustering allows multiple application servers to share their session data, so it is transparent to the user which server they are using at any time

Why would you want to do this ? well for two reasons, firstly to enhance performance, on sites with large volumes (two servers are better then one), giving you more capacity. Secondly for business critical applications distributed server gives fail over, I.e. if one server goes down you have a seconds server there ready to take the load (just like those skeletons in Jason and the Argonauts).

We’ve been doing some work on ResponseMX with MapXtreme to run over load balanced servers with clustered tomcats. ResponseMX uses session persistence quite heavily to store the state of its geographic maps. Clustered applications can work in a few ways, such as persisting its session data to a database, or broadcasting its sessions data to any other application servers, to do this we need all session data to serialisable, this is allowing data to be streamed to a string or suitable format to be wrote to disk. The changes required to ResponseMX will be included in the 3.0 release, however if you would like to get hold of a Beta version of this functionality in 2.6 please contact a member of the Dotted Eyes team who will be happy to oblige.

To set up our environment we shall be using 2 Tomcat 5 Application servers clustered both running ResponseMX and Mapxtreme. We shall also use Apache HTTP server in front on them to act as our load balancer.

1. Setting up Apache as a load balance (Many thanks to Pascal Forget and Matt Raible) for a great aticle on setting up Apache as load balancer http://raibledesigns.com/tomcat/)

1.1. Download and install Apache2.2

1.2. Download and install the mod_jk module into the modules directory of Apache

1.3. Add the following text to the Dynamic Shared Object (DSO) Support section of the httpd.conf file to install mod_jk

<br />
  LoadModule jk_module modules/mod_jk.so<br />
  JkWorkersFile conf/workers.properties<br />
  JkLogFile logs/mod_jk.log<br />
  JkLogLevel info<br />


1.4. Add the the following text to the httpd.conf file telling Apache what requests are for ResponseMX and to forward to the Tomcat application server.

<br />
  JkMount /rmx26* loadbalancer<br />
  JkMount /mapxtreme* loadbalancer<br />


1.5. Create a workers.properties file in the conf folder of your Apache installation, with the following text

<br />
  ps=/<br />
  # list the workers by name<br />
  worker.list=tomcat1, tomcat2, loadbalancer<br />
  # ------------------------<br />
  # First tomcat server<br />
  # ------------------------<br />
  worker.tomcat1.port=11009<br />
  worker.tomcat1.host=localhost<br />
  worker.tomcat1.type=ajp13<br />
  worker.tomcat1.lbfactor=100<br />
  # ------------------------<br />
  # Second tomcat server<br />
  # ------------------------<br />
  worker.tomcat2.port=12009<br />
  worker.tomcat2.host=localhost<br />
  worker.tomcat2.type=ajp13<br />
  worker.tomcat2.lbfactor=100<br />
  # ------------------------<br />
  # Load Balancer worker<br />
  # ------------------------<br />
  worker.loadbalancer.type=lb<br />
  worker.loadbalancer.balanced_workers=tomcat1, tomcat2<br />


2. Set up 2 clustered tomcat instances, for this example we shall be running two instances of tomcat on the same server. We shall also be using the in memory broadcasting clustering setup.

2.1. Download Apache Tomcat, I’ve tested this configuration on Tomcat 5.5.25 and Java 1.5

2.2. Install it twice in two separate directories.

2.3. Change the server.xml file to make sure tomcat is listening on 2 different ports to avoid those annoying port conflict errors. I’ve chosen to run mine on ports 9080 and 10080

<br />
    <Connector port="9080" maxHttpHeaderSize="8192"<br />
           maxThreads="150" minSpareThreads="25" maxSpareThreads="75"<br />
           enableLookups="false" redirectPort="8443" acceptCount="100"<br />
           connectionTimeout="20000" disableUploadTimeout="true" /><br />


2.4. Change the AJP13 connector ports so they correspond the the ports in the workers.properties file in the apache installation

<br />
      <Connector port="11009"<br />
           enableLookups="false" redirectPort="8443" protocol="AJP/1.3" /><br />

2.5. Add the following cluster Element to the server.xml in the Engine element
<br />
         <Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"<br />
       managerClassName="org.apache.catalina.cluster.session.DeltaManager"<br />
       expireSessionsOnShutdown="false"<br />
       useDirtyFlag="true"<br />
       notifyListenersOnReplication="true"><br />
        <Membership<br />
      className="org.apache.catalina.cluster.mcast.McastService"<br />
      mcastAddr="228.0.0.4"<br />
      mcastPort="45564"<br />
      mcastFrequency="500"<br />
      mcastDropTime="3000"/><br />
        <Receiver<br />
      className="org.apache.catalina.cluster.tcp.ReplicationListener"<br />
      tcpListenAddress="auto"<br />
      tcpListenPort="4001"<br />
      tcpSelectorTimeout="100"<br />
      tcpThreadCount="6"/><br />
        <Sender<br />
      className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"<br />
      replicationMode="pooled"<br />
      ackTimeout="15000"<br />
      waitForAck="true"/><br />
        <Valve className="org.apache.catalina.cluster.tcp.ReplicationValve"<br />
         filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/><br />
        <Deployer className="org.apache.catalina.cluster.deploy.FarmWarDeployer"<br />
            tempDir="/tmp/war-temp/"<br />
            deployDir="/tmp/war-deploy/"<br />
            watchDir="/tmp/war-listen/"<br />
            watchEnabled="false"/><br />
        <ClusterListener className="org.apache.catalina.cluster.session.ClusterSessionListener"/><br />
    </Cluster><br />


3. Install Mapxtreme471 and ResponseMX on both tomcat instances.

3.1. Some configuration is needed on where the ResponseMX server will write its map images to, and where the client will request them from. The problem with having the application load balanced is you never know which server is going to get the request, and a request for a map from ResponseMX actually means two requests to the server, the xml map request telling the server what geoset to load, the view etc, which will respond with a URL to a map image, and the the second request to get that map image. However on load balanced and even clustered servers the image will only be wrote on that server, so if the second request goes to another server you’ll get a 404 error. The answer is to write the image to a shared directory (perhaps where your map data is being held). On my example I wrote them to a separate directory outside the ResponseMX application so the images could be requested straight from apache. The parameters that need changing are in the rmx.properties file

<br />
  servername  =  rmx1<br />
  image_url  =  http://<server name>/images<br />
  image_dir  =  C:/Program Files/Apache Software Foundation/Apache2.2/htdocs/images<br />


The servername property needs to be different for each clustered tomcat instance. This parameter is appended the the image name to avoid naming conflicts between servers.

3.2. The server.xml in each ResponseMX application needs to be changed; the following line should be added to the web.xml file in the WEB-INF folder.

This tell tomcat that the session objects for this application should be shared with any other application servers in the cluster

4. Have fun with maps

4.1. Start both your tomcat instances and apache

4.2. Open a browser and hit the ResponseMX application through Apache on port 80 as opposed to going straight to tomcat (http://(server name)/rmx20 )

4.3. By looking at the ResponseMX log files you should be able to tell which tomcat instance is is receiving the requests, and if all is working it should be all of them.

Have fun clustering everyone.

Here’s a tip for mapserver users which allows you read and display vector data stored within a regular database in ResponseMX.

You will need ODBC access to your database. I’ve used mysql on windows for this example but other databases should work just the same.

The tip takes advantage of OGR ability to understand geometry in Well Known Text (WKT) format, via its Virtual Format mechanism.

So to be of use, you’ll need to be able to create WKT feature descriptions and store them in a text column in your non-spatial database.

To do this you need to do the following:

1. Create a database containing your vector data
2. Create a system DSN to give ODBC access to a data source
3. Create an .ovf file
4. Test the connection through ogrinfo to prove access to the data
5. Modify map file
6. Modify the ResponseMX geoset

Here’s the detailed steps:

1. Create a database containing your vector data

Create a database table


  DROP TABLE IF EXISTS `test`.`polygon_test`;
  CREATE TABLE  `test`.`polygon_test` (
  `Record_Id` int(10) unsigned NOT NULL auto_increment,
  `Polygon` text NOT NULL,
  `Info` varchar(45) NOT NULL,
  PRIMARY KEY  (`Record_Id`)
  ) ENGINE=InnoDB DEFAULT CHARSET=latin1;


Now add some data to it. The Polygon column is going to hold the geometry which can be a very long string, so it needs to be a text type column.


insert into polygon_test values (DEFAULT, 'POLYGON (( 350600 420200, 350600 440200, 330100 440200, 330100 420200, 350600 420200 ))', 'A square');
insert into polygon_test values (DEFAULT, 'POLYGON (( 436909.520549625 346979.908498395, 428048.086151056 341231.951050675, 431401.061328893 329975.534382221, 443495.721791806 330694.029063187, 446249.951402172 340752.954596698, 446249.951402172 340752.954596698, 436909.520549625 346979.908498395 ))', 'A Pentagon');


2 Create a system DSN to give ODBC access to the data source

On Windows, Go to Control panel -> Administrative tools -> Data Sources (ODBC) then select the System DSN tab.

Click Add and select the correct driver for whatever database is being used, e.g. SQL Server, MySQL

From this point the set up will be specific to the database server, but typically you will have to configure the database server connection details, such as server name, database name, username and password, default database to connect to.

Test the connection once you have entered the necessary details.

Once you have a working ODBC connection to your database you’re ready for step 2.

3. Create an .ovf file

The ovf file provides mapserver with the connection details it needs to retrieve geometry and feature information from the database.

Example .ovf file:

<OGRVRTDataSource>
<OGRVRTLayer name=”boundaries”>
<SrcDataSource>ODBC:TestDbUser/secret@testdb</SrcDataSource>
<SrcLayer>geometryTable</SrcLayer>
<FID>Record_Id</FID>
<GeometryType>wkbPolygon</GeometryType>
<GeometryField encoding=”WKT” field=”POLYGON”/>
</OGRVRTLayer>
</OGRVRTDataSource>>

Things you will have to configure:

<OGRVRTLayer name=”boundaries”> – this needs to be the name of your data source, in this case ‘boundaries’

<SrcDataSource>ODBC:TestDbUser/secret@testdb</SrcDataSource>

This is the odbc connection string which is in the format

ODBC:username/password@database

or

ODBC:username/password@database,tableName

<SrcLayer>geometryTable</SrcLayer> – this is where you specify the table that holds your vector data.

<FID>Record_Id</FID> – this may or may not be required. It is used to specify the column that contains the unique key for the row (the Feature Identifier, hence FID). Mapserver will have a go at guessing but can be confused by some data structures, hence this tag.

For point data you would leave <GeometryType></GeometryType> set to wkbPoint and leave the encoding attribute of the <GeometryField/> tag set to “PointFromColumns”.
Then you will need to specify the columns where the x and y coordinates are drawn from in the database table as the x and y attributes of the <GeometryField/> tag.

However, in this example we’re storing polygons in our database, so you would set
<GeometryType></GeometryType> to wkbPolygon and set the GeometryField tag thus:
<GeometryField encoding=”WKT” field=”POLYGON”/>

This tells OGR to expect the geometry of our polygons to be stored in a column called POLYGON and encoded in WKT format.

You’ll need to store your .ovf file in the same folder as your mapserver’s .map file. To avoid confusion, give the file a name that matches the layer name you’re going to give the layer in your .MAP file. e.g. boundaries.ovf

Be aware that although the .ovf file format looks like xml, it isn’t actually fully-compliant xml, so don’t put an xml declaration at the top of the file and if you have problems when you test the connection, make sure you don’t have blank lines or hidden characters in the file.

4. Test the connection through ogrinfo to prove access to the data

Having configured your .ovf file, test it using the following ogrinfo command:

ogrinfo -al boundaries.ovf

ogrinfo will warn you that the layer is read only with the following message:

ERROR 4: Update access not supported for VRT datasources.

This is ok - unfortunately virtual format layers can’t be updated (yet). If your configuration is correct you will be rewarded with a list of the features in your database.

5. Modify map file

Now you’ll need to modify your mapserver’s .map file to include your new layer. Note that you’ll need to provide styling for the layer as there won’t be any styling information in your database.


LAYER
NAME "boundaries"
STATUS ON
CONNECTIONTYPE OGR
CONNECTION "boundaries.ovf"
DATA "polygon_test"
TYPE POLYGON