Auto-Clustering Configuration

Last Updated: Mar 4, 2021
documentation for the dotCMS Content Management System

The Auto-clustering feature allows you to automatically add servers to a cluster, and automatically manages the cluster configuration for you. The following configuration steps are required to set up auto-clustering.

Note: You must complete all installations steps in the Cluster Setup document before performing any of these installation steps.

Download and Apply a License Pack

Auto-clustering requires a valid Enterprise license pack. Enterprise clients and customers wishing to trial dotCMS can request license packs from dotCMS website.

License packs are a zip file of multiple license designed for auto-clustering. The entire .zip file can be uploaded to any server that is part of the cluster. Once the license pack is uploaded, a license can be chosen for the current server.

For more information on applying licenses, see the License Management documentation.

Add Additional Servers to the Cluster

The rest of the server nodes in the cluster do not need their own database or assets folder, and do not need separate licenses. Instead, each additional server in the cluster must only:

  • Have a symbolic link to the same asset directory as the first server.
  • Share a connection to the same database.

No other configuration is required. However the following settings found in the dotcms-config-cluster.properties file can be used to modify Auto-clustering behavior:

PropertyDefaultDescription
HEARTBEAT_TIMEOUT1800
(30 minutes)
Timeout (in seconds) before a node will be considered unresponsive (please see Unresponsive Nodes, below).
Note: 30 minutes without a heart beat provides a reasonable time window to perform normal server changes without dropping the node from the cluster.
HEARTBEAT_CRON_EXPRESSION0 0/1 * * * ?
(once every minute)
The frequency to check for dead servers in the cluster.

As each additional node starts up, it will recognize that it shares the same database and assets directory as the first node in the cluster, and will automatically search for an available license from the license pack already uploaded to the first server. If a license is available, it will automatically be assigned, as long as there are enough valid licenses in the license pack for each node. Additional license packs can always be uploaded to the first server, allowing scaling of the cluster over time.

Unresponsive Nodes

If a node's “heartbeat” is not detected before the HEARTBEAT_TIMEOUT expires, then the node will be dropped from the cluster. If a node is dropped, dotCMS will:

  1. Free the license.
  2. Update the Elasticsearch replica count.
  3. Rewire the cache and Elasticsearch.

If a temporarily disconnected server is re-discovered within 30 minutes, the server can still re-join the cluster normally.

Elasticsearch Replica Management

By default, the Auto-clustering feature handles the replicas for Elasticsearch. The default configuration setting ES_INDEX_REPLICAS=autowire. The AUTOWIRE_CLUSTER_ES is also set to true by default.

When AUTOWIRE_CLUSTER_ES is set to true OR if ES_INDEX_REPLICAS is set to anything other than a fixed integer, then backend UI manual management of clustered elastic search replicas is disabled.

However, if AUTOWIRE_CLUSTER_ES=false AND ES_INDEX_REPLICAS is set to a fixed INTEGER (not a range), then this enables the backend UI manual management of clustered elastic search replicas

The possible values for ES_INDEX_REPLICAS are:

ValueSynposisDetailsAutomatically sets
auto_expand_replicas to
autowireUse the dotCMS autowire logic to set the replicas
based on the current alive dotCMS instances.
  • Should only be used with local Elasticsearch indexes.
  • Lets the autowire logic set the replicas based on the alive-node count.
false
{integer}
(any integer value)
Use the specified number of static replicas.
  • Useful if you have a static set of external Elasticsearch servers.
  • If there are not enough instances/replicas available to satisfy the configuration, the index will go into the “yellow” (unhealthy) state until additional instances/replicas are available.
false
{low}-{high}Sets a minimum number of replicas, and an
upper boundary on the number the index can expand to.
  • Please contact dotCMS Support before implementing this option.
    • This option allows for flexible Elasticsearch cluster scalability, but can cause data loss in certain situations.
  • Allows the index replicas to auto-expand and contract based on the current Elasticsearch cluster topology.
    • The {low} can be any integer 0 or higher.
    • The {high} can be any integer equal to or larger than {low}, OR can be the keyword all (e.g. 0-all).
Based on the specified upper boundary

For more information on setting the number of replicas, please see the Manual Cluster Configuration documentation.

Auto-Cluster Configuration Checklist

The following checklist outlines the steps required to establish an auto-cluster environment:

  1. Have a valid Enterprise contract with available server licenses.
  2. Request a license pack by submitting a ticket on the dotCMS Support website.
  3. Start up the first dotCMS server (server #1).
  4. Upload the license pack to the License Manager of server #1.
  5. Select a license from the pack to be used by server #1.
  6. Configure the second dotCMS server to share the same assets and database.
  7. Start up the second server (it will automatically join the cluster).

To add additional server nodes to the cluster, repeat steps 6 & 7 and each node will automatically be joined to the cluster (as long as unused licenses are still available*).

* Warning: If multiple nodes in the cluster are on the same physical server, then the Tomcat port and shutdown ports will need to be different for each node in the cluster. Ports can be configured in the server.xml file (/dotserver/tomcat-X.xx/conf/server.xml), to avoid conflicts.

Verifying Your Cluster

After adding all nodes to your cluster, please see the Testing Your Cluster section in the Cluster Setup documentation for steps to verify the proper operation of your cluster.

On this page