DSE Advanced Replication metrics

Collect metrics on each source node to review the current status of that node in the source cluster. A working source and destination configuration is required to use the metrics feature. See Getting started.

Ensure JMX access

Metrics are stored in the DataStax Enterprise JMX system. JMX access is required.

  • For production, DataStax recommends authenticating JMX users, see Enabling DSE Unified Authentication.

  • Use these steps to enable local JMX access. Localhost access is useful for test and development.

    1. On the source node, edit cassandra-env.sh and enable local JMX:

      JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=localhost"
      LOCAL_JMX=yes
      Where is the cassandra-env.sh file?

      The location of the cassandra-env.sh file depends on the type of installation:

      Installation Type Location

      Package installations + Installer-Services installations

      /etc/dse/cassandra/cassandra-env.sh

      Tarball installations + Installer-No Services installations

      <installation_location>/resources/cassandra/conf/cassandra-env.sh

    2. On the source node, stop and restart DataStax Enterprise to recognize the local JMX change.

Display metrics on the command line

Use the dse advrep command line tool to display metrics on the command line. Ensure that the source node meets the command line prerequisites.

  1. On the source node:

    dse advrep --jmx-port 7199 metrics list
    ------------------------------------------
    |Group         |Type               |Count|
    ------------------------------------------
    |Tables        |MessagesDelivered  |1002 |
    ------------------------------------------
    |ReplicationLog|CommitLogsToConsume|1    |
    ------------------------------------------
    |Tables        |MessagesReceived   |1002 |
    ------------------------------------------
    |ReplicationLog|MessageAddErrors   |0    |
    ------------------------------------------
    |ReplicationLog|CommitLogsDeleted  |0    |
    ------------------------------------------
    
    -------------------------------------------------------------------------------------------------------------------------------------------------
    |Group         |Type                 |Count|RateUnit     |MeanRate            |FifteenMinuteRate    |OneMinuteRate      |FiveMinuteRate         |
    -------------------------------------------------------------------------------------------------------------------------------------------------
    |ReplicationLog|MessagesAdded        |1002 |events/second|0.012688461014514603|9.862886141388435E-39|2.964393875E-314   |2.322135514219019E-114 |
    -------------------------------------------------------------------------------------------------------------------------------------------------
    |ReplicationLog|MessagesDeleted      |0    |events/second|0.0                 |0.0                  |0.0                |0.0                    |
    -------------------------------------------------------------------------------------------------------------------------------------------------
    |ReplicationLog|MessagesAcknowledged |1002 |events/second|0.012688456391385135|9.86403600116801E-39 |2.964393875E-314   |2.3230339468969963E-114|
    -------------------------------------------------------------------------------------------------------------------------------------------------
    |ReplicationLog|CommitLogMessagesRead|16873|events/second|0.21366497971804438 |0.20580430240786005  |0.39126032533612265|0.2277227124698431     |
    -------------------------------------------------------------------------------------------------------------------------------------------------
    
    -------------------------------------
    |Group       |Type            |Value|
    -------------------------------------
    |Transmission|AvailablePermits|30000|
    -------------------------------------

Accessing the metrics

Use JMX to access the metrics. Any JMX tool, such as jconsole, can access the MBeans for advanced replication. The port listed above, 7199, is used with the hostname or IP address:

JConsole:New Connection Window

Choose the MBeans tab and find com.datastax.bdp.advrep.v2.metrics in the left-hand navigation frame:

Attribute values in MBeans tab

The example shown here displays the attributes for com.datastax.bdp.advrep.v2.metrics:type=ReplicationLog,name=MessagesAdded.

Performance metrics

Metrics are exposed as JMX MBeans under the com.datastax.bdp.advrep.v2.metrics path and are logically divided into main groups. Each group refers to an architecture component. Metrics types are:

Counter

A simple incrementing and decrementing 64-bit integer.

Meter

Measures the rate at which a set of events occur.

Histogram

Measures the distribution of values in a stream of data.

Timer

A histogram of the duration of a type of event and a meter of the rate of its occurrence.

Gauge

A gauge is an instantaneous measurement of a value.

Metrics are available for the following groups:

Metrics are also available per table:

Descriptions of each metric is provided.

Metrics for DSE 5.0 (V1) are still present; see the DSE 5.0 documentation for those metrics.

ReplicationLog

Metrics for the ReplicationLog group
Metric name Description Metric type

MessagesAdded

The number of messages that were added to the replication log, and the rate that the messages were added, per replica.

Meter

MessagesAcknowledged

The number of messages that were acknowledged (and removed) from the replication log. Acknowledgement can be 1 or 1+n if errors occur.

Meter

MessagesDeleted

The number of messages that were deleted from the replication log, including invalid messages and messages that were removed after a channel truncate operation.

Meter

MessageAddErrors

The number of errors that occurred when adding a message to the replication log.

Counter

CommitLogsToConsume

The number of commit logs that need to be consumed that have advanced replication messages.

Counter

CommitLogMessagesRead

The number of commit log messages added to the replication log. The commit log messages are read if a message pertains to a source table that has collection enabled.

Meter

CommitLogMessagesDeleted

The number of commit log messages deleted from the commit log after adding to the replication log. Like CommitLogMessagesRead, this metric only pertains to messages in tables that are enabled for advanced replication.

Meter

Transmission

Metrics for the Transmission group
Metric name Description Metric type

AvailablePermits

The current number of available global permits for transmission.

Gauge

AdvancedReplicationHub-[destinationName]-metrics

Metrics for the AdvancedReplicationHub-[destinationName]-metrics group are provided automatically by the DSE Java driver.

advRepMetrics4

Incomplete examples of per-destination-metrics are:

Metric name Metric type

known-hosts

Counter

connected-to

Counter

open-connections

Counter

requests-timer

Timer

connection-errors

Counter

write-timeouts

Counter

read-timeouts

Counter

unavailables

Counter

other-errors

Counter

retries

Counter

ignores

Counter

For details, see the DSE Java driver documentation.

Performance metrics per table

Use JMX to find performance metrics per table, look under the com.datastax.bdp.advrep.v2.metrics tab in the left-hand navigation frame for Tables, select a table and inspect the metrics:

advRepMetrics3

For example, to access the MessagesReceived metric for the table sensor_readings in the keyspace demo look at the following path:

$ com.datastax.bdp.advrep.v2.metrics:type=Tables,scope=demo.sensor_readings,name=MessagesReceived

The following metrics are provided per table:

Metrics
Metric name Description Metric type

MessagesReceived

The number of messages received from the source cluster for this table.

Counter

MessagesDelivered

The number of messages for the source table that were replicated to the destination.

Counter

MessagesDeleted

The number of messages that were deleted from the replication log, including invalid messages and messages that were removed after a channel truncate operation.

Counter

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com