Metrics
Mission Control collects metrics across all components and aggregates them across projects and clusters. Review this unified observability data in the centralized user interface. Mission Control installs and configures metrics components at the same time as the Mission Control control plane and scales those components independently.
These components enable you to monitor metrics from many sources within Mission Control, including:
-
Platform services
-
Operators
-
Observability components
-
Reaper
-
-
Database instances
Mission Control only deploys observability components to Platform instances.
Metrics are read from the database as the scraping occurs, providing the most up-to-date metrics. Metrics are scraped every 30 seconds and logs are collected as they are written to disk. This data is then pushed to aggregator instances. The aggregator handles applying configured transforms and sinks to the data stream.
See collected metrics with Mission Control’s graphical metrics view.
You can use Mission Control to push metrics to existing monitoring stacks. Manipulate and send observability data externally by adding custom transforms and sinks in the Mission Control configuration.
Prerequisites
You must provide an AWS S3 or S3-compatible, Google Cloud Storage, or an Azure Blob Storage object store during installation and configuration. All metrics are stored within an object store, providing long-term storage for metrics.
When cloud-based metrics storage is a concern, for example if you don’t use a cloud provider, you can use an S3 API to store objects. For environments without an S3 endpoint, you can use Red Hat OpenShift Data Foundation Object Bucket Claim or SeaweedFS to provide an S3-compatible object store within the Mission Control platform.
Metrics collection
Vector is an observability pipeline framework from Datadog that collects metrics and logs from various Mission Control services.
Vector transforms and sends the metrics to destinations like Loki and Mimir.
The mission-control-aggregator ConfigMap in the mission-control namespace stores configuration data for Vector, including the vector.yaml file.
The vector.yaml file defines Vector’s behavior, such as configured transforms and sinks.
Each database instance within a control or data plane includes a server-system-logger sidecar container. The server-system-logger collects metrics and logs generated by the local database instance.
This configuration is part of a larger Vector configuration that defines other components, such as sources, transforms, and other sinks.
Vector configuration is managed through the aggregator.customConfig section in your values.yaml file.
The key components for metrics collection are:
-
aggregator: Configures the Vector aggregator that processes and forwards metrics -
mimir: Configures the metrics storage backend -
agent: Configures the Vector agent that collects metrics from nodes For detailed configuration options, see Install Mission Control with Helm.
For example, the following TOML configuration defines two sinks within a Vector configuration.
The configuration specifies how to send data to the vector_aggregator and the console.
aggregator:
enabled: true
customConfig:
sinks:
vector_aggregator:
type: "vector"
inputs: ["cassandra_metrics", "enrich_host_metrics", "add_source_to_systemlog", "gclog_parser"]
address: "mission-control-aggregator.mission-control.svc:6000"
console_log:
type: "console"
inputs: ["systemlog"]
target: "stdout"
encoding:
codec: "text"
Functionally, this configuration instructs Vector to do the following:
-
Collect data from the specified input sources.
-
cassandra_metrics -
enrich_host_metrics -
add_source_to_systemlog -
gclog_parser
-
-
Forward the collected data to the
vector_aggregatorservice at the specified address. -
Forward the systemlog data to the console for immediate inspection:
-
[sinks.vector_aggregator]: Defines thevector_aggregatorsink. The aggregator forwards direct sources, enriched sources, and parsed logs to the defined address.-
type = "vector": Specifies that this sink is of type "vector", indicating that it forwards data to another Vector instance. -
inputs = ["cassandra_metrics", "enrich_host_metrics", "add_source_to_systemlog", "gclog_parser"]: Specifies the input sources that are forwarded to thevector_aggregator. These sources represent different types of metrics or logs collected by Vector. -
address = "mission-control-aggregator.mission-control.svc:6000": Sets the address of thevector_aggregatorservice. In this example, themission-control-aggregatorservice in themission-controlnamespace listening on port 6000.
-
-
[sinks.console_log]: Defines theconsole_logsink.-
type = "console": Specifies that this sink is of type"console", indicating that it will output data to the console. -
inputs = ["systemlog"]: Specifies that thesystemloginput source will be forwarded to the console. -
target = "stdout": Sets the target output stream to the standard output (stdout).encoding.codec = "text": Specifies that the output data must be encoded in text format.
-
-
View metrics
Use the Mission Control UI to view metrics.
- Overview view
-
-
In the Mission Control UI, go to Home, and then select your target cluster’s project.
-
In the Overview tab, the Mission Control Overview view shows datacenter and node information.
-
- Node view
-
-
In the Mission Control UI, go to Home, and then select your target cluster’s project.
-
In the Overview tab, find the the Nodes section, and then click the name of the node that you want to inspect.
-
View the following details for the selected node:
-
Availability: The status is next to the node name
-
Type: HCD, DSE, or Cassandra
-
Storage Capacity: Typically measured in gigabytes (GB)
-
Load: The current read/write load on the node
-
Memory Usage: With System, Heap, and In Memory
-
Gossip: Activity status
-
Pending Tasks: Number of tasks pending execution
-
Native Clients: Number of native clients connected
-
Uptime: Number of days the node has been running
-
Running Tasks: With Type, SSTable, and Progress
-
Incoming Streams: With Operation, Peer, and Progress
-
Outgoing Streams: With Operation, Peer, and Progress
-
Thread Pool Stats: With Name, Active, Pending, Completed, Blocked, and Total Blocked
-
-
- Observability view
-
-
In the Mission Control UI, go to Home, and then select your target cluster’s project.
-
Click Observability, and then go to the Health Metrics tab.
You can filter and drill down into charts:
-
Hover over on any part of a chart to view more details.
-
Scrub along the horizontal time line to examine metrics at different points in time, or select a time period from the Time Period list.
-
In the Filter list, select the datacenter to monitor.
-
In the Frequency list, select the duration in which to refresh the metrics.
The following metrics are available:
-
Read/Write Throughput
-
Read/Write Latencies
-
Other Latencies
-
Errors
-
CPU Utilization
-
Unix Load
-
Garbage Collection Time
-
Disk Read Throughput
-
Disk Write Throughput
-
Network IO with Receive (RX) and Transmit (TX)
-