Collecting node health and indexing status scores

Steps to configure node health data collection, and commands to retrieve health and indexing scores.

Node health options are always enabled for all nodes. Node health is a score-based representation of how fit a node is to handle search queries.

The node health score is a composite score based on dropped mutations and uptime. A dynamic health score between 0 and 1 describes the health of the specified DataStax Enterprise node:

A higher score indicates better node health. 1 is the highest score.
A lower score applies to nodes that have a large number of dropped mutations and nodes that are just started.

On DSE Search nodes, the shard selection algorithm uses account proximity and secondary factors such as active and indexing statuses.

You can enable and configure node health data collection, and examine node health scores and indexing status. The indexing status is INDEXING, FINISHED, or FAILED.

Replication selection for distributed Solr queries can be configured to consider node health when multiple candidates exist for a particular token range. This health-based routing enables a trade-off between index consistency and query throughput. When the primary concern is performance, do not enable health-based routing.

The location of the dse.yaml file depends on the type of installation:

Installer-Services	/etc/dse/dse.yaml
Package installations	/etc/dse/dse.yaml
Installer-No Services	`install_location`/resources/dse/conf/dse.yaml
Tarball installations	`install_location`/resources/dse/conf/dse.yaml

Procedure

In the dse.yaml file:
1. Enable and customize node health options:
```
node_health_options:
    enabled: false
    refresh_rate_ms: 60000
    uptime_ramp_up_period_seconds: 86400
    dropped_mutation_window_minutes: 30
```
  - enabled
    Enable node health data collection. Default: false
  - ttl_seconds
    Default: 60000 seconds
  - uptime_ramp_up_period_seconds
    The amount of continuous uptime required for the node's uptime score to advance the node health score from 0 to 1 (full health), assuming there are no recent dropped mutations. The health score is a composite score based on dropped mutations and uptime. Tip: If a node is repairing after a period of downtime, you might want to increase the uptime period to the expected repair time. Default: 86400 (1 day)
  - dropped_mutation_window_minutes
    The historic time window over which the rate of dropped mutations affect the node health score. Default: 30
  Tip: If a node is repairing after a period of downtime, you might want to increase the uptime_ramp_up_period_seconds value to the expected repair time.
2. To enable replication selection for distributed Solr queries to consider node health, enable health-based routing:
```
enable_health_based_routing: true
```
  Health-based routing enables a trade-off between index consistency and query throughput. When the primary concern is performance, do not enable health-based routing.
To retrieve a dynamic health score between 0 and 1 that describes the specified DataStax Enterprise node, use the dsetool node_health command.
For example:
```
$ dsetool -h 200.192.10.11 node_health 
Node Health: 0.7
```
If you do not specify the IP address, the default is the local DataStax Enterprise node.