Collecting node health and indexing status scores

Steps to configure node health options, and commands to retrieve health and indexing scores.

Node health options are always enabled for all nodes. Node health is a score-based representation of how fit a node is to handle search queries. The node health composite score is based on dropped mutations and uptime. A dynamic health score between 0 and 1 describes the health of the specified DataStax Enterprise node:

A higher score indicates better node health. The highest score is 1.
A lower score applies to nodes that have a large number of dropped mutations and nodes that are just started.

On DSE Search nodes, the shard selection algorithm uses account proximity and secondary factors such as active and indexing statuses. You can examine node health scores and indexing status. The indexing status is INDEXING, FINISHED, or FAILED.

Replication selection for distributed search queries can be configured to consider node health when multiple candidates exist for a particular token range. This health-based routing enables a trade-off between index consistency and query throughput. When the primary concern is performance, do not enable health-based routing.

dse.yaml

The location of the dse.yaml file depends on the type of installation:

Package installations	/etc/dse/dse.yaml
Tarball installations	`installation_location`/resources/dse/conf/dse.yaml

Procedure

In the dse.yaml file:
1. Customize node health options to increase the node health score from 0 to 1 (full health):
```
node_health_options:
    refresh_rate_ms: 50000    
    uptime_ramp_up_period_seconds: 10800    
    dropped_mutation_window_minutes: 30
```
  node_health_options
  
  Node health options are always enabled.
  
  Tip: If a node is repairing after a period of downtime, try increasing the uptime_ramp_up_period_seconds value to the expected repair time.
2. To enable replication selection for distributed search queries to consider node health, enable health-based routing:
```
enable_health_based_routing: true
```
  Health-based routing enables a trade-off between index consistency and query throughput. When the primary concern is performance, do not enable health-based routing.
To retrieve a dynamic health score between 0 and 1 that describes the specified DataStax Enterprise node, use the dsetool node_health command.
For example:
```
dsetool -h 200.192.10.11 node_health 
```
```
Node Health [0,1]: 0.7
```
If you do not specify the IP address, the default is the local DataStax Enterprise node.
Specify dsetool node_health -all to retrieve the node health scores for all nodes.

You can also see node health scores with dsetool status.
To retrieve the dynamic indexing status (INDEXING, FINISHED, or FAILED) of the specified core on a node, use the dsetool core_indexing_status command.
For example:
```
dsetool -h 200.192.10.11 core_indexing_status wiki.solr
```
```
wiki.solr: INDEXING
```