DSE Search initial data migration

Best practices and guidelines for loading data into DSE Search.

When you initially load data into DataStax Enterprise (DSE) resource contention requires planning to ensure performance.

These two activities compete for resources, so proper resource allocation is critical to maximize efficiency for initial data load.

Recommendations

For maximum throughput, store the search index data and DataStax Enterprise (Cassandra) data on separate physical disks.
If you are unable to use separate disks, DataStax recommends that SSDs have a minimum of 500 MB/s read/write speeds (bandwidth).
Enable OpsCenter 6.1 repair service.

Also see memory recommendations in the planning guide.

DataStax recommends following this high-level procedure:

Install DSE and configure nodes for search workloads.
Use the CQL command to create search indexes.
Tune the index for maximum indexing throughput.
Load data into the database using best practices for data loading. For example, load data with the driver with the consistency level at LOCAL_ONE (CL.LOCAL_ONE) and a sufficiently high write timeout.
After data loading is completed, there might be lag time because indexing is asynchronous.
Verify the indexing QueueSizeQueueSize with the IndexPool MBean. After the index queue size has receded, run this CQL query to verify that the number of records is as expected:
```
SELECT count(*) FROM ks.table WHERE solr_query = '*:*';
```

New data is automatically indexed.

If the record count does not stabilize:

If dropped mutations exist in the nodetool tpstats output for some nodes, and OpsCenter repair service is not enabled, run manual repair on those nodes.
If dropped mutations do not exist, check the system.log and the Solr validation logfor indexing errors.