DSE Search initial data migration
Best practices and guidelines for loading data into DSE Search.
When you initially load data into DataStax Enterprise (DSE) resource contention requires planning to ensure performance.
-
DSE is performant when writing data.
-
Apache Solr™ is resource intensive when creating a search index.
These two activities compete for resources, so proper resource allocation is critical to maximize efficiency for initial data load.
Recommendations
-
For maximum throughput, store the search index data and DataStax Enterprise (Cassandra) data on separate physical disks.
If you are unable to use separate disks, DataStax recommends that SSDs have a minimum of 500 MB/s read/write speeds (bandwidth).
-
Enable OpsCenter 6.1 repair service.
Initial bulk loading
DataStax recommends following this high-level procedure:
-
Install DSE and configure nodes for search workloads.
-
Use the CQL
CREATE SEARCH INDEXcommand to create search indexes. -
Tune the index for maximum indexing throughput.
-
Load data into the database with the index in place. For example, load data with the driver with the consistency level at
LOCAL_ONE(CL.LOCAL_ONE) and a sufficiently high write timeout. Use best practices for data loading.Use the DataStax Bulk Loader.
After data loading is completed, there might be lag time because indexing is asynchronous.
-
Verify the indexing
QueueSizewith theIndexPool MBean. After the index queue size has receded, run this CQL query to verify that the number of records is as expected:SELECT count(*) FROM ks.table WHERE solr_query = '*:*';
|
The |
New data is automatically indexed.
Troubleshooting
If the record count does not stabilize:
-
If dropped mutations exist in the
nodetool tpstatsoutput for some nodes, and OpsCenter repair service is not enabled, runmanual repairon those nodes. -
If dropped mutations do not exist, check the
system.logand the Solr validation logfor indexing errors.
