DSE Search initial data migration
All writes to DataStax Enterprise (DSE) are dependent on available system resources, such as CPU and memory.
DSE Search is an advanced workload that introduces additional resource contention.
Make sure your DSE cluster has enough resources to support both performant writes to your databases and Apache Solr™ search index creation.
This is especially critical during initial data loading for DSE Search workloads, which requires a large volume of writes and creation of all necessary indexes.
These two activities compete for resources, so proper resource allocation is critical to maximize efficiency for initial data load.
Recommendations
-
For maximum throughput, store the search index data and DSE (Apache Cassandra) data on separate physical disks. If you are unable to use separate disks, DataStax recommends high-performance SSDs.
-
Required CPU and memory are directly related to the number of indexes you plan to create.
For more information, see Capacity planning and hardware selection for DSE deployments.
-
If using DSE OpsCenter, enable the Repair Service.
Initial bulk loading
DataStax recommends following this high-level procedure:
-
Install DSE and configure nodes for search workloads.
-
Use the CQL CREATE SEARCH INDEX command to create search indexes.
-
Tune the index for maximum indexing throughput.
-
Load data into the database using best practices for data loading. For example, load data with the driver with the consistency level at LOCAL_ONE (CL.LOCAL_ONE) and a sufficiently high write timeout.
After data loading is completed, there might be lag time because indexing is asynchronous.
-
Verify the indexing
QueueSizewith theIndexPoolMBean. After the index queue size has receded, run this CQL query to verify that the number of records is as expected:SELECT count(*) FROM ks.table WHERE solr_query = '*:*';
New data is automatically indexed.
Troubleshooting
If the record count does not stabilize:
-
If dropped mutations exist in the nodetool tpstats output for some nodes, and DSE OpsCenter Repair Service is not enabled, run manual repair on those nodes.
-
If dropped mutations do not exist, check the
system.logand the Solr validation log for indexing errors.
