• Glossary
  • Support
  • Downloads
  • DataStax Home
Get Live Help
Expand All
Collapse All

DataStax Enterprise 6.8 Documentation

    • Overview
      • Release notes
        • DSE release notes
        • Cass Operator release notes
        • Studio release notes
        • Bulk loader release notes
        • Kafka Connector release notes
    • Architecture
      • Architecture FAQ
      • Database architecture
        • Architecture in brief
        • Internode communications (gossip)
        • Data distribution and replication
          • Data distribution to nodes
          • Consistent hashing
          • Virtual nodes
          • Data replication
          • Partitioners
          • Snitches
            • Dynamic snitching
            • Types of snitches
        • Node repair
          • NodeSync: Continuous background repair
          • Hinted handoff: repair during write path
          • Read Repair: repair during read path
          • Anti-entropy repair
      • Component architecture
        • DSE Analytics
        • DSE Search
        • DSE Graph
          • When to use DSE Graph
          • OLTP and OLAP
          • Comparing DSE Graph and relational databases
          • Migrating to DSE Graph from a relational database
          • Migrating to DSE Graph from Apache Cassandra
      • Database internals
        • Storage engine
        • About reads and writes
          • How is data written?
          • How is data maintained?
          • How is data updated?
          • How is data deleted?
          • What are tombstones?
          • How are indexes stored and updated?
          • How is data read?
          • How do write patterns affect reads?
        • Data consistency
          • Read and write consistency
          • Differences between DSE and RDBMS transactions
          • Using lightweight transactions
          • Consistency level performance
          • Consistency level configuration
          • Configuring serial consistency
          • Read requests
            • Read consistency levels examples
          • Write requests
            • Multiple datacenter write requests
    • Planning
      • Initializing a cluster
        • Initializing datacenters
          • Initializing a single datacenter per workload type
          • Initializing multiple datacenters per workload type
        • Setting seed nodes for a single datacenter
        • Use cases for listen address
      • Initializing single-token architecture datacenters
        • Calculating tokens for single-token architecture nodes
    • Getting started
    • Installing
      • Which install method should I use?
      • Install on a single node
      • Installing supporting software
      • Installing a cluster using Lifecycle Manager 6.8
      • Installing from the Yum package
      • RedHat systemd configuration
      • Installing from the Debian package
      • Install from the tarball on any Linux distribution
      • Installing patch releases
      • Installing on cloud providers
      • Installing on Docker
      • Uninstalling DSE
      • Default DSE file locations
        • Package installations
        • Tarball installations
      • Installing DSE 6.8 Tools
        • Installing CQLSH
        • Installing DataStax Studio 6.8
        • Installing DSE Graph Loader
        • Installing DataStax Bulk Loader
        • Installing DataStax Apache Kafka Connector
      • Installing DSE OpsCenter 6.8
        • Installing from the RPM package
        • Installing from the Debian package
        • Installing from the tarball on any Linux distribution
        • Installing on Docker
        • Uninstalling OpsCenter
        • Installing DataStax Agents 6.8
          • Installing DataStax Agents automatically
          • Installing DataStax Agents manually
            • From the RPM package
            • From the Debian package
            • From a tarball
          • Setting Agent permissions to run as the DSE user
          • Configuring JAVA_HOME
    • Managing
      • Configuration
        • Recommended production settings
        • YAML and configuration properties
          • cassandra.yaml
          • dse.yaml
          • remote.yaml
          • cassandra-rackdc.properties
          • cassandra-topology.properties
        • Cloud provider snitches
          • Amazon EC2 single-region snitch
          • Amazon EC2 multi-region snitch
          • Google Cloud Platform
          • Apache CloudStack snitch
        • JVM system properties
          • Cassandra
          • JMX
          • DSE Search
          • TPC
          • LDAP
          • Kerberos
          • NodeSync
          • DSE Metrics Collector
        • Choosing a compaction strategy
        • NodeSync service
          • About NodeSync
          • Starting and stopping the NodeSync service
          • Enabling NodeSync validation
          • Tuning NodeSync validations
            • Setting the NodeSync rate
            • Setting the NodeSync deadline
          • Manually starting NodeSync validation
        • Using multiple network interfaces
        • Configuring gossip settings
        • Configuring the heap dump directory
        • Configuring Virtual Nodes
          • Virtual node (vnode) configuration
          • Enabling virtual nodes on an existing production cluster
        • Logging configuration
          • Changing logging locations
          • Configuring logging
          • Commit log archive configuration
          • Change Data Capture (CDC) logging
      • Tools
        • DSE Metrics Collector
        • nodetool
          • Get information
            • clientstats
            • describecluster
            • describering
            • getbatchlogreplaythrottle
            • getcachecapacity
            • getcachekeystosave
            • getconcurrentviewbuilders
            • getendpoints
            • getinterdcstreamthroughput
            • getlogginglevels
            • getseeds
            • getsstables
            • getstreamthroughput
            • gettimeout
            • gettraceprobability
            • help
            • info
            • inmemorystatus
            • rangekeysample
            • ring
            • status
            • version
          • Collect metrics
            • gcstats
            • netstats
            • proxyhistograms
            • tablehistograms
            • tablestats
            • toppartitions
            • tpstats
          • Perform operations
            • assassinate
            • bootstrap resume
            • decommission
            • disablebinary
            • disablegossip
            • drain
            • enablebinary
            • enablegossip
            • gossipinfo
            • invalidatecountercache
            • invalidatekeycache
            • invalidaterowcache
            • import
            • join
            • move
            • refresh
            • reloadtriggers
            • relocatesstables
            • removenode
            • replaybatchlog
            • sequence
            • sjk
            • statusbinary
            • statusgossip
            • stopdaemon
            • upgradesstables
          • Adjust settings
            • reloadseeds
            • setbatchlogreplaythrottle
            • setcachecapacity
            • setcachekeystosave
            • setconcurrentviewbuilders
            • setinterdcstreamthroughput
            • setlogginglevel
            • setstreamthroughput
            • settimeout
            • settraceprobability
          • Diagnose issues
            • failuredetector
            • leaksdetection
          • Manage backup commands
            • clearsnapshot
            • disablebackup
            • enablebackup
            • listsnapshots
            • snapshot
            • statusbackup
          • Ensure data consistency
            • abortrebuild
            • cleanup
            • flush
            • mark_unrepaired
            • rebuild
            • rebuild_index
            • rebuild_view
            • resetlocalschema
            • repair
            • scrub
            • verify
          • Manage compaction
            • compact
            • compactionhistory
            • compactionstats
            • disableautocompaction
            • enableautocompaction
            • garbagecollect
            • getcompactionthreshold
            • getcompactionthroughput
            • getconcurrentcompactors
            • setcompactionthreshold
            • setcompactionthroughput
            • setconcurrentcompactors
            • stop
          • Manage NodeSync service
            • nodesyncservice enable
            • nodesyncservice disable
            • nodesyncservice getrate
            • nodesyncservice ratesimulator
            • nodesyncservice setrate
            • nodesyncservice status
          • Manage hints
            • disablehandoff
            • disablehintsfordc
            • enablehandoff
            • enablehintsfordc
            • gethintedhandoffthrottlekb
            • getmaxhintwindow
            • handoffwindow
            • listendpointspendinghints
            • pausehandoff
            • resumehandoff
            • sethintedhandoffthrottlekb
            • setmaxhintwindow
            • statushandoff
            • truncatehints
        • dse commands
          • dse connection options
          • Perform routine DSE operations
            • add-node
            • cassandra
            • cassandra-stop
            • list-nodes
            • remove-node
            • -v
          • Manage Spark
            • exec
            • pyspark
            • spark
            • spark-class
            • spark-jobserver
            • spark-history-server
            • spark-sql
            • spark-sql-thriftserver
            • spark-submit
            • SparkR
          • Connect to development consoles
            • beeline
            • fs
            • gremlin-console
          • Connect external client to DSE node
            • dse client-tool help
            • client-tool connection options
            • cassandra
            • configuration export
            • configuration byos-export
            • configuration import
            • spark
            • alwayson-sql
            • graph-olap
          • Modifies CQL nodesync
            • disable
            • enable
            • help
            • tracing
              • disable
              • enable
              • show
              • status
            • validation
      • dsefs shell commands
        • Get information
          • df
          • du
          • echo
          • ls
          • pwd
          • realpath
          • stat
        • Navigate DSEFS
          • cd
          • exit
        • Manage files
          • append
          • cat
          • cp
          • fsck
          • get
          • mkdir
          • mv
          • put
          • rename
          • rm
          • rmdir
          • truncate
          • umount
        • Manage permissions
          • chgrp
          • chmod
          • chown
      • dsetool
        • Connection options
        • Get information
          • help
          • inmemorystatus
          • list_subranges
          • listjt
          • node_health
          • partitioner
          • ring
          • status
          • tieredtablestats
        • Perform operations
          • infer_solr_schema
          • perf
          • sparkmaster cleanup
          • sparkworker restart
          • tsreload
        • Configure DSE Metrics Collector
          • insights_config
          • insights_filters
        • Manage security
          • createsystemkey
          • encryptconfigvalue
          • managekmip list
          • managekmip expirekey
          • managekmip revoke
          • managekmip destroy
        • Manage search index
          • core_indexing_status
          • create_core
          • get_core_config
          • get_core_schema
          • index_checks
          • list_index_files
          • list_core_properties
          • read_resource
          • rebuild_indexes
          • reload_core
          • set_core_property
          • stop_core_reindex
          • unload_core
          • upgrade_index_files
          • write_resource
      • SSTable tools
        • Get information
          • sstabledump
          • sstableexpiredblockers
          • sstablemetadata
          • sstablepartitions
          • sstableutil
        • Perform operations
          • sstabledowngrade
          • sstablelevelreset
          • sstableloader
          • sstableofflinerelevel
          • sstablesplit
          • sstableupgrade
        • Ensure data consistency
          • sstablerepairedset
          • sstablescrub
          • sstableverify
      • Preflight check tool
      • Compare yaml files
        • yaml_diff
        • cluster_check
      • Operations
        • Starting and stopping DSE
          • Starting as a service
          • Starting as a stand-alone process
          • Stopping a node
        • Adding or removing nodes, datacenters, or clusters
          • Adding nodes to vnode-enabled cluster
          • Adding a datacenter to a cluster using a designated datacenter as a data source
          • Replacing a dead node or dead seed node
          • Replacing a running node
            • Adding a node and then decommissioning the old node
            • Replacing a running node
          • Moving a node from one rack to another
          • Decommissioning a datacenter
          • Removing a node
          • Changing the IP address of a node
          • Switching snitches
          • Changing keyspace replication strategy
          • Migrating or renaming a cluster
          • Adding single-token nodes to a cluster
          • Adding a datacenter to a single-token architecture cluster
          • Replacing a dead node in a single-token architecture cluster
        • Backing up and restoring data using the DSE Backup and Restore Service
          • About the DSE Backup and Restore Service
          • Enabling and configuring the DSE Backup and Restore Service
          • Creating and managing backup stores
          • Creating and managing backup configurations
          • Managing backups
          • Restoring backups
          • Backup and Restore Service CQL command reference
            • ALTER BACKUP CONFIGURATION
            • ALTER BACKUP STORE
            • CANCEL BACKUP
            • CANCEL RESTORE
            • CLEAN BACKUPS
            • CREATE BACKUP CONFIGURATION
            • CREATE BACKUP STORE
            • DROP BACKUP CONFIGURATION
            • DROP BACKUP STORE
            • FORCE RESTORE
            • LIST BACKUP CONFIGURATIONS
            • LIST BACKUPS FROM KEYSPACE
            • LIST BACKUP STORES
            • RESTORE
            • RUN BACKUP
            • VERIFY BACKUP STORE
        • Backing up and restoring data using snapshots
          • About snapshots
          • Taking a snapshot
          • Deleting snapshot files
          • Enabling incremental snapshot backups
          • Restoring from a snapshot
          • Restoring a snapshot into a new cluster
          • Recovering from a single disk failure using JBOD
        • Repairing nodes
          • Manual repair: Anti-entropy repair
          • When to run anti-entropy repair
          • Changing repair strategies
            • Migrating to full repairs
            • Migrating to incremental repairs
        • Monitoring a DSE cluster
        • Tuning the database
          • Tuning Java Virtual Machine
            • Changing heap size parameters
            • Configuring the garbage collector
              • G1 MaxGCPauseMillis
              • CMS parameters
          • Tuning Bloom filters
          • Configuring memtable thresholds
        • Data caching
          • Configuring data caches
            • Enabling caching globally
            • Tips for efficient cache use
          • Monitoring and adjusting caching
        • Compacting and compressing
          • Configuring compaction
          • Compression
            • When to compress data
            • Configuring compression
          • Testing compaction and compression
        • Materialized views maintenance guidelines
        • Migrating data to DSE
        • Collecting node health and indexing scores
        • Clearing data from DSE
      • DSE Management Services
        • Performance Service
          • Performance Service
          • Configuring Performance Service replication strategy
          • Collecting data
            • Collecting slow queries
            • Collecting system level diagnostics
            • Collecting object I/O level diagnostics
            • Statistics gathered for objects
            • Collecting database summary diagnostics
            • Collecting cluster summary diagnostics
            • Collecting histogram diagnostics
            • Collecting user activity diagnostics
            • Statistics gathered for user activity
          • Collecting search data
            • Collecting slow search queries
            • Collecting Apache Solr performance statistics
            • Collecting cache statistics
            • Collecting index statistics
            • Collecting handler statistics
            • Collecting request handler metrics
          • Monitoring Spark with Spark Performance Objects
          • Diagnostic table reference
          • Solr diagnostic table reference
            • Frequently asked questions
            • Slow sub-query log for search
            • Indexing error log
            • Query latency snapshot
            • Update latency snapshot
            • Commit latency snapshot
            • Merge latency snapshot
            • Filter cache statistics
            • Query result cache statistics
            • Index statistics
            • Update handler statistics
            • Update request handler statistics
            • Search request handler statistics
      • DSE In-Memory
        • Creating or altering tables to use DSE In-Memory
        • Verifying table properties
        • Managing memory
        • Backing up and restoring data
      • DSE Tiered Storage
        • About DSE Tiered Storage
        • Configuring DSE Tiered Storage
        • Testing configurations
      • DSE Multi-Instance
        • About DSE Multi-Instance
        • DSE Multi-Instance architecture
        • Adding nodes to DSE Multi-Instance
        • DSE Multi-Instance commands
    • Securing
      • Security FAQ
      • Security checklists
      • Securing the environment
        • Securing ports
        • Securing the TMP directory
      • Authentication and authorization
        • Configuring authentication and authorization
          • About DSE Unified Authentication
            • Steps for new deployment
            • Steps for production environments
          • Configuring security keyspaces
          • Setting up Kerberos
            • Kerberos guidelines
            • Enabling JCE Unlimited
              • Removing AES-256
            • Preparing DSE nodes for Kerberos
              • DNS and NTP
              • krb5.conf
              • Principal
              • Keytab
          • Enabling authentication and authorization
            • Defining a Kerberos scheme
            • Defining an LDAP scheme
          • Configuring JMX authentication
          • Configuring cache settings
          • Securing schema information
        • Managing database access
          • About RBAC
          • Setting up logins and users
            • Adding a superuser login
            • Adding database users
            • LDAP users and groups
              • LDAP logins
              • LDAP groups
            • Kerberos principal logins
            • Setting up roles for applications
            • Binding a role to an authentication scheme
          • Assigning permissions
            • Database object permissions
              • Data resources
              • Functions and aggregate resources
              • Search indexes
              • Roles
              • Proxy login and execute
              • Authentication schemes
              • DSE Utilities (MBeans)
              • Analytic applications
              • Remote procedure calls
            • Separation of duties
            • Keyspaces and tables
            • Row Level Access Control (RLAC)
            • Search index permissions
            • DataStax Graph keyspace
            • Spark application permissions
            • DataStax Studio permissions
            • Remote procedure calls
            • DSE client-tool spark
            • JMX MBean permissions
            • Deny (denylist) db object permission
            • Restricting access to data
        • Providing credentials from DSE tools
          • About clients
          • Internal and LDAP authentication
            • Command line
            • File
            • Environment variables
            • Using CQLSH
          • Kerberos
            • JAAS configuration file location
            • Keytab
            • Ticket Cache
            • Spark jobs
            • SSTableLoader
            • Graph and gremlin-console
            • dsetool
            • CQLSH
          • Nodetool
          • JConsole
      • Auditing database activity
        • Enabling database auditing
        • Capturing DSE Search HTTP requests
        • Log formats
        • View events from DSE audit table
      • Transparent data encryption
        • About Transparent Data Encryption
        • Configuring local encryption
          • Setting up local encryption keys
          • Encrypting configuration file properties
          • Encrypting system resources
          • Encrypting tables
          • Rekeying existing data
          • Using tools with TDE-encrypted SSTables
          • Troubleshooting encryption key errors
        • Configuring KMIP encryption
        • Encrypting Search indexes
          • Encrypting new Search indexes
          • Encrypting existing Search indexes
          • Tuning encrypted Search indexes
        • Migrating encrypted tables from earlier versions
        • Bulk loading data between TDE-enabled clusters
      • Configuring SSL
        • Steps for configuring SSL
        • Creating SSL certificates, keystores, and truststores
          • Remote keystore provider
          • Local keystore files
        • Securing node-to-node connections
        • Securing client-to-node connections
          • Configuring JMX on the server side
          • nodetool, nodesync, dsetool, and Advanced Replication
          • JConsole (JMX)
          • SSTableloader
          • Connecting to SSL-enabled nodes using cqlsh
        • Enabling SSL encryption for DSEFS
        • Reference: SSL instruction variables
      • Securing Spark connections
    • Tooling Resources
      • Stress tools
        • cassandra-stress tool
          • About the cassandra-stress tool
          • Interpret output
          • counter_read
          • counter_write
          • help
          • legacy
          • mixed
          • print
          • read
          • user
          • version
          • write
      • fs-stress tool
      • OpsCenter services
        • Best Practice Service
        • Capacity Service
        • Repair Service
    • DSE Advanced Replication
      • About DSE Advanced Replication
      • Architecture
      • Traffic between the clusters
      • Terminology
      • Getting started
      • Keyspaces
      • Data types
      • Operations
      • CQL queries
      • Metrics
      • Managing invalid messages
      • Managing audit logs
      • Command line tool
        • connection options
        • channel create
        • channel update
        • channel delete
        • channel pause
        • channel resume
        • channel status
        • channel truncate
        • conf list
        • conf remove
        • conf update
        • destination create
        • destination update
        • destination delete
        • destination list
        • destination list-conf
        • destination remove-conf
        • help
        • metrics list
        • replog count
        • replog analyze-audit-log
    • DSE Analytics
      • Setting the replication factor for analytics keyspaces
      • DSE Analytics and Search integration
        • Using predicate push down on search indexes in Spark SQL
      • About DSE Analytics Solo
      • DSEFS (DataStax Enterprise file system)
        • About DSEFS
        • Enabling DSEFS
        • Disabling DSEFS
        • Configuring DSEFS
        • Commands DSEFS
        • DSEFS compression
        • DSEFS authentication
        • DSEFS authorization
        • Using the DSEFS REST interface
        • Programmatic access to DSEFS
        • Hadoop FileSystem interface implemented by DseFileSystem
        • Using JMX to read DSEFS metrics
    • DSE Graph
      • About Graph
      • What’s new
      • Graph QuickStart
      • CQL as Graph
      • Convert CQL to Graph
      • Graph OLTP and OLAP
      • Graph data modeling
        • Data modeling introduction
        • Basic data modeling
        • Data modeling design
        • Advanced data modeling
      • Manage graph
        • Create a graph
        • Exanube a Graph
        • Drop a Graph
      • Manage schema
        • Create a Graph schema
        • Examine a schema
        • Create UDT schema
        • Create collection and tuple schema
        • Create vertex lable schema
        • Create edge lable schema
        • Indexing
        • Create index schema
        • Drop Graph schema
        • Vertex and edge IDs
      • Manage Graph data
        • Data formats
        • Insert data with Graph traversal API
        • DataStax Bulk Loader for Graph
          • Install DataStax Bulk Loader
          • DataStax Bulk Loader Examples
        • Load data with DseGraphFrames
        • Drop graph data
      • Discovering properties
      • Creating queries using traversals
        • Anatomy of a graph traversal
        • Use indexes
        • Use search indexes
        • Simple traversals
        • Geospatial traversals
        • Branching traversals
        • Recursive traversals
        • Path traversals
      • Graph analysis with DSE Analytics
        • DseGraphFrame overview
          • TinkerPop API support in DseGraphFrame
          • Mapping rules for DseGraphFrame
          • DseGraphFrame API reference
        • Export graphs to DSEFS
        • Import graphs
        • Northwind demo graph with Spark OLAP jobs
      • DSE Graph Operations
        • Configuring DSE Graph
          • Specifying DSE database and graph settings
          • Configuring security
        • Graph backup and restore
        • Graph import/export
        • Graph JMX metrics
      • Graph tools
      • Start Gremlin console
      • Graph Reference
        • Graph traversal API
          • addE
          • addV
          • io
          • property
          • with
        • Schema API
          • drop
          • describe
          • edgeLabel
          • type
          • vertexLabel
        • System API
          • Graph
          • GraphClassic
            • config
            • option
          • graphs
          • list
        • TinkerPop traversal API
          • TinkerPop framework
          • TinkerPop general information
          • TinkerPop predicates
            • eq
            • neq
            • lt
            • lte
            • gt
            • gte
            • inside
            • outside
            • between
            • within
            • without
            • Step-modulators
            • as
            • by
            • emit
            • from
            • option
            • times
            • to
            • until
            • Vertex step
            • out
            • in
            • both
            • outE
            • inE
            • bothE
            • outV
            • inV
            • bothV
            • otherV
            • addV
            • addE
            • property
            • mid-traversal V()
            • aggregate
            • and
            • barrier
            • branch
            • cap
            • choose
            • coalesce
            • constant
            • count
            • cyclicPath
            • dedup
            • drop
            • explain
          • fill
          • filter
          • flatMap
          • fold
          • group
          • groupCount
          • has
          • hasId
          • hasKey
          • hasLabel
          • hasNext
          • hasNot
          • hasValue
          • id
          • inject
          • is
          • key
          • label
          • limit
          • local
          • loops
          • map
          • match
          • math
          • max
          • mean
          • min
          • next
          • not
          • optional
          • or
          • order
          • pageRank
          • path
          • peerPressure
          • profile
          • project
          • properties
          • propertyMap
          • range
          • repeat
          • sack
          • sample
          • select
          • sideEffect
          • simplePath
          • skip
          • store
          • subGraph
          • sum
          • tail
          • timeLimit
          • toBulkSet
          • toList
          • toSet
          • tree
          • unfold
          • union
          • value
          • valueMap
          • values
          • where
        • DataStax Graph data types
        • Graph storage in Cassandra keyspace and table
    • DSE Search
      • About Search
        • Solr OSS differences
        • Unsupported search features
        • Solr Lucene limitations
      • Configuring Search
      • Search Reference
      • Search index configuration
      • Search index schema
      • Search config.yaml options
      • Adding/viewing index resources
      • Initial data migration
      • Shard routing for distributed queries
      • Deleting Solr data
      • Verifying index status
      • Backing up search indexes
      • Restoring a search node
      • Metrics (MBEANS)
      • Uploading custom index resources
      • Solr admin UI configuration
      • Configuring Solr connector port
      • reqPerm Solr admin UI
      • Changing Tomcat settings
      • Configuring Solr library path
      • Using the Solr HTTP API
      • Configuring HTTP for AJP
      • aboutUpdateRequestProcessorAndFieldTransformer
      • Field Input/Output Transformer (FIT) API
      • FIT class examples
      • Custom URP example
      • Interface custom field types
      • Deleting by query - best practice
      • Monitoring segments
      • Solr clients
      • Tutorials and demos
        • Creating a healthcare keyspace for tutorials
        • Multi-faceted search using healthcare data
        • Term and phrase searches using the wikipedia demo
        • Using secure cluster
        • Indexing and querying polygons
    • DSE Spark
      • About Spark
      • Using Spark with DataStax Enterprise
        • Starting Spark
        • Running Spark commands against a remote cluster
        • Accessing database data from Spark
          • Using the Spark session
          • Using the Spark context
          • Controlling automatic direct join optimizations in queries
          • Accessing the Spark session and context for applications running outside of DSE Analytics
          • Saving RDD data to DSE
          • Spark supported types
          • Loading external HDFS data into the database using Spark
        • Monitoring Spark with the web interface
        • Getting started with the Spark Cassandra Connector
        • Using DSE Spark with third party tools and integrations
      • Configuring Spark nodes
        • Automatic Spark Master election
        • Configuring Spark logging options
        • Running Spark processes as separate users
        • Configuring the Spark history server
        • Setting Spark Cassandra Connector-specific properties
        • Creating a DSE Analytics Solo datacenter
        • Spark JVMs and memory management
      • Using Spark modules with DataStax Enterprise
        • Getting started with Spark Streaming
          • Creating a Spark Structured Streaming sink using DSE
        • Using Spark SQL to query data
          • Querying database data using Spark SQL in Scala
          • Querying database data using Spark SQL in Java
          • Querying DSE Graph vertices and edges with Spark SQL
          • Using Spark predicate push down in Spark SQL queries
          • Supported syntax of Spark SQL
          • Inserting data into tables with static columns using Spark SQL
          • Running HiveQL queries using Spark SQL
          • Using the DataFrames API
          • Using the Spark SQL Thriftserver
        • Using SparkR with DataStax Enterprise
      • Using AlwaysOn SQL service
        • Enabling SSL for AlwaysOn SQL
        • Using authentication with AlwaysOn SQL
        • Simba JDBC Driver for Apache Spark
        • Simba ODBC Driver for Apache Spark
        • Connecting to AlwaysOn SQL server using Beeline
      • Accessing DataStax Enterprise data from external Spark clusters
        • Overview of BYOS support in DataStax Enterprise
        • Generating the BYOS configuration file
        • Connecting to DataStax Enterprise using the Spark shell on an external Spark cluster
        • Generating Spark SQL schema files
        • Starting Spark SQL Thrift Server with Kerberos
      • Using the Spark Jobserver
      • Spark examples
        • Portfolio Manager demo using Spark
        • Running the Weather Sensor demo
        • Running the Wikipedia demo with SearchAnalytics
        • Running the Spark MLlib demo application
        • Running the http_receiver demo
        • Using DSE geometric types in Spark
        • Importing a text file into a table
        • Running spark-submit job with internal authentication
      • DSE Spark Connector API documentation
  • DataStax Enterprise 6.8 Documentation
  • DSE Spark
  • Configuring Spark nodes

Configuring Spark nodes

Modify the settings for Spark nodes security, performance, and logging.

To manage Spark performance and operations:

  • Set the replication factor for DSE Analytics keyspaces

  • Set environment variables

  • Protect Spark directories

  • Grant access to default Spark directories

  • Secure Spark nodes

  • Configure Spark memory and cores

  • Configure Spark logging options

Set environment variables

DataStax recommends using the default values of Spark environment variables unless you need to increase the memory settings due to an OutOfMemoryError condition or garbage collection taking too long. Use the Spark memory configuration options in the dse.yaml and spark-env.sh files.

You can set a user-specific SPARK_HOME directory if you also set ALLOW_SPARK_HOME=true in your environment before starting DSE.

For example, on Debian or Ubuntu using a package installation:

export SPARK_HOME=$HOME/spark &&
export ALLOW_SPARK_HOME=true &&
sudo service dse start

The temporary directory for shuffle data, RDDs, and other ephemeral Spark data can be configured for both the locally running driver and for the Spark server processes managed by DSE (Spark Master, Workers, shuffle service, executor and driver running in cluster mode).

For the locally running Spark driver, the SPARK_LOCAL_DIRS environment variable can be customized in the user environment or in spark-env.sh. By default, it is set to the system temporary directory. For example, on Ubuntu it is /tmp/. If there’s no system temporary directory, then SPARK_LOCAL_DIRS is set to a .spark directory in the user’s home directory.

For all other Spark server processes, the SPARK_EXECUTOR_DIRS environment variable can be customized in the user environment or in spark-env.sh. By default it is set to /var/lib/spark/rdd.

The default SPARK_LOCAL_DIRS and SPARK_EXECUTOR_DIRS environment variable values differ from non-DSE Spark.

To configure worker cleanup, modify the SPARK_WORKER_OPTS environment variable and add the cleanup properties. The SPARK_WORKER_OPTS environment variable can be set in the user environment or in spark-env.sh. For example, the following enables worker cleanup,.sets the cleanup interval to 30 minutes (i.e. 1800 seconds), and sets the length of time application worker directories will be retained to 7 days (i.e. 604800 seconds).

export SPARK_WORKER_OPTS="$SPARK_WORKER_OPTS \
  -Dspark.worker.cleanup.enabled=true \
  -Dspark.worker.cleanup.interval=1800 \
  -Dspark.worker.cleanup.appDataTtl=604800"

Protect Spark directories

After you start up a Spark cluster, DataStax Enterprise creates a Spark work directory for each Spark Worker on worker nodes. A worker node can have more than one worker, configured by the SPARK_WORKER_INSTANCES option in spark-env.sh. If SPARK_WORKER_INSTANCES is undefined, a single worker is started. The work directory contains the standard output and standard error of executors and other application specific data stored by Spark Worker and executors; the directory is writable only by the DSE user.

By default, the Spark parent work directory is located in /var/lib/spark/work, with each worker in a subdirectory named worker-<number>, where the number starts at 0. To change the parent worker directory, configure SPARK_WORKER_DIR in the spark-env.sh file.

The Spark RDD directory is the directory where RDDs are placed when executors decide to spill them to disk. This directory might contain the data from the database or the results of running Spark applications. If the data in the directory is confidential, prevent access by unauthorized users. The RDD directory might contain a significant amount of data, so configure its location on a fast disk. The directory is writable only by the cassandra user. The default location of the Spark RDD directory is /var/lib/spark/rdd. The directory should be located on a fast disk. To change the RDD directory, configure SPARK_EXECUTOR_DIRS in the spark-env.sh file.

Grant access to default Spark directories

Before starting up nodes on a tarball installation, you need permission to access the default Spark directory locations: /var/lib/spark and /var/log/spark. Change ownership of these directories as follows:

sudo mkdir -p /var/lib/spark/rdd; sudo chmod a+w /var/lib/spark/rdd; sudo chown -R  $USER:$GROUP /var/lib/spark/rdd &&
sudo mkdir -p /var/log/spark; sudo chown -R  $USER:$GROUP /var/log/spark

In multiple datacenter clusters, use a virtual datacenter to isolate Spark jobs. Running Spark jobs consume resources that can affect latency and throughput.

DataStax Enterprise supports the use of virtual nodes (vnodes) with Spark.

Secure Spark nodes

Client-to-node SSL

Ensure that the truststore entries in cassandra.yaml are present as described in Client-to-node encryption, even when client authentication is not enabled.

Enabling security and authentication

Security is enabled using the spark_security_enabled option in dse.yaml. Setting it to enabled turns on authentication between the Spark Master and Worker nodes, and allows you to enable encryption. To encrypt Spark connections for all components except the web UI, enable spark_security_encryption_enabled. The length of the shared secret used to secure Spark components is set using the spark_shared_secret_bit_length option, with a default value of 256 bits. These options are described in DSE Analytics. For production clusters, enable these authentication and encryption. Doing so does not significantly affect performance.

Authentication and Spark applications

If authentication is enabled, users need to be authenticated in order to submit an application.

Authorization and Spark applications

If DSE authorization is enabled, users needs permission to submit an application. Additionally, the user submitting the application automatically receives permission to manage the application, which can optionally be extended to other users.

Database credentials for the Spark SQL Thrift server

In the hive-site.xml file, configure authentication credentials for the Spark SQL Thrift server. Ensure that you use the hive-site.xml file in the Spark directory:

  • Package installations: /etc/dse/spark/hive-site.xml

  • Tarball installations: <installation_location>/resources/spark/conf/hive-site.xml

Kerberos with Spark

With Kerberos authentication, the Spark launcher connects to DSE with Kerberos credentials and requests DSE to generate a delegation token. The Spark driver and executors use the delegation token to connect to the cluster. For valid authentication, the delegation token must be renewed periodically. For security reasons, the user who is authenticated with the token should not be able to renew it. Therefore, delegation tokens have two associated users: token owner and token renewer.

The token renewer is none so that only a DSE internal process can renew it. When the application is submitted, DSE automatically renews delegation tokens that are associated with Spark application. When the application is unregistered (finished), the delegation token renewal is stopped and the token is cancelled.

Set Kerberos options, see Defining a Kerberos scheme.

Configure Spark memory and cores

Spark memory options affect different components of the Spark ecosystem:

Spark History server and the Spark Thrift server memory

The SPARK_DAEMON_MEMORY option configures the memory that is used by the Spark SQL Thrift server and history-server. Add or change this setting in the spark-env.sh file on nodes that run these server applications.

Spark Worker memory

The memory_total option in resource_manager_options.worker_options section of dse.yaml configures the total system memory that you can assign to all executors that are run by the work pools on the particular node. The default work pool will use all of this memory if no other work pools are defined. If you define additional work pools, you can set the total amount of memory by setting the memory option in the work pool definition.

Application executor memory

You can configure the amount of memory that each executor can consume for the application. Spark uses a 512MB default. Use either the spark.executor.memory option, described in "Spark Available Properties", or the --executor-memory <mem> argument to the dse spark command.

Application memory

You can configure additional Java options that are applied by the worker when spawning an executor for the application. Use the spark.executor.extraJavaOptions property, described in Spark 1.6.2 Available Properties. For example: spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

Core management

You can manage the number of cores by configuring these options.

  • Spark Worker cores

    The cores_total option in the resource_manager_options.worker_options section of dse.yaml configures the total number of system cores available to Spark Workers for executors. If no work pools are defined in the resource_manager_options.workpools section of dse.yaml the default work pool will use all the cores defined by cores_total. If additional work pools are defined, the default work pool will use the cores available after allocating the cores defined by the work pools.

    A single executor can borrow more than one core from the worker. The number of cores used by the executor relates to the number of parallel tasks the executor might perform. The number of cores offered by the cluster is the sum of cores offered by all the workers in the cluster.

  • Application cores

    In the Spark configuration object of your application, you configure the number of application cores that the application requests from the cluster using either the spark.cores.max configuration property or the --total-executor-cores <cores> argument to the dse spark command.

See the Spark documentation for details about memory and core allocation.

DataStax Enterprise can control the memory and cores offered by particular Spark Workers in semi-automatic fashion. The resource_manager_options.worker_options section in the dse.yaml file has options to configure the proportion of system resources that are made available to Spark Workers and any defined work pools, or explicit resource settings. When specifying decimal values of system resources the available resources are calculated in the following way:

  • Spark Worker memory = memory_total * (total system memory - memory assigned to DSE)

  • Spark Worker cores = cores_total * total system cores

This calculation is used for any decimal values. If the setting is not specified, the default value 0.7 is used. If the value does not contain a decimal place, the setting is the explicit number of cores or amount of memory reserved by DSE for Spark.

Setting cores_total or a workpool’s cores to 1.0 is a decimal value, meaning 100% of the available cores will be reserved. Setting cores_total or cores to 1 (no decimal point) is an explicit value, and one core will be reserved.

The lowest values you can assign to a named work pool’s memory and cores are 64 MB and 1 core, respectively. If the results are lower, no exception is thrown and the values are automatically limited.

The following example shows a work pool named workpool1 with 1 core and 512 MB of RAM assigned to it. The remaining resources calculated from the values in worker_options are assigned to the default work pool.

resource_manager_options:
    worker_options:
        cores_total: 0.7
        memory_total: 0.7

        workpools:
              - name: workpool1
              cores: 1
              memory: 512M

Running Spark clusters in cloud environments

If you are using a cloud infrastructure provider like Amazon EC2, you must explicitly open the ports for publicly routable IP addresses in your cluster. If you do not, the Spark workers will not be able to find the Spark Master.

One work-around is to set the prefer_local setting in your cassandra-rackdc.properties snitch setup file to true:

# Uncomment the following line to make this snitch prefer the internal ip when possible, as the Ec2MultiRegionSnitch does.
prefer_local=true

This tells the cluster to communicate only on private IP addresses within the datacenter rather than the public routable IP addresses.

Configuring the number of retries to retrieve Spark configuration

When Spark fetches configuration settings from DSE, it will not fail immediately if it cannot retrieve the configuration data, but will retry 5 times by default, with increasing delay between retries. The number of retries can be set in the Spark configuration, by modifying the spark.dse.configuration.fetch.retries configuration property when calling the dse spark command, or in spark-defaults.conf.

Disabling continuous paging

Continuous paging streams bulk amounts of records from DSE to the DataStax Java Driver used by DSE Spark. By default, continuous paging in queries is enabled. To disable it, set the spark.dse.continuous_paging_enabled setting to false when starting the Spark SQL shell or in spark-defaults.conf. For example:

dse spark-sql --conf spark.dse.continuous_paging_enabled=false

Using continuous paging can potentially improve performance up to 3 times, though the improvement will depend on the data and the queries. Some factors that impact the performance improvement are the number of executor JVMs per node and the number of columns included in the query. Greater performance gains were observed with fewer executor JVMs per node and more columns selected.

Configuring the Spark web interface ports

By default the Spark web UI runs on port 7080. To change the port number, do the following:

  1. Open the spark-env.sh file in a text editor.

  2. Set the SPARK_MASTER_WEBUI_PORT variable to the new port number. For example, to set it to port 7082:

    export SPARK_MASTER_WEBUI_PORT=7082
  3. Repeat these steps for each Analytics node in your cluster.

  4. Restart the nodes in the cluster.

Enabling Graphite Metrics in DSE Spark

Users can add third party JARs to Spark nodes by adding them to the Spark lib directory on each node and restart the cluster. Add the Graphite Metrics JARs to this directory to enable metrics in DSE Spark.

The default location of the Spark lib directory depends on the type of installation:

  • Package installations: /usr/share/dse/spark/lib

  • Tarball installations: /var/lib/spark

To add the Graphite JARs to Spark in a package installation, copy them to the Spark lib directory:

cp metrics-graphite-3.1.2.jar /usr/share/dse/spark/lib/ &&
cp metrics-json-3.1.2.jar /usr/share/dse/spark/lib/

Setting Spark properties for the driver and executor

Additional Spark properties for the Spark driver and executors are set in spark-defaults.conf. For example, to enable Spark’s commons-crypto encryption library:

spark.network.crypto.enabled true

Using authorization with Spark

See Analytic applications and Setting up DSE Spark application permissions.

Spark server configuration

The spark-daemon-defaults.conf file configures DSE Spark Masters and Workers.

Table 1. Spark server configuration properties
Option Default value Description

dse.spark.application.timeout

30

The duration in seconds after which the application will be considered dead if no heartbeat is received.

spark.dseShuffle.sasl.port

7447

The port number on which a shuffle service for SASL secured applications is started. Bound to the listen_address in cassandra.yaml.

spark.dseShuffle.noSasl.port

7437

The port number on which a shuffle service for unsecured applications is started. Bound to the listen_address in cassandra.yaml.

By default Spark executor logs, which log the majority of your Spark Application output, are redirected to standard output. The output is managed by Spark Workers. Configure logging by adding spark.executor.logs.rolling.* properties to spark-daemon-defaults.conf file.

spark.executor.logs.rolling.maxRetainedFiles 3
spark.executor.logs.rolling.strategy size
spark.executor.logs.rolling.maxSize 50000

Additional Spark properties that affect the master and driver can be added to spark-daemon-defaults.conf. For example, to enable Spark’s commons-crypto encryption library:

spark.network.crypto.enabled true
Using DSE Spark with third party tools and integrations Automatic Spark Master election

General Inquiries: +1 (650) 389-6000 info@datastax.com

© DataStax | Privacy policy | Terms of use

DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Kubernetes is the registered trademark of the Linux Foundation.

landing_page landingpage