dse client-tool spark

Connect an external client to a DataStax Enterprise node and perform operations related to integrated Spark.

Perform operations related to integrated Spark.


dse client-tool connection_options spark 
(master-address | leader-address | version | 
sql-schema (--exclude | --keyspace | --table | --decimal | --all)
metastore-migrate --from_version --to_version)
Table 1. Legend
Syntax conventions Description
UPPERCASE Literal keyword.
Lowercase Not literal.
Italics Variable value. Replace with a valid option or user-defined value.
[ ] Optional. Square brackets ( [ ] ) surround optional command arguments. Do not type the square brackets.
( ) Group. Parentheses ( ( ) ) identify a group to choose from. Do not type the parentheses.
| Or. A vertical bar ( | ) separates alternative elements. Type any one of the elements. Do not type the vertical bar.
... Repeatable. An ellipsis ( ... ) indicates that you can repeat the syntax element as often as required.
'Literal string' Single quotation ( ' ) marks must surround literal strings in CQL statements. Use single quotation marks to preserve upper case.
{ key:value } Map collection. Braces ( { } ) enclose map collections or key value pairs. A colon separates the key and the value.
<datatype1,datatype2> Set, list, map, or tuple. Angle brackets ( < > ) enclose data types in a set, list, map, or tuple. Separate the data types with a comma.
cql_statement; End CQL statement. A semicolon ( ; ) terminates all CQL statements.
[ -- ] Separate the command line options from the command arguments with two hyphens ( -- ). This syntax is useful when arguments might be mistaken for command line options.
' <schema> ... </schema> ' Search CQL only: Single quotation marks ( ' ) surround an entire XML schema declaration.
@xml_entity='xml_entity_type' Search CQL only: Identify the entity and literal value to overwrite the XML element in the schema and solrconfig files.
Returns the IP address of the currently selected Spark Master for the datacenter.
Returns the localhost IP address used to configure Spark applications. The address is returned as URI:

The connection.host=cs_list_contactpoints option is a comma separated list of IP addresses of additional contact points. The additional contact points are up to five randomly selected nodes from the datacenter.

Note: DSE automatically connects Spark applications to the Spark Master. You do not need to use the IP address of the current Spark Master in the connection URI.
metastore-migrate --from_version --to_version
Migrate Spark SQL metastore from one DSE version to another DSE version.
  • --from_version - the version to migrate metastore from
  • --to_version - the version to migrate metastore to
Returns the version of Spark that is bundled with DataStax Enterprise.
sql-schema (--exclude | --keyspace | --table | --decimal | --all)
Exports the SQL table creation query with these options:
  • --table tablename - comma-separated list of tables to include
  • --exclude csvlist - comma-separated list of tables to exclude
  • --all - includes all keyspaces
  • --keyspace csvlist - comma-separated list of keyspaces to include


View the Spark connection URL for this datacenter:

dse client-tool spark master-address

View the IP address of the current Spark Master in this datacenter:

dse client-tool spark leader-address

Generate Spark SQL schema files

You can use the generated schema files with Spark SQL on external Spark clusters.

dse client-tool --use-server-config spark sql-schema --all > output.sql

Migrate Spark metastore

To map custom external tables from DSE 5.0.11 to the DSE 6.7.0 release format of the Hive metastore used by Spark SQL after upgrading:

dse client-tool spark metastore-migrate --from 5.0.11 --to 6.7.0