Monitoring Apache Spark™ with the web interface

To use the Spark web interface:

If the Spark Master is not available, the UI keeps polling for the Spark Master every 10 seconds until the Master is available.

The Spark web interface can be secured using SSL. SSL encryption of the web interface is enabled by default when client encryption is enabled.

If authentication is enabled, and plain authentication is available, a prompt for authentication credentials when accessing the web UI appears. We recommend using SSL with authentication.

DSE SSL encryption and authentication only apply to the Spark Master and Worker UIs, not the Spark Driver UI. To use encryption and authentication with the Driver UI, refer to the Spark security documentation.

Authorization is not supported in the Spark web UI. Any authenticated user can monitor and control any Spark applications within the UI.

See the Spark documentation for information on using the Spark web UI.

To display fully qualified domain names (FQDNs) in the Spark web UI, set the SPARK_PUBLIC_DNS variable in spark-env.sh on each Analytics node.

Set SPARK_PUBLIC_DNS to the FQDN of the node if you have SSL enabled for the web UI.

Set the SPARK_LOCAL_IP or SPARK_LOCAL_HOSTNAME in the spark-env.sh file on each node to the fully qualified domain name (FQDN) of the node to force any redirects to the web UI using the FQDN of the Spark master. This is useful when enabling SSL in the web UI.

The Spark Driver UI has an Environment tab that lists the Spark configuration and system properties used by Spark. This can include sensitive information like passwords and security tokens. DSE Spark filters these properties and mask their values with sequences of asterisks. The spark.ui.confidentialKeys filter is configured as a comma separated list of regular expressions that by default includes all properties that contain the string "token" or "password". To modify the filter, edit the spark.ui.confidentialKeys property in spark-defaults.conf in the Spark configuration directory.