Configuring Apache Spark™ logging options
You can configure Spark logging options for the Spark logs.
Log directories
The Spark logging directory is the directory where the Spark components store individual log files. DataStax Enterprise places logs in the following locations:
- Executor logs
-
-
SPARK_WORKER_DIR/worker-n/application_id/executor_id/stderr -
SPARK_WORKER_DIR/worker-n/application_id/executor_id/stdout
-
- Spark master/worker logs
-
Spark master: the global
system.logSpark worker:
SPARK_WORKER_LOG_DIR/worker-n/worker.logThe default
SPARK_WORKER_LOG_DIRlocation is/var/log/spark/worker. - Default log directory for Spark SQL Thrift server
-
The default log directory for starting the Spark SQL Thrift server is
$HOME/spark-thrift-server. - Spark shell and application logs
-
Spark shell and application logs are output to the console.
- SparkR shell log
-
The default location for the SparkR shell is
$HOME/.sparkR.log - Log configuration file
-
Log configuration files are located in the same directory as
spark-env.sh.The default location of the
spark-env.shfile depends on the type of installation:-
Package installations and Installer-Services installations:
/etc/dse/spark/spark-env.sh -
Tarball installations and Installer-No Services installations:
<installation_location>/resources/spark/conf/spark-env.sh
-
Log configuration files
Configure logging options, such as log levels, in the following files:
| Option | Description |
|---|---|
Executors |
|
Spark master |
|
Spark worker |
|
Spark driver (Spark shell, Spark applications) |
|
SparkR |
The location of the
|
Rolling logging for executors
If you want to enable rolling logging for Spark executors, add the following options to spark-daemon-defaults.conf.
Enable rolling logging with 3 log files retained before deletion. The log files are broken up by size with a maximum size of 50,000 bytes.
spark.executor.logs.rolling.maxRetainedFiles 3
spark.executor.logs.rolling.strategy size
spark.executor.logs.rolling.maxSize 50000The default location of the Spark configuration files depends on the type of installation:
-
Package installations and Installer-Services:
/etc/dse/spark/ -
Tarball installations and Installer-No Services:
<installation_location>/resources/spark/conf
Authentication
|
When user credentials are specified in plain text on the dse command line, like |
Configure a safe communication channel to access the Spark user interface.
The Spark master, Spark worker, executor, and driver logs might include sensitive information. Sensitive information includes passwords and digest authentication tokens for Kerberos guidelines mode that are passed in the command line or Spark configuration. DataStax recommends using only safe communication channels like VPN and SSH to access the Spark user interface.
|
Authentication credentials can be provided in several ways, see Connecting to authentication enabled clusters. |
