Configuring Spark logging options
Configure Spark logging options.
Log directories
The location of the
logback.xml file depends on the type of
installation:
Installer-Services and Package installations | /etc/dse/cassandra/logback.xml |
Installer-No Services and Tarball installations | install_location/resources/cassandra/logback.xml |
The Spark logging directory is the directory where the Spark components store
individual log files. DataStax Enterprise places logs in the following locations:
- Executor logs
-
- SPARK_WORKER_DIR/worker-n/application_id/executor_id/stderr
- SPARK_WORKER_DIR/worker-n/application_id/executor_id/stdout
- Spark Master/Worker logs
- Spark Master: the global system.log
- Default log directory for Spark CQL Thrift server
- The default log directory for starting the Spark CQL Thrift server is $HOME/spark-thrift-server.
- Spark Shell and application logs
- Spark Driver Shell and application logs are output to the console.
- Log configuration file
- Log configuration files are located in the same directory as spark-env.sh.
Procedure
To configure Spark logging options: