Adding or removing a Spark application, driver, or worker fails
Adding or removing a Spark application, driver, or worker fails if it cannot be written to recovery storage.
Adding or removing a Spark application, driver, or worker fails if it cannot be written to recovery storage.
When the Spark Master first starts it updates its transient state and begins process management tasks, like communicating with workers. Then the Master tries to store changes in the persistence engine. If that task fails, the Master is restarted and its transient state is rebuilt from recovery storage, from the last state that was successfully persisted. To avoid an inconsistent state between these stages, any registration or removal of Spark applications, drivers, or workers will fail until the Master can write the data to recovery storage.
If you encounter a failure, wait until the Master is running and has successfully written to recovery storage before adding or removing applications, drivers, or workers.