About DSEFS

DSEFS (DataStax Enterprise file system) is a new distributed file system within DataStax Enterprise that is intended primarily for Spark streaming use cases and Write Ahead Logging (WAL).

DSEFS (DataStax Enterprise file system) is a new distributed file system within DataStax Enterprise that is intended primarily to provide fault tolerance for Spark streaming use cases and Write Ahead Logging (WAL). DSEFS is more performant than CFS (Cassandra File System).

CFS is always enabled by default on analytics nodes. To use DSEFS, you must enable DSEFS on all analytics nodes. However, CFS is deprecated and will be replaced by DSEFS in a future release.

DSEFS splits the stored data into metadata and file data blocks:
  • Metadata is stored in the Cassandra database.
  • File data blocks are stored locally on each node and are replicated onto multiples nodes.

    The redundancy factor is set at the DSEFS directory or file level, which is more granular than the replication factor that is set at the keyspace level in the Cassandra database.

You must store the DSEFS data on physical devices that are separate from the Cassandra database.

Deployment overview

  • The DSEFS server runs in the same JVM as DataStax Enterprise. Similar to the Cassandra database, there is no master node. All nodes running DSEFS are equal.
  • A single DSEFS cannot span multiple datacenters. To deploy DSEFS in multiple datacenters, you can create a separate instance of DSEFS for each datacenter.
  • You can use different keyspaces to configure multiple DSEFS file systems in a single datacenter.
  • For optimal performance, locate the local DSEFS data on a different physical drive than the Cassandra database.
  • Authentication, authorization, and encryption are not supported. Use operating system access controls to protect the local DSEFS data directories. Other limitations apply.
  • DSEFS is intended for Spark streaming use cases and is not appropriate for long term data storage or for exclusive storage of any data.
  • DSEFS uses the LOCAL_QUORUM consistency level.