DSEFS compression

DSEFS ships with the lz4 encoder, which works out of the box.

To compress files use the -c or --compression-encoder parameter for put command. The parameter specifies the compression encoder to use for the file that is about to get uploaded.

Optionally, set the frame size with the -f, --compression-frame-size option.

Set the maximum frame size in bytes with the compression_frame_max_size option in dse.yaml. Setting the frame size to a value greater than compression_frame_max_size when using put -f results in an error, and the command fails. Modify the compression_frame_max_size setting based on the available memory of the node.

Append to compressed files just as you would to uncompressed files. Compressed files append data by transparent compression using the same file encoder specified for the initial put operation.

Directories can have a default compression encoder specified during directory creation with the mkdir command. Newly added files with the put command inherit the default compression encoder from containing directory. You can override the default compression encoder with the c parameter during put operations.

Automatic decompression occurs for all commands that transport data to the client. There is no need for additional configuration to retrieve the original, decompressed file content.

Enabling compression creates a distinction between the logical and physical file size.

The logical size is the size of a file before uploading it to DSEFS, where it is then compressed. Show the logical size with the stat command. Look for results in Size.

The physical size is the actual size of a data stored on the storage device. Show the physical size with the df command and the stat -v command for each block separately. Look for results in the Compressed length column.

Truncating compressed files is not possible.