Formats of DataStax Enterprise logs

The log format is a simple set of pipe-delimited name/value pairs. A name/value pair, or field, is only included in the log line if a value exists for that particular event.

The log format is a simple set of pipe-delimited name/value pairs. The pairs themselves are separated by the pipe symbol ("|"), and the name and value portions of each pair are separated by a colon. A name/value pair, or field, is only included in the log line when a value exists for that particular event. Some fields always have a value, and are always present. Others might not be relevant for a given operation. To make parsing with automated tools easier, the order in which fields appear (when present) in the log line is predictable. For example, the text of CQL statements is unquoted, but if present, is always the last field in the log line.

Field Label Field Value Optional
host dse node address no
source client address no
user authenticated user no
timestamp system time of log event no
category DML/DDL/QUERY for example no
type API level operation no
batch batch id yes
ks keyspace yes
cf column family yes
operation textual description yes

The textual description value for the operation field label is currently only present for CQL.

Auditing is completely separate from authorization, although the data points logged include the client address and authenticated user, which may be a generic user if the default authenticator is not overridden. Logging of requests can be activated for any or all of the list of categories described in Enabling data auditing in DataStax Enterprise.

CQL logging examples

Generally, SELECT queries are placed into the QUERY category. The INSERT, UPDATE, and DELETE statements are categorized as DML. CQL statements that affect schema, such as CREATE KEYSPACE and DROP KEYSPACE are categorized as DDL.

CQL USE

USE dsp904;

host:/192.168.56.1|source:/192.168.56.101|user:#User allow_all groups=[]
  |timestamp:1351003707937|category:DML|type:SET_KS|ks:dsp904|operation:use dsp904;

CLI USE

USE dsp904;

host:/192.168.56.1|source:/192.168.56.101|user:#User allow_all groups=[]
  |timestamp:1351004648848|category:DML|type:SET_KS|ks:dsp904

CQL query

SELECT * FROM t0;

host:/192.168.56.1|source:/192.168.56.101|user:#User allow_all groups=[]
  |timestamp:1351003741953|category:QUERY|type:CQL_SELECT|ks:dsp904|cf:t0|operation:select * from t0;

CQL BATCH

BEGIN BATCH
  INSERT INTO t0(id, field0) VALUES (0, 'foo')
  INSERT INTO t0(id, field0) VALUES (1, 'bar')
  DELETE FROM t1 WHERE id = 2
APPLY BATCH;

host:192.168.56.1|source:/192.168.56.101|user:#User allow_all groups=[]
  |timestamp:1351005482412|category:DML|type:CQL_UPDATE
  |batch:fc386364-245a-44c0-a5ab-12f165374a89|ks:dsp904|cf:t0
  |operation:INSERT INTO t0 ( id , field0 ) VALUES ( 0 , 'foo' )

host:192.168.56.1|source:/192.168.56.101|user:#User allow_all groups=[]
  |timestamp:1351005482413|category:DML|type:CQL_UPDATE
  |batch:fc386364-245a-44c0-a5ab-12f165374a89|ks:dsp904|cf:t0
  |operation:INSERT INTO t0 ( id , field0 ) VALUES ( 1 , 'bar' )

host:192.168.56.1|source:/192.168.56.101|user:#User allow_all groups=[]
  |timestamp:1351005482413|category:DML|type:CQL_DELETE
  |batch:fc386364-245a-44c0-a5ab-12f165374a89|ks:dsp904|cf:t1
  |operation:DELETE FROM t1 WHERE id = 2

CQL DROP KEYSPACE

DROP KEYSPACE dsp904;

host:/192.168.56.1|source:/192.168.56.101|user:#User allow_all groups=[]
  |timestamp:1351004777354|category:DDL|type:DROP_KS
  |ks:dsp904|operation:drop keyspace dsp904;

CQL prepared statement

host:/10.112.75.154|source:/127.0.0.1|user:allow_all
  |timestamp:1356046999323|category:DML|type:CQL_UPDATE
  |ks:ks|cf:cf|operation:INSERT INTO cf (id, name) VALUES (?, ?) [id=1,name=vic]

Thrift batch_mutate

host:/192.168.56.1|source:/192.168.56.101|user:#User allow_all groups=[]
  |timestamp:1351005073561|category:DML|type:INSERT
  |batch:7d13a423-4c68-4238-af06-a779697088a9|ks:Keyspace1|cf:Standard1

host:/192.168.56.1|source:/192.168.56.101|user:#User allow_all groups=[]
  |timestamp:1351005073562|category:DML|type:INSERT
  |batch:7d13a423-4c68-4238-af06-a779697088a9|ks:Keyspace1|cf:Standard1

host:/192.168.56.1|source:/192.168.56.101|user:#User allow_all groups=[]
  |timestamp:1351005073562|category:DML|type:INSERT
  |batch:7d13a423-4c68-4238-af06-a779697088a9|ks:Keyspace1|cf:Standard1

DataStax Java Driver queries

host:ip-10-85-22-245.ec2.internal/10.85.22.245|source:/127.0.0.1|user:anonymous
  |timestamp:1370537557052|category:DDL|type:ADD_KS
  |ks:test|operation:create keyspace test with replication = {'class':'NetworkTopologyStrategy', 'Analytics': 1};

host:ip-10-85-22-245.ec2.internal/10.85.22.245|source:/127.0.0.1|user:anonymous
  |timestamp:1370537557208|category:DDL|type:ADD_CF
  |ks:test|cf:new_cf|operation:create COLUMNFAMILY test.new_cf ( id text PRIMARY KEY , col1 int, col2 ascii, col3 int);

host:ip-10-85-22-245.ec2.internal/10.85.22.245|source:/127.0.0.1|user:anonymous
  |timestamp:1370537557236|category:DML|type:CQL_UPDATE
  |ks:test|cf:new_cf|operation:insert into test.new_cf (id, col1, col2, col3) values ('test1', 42, 'blah', 3);

host:ip-10-85-22-245.ec2.internal/10.85.22.245|source:/127.0.0.1|user:anonymous
  |timestamp:1370537704885|category:QUERY|type:CQL_SELECT
  |ks:test|cf:new_cf|operation:select * from test.new_cf;

Batch updates

Batch updates, whether received via a Thrift batch_mutate call, or in CQL BEGIN BATCH....APPLY BATCH block, are logged in the following way: A UUID is generated for the batch, then each individual operation is reported separately, with an extra field containing the batch id.