Missing field error

If a Kafka record is missing fields that are set in the topic-table mapping data parsing fails.

When the Kafka Connect log contains a missing field error message, check for the following issues:
For example, the log contains the following error message.
Error inserting/updating row for Kafka record … ConnectRecord{topic='inf1-src',kafkaPartition=0, key=null, value={set=[37, 96, 90], udt={udtmem2=90, udtmem1=47}}, ...}: Required field 'value.bigint' (mapped to column bigintcol) was missing from record. Pleaseremove it from the mapping.

Field is missing from record

All fields referenced in the mapping specification are required to process the record.

All fields referenced in the mapping specification are required to process the record. If a Kafka record does not contain one or more fields that are set in the topic-table mapping, the data parsing fails and an error message is recorded in the Kafka Connect log.

For example, the log contains the following error message.
Error inserting/updating row for Kafka record … ConnectRecord{topic='inf1-src',kafkaPartition=0, key=null, value={set=[37, 96, 90], udt={udtmem2=90, udtmem1=47}}, ...}: Required field 'value.bigint' (mapped to column bigintcol) was missing from record. Pleaseremove it from the mapping.

Remediation

Remove the field-column mapping from the connector configuration file or add the missing field it to the Kafka records.

Converter data type issue

Missing fields reported when the connector uses the wrong convertor for the record data type.

When the wrong converter is used, an error message reports missing fields and the fields are in the Kafka record. For example, if the JsonConverter is used to parse Kafka records that are actually strings.

Remediation

Change the following in the connect-distributed.properties or connect-standalone .properties files:
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter

Invalid JSON

Kafak record contains invalid JSON and cannot be parsed.

If the raw value is not valid JSON, the record cannot be parsed.

Remediation

Use a JSON validator to make sure that the JSON records are valid.