Change the topic or table schema

You must update the DataStax Apache Pulsar™ connector configuration when your field or table schema changes.

Add a field

Use these steps to write a new Pulsar field to an existing table column:

In the connector’s configuration file, add the field-to-column mapping.
Use pulsar-admin sinks update to apply the new configuration.
Start the Pulsar producer that had the schema change if it was stopped.

Remove a field

Connector processing fails if any mapped Pulsar topic fields are missing from the record. If a field is missing or if the schema change is known ahead, remove or edit the topic-column mapping.

Use the cqlsh command DESCRIBE TABLE to verify that the field isn’t mapped to a primary key (PK) column.

If the field is mapped to a PK column, you must map a different field to the column because PK columns cannot contain NULL values. PK columns must always have a mapping.

If the field isn’t mapped to a PK column, you can remove or edit the mapping, according to your preference and use case.
Remove or edit the invalid definition in your field-to-column mapping.
Use pulsar-admin sinks update to apply the new configuration.
Start the Pulsar producer that had the schema change if it was stopped.

Add a column to a table and synchronize an existing field

You can add a column to your database table, and then synchronize the data from an existing field in a Pulsar topic:

Use ALTER TABLE to add the column to the table in your database.
For DSE, confirm that the schema change has been pushed to all nodes using nodetool describecluster.

Data synchronization can fail if the schema change is inconsistent across the entire cluster.
In your connector’s configuration file, add the new column to your field-to-column mapping.
Use pulsar-admin sinks update to apply the new configuration.

Remove a column

You can only remove non-primary key columns because primary key columns must always have a mapping.

Remove the column definition from your field-to-column mapping.
Use ALTER TABLE to remove the column from the table on your database.
For DSE, confirm that the schema change has been pushed to all nodes using nodetool describecluster.

Data synchronization can fail if the schema change is inconsistent across the entire cluster.

Change the topic or table schema

Add a field

Remove a field

Add a column to a table and synchronize an existing field

Remove a column

Was this helpful?

Give Feedback