Invalid Avro schema generation when used with debezium outbox pattern

58 Views Asked by At

I'm having trouble generating proper schemas when I use Debezium with outbox pattern. It generates schemas but it's not what I would expect it to generate.

below is my connector config.

table.include.list: public.events_outbox
plugin.name: pgoutput

topic.prefix: cdc.keycloak.service
topic.creation.enable: true
topic.creation.default.replication.factor: 3
topic.creation.default.partitions: 3
topic.creation.default.cleanup.policy: compact
topic.creation.default.compression.type: lz4

decimal.handling.mode: double

# confluent schema registry
key.converter.schemas.enable: false
value.converter.schemas.enable: false
key.converter: org.apache.kafka.connect.storage.StringConverter

value.converter: io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url: http://schema-registry:8081
transforms: "outbox"

transforms.outbox.type: io.debezium.transforms.outbox.EventRouter
transforms.outbox.table.field.event.key: aggregate_id
transforms.outbox.route.by.field: aggregate_type
tombstones.on.delete: false
transforms.outbox.route.topic.replacement: cdc.keycloak.${routedByValue}

value.converter.delegate.converter.type.schemas.enable: true
transforms.outbox.debezium.expand.json.payload: true

and below is my database structure

Column        |          Type          | Modifiers
--------------+------------------------+-----------
id            | uuid                   | not null
aggregate_type | character varying(255) | not null
aggregate_id   | character varying(255) | not null
type          | character varying(255) | not null
payload       | jsonb                  |

Whenever a new event gets created on the outbox table if a schema does not exist it create a new schema like below

[
    "null",
    {
        "type": "string",
        "connect.version": 1,
        "connect.name": "io.debezium.data.Json"
    }
]

and below is my payload data (jsonb)

{
  "id": "87848159-6478-4496-8646-fddff2700ada",
  "email": "[email protected]",
  "isActive": true,
  "lastName": "Doe",
  "firstName": "Joe",
  "isEmailVerified": false
}

My question is am I doing anything wrong to generate a schema like above or it is the expected behavior? I would like to like to generate a proper avro schema based on the content (payload structure).

0

There are 0 best solutions below