I'm having trouble generating proper schemas when I use Debezium with outbox pattern. It generates schemas but it's not what I would expect it to generate.
below is my connector config.
table.include.list: public.events_outbox
plugin.name: pgoutput
topic.prefix: cdc.keycloak.service
topic.creation.enable: true
topic.creation.default.replication.factor: 3
topic.creation.default.partitions: 3
topic.creation.default.cleanup.policy: compact
topic.creation.default.compression.type: lz4
decimal.handling.mode: double
# confluent schema registry
key.converter.schemas.enable: false
value.converter.schemas.enable: false
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url: http://schema-registry:8081
transforms: "outbox"
transforms.outbox.type: io.debezium.transforms.outbox.EventRouter
transforms.outbox.table.field.event.key: aggregate_id
transforms.outbox.route.by.field: aggregate_type
tombstones.on.delete: false
transforms.outbox.route.topic.replacement: cdc.keycloak.${routedByValue}
value.converter.delegate.converter.type.schemas.enable: true
transforms.outbox.debezium.expand.json.payload: true
and below is my database structure
Column | Type | Modifiers
--------------+------------------------+-----------
id | uuid | not null
aggregate_type | character varying(255) | not null
aggregate_id | character varying(255) | not null
type | character varying(255) | not null
payload | jsonb |
Whenever a new event gets created on the outbox table if a schema does not exist it create a new schema like below
[
"null",
{
"type": "string",
"connect.version": 1,
"connect.name": "io.debezium.data.Json"
}
]
and below is my payload data (jsonb)
{
"id": "87848159-6478-4496-8646-fddff2700ada",
"email": "[email protected]",
"isActive": true,
"lastName": "Doe",
"firstName": "Joe",
"isEmailVerified": false
}
My question is am I doing anything wrong to generate a schema like above or it is the expected behavior? I would like to like to generate a proper avro schema based on the content (payload structure).