Log-based CDC with a query to stream data from source

218 Views Asked by At

I'm currently exploring on both Query based CDC and Log based CDC using Debezium connectors with Kafka connect. With Query based CDC I could stream data when I specify specific tables or a query with joins on multiple tables in the connector configuration.

Now, trying out Log based CDC. I'm able to stream data to topics by specifying table name in the connector config but unable to stream when I use query in the connector. Mentioned below is the connector config that I'm using.

kafka connect config :-

{
  "name": "Oracle_CDC_Customer_Ind",
  "config": {
    "connector.class": "io.debezium.connector.oracle.OracleConnector",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "database.server.name" : "Customer-Ind",
    "database.oracle.version": "12+",
    "database.hostname" : "localhost",
    "database.port" : "1521",
    "database.user" : "C##DBZUSER",
    "database.password" : "*****",
    "oracle.jdbc.useFetchSizeWithLongColumn": "true",
    "database.schema": "c##dbzuser",
    "database.dbname" : "*****",
    "database.out.server.name" : "dbzxout",
    "database.connection.adapter": "logminer",
    "database.tablename.case.insensitive": "true",
    "database.history.kafka.bootstrap.servers" : "localhost:9092",
    "database.history.kafka.topic": "schema-changes.customer_ind",
    "database.history.skip.unparseable.ddl": "true",
    "include.schema.changes": "true",
    "table.include.list": "C##DBZUSER.customers",
    "query": "Select * from C##DBZUSER.customers where country == 'India'",
    "snapshot.mode": "initial",
    "errors.log.enable": "true"
  }
}

I dont see any error in the connector logs. Should I look into any other logs or enable some logging?

Do I need to change/add properties in the connector config to stream data using query with log based CDC ? Any pointers would be of great help.

0

There are 0 best solutions below