Warning: Long Post
I'm using Debezium version 1.9 and I've been facing a problem with getting topics in kafka for some of the topics that have been deleted before. I'm using the mysql connector to connect to the MariaDB database and it seems to be working flawlessly. But I'm currently trying to implement the signal table functionality and thats when I get this problem: When I add new tables to the connector's include list and add it to the debezium_signal table, it shows up in kafka immediately BUT if i delete this topic, remove it from the include list and from the debezium_signal table and try to add this table again to the include list and debezium table, it does not create a topic in kafka and gives the below message in logs:
Requested 'INCREMENTAL' snapshot of data collections '[dbname.tablename]' (io.debezium.pipeline.signal.ExecuteSnapshot)
2023-10-04T14:41:51+02:00 2023-10-04 12:41:51,187 WARN [debezium-sampleconnector|task-0] Schema not found for table 'dbname.tablename', known tables.
It then stops streaming shuts down and restarts connector and after a set of logs i get the initial info that I started this thread with.:
A previous offset indicating a completed snapshot has been found. Neither schema nor data will be snapshotted
In my codebase I'm updating the include list with tablenames dynamically and then I check if the status is Running and only if its running i update the signal table... all in the same thread. but it still does all this and then restarts the connector and i think thats the reason why it isnt finding the Schema for the table because the signal table is getting updated and the connector is getting the updated include list afterwards.
Is my assumption true? if yes what would I need to do as Im already checking the status of the connector if its Running or not.
How do I circumvent this issue? I also tried to add the snapshot mode to "when_needed" along with adding tables to the include list. but it made no difference.
How do I get the previously deleted topics back as new topics in kafka using the signal table? Kindly help. Also let me know in case any further information is required in this regard.