Clickhouse kafka integration tables cleaning

116 Views Asked by At

I use kafka integration mechanism to transfer data via chain (kafka engine)->(materialized view with TO)->(destination table)

materialized view reads any data block coming to kafka table , and use view source select to add data to destination table and also the data is materialized inside the mat view inself. Sometimes I need to remove data from materilized view and remove data from dest table. I plan to remove old version rows based on two version keys with specific order. I have specific delete statement that working well to remove old version rows based mention keys (using PARTITION BY). So overall plan is DETACH mat view from dest table to stop kafka message streaming into mat view -> destination table , do my delete statement , delete data from mat view , attach mat view back after all. I use standard mergetree engine for dest table and I do not want any other engine (aggmergetre, replacingmergeetre or others) because of standard mergetree logic is better for me. I know that by using CREATE MATERIALIZED VIEW with "TO" make mat view engine to be inherited from dest table.

Question is it safely to remove data from mat view during DETACH active time ? I mean will I not hurt integration mechanism ? Probably some rows came to mat view but not dest table because of my DETACH statement during same time , and I would probably delete data that not transfered to dest table else.

0

There are 0 best solutions below