Postgresql Replication Slot grows in GB suddenly

54 Views Asked by At

We are using Qlik Replicate for replicating data from RDS Postgresql to Synapse.

There is logical replication slot involved in this process, so here is the case when we start the replication, the slots size be in MBs and then suddenly slot size spurts to GBs and grow immensely like 1gb/min

There is highly transaction table in this process, ~ 80million transaction a day. this table also get auto vacuumed every 5 mins.

We skipped this table in order to optimize the replication, still there was latency.

We checked at the target, there everything was fine as concluded by the azure engineer.

1

There are 1 best solutions below

0
manjunath On

For every logical replication slot,we will have a file called state file which is present as /PGdata/pg_replslot/<slot_name>/state.This state file is in binary format,which will be around 200 bytes approx.

This state file size doesn't increase above specified value.I think when very large transactions are running/ongoing walSender process will serialise the contents of reorder buffer into the /PGdata/pg_replslot/<slot_name> directory,so you will find size of the directory increasing abruptly.

WalSender process will read all the transaction changes into reorder buffer and serialise them based on transaction size,check to send them to client is decided in output plugin code,so therefore you find size of directory being increased even you didn't subscribed for those changes.