I want to capture data changes from few tables in a huge PostgreSQL database. Initially I planned to use the logical decoding feature with Debezium. But this solution has significant overhead since it's necessary to decode the entire WAL. Another solution uses triggers and PgQ. Is there any general way to integrate PgQ with Kafka or perhaps a Kafka connector for this purpose?
What is the best way to read from PgQ to Apache Kafka?
357 Views Asked by Iskuskov Alexander At
1
There are 1 best solutions below
Related Questions in POSTGRESQL
- Only the first SQL script gets executed inside Docker Postgres container
- Compare fields in two tables
- Hibernate ClobJdbcType bindings: what are the diferences?
- Postgres && statement Error in Mybatis Mapper?
- Can this query be optimized? (Choosing a random row to insert, that excludes previously inserted Rows)
- Connection terminated unexpectedly while performing multi row insert using pg-promise
- Processing multiple forms in nodejs and postgresql
- How to copy data from SQLite to postgreSQL?
- PGAdmin4 configured behind a reverse proxy but unable to connect to Postgresql server
- Updates to pgsodium encrypted values don't use specified key_id
- Connecting to Postgres running in a Docker container using psql
- Can't connect to local postgresql server from my docker container
- Django Arrayfield migration to cloud sql (Postgresql) not creating the column
- Get list of matching keywords for each post
- docker-compose can't reset postgresql database
Related Questions in APACHE-KAFKA
- No method found for class java.lang.String in Kafka
- How to create beans of the same class for multiple template parameters in Spring
- Troubleshoot .readStream function not working in kafka-spark streaming (pyspark in colab notebook)
- Handling and ignore UNKNOWN_TOPIC_OR_PARTITION error in Kafka Streams
- Connect Apache Flink with Apache kudu as sink using Pyflink
- Embedded Kafka Failed to Start After Spring Starter Parent Version 3.1.10
- Producer Batching Service Bus Vs Kafka
- How to create a docker composer environment where containers can communicate each other?
- Springboot Kafka Consumer unable to maintain connect to kafka cluster brokers
- Kafka integration between two micro service which can respond back to the same function initiated the request
- Configuring Apache Spark's MemoryStream to simulate Kafka stream
- Opentelemetry Surpresses Kafka Produce Message Java
- Kafka: java.lang.NoClassDefFoundError: Could not initialize class org.apache.logging.log4j.core.appender.mom.kafka.KafkaManager
- MassTransit Kafka producers configure to send several events to the same Kafka topic
- NoClassDefFoundError when running JAR file with Apache Kafka dependencies
Related Questions in APACHE-KAFKA-CONNECT
- Transform Load pipeline for a logs system: Apache Airflow or Kafka Connect?
- Kafka Connect JDBC Connector not working with teradata
- How can I change index name that created by kafka-connect with connect config
- Advantages of using two Kafka Connect Sink connectors for topics with similar data and workload?
- Neo4j Source Connectors Failing to build the Schema where the source query returns null for some of the fields
- kafka connect to minio
- MANAGE LIFECYCLE OF VAULT SECRET WITH CUSTOM ["org.apache.kafka.common.config.provider"]
- Debezium sink connector doesn't delele rows in destination postgres
- Is it possible to drop a field that it's value is null from a kafka event in kafka connector?
- MSK Connector w/ SMT
- How to make one reroute for all topics in debezium Kafka Connect PostgresConnector
- Kafka Connect JDBC Sink Connector giving WorkerSinkTask ERROR: JsonConverter with schemas.enable requires "schema" and "payload"
- JAAS configuration for Managed Identity for Kafka-Connect in Azure
- Current parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as 'PERMISSIVE'
- Kafka Connect - Redis Sink Connector doing String upsert with last consumed message offset
Related Questions in DEBEZIUM
- Error reading MySQL incremental logs using Debezium
- Throughput issue on Target Kafka topic when using Oracle CDC confluent connector
- How to change topic names and schema names along with namespaces to format topicPrefix.tableName in debezium
- Kafka connect error while loading debezium - "Unable to instantiate " " Failed to connect to and describe Kafka cluster"
- dabezium mongo connector in local
- Debezium sink connector doesn't delele rows in destination postgres
- Debezium SQL DB connector on AKS - InconsistentGroupProtocolException
- Kafka Connect JDBC Sink Connector giving WorkerSinkTask ERROR: JsonConverter with schemas.enable requires "schema" and "payload"
- is there any solution Error while connecting to kafka topic
- Kafka Connector : Debezium mongodb source connector got error if field in object that in array can be null
- Debezium can not authorize the Mongodb shards
- What is the difference between debezium vs confluent JdbcSinkConnector?
- how postgres bulk insert affect Debezium?
- Debezium connector for PostgreSQL Connector configuration is invalid
- CREATE PUBLICATION perfomance
Related Questions in LOGICAL-DECODING
- logical_decoding_work_mem and max_logical_worker_processes tunning
- How to use exported snapshot in Postgres replication slot
- What is the difference between Wal2json and Pgoutput?
- Proper standby status update in streaming replication protocol
- How to do logical replication on cascading standby server on postgres?
- Encoding 0 to 255(1 Byte) range of 10 different values into 3 Byte length value and retrive it to original values
- How to interpret executable binary file
- Logical decoding - postgres - multiple output formats
- PostgresQL: Is an entire transaction always written to a replication slot?
- What is the best way to read from PgQ to Apache Kafka?
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular # Hahtags
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
You either go transaction log, or you go query-based.
Which you use depends on your use of the data. Query-based polls the DB, log-based uses the log (WAL).
I'm interested in your assertion that Debezium has "significant overhead"—have you quantified this? I know there are lots of people using it and it's not usually raised as an issue.
For query-based capture use the Kafka Connect JDBC source connector.
You can see pros and cons of each approach here: http://rmoff.dev/ksny19-no-more-silos