DBs-ium, CDC and Streaming
An airhacks.fm conversation with Gunnar Morling (@gunnarmorling) about: The first Debezium commit, Randal Hauch, DBs-iuim, Java Content Repository JCR / modshape, exploring the Change Data Capture (CDC), how Debezium started, the MySQL binlog, the logical decoding in Postgres, Oracle Advanced Queuing, update triggers, Java Message System (JMS), there is no read detection, switching the current user at JDBC connection for audit purposes, helping Debezium with additional metadata table, using Kafka Streams to join the metadata and the payload, installing the logical decoding plugins into PostgreSQL, logical decoding plugin exposes the data from the write ahead log, decoding into protocol buffers with decoderbufs, in cloud environments like e.g. Amazon RDS you are not allowed to install any plugins, wal2json is verbose but comes preinstalled on RDS, pgoutput is responsible for the actual decoding of the events, debezium only sees committed transactions, debezium is mainly written in Java, decoderbufs was written by community and included to debezium, Debezium communicates with Postgres via the JDBC / Postgres API, pgoutput format is converted into Kafka Connector source format, Kafka Connect is a framework for running connectors, Kafka Connect comes with sink and source connectors, Kafka Connect comes with connector specific connectors like e.g. StringConverter, Converters are not Serializers, Debezium ships as Kafka Connect plugin, Kafka Connector runs as standalone process, running Debezium in embedded mode, JPA cache invalidation with Debezium, converting Debezium events into CDI events, converting database changes to WebSockets events, database polling vs the Debezium approach, DB2 will support Debezium, Oracle support is "on the horizon", Oracle LogmMiner, Oracle XStream, Debezium supports Microsoft SQL Server (starting with Enterprise license), Apache Pulsar comes with Debezium out-of-the-box, Pulsar IO, running Debezium as standalone service with outbounds APIs, MongoDB supports the "Debezium Change Event Format", Kafka Sink connectors are easy to implement, Debezium embedded mode and offsets, embedded connector has to remember the offset, an offset API is available for embedded Debezium connectors, combining CDC with Kafka Streams, Quarkus supports Kafka Streams and Reactive Messaging, Quarkus and Kafka Streams, Quarkus supports Kafka Streams in dev mode, replacing Hibernate Envers with Debezium, Messaging vs. Streaming or JMS vs. Kafka, Kafka is a database, the possible Debezium features, Cassandra support is coming, Outbox pattern is going to be better supported, transactional event grouping, dedicated topic for transaction demarcations, commercial support for Debezium, Debezium exposes JMX metrics, Five Advantages of Log-Based Change Data Capture, Reliable Microservices Data Exchange With the Outbox Pattern, Automating Cache Invalidation With Change Data Capture Gunnar Morling on twitter: @gunnarmorling and github: https://github.com/gunnarmorling. Gunnar's blog: https://morling.dev/.