Change Data Capture

Change Data Capture (CDC) extracts and streams database changes to downstream systems in real-time. Rather than polling databases or maintaining dual-write logic, CDC reads directly from the database’s internal change mechanisms—transaction logs, replication streams, or triggers—providing a reliable, non-invasive way to propagate data changes across systems.

This article covers CDC approaches, log-based implementation internals, production patterns, and when each variant makes sense.

Mental Model

CDC provides eventually-consistent data propagation without application-level dual writes. The insight that makes it work: databases already record every change internally — for crash recovery and physical replication — so CDC is “expose that internal stream as a public, consumer-facing API.”

Three approaches, ordered by how much database cooperation they require:

• Log-based CDC tails the database’s transaction log (PostgreSQL WAL, MySQL binlog, MongoDB oplog, SQL Server transaction log). Non-invasive on the write path, captures every committed change including direct SQL and migrations, preserves commit order. The production default.
• Trigger-based CDC installs AFTER INSERT/UPDATE/DELETE triggers that copy mutations to a shadow table. Works on locked-down or legacy databases without log access, at the cost of write-path latency and shadow-table contention.
• Polling-based CDC runs SELECT ... WHERE updated_at > :hwm against a read replica. No special database privileges, but it cannot see hard deletes and pays a perpetual query tax.

The decision axis is source access vs. operational footprint: log-based asks the DB team to expose replication slots / binlog access in exchange for minimal runtime cost; polling avoids that conversation but loses fidelity. Trigger-based is the in-between option for environments where neither extreme is available.

Production reality: log-based CDC dominates. The two anchor tools are Debezium (self-managed, sub-second, runs on Kafka Connect) and AWS DMS (managed, seconds-to-minutes, AWS-native sinks). Kafka is the default transport — partly because it preserves order per partition, partly because Schema Registry + Kafka Connect Sinks form a pre-built fan-out into search indexes, caches, warehouses, and downstream services.

The Problem CDC Solves

Why Naive Solutions Fail

Approach 1: dual writes in application code.

Show 4 hidden lines

Fails because:

• Partial failures: Database commits but Kafka publish fails. Data is now inconsistent.
• Distributed transaction complexity: XA/2PC across an RDBMS and Kafka exists but is slow, fragile, and not supported by Kafka’s open-source brokers.
• Missed changes: Direct SQL updates, migrations, replicas, and other services bypass the publish logic entirely.
• Ordering: Kafka messages may interleave or arrive out of database commit order.

Approach 2: polling with timestamps.

1SELECT * FROM users WHERE updated_at > :last_poll_time

Fails because:

• Misses hard deletes: Deleted rows don’t appear in query results.
• Clock skew: updated_at may not reflect commit order — especially across replicas, or under multi-statement transactions where now() is captured at statement start.
• Polling interval trade-off: frequent polling adds DB load; infrequent polling adds end-to-end latency.
• Transaction visibility: may read mid-transaction state if isolation isn’t tight.

Approach 3: trigger-based capture.

1CREATE TRIGGER user_changes AFTER INSERT OR UPDATE OR DELETE ON users2FOR EACH ROW EXECUTE FUNCTION capture_change();

Fails at scale because:

• Transaction overhead: trigger runs synchronously within the transaction, adding latency to every write.
• Lock contention: writing to a shadow table from every transaction concentrates contention on a single hot relation.
• Operational burden: triggers must be re-applied on every schema change and replicated to every shard.

The Core Challenge

The fundamental tension: application code cannot reliably capture all database changes without the database’s cooperation. Direct SQL, stored procedures, migrations, and multiple services all modify data outside application control.

CDC resolves this by reading changes where they’re already reliably recorded—the database’s transaction log. This log exists for durability and replication; CDC treats it as a public API.

CDC Approaches

Log-Based CDC (Primary Approach)

How it works:

1. CDC connector acts as a replica consumer for the database’s transaction log
2. Connector maintains position (LSN, binlog coordinates, or GTID) for resumability
3. Changes parsed from binary log format into structured events
4. Events published to message broker, maintaining transaction boundaries

Database-specific mechanisms:

Why log-based is preferred:

• Complete capture: Every committed change, including DDL, is in the log
• Minimal overhead: Reading the log adds no load to write path
• Transactional boundaries: Changes can be grouped by transaction
• Ordering guarantees: Log order matches commit order

Trade-offs:

Trigger-Based CDC

How it works:

1. Create triggers on source tables for INSERT, UPDATE, DELETE
2. Triggers write change records to shadow tables
3. Separate process polls shadow tables and publishes events
4. Shadow table records deleted after successful publish

When to choose:

• Log-based access unavailable (managed databases, permission restrictions)
• Only specific tables need capture (trigger overhead is localized)
• Legacy databases without logical replication support

Trade-offs:

Polling-Based CDC

How it works:

1. Query source tables periodically for changes since last poll
2. Use updated_at timestamp or sequence column to identify changes
3. Mark captured rows or track high-water mark
4. Publish changes to downstream systems

When to choose:

• Read replica available for polling (isolates from production writes)
• Soft deletes only (hard deletes not used)
• Near-real-time acceptable (seconds to minutes latency)

Limitations:

• Cannot capture hard deletes without tombstone markers
• Timestamp precision issues (multiple changes within same timestamp)
• Must poll frequently to approach real-time
• No transaction grouping

Decision Framework

Log-Based CDC Internals

PostgreSQL: WAL and Logical Replication

PostgreSQL’s CDC uses logical replication, which decodes the physical WAL into logical change events.

Architecture:

Configuration requirements:

1-- postgresql.conf2wal_level = logical                    -- Required for logical replication3max_replication_slots = 4              -- One per CDC connector4max_wal_senders = 4                    -- Connections for replication56-- Create replication slot (done by Debezium automatically)7SELECT pg_create_logical_replication_slot('debezium', 'pgoutput');

Output plugins:

Critical operational concern—slot bloat:

PostgreSQL retains WAL as long as a replication slot hasn’t consumed it. If a CDC connector goes down:

1-- Monitor slot lag2SELECT slot_name,3       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS lag4FROM pg_replication_slots;56-- Set maximum retained WAL (PostgreSQL 13+)7ALTER SYSTEM SET max_slot_wal_keep_size = '10GB';

Without max_slot_wal_keep_size, an inactive slot can fill the disk. This is the most common CDC production incident.

Version evolution:

MySQL: Binary Log

MySQL’s CDC reads the binary log, which records all data modifications.

Configuration requirements:

1# my.cnf2server-id = 1                          # Unique across replication topology3log_bin = mysql-bin                    # Enable binary logging4binlog_format = ROW                    # Required: ROW format (not STATEMENT)5binlog_row_image = FULL                # Capture before and after state6binlog_expire_logs_seconds = 259200    # Retention (3 days). expire_logs_days is deprecated since MySQL 8.0.

GTID (Global Transaction ID):

GTIDs uniquely identify transactions across the replication topology, enabling position-independent replication.

1-- Enable GTID mode2gtid_mode = ON3enforce_gtid_consistency = ON45-- Format: server_uuid:transaction_id6-- Example: 3E11FA47-71CA-11E1-9E33-C80AA9429562:23

Why GTID matters for CDC:

• Resumability: CDC connector can resume from GTID regardless of binlog file rotation
• Failover: After primary failover, GTID identifies exactly which transactions to resume from
• Multi-source: When capturing from multiple MySQL instances, GTIDs prevent duplicate processing

Binlog format comparison:

MongoDB: Change Streams

MongoDB provides Change Streams, a high-level API over the oplog (operations log).

Show 3 hidden lines4// Watch collection-level changes5const changeStream = db.collection("users").watch([], {6  fullDocument: "updateLookup", // Include full document on updates7  fullDocumentBeforeChange: "whenAvailable", // Include before-image (MongoDB 6.0+)8})910changeStream.on("change", (change) => {11  // change.operationType: 'insert' | 'update' | 'delete' | 'replace'12  // change.fullDocument: current document state13  // change.fullDocumentBeforeChange: previous state (if configured)14  // change._id: resume token for resumability15})

Key differences from relational CDC:

• Schema-free: Documents can vary; change events reflect actual structure
• Nested changes: Updates to nested fields captured as partial updates
• Resume tokens: Opaque tokens for resumability (vs. LSN/GTID)

Limitation: Change Streams require replica set or sharded cluster. Single-node MongoDB doesn’t support CDC.

Design Paths

Path 1: Debezium + Kafka Connect

Context: Open-source CDC platform. Most popular choice for self-managed CDC.

Architecture:

When to choose this path:

• Self-managed infrastructure with Kafka already in place
• Need sub-second latency
• Require full control over configuration and schema handling
• Multi-database environments

Key characteristics:

• One Kafka topic per table (configurable)
• Schema Registry integration for Avro/Protobuf/JSON Schema
• Exactly-once semantics with Kafka 3.3.0+ and KRaft
• Snapshot for initial data load, then streaming

Configuration example:

Show 2 hidden lines3  "config": {4    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",5    "database.hostname": "db.example.com",6    "database.port": "5432",7    "database.user": "debezium",8    "database.password": "${secrets:postgres/password}",9    "database.dbname": "myapp",10    "topic.prefix": "myapp",11    "table.include.list": "public.users,public.orders",12    "slot.name": "debezium_users",13    "publication.name": "dbz_publication",14    "snapshot.mode": "initial",Show 10 hidden lines

Trade-offs vs other paths:

Real-world: Shopify

Shopify retired their batch extraction service (“Longboat”) in favor of log-based CDC on Debezium + Kafka Connect, running ~150 connectors on Kubernetes against their sharded MySQL monolith. Schema evolution is mediated by Confluent Schema Registry; large tables use a custom snapshot mode that does not block binlog tailing.

For scale context, Tobi Lütke reported Shopify’s 2024 Black Friday peak at 284M edge requests/min and 66M Kafka messages/sec — that Kafka layer is downstream of the same CDC pipeline. The CDC stream itself is a fraction of total edge traffic, but the design point is “every committed write reaches Kafka without application participation.”

Path 2: AWS Database Migration Service

Context: Managed CDC service integrated with AWS ecosystem.

Architecture:

When to choose this path:

• AWS-centric infrastructure
• Prefer managed over self-managed
• Target is AWS service (S3, Redshift, DynamoDB)
• Batch/near-real-time acceptable (not sub-second)

Key characteristics:

• Full load + ongoing CDC in single task
• Automatic schema migration (optional)
• Built-in monitoring via CloudWatch
• No Kafka required (direct to S3/Redshift)

Limitations:

• Tables without primary keys: Skipped during CDC (critical gap)
• Latency: Seconds to minutes, not sub-second
• Large transactions: Can cause significant lag
• DDL propagation: Limited support; may require manual intervention

Cost model:

Path 3: Maxwell’s Daemon (MySQL-Specific)

Context: Lightweight MySQL CDC tool. Simpler than Debezium for MySQL-only environments.

Architecture:

When to choose:

• MySQL only
• Want simpler deployment than full Kafka Connect
• JSON output acceptable (no schema registry)
• Lower operational overhead priority

Output format:

1{2  "database": "myapp",3  "table": "users",4  "type": "update",5  "ts": 1706745600,6  "data": { "id": 1, "name": "Alice", "email": "alice@example.com" },7  "old": { "name": "Old Name" }8}

Trade-offs:

Comparison Matrix

Production Implementations

LinkedIn: Databus

Context: LinkedIn built Databus (2012) as one of the first production CDC systems. Open-sourced; influenced later designs.

Architecture:

Implementation details:

• Relay pattern: Relays pull from OLTP database, deserialize to Avro, store in circular memory buffer
• Bootstrap service: Provides full data snapshots for new consumers or catch-up
• Infinite lookback: New consumers can request full dataset without stressing production database
• Transactional ordering: Preserves commit order within source

Scale:

• Thousands of events/second per relay server
• Millisecond end-to-end latency
• Powers: Social Graph Index, People Search Index, member profile replicas

Key insight from LinkedIn:

“The relay maintains a sliding time window of changes in memory. Consumers that fall behind can catch up from the relay; consumers that fall too far behind bootstrap from a snapshot and then resume streaming.”

Airbnb: SpinalTap + Riverbed

Context: Airbnb uses CDC for their materialized views framework, processing billions of events daily.

SpinalTap (CDC layer):

• Scalable CDC across MySQL, DynamoDB, and internal storage
• Kafka as event transport
• Handles sharded monolith with transactional consistency

Riverbed (materialized views):

Scale (2024):

• 2.4 billion CDC events per day
• 350 million documents written daily to materialized views
• 50+ materialized views (search, payments, reviews, itineraries)
• Lambda architecture: Kafka (online) + Spark (offline)

What worked:

• GraphQL DSL for declarative view definitions
• Automatic schema evolution handling
• Real-time search index updates

Netflix: DBLog

Context: Netflix developed DBLog for CDC across heterogeneous databases.

Key innovation—incremental snapshots:

Traditional CDC: Full snapshot (locks table) → Start streaming

DBLog approach:

11. Start CDC streaming (no snapshot)22. Incrementally snapshot in chunks:3   - Select small range by primary key4   - Emit snapshot events5   - Continue streaming concurrently63. Reconcile snapshot with streaming at consumer

Benefits:

• No long-running locks or table copies
• Snapshot can be paused/resumed
• Works alongside live traffic

The watermark technique looks like this in practice — chunk-by-chunk SELECTs interleave with the live log via two marker writes that bracket each chunk:

The original algorithm is described in the DBLog paper (Andreakis et al., 2020); Debezium adopted it as its default ad-hoc snapshot mode.

Production since 2018:

• Powers Netflix’s Delta platform (data synchronization) and the broader Data Mesh movement / processing layer
• Studio applications event processing
• DBLog itself is RDBMS-only (MySQL, PostgreSQL); CockroachDB, Cassandra, and other non-relational stores feed Data Mesh via separate, source-specific connectors (e.g. CockroachDB changefeeds)

WePay: Cassandra CDC

Context: WePay (now part of Chase) built CDC for Cassandra, which lacks native CDC support.

Implementation:

Key design decisions:

• Agent per node: Each Cassandra node has a local CDC agent reading commit logs
• Primary agent pattern: Each agent is “primary” for a subset of partition keys, avoiding duplicates
• Exactly-once: Achieved at agent level through offset tracking

Open-sourced: Donated upstream and now lives as the Debezium Cassandra connector (still flagged as incubating). Unlike most Debezium connectors, it runs as a standalone JVM agent on each Cassandra node rather than as a Kafka Connect task — there is no central process that can read commit logs from a remote node.

Implementation Comparison

Schema Evolution

The Schema Challenge

CDC events must carry schema information. When source schema changes, downstream consumers must handle the evolution.

Problem scenarios:

1. Column added: New events have field; old events don’t
2. Column removed: Old events have field; new events don’t
3. Column renamed: Appears as remove + add
4. Type changed: INT → BIGINT, VARCHAR(50) → VARCHAR(100)

Schema Registry Integration

How it works:

1. CDC connector serializes event with schema
2. Schema registered in Schema Registry (if new)
3. Event includes schema ID reference (not full schema)
4. Consumer fetches schema by ID, caches locally
5. Consumer deserializes using fetched schema

Compatibility modes:

Recommended approach: BACKWARD_TRANSITIVE (all previous versions readable by latest)

Handling DDL Changes

Safe operations (backward compatible):

• Add nullable column
• Add column with default value
• Increase column size (VARCHAR(50) → VARCHAR(100))

Breaking operations (require coordination):

• Remove column
• Rename column
• Change column type
• Add NOT NULL column without default

Migration pattern for breaking changes:

The order matters — every intermediate state must be readable by both the previous and the next code version, otherwise the CDC stream serializes a state nobody can decode.

Debezium Schema Handling

Debezium can be configured to:

1{2  "schema.history.internal.kafka.topic": "schema-changes.myapp",3  "schema.history.internal.kafka.bootstrap.servers": "kafka:9092",4  "include.schema.changes": "true"5}

Schema change events:

1{2  "source": { "table": "users", "db": "myapp" },3  "ddl": "ALTER TABLE users ADD COLUMN phone VARCHAR(20)",4  "databaseName": "myapp",5  "tableChanges": [{6    "type": "ALTER",7    "id": "myapp.users",8    "table": {9      "columns": [...]10    }11  }]12}

Exactly-Once Semantics

The Delivery Challenge

CDC involves multiple hops where failures can occur:

1Database → CDC Connector → Kafka → Consumer → Target System

Each transition can fail after partial completion.

Kafka Exactly-Once (Since 0.11.0)

Idempotent producer:

1enable.idempotence=true

Producer assigns sequence number to each message. Broker deduplicates by (producer_id, sequence).

Transactional writes:

1producer.initTransactions();2producer.beginTransaction();3producer.send(record1);4producer.send(record2);5producer.commitTransaction(); // Atomic: all or nothing

Consumer isolation:

1isolation.level=read_committed

Consumer only sees committed transactional messages.

Debezium EOS (Kafka Connect 3.3+, KIP-618)

KIP-618 landed in Kafka 3.3.0 (Oct 2022) and exposed exactly-once semantics to Kafka Connect source connectors. Debezium opted in incrementally; Debezium 3.3.0 (Oct 2025) extended EOS to all core connectors (MariaDB, MongoDB, MySQL, Oracle, PostgreSQL, SQL Server).

Prerequisites per the Debezium EOS reference:

1. Kafka Connect 3.3+ in distributed mode (standalone mode is not supported).
2. Worker config exactly.once.source.support=enabled.
3. Connector config exactly.once.support=required.
4. Connector offset topic stored in Kafka (the default).

1exactly.once.source.support=enabled

1exactly.once.support=required

KRaft is not a hard requirement — EOS works against ZooKeeper-backed brokers too — but new Kafka 3.x clusters generally run KRaft, and ZooKeeper mode is removed in Kafka 4.0.

How it works:

1. Connector reads changes and writes the source offset inside a Kafka transaction.
2. Records + offset commit are written atomically; partial failures roll back.
3. On restart, the connector resumes from the last committed offset (any aborted transaction’s records are filtered out by read_committed consumers).

End-to-End Exactly-Once

For true end-to-end exactly-once:

Consumer-side idempotency:

Show 5 hidden lines6  await target.upsert(7    {8      id: key,9      data: change.after,10      _version: version,11    },12    {13      where: { _version: { lt: version } }, // Only apply if newer14    },15  )16}

CDC Consumer Patterns

Transactional Outbox Integration

The transactional outbox pattern (Chris Richardson) ensures reliable event publishing by writing events to a database table (outbox) within the same transaction as business data. CDC tails the outbox and replays it onto Kafka, replacing the dual-write anti-pattern from §“Why Naive Solutions Fail” with a single atomic commit.

CDC as outbox relay:

1-- Outbox table2CREATE TABLE outbox (3    id UUID PRIMARY KEY,4    aggregate_type VARCHAR(255),5    aggregate_id VARCHAR(255),6    type VARCHAR(255),7    payload JSONB,8    created_at TIMESTAMP DEFAULT NOW()9);1011-- Application writes to outbox in same transaction12BEGIN;13UPDATE users SET email = 'new@example.com' WHERE id = 123;14INSERT INTO outbox (id, aggregate_type, aggregate_id, type, payload)15VALUES (gen_random_uuid(), 'User', '123', 'EmailChanged', '{"email": "new@example.com"}');16COMMIT;

Debezium outbox transform:

1{2  "transforms": "outbox",3  "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",4  "transforms.outbox.table.field.event.key": "aggregate_id",5  "transforms.outbox.table.field.event.payload": "payload",6  "transforms.outbox.route.topic.replacement": "events.${routedByValue}"7}

Cache Invalidation

CDC enables event-driven cache invalidation without TTL guessing:

Implementation:

Show 8 hidden lines9  const table = change.source.table10  const key = change.after?.id ?? change.before?.id1112  // Invalidate cache entry13  await redis.del(`${table}:${key}`)1415  // Optional: warm cache with new value16  if (change.op !== "d" && change.after) {17    await redis.setex(`${table}:${key}`, 3600, JSON.stringify(change.after))18  }19}

Benefits over TTL:

• Immediate invalidation (sub-second vs. minutes/hours)
• No stale reads from long TTLs
• No thundering herd from short TTLs

Search Index Synchronization

CDC keeps search indices in sync with source of truth:

Kafka Connect Elasticsearch sink:

1{2  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",3  "topics": "myapp.public.products",4  "connection.url": "http://elasticsearch:9200",5  "type.name": "_doc",6  "key.ignore": "false",7  "schema.ignore": "true",8  "behavior.on.null.values": "delete"9}

Handling deletions:

• Debezium emits tombstone (null value) for deletes
• Sink connector translates tombstone to Elasticsearch delete
• Index stays synchronized including deletions

Analytics Pipeline Feeding

CDC enables real-time analytics without batch ETL:

Lambda architecture simplification:

Common Pitfalls

1. Replication Slot Disk Bloat (PostgreSQL)

What happens: CDC connector goes down or can’t keep up. PostgreSQL retains all WAL since the slot’s restart_lsn. Disk fills. Database crashes.

Example: Connector had a 2-hour network partition. 50 GB of WAL accumulated. Recovery required manual slot deletion and a full re-snapshot.

Solutions:

1-- Monitor slot lag2SELECT slot_name,3       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS lag,4       active5FROM pg_replication_slots;67-- Cap retained WAL per slot (PostgreSQL 13+ — default is -1 / unlimited)8ALTER SYSTEM SET max_slot_wal_keep_size = '10GB';910-- Alert on inactive slots11SELECT slot_name FROM pg_replication_slots WHERE NOT active;

Setting max_slot_wal_keep_size trades durability of the CDC stream for primary availability: once a slot exceeds the cap, PostgreSQL invalidates it and the connector must re-snapshot. Pick the value such that it covers your worst expected connector outage but leaves disk headroom — Gunnar Morling’s replication slot deep-dive is the best operational reference.

2. Tables Without Primary Keys

The mistake: Creating tables without primary keys, then adding them to CDC.

What happens: AWS DMS skips these tables entirely during CDC. Debezium can capture but updates/deletes can’t be keyed properly.

Example: Legacy table audit_log had no PK. Added to CDC scope. All changes captured as creates; updates appeared as new rows.

Solutions:

• Add primary keys to all tables before enabling CDC
• Use composite key if no natural key exists
• For truly keyless tables, add surrogate key column

3. Large Transaction Handling

The mistake: Running batch updates (millions of rows) during CDC operation.

What happens: Debezium buffers changes until transaction commits. Memory pressure. Downstream lag. Potential OOM.

Example: Nightly job updating 5M rows in single transaction. CDC connector memory spiked to 8GB, causing restart. Other tables’ CDC delayed by 30 minutes.

Solutions:

• Break large updates into batches with commits
• Configure Debezium memory limits
• Schedule large batch jobs during low-traffic windows
• Use incremental.snapshot for backfills

4. Snapshot + Streaming Race Conditions

The mistake: Not understanding snapshot isolation during initial load.

What happens: Snapshot reads table at point-in-time. Streaming starts from “after snapshot.” Changes during snapshot can be missed or duplicated.

Example:

1. Snapshot starts at LSN 100
2. Row inserted at LSN 150
3. Snapshot reads row (sees insertion)
4. Streaming starts at LSN 100
5. Streaming also captures insertion at LSN 150
6. Duplicate row in target

Solutions:

Debezium handles this correctly when configured properly:

1{2  "snapshot.mode": "initial",3  "snapshot.locking.mode": "minimal"4}

Consumer must be idempotent to handle potential duplicates during snapshot-to-streaming transition.

5. Schema Change During CDC

The mistake: Assuming DDL changes propagate seamlessly.

What happens:

• Column added: Old consumers fail parsing
• Column removed: Data loss if not handled
• Type changed: Deserialization errors

Example: Added phone column to users table. CDC captured the DDL. Downstream consumer’s Avro schema didn’t have phone. Consumer crashed with schema mismatch error.

Solutions:

• Use Schema Registry with BACKWARD compatibility
• Test schema changes in staging with CDC running
• Coordinate consumer deployments with schema changes
• Monitor for schema change events before production DDL

Implementation Guide

Starting Point Decision

Checklist for Production CDC

Database preparation:

• Enable logical replication/binary logging
• Create dedicated CDC user with minimal permissions
• Configure log retention appropriately
• Add primary keys to all tables in scope
• Test DDL change impact

Infrastructure:

• Kafka cluster sized for CDC throughput
• Schema Registry deployed and accessible
• Monitoring dashboards for connector lag
• Alerting on replication slot lag (PostgreSQL)
• Alerting on connector failures

Operational:

• Runbook for connector restart
• Runbook for re-snapshot after extended downtime
• Backup strategy for connector offsets
• Schema change coordination process
• Large transaction handling policy

Capacity Planning

Throughput estimation:

1CDC messages/sec ≈ (writes/sec to source tables) × (avg columns per table / 10)

Each CDC message size depends on row size and change type (update includes before/after).

Kafka sizing:

Conclusion

CDC transforms database changes into reliable event streams, enabling real-time data propagation without application-level dual writes. Log-based CDC—reading from WAL, binlog, or oplog—is the production standard, capturing all changes with minimal database impact.

Key decisions:

1. Log-based vs. polling: Log-based captures everything including deletes; polling is simpler but misses hard deletes and adds latency
2. Debezium vs. managed: Debezium offers sub-second latency and full control; managed services (DMS, Fivetran) reduce operational burden
3. Schema evolution strategy: Schema Registry with BACKWARD compatibility prevents consumer breakage

Critical operational concerns:

• PostgreSQL replication slot bloat is the most common production incident
• Large transactions can cause memory pressure and downstream lag
• Tables without primary keys create CDC gaps

Start simple: Single database → Debezium → Kafka → single consumer. Add complexity (schema registry, multiple sources, complex routing) as requirements demand.

Appendix

Prerequisites

• Database administration fundamentals (replication, transaction logs)
• Message broker concepts (Kafka topics, partitions, consumer groups)
• Distributed systems basics (eventual consistency, exactly-once semantics)

Terminology

Summary

• CDC extracts database changes from transaction logs without impacting write performance
• Log-based CDC (Debezium, DMS) is the production standard—captures all operations including deletes and DDL
• PostgreSQL uses logical replication slots; monitor max_slot_wal_keep_size to prevent disk bloat
• MySQL requires binlog_format=ROW and benefits from GTID for resumability across failover
• Exactly-once semantics require Kafka Connect 3.3+ in distributed mode (KIP-618); consumer-side idempotency keyed on source LSN/GTID is what closes the end-to-end loop
• Schema evolution needs Schema Registry with BACKWARD compatibility; coordinate schema changes with consumer deployments
• Transactional outbox pattern integrates naturally with CDC for reliable event publishing

References

Official Documentation:

• Debezium Documentation - Architecture, connectors, and configuration
• Debezium Exactly-Once Delivery - EOS prerequisites and connector configuration
• Debezium 3.3.0.Final release notes - EOS extended to all core connectors
• Debezium Incremental Snapshots - Watermark-based snapshot, post-Netflix DBLog
• PostgreSQL Logical Replication - Native PostgreSQL replication
• PostgreSQL Logical Decoding - WAL decoding internals
• PostgreSQL 17 Logical Replication Failover - failover = true slot synchronization
• MySQL Binary Log - Binlog configuration and format
• MySQL GTID - Global Transaction ID concepts
• MongoDB Change Streams - Change Stream API reference
• AWS DMS CDC - DMS ongoing replication
• AWS DMS pricing - On-demand replication instances and DMS Serverless DCU model

Engineering Blogs:

• LinkedIn: Open Sourcing Databus - Original Databus architecture
• Shopify: Capturing Every Change - CDC at Shopify scale
• Netflix: DBLog - Incremental snapshot approach
• Airbnb: SpinalTap - CDC for materialized views

Patterns and Best Practices:

• Transactional Outbox Pattern - Reliable event publishing pattern
• AWS Transactional Outbox - AWS implementation guide
• PostgreSQL Replication Slots Deep Dive - Operational guidance
• Advantages of Log-Based CDC - Comparison with other approaches

Kafka Exactly-Once:

• Kafka Exactly-Once Semantics - Confluent explanation
• KIP-98: Exactly Once Delivery - Original Kafka transactional producer proposal
• KIP-618: Exactly-Once Support for Source Connectors - Source-side EOS in Kafka Connect 3.3+