Change Data Capture (CDC)

Introduction

Modern distributed systems rarely consist of a single database and a single application. Instead, large-scale systems include multiple services that must react to changes in data across the system.

For example:

• When a user updates their profile, several systems might need to know:
  • Search indexes
  • Analytics systems
  • Recommendation engines
  • Notification services
  • Cache layers

If every service constantly queries the database for updates, the database becomes overwhelmed.

This is where Change Data Capture (CDC) becomes essential.

Change Data Capture (CDC) is a technique that detects changes in a database and streams those changes to other systems in real time.

Instead of repeatedly asking:

“Has anything changed?”

CDC allows systems to say:

“Notify me whenever something changes.”

The Problem CDC Solves

Imagine a simple e-commerce system.

We have:

• Orders database
• Inventory system
• Analytics pipeline
• Notification service
• Search index

When a new order is created, several downstream systems must update.

Without CDC, the system might look like this.

Problems:

CDC solves this by streaming database changes once and distributing them to multiple consumers.

CDC Architecture Overview

With CDC, the system becomes event-driven.

Flow:

1. A change happens in the database.
2. CDC captures the change.
3. CDC converts the change into an event.
4. The event is published to a streaming platform.
5. Multiple services consume the event.

This creates a loosely coupled architecture.

Types of Database Changes Captured

CDC typically captures three kinds of operations.

Example table:

Users

If Alice changes her email:

UPDATE Users SET email="alice@newmail.com" WHERE id=1;

CDC emits an event:

{
  "operation": "UPDATE",
  "table": "Users",
  "before": {
    "id": 1,
    "email": "alice@email.com"
  },
  "after": {
    "id": 1,
    "email": "alice@newmail.com"
  }
}

This event can be consumed by many systems.

Where CDC Is Used

CDC is heavily used in modern architectures.

Example:

CDC Implementation Strategies

There are several ways to implement CDC.

1. Polling Based CDC

The simplest approach is periodic polling.

A service repeatedly queries the database for changes.

Example:

SELECT * FROM orders
WHERE updated_at > last_checked_time;

Diagram

Problems

Because of these limitations, large systems rarely rely on polling.

2. Trigger Based CDC

Another approach uses database triggers.

A trigger runs automatically when a row changes.

Example:

CREATE TRIGGER order_update_trigger
AFTER UPDATE ON orders
FOR EACH ROW
INSERT INTO order_events VALUES (NEW.id, NOW());

Architecture

Advantages

• Immediate capture of changes
• Simple implementation

Disadvantages

Large-scale systems often avoid heavy trigger usage.

3. Log-Based CDC (Most Scalable)

The most powerful CDC method reads the database transaction log.

Every database maintains a log of changes.

Examples:

Instead of polling tables, CDC tools read the log stream.

Architecture

Advantages:

This approach is used in production systems.

CDC Event Pipeline

A typical CDC pipeline looks like this.

Components:

Data Flow Example

Consider an order creation event.

Step-by-step flow.

One change can trigger multiple downstream updates.

CDC Event Structure

A CDC event typically contains metadata.

Example:

{
  "event_id": "9821",
  "table": "orders",
  "operation": "INSERT",
  "timestamp": "2026-01-10T10:30:00Z",
  "data": {
    "order_id": 1001,
    "user_id": 42,
    "amount": 120
  }
}

Common fields:

Handling Ordering and Consistency

CDC systems must maintain event ordering.

Why?

Because updates must occur in the same sequence.

Example:

1. Order created
2. Order shipped
3. Order delivered

If delivered arrives before shipped, systems break.

To ensure order:

• Use log offsets
• Use partitioning keys
• Use ordered message streams

Idempotency in CDC

Consumers may receive duplicate events.

Example causes:

• Retry
• Consumer restart
• Network failure

Consumers must process events idempotently.

Example strategy:

if event_id already_processed:
    ignore
else:
    process_event()

Handling Schema Changes

Database schemas evolve.

Example:

ALTER TABLE users ADD COLUMN phone;

CDC pipelines must handle:

Real World Example

Consider an online marketplace.

When a product changes:

• Search index must update
• Recommendation engine must retrain
• Analytics must record metrics
• Cache must invalidate

Architecture:

Instead of services querying the database, they react to events.

Benefits of CDC

Challenges of CDC

CDC introduces complexities.

These challenges require careful system design.

Best Practices

Use Log-Based CDC

Prefer transaction log based CDC over polling.

Ensure Idempotent Consumers

Consumers must handle duplicate events safely.

Maintain Event Ordering

Partition events using a consistent key.

Use Schema Versioning

Support backward compatibility.

Monitor the CDC Pipeline

Track:

• event lag
• processing delays
• failed consumers

Summary

Change Data Capture enables real-time propagation of database changes across distributed systems.

Instead of services repeatedly querying databases, CDC allows systems to stream events whenever data changes.

Key ideas:

CDC is a foundational technology for building event-driven, scalable, and loosely coupled architectures in modern distributed systems.

Without CDC, large-scale systems would struggle to keep multiple services synchronized with continuously changing data.