peekaboo51490/cdc-pg-connect

By peekaboo51490

•Updated 3 months ago

Prebuilt Kafka Connect image with Debezium PostgreSQL CDC.

Image

Message queues

Databases & storage

2.7K

Overview Tags

peekaboo51490/cdc-pg-connect repository overview

⁠cdc-pg-connect

A production-ready Kafka Connect image for PostgreSQL Change Data Capture (CDC) and PostgreSQL JDBC integrations.

The image comes pre-packaged with:

Debezium PostgreSQL Connector
Confluent JDBC Connector
PostgreSQL JDBC Driver

This eliminates the need to download or install connectors during container startup, making deployments faster, reproducible, and CI/CD friendly.

⁠Included Components

Component	Version
Confluent Platform Kafka Connect	6.2.0
Kafka Connect JDBC Connector	10.7.4
Debezium PostgreSQL Connector	1.9.7
PostgreSQL JDBC Driver	42.6.0

⁠Supported Use Cases

⁠PostgreSQL CDC

Capture database changes directly from PostgreSQL Write-Ahead Logs (WAL) using Debezium.

Examples:

User events
Audit streams
Event sourcing
Data lake ingestion
Search indexing pipelines

⁠PostgreSQL JDBC Source

Read data from PostgreSQL tables and publish records to Kafka topics.

⁠PostgreSQL JDBC Sink

Consume Kafka topics and write records into PostgreSQL tables.

⁠Docker Image

Pull the image from Docker Hub:

docker pull peekaboo51490/cdc-pg-connect:latest

⁠Running with Docker Compose

Example Kafka Connect deployment using Redpanda as the Kafka-compatible broker.

connect:
  image: peekaboo51490/cdc-pg-connect:latest
  container_name: debezium-connect

  depends_on:
    postgres:
      condition: service_healthy

    redpanda:
      condition: service_healthy

  environment:
    CONNECT_BOOTSTRAP_SERVERS: "redpanda:9092"

    CONNECT_GROUP_ID: "kafka-connect-group"

    CONNECT_CONFIG_STORAGE_TOPIC: "connect-config"
    CONNECT_OFFSET_STORAGE_TOPIC: "connect-offsets"
    CONNECT_STATUS_STORAGE_TOPIC: "connect-status"

    CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
    CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
    CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"

    CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
    CONNECT_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"

    CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
    CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"

    CONNECT_REST_ADVERTISED_HOST_NAME: "connect"
    CONNECT_REST_PORT: "8083"

    CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"

  ports:
    - "8083:8083"

  restart: unless-stopped

⁠Verify Installed Connectors

After Kafka Connect starts, verify the available plugins:

curl http://localhost:8083/connector-plugins | jq

Expected connectors:

io.debezium.connector.postgresql.PostgresConnector
io.confluent.connect.jdbc.JdbcSourceConnector
io.confluent.connect.jdbc.JdbcSinkConnector

⁠Example: PostgreSQL CDC Connector

curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d @postgres-cdc.json

Example connector configuration:

{
  "name": "ab-users-source",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.dbname": "identity_db",
    "database.user": "debezium",
    "database.password": "debezium",
    "database.server.name": "ab-identity-svc",
    "table.include.list": "public.ab_users",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_ab_users",
    "publication.name": "debezium_pub_ab_users",
    "publication.autocreate.mode": "disabled",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "true",
    "errors.log.enable": "true",
    "errors.log.include.messages": "true"
  }
}

⁠Health Check

Verify Kafka Connect is healthy:

curl http://localhost:8083/

Expected response:

{
  "version": "...",
  "commit": "...",
  "kafka_cluster_id": "..."
}

⁠Notes

Connectors are installed during image build time.
No runtime plugin downloads are required.
Optimized for PostgreSQL CDC workloads.
Compatible with Apache Kafka and Redpanda.
Additional databases require their respective JDBC drivers to be added to the image.

Tag summary

Recent tags

Content type

Image

Digest

sha256:902870dc5…

Size

870.4 MB

Last updated

3 months ago

docker pull peekaboo51490/cdc-pg-connect