Sign inSign up

spoud/kafka-toolbox

By spoud

•Updated about 1 month ago

Image
0

10K+

spoud/kafka-toolbox repository overview

⁠SPOUD Kafka toolbox

There are 2 variants one is slim and can be used for producing / consuming messages and the other one is the full version which includes also build tools maven and gradle.

⁠Run the slim version

docker run -it --rm spoud/kafka-toolbox:slim
⁠Tools on slim version

⁠Run the full version

docker run -it --rm spoud/kafka-toolbox:latest
⁠Tools on full version
  • All tools from slim version
  • confluent
  • rpk
  • kafi
  • tmux
  • Maven
  • Gradle
  • Vim
  • Python
  • gotty
  • tcpdump
  • ksql tools / datagen

⁠Environment Variables

When environment variables are set, they will be used by the various tools in the toolbox to connect to the Kafka cluster and perform their operations.

VariableMeaning
BOOTSTRAP_SERVER / BOOTSTRAP_SERVERSHostname:port or comma-separated list of brokers used to connect to the Kafka cluster
PROTOCOLProtocol to be used for the connection (e.g. PLAINTEXT, SASL_SSL)
KAFKA_SASL_MECHANISMSASL mechanism to be used (e.g. PLAIN, SCRAM-SHA-256)
KAFKA_USERUsername for SASL authentication
KAFKA_PASSWORDPassword for SASL authentication
TLS_CAOptional path or inline content for TLS CA to configure mTLS / TLS trust for clients (applied to kafkactl, kcat, kaf, kafi)
TLS_CERTOptional path or inline content for client TLS certificate (applied to kafkactl, kcat, kaf, kafi)
TLS_KEYOptional path or inline content for client TLS private key (applied to kafkactl, kcat, kaf, kafi)
TLS_INSECUREOptional boolean (true/false). If true, skip TLS verification for tools that support this option (use with caution)
CCLOUD_API_KEYConfluent Cloud API Key
CCLOUD_API_SECRETConfluent Cloud API Secret
SCHEMA_REGISTRY_URLURL to the schema registry
SR_API_KEYSchema Registry API Key
SR_API_SECRETSchema Registry API Secret
CONNECT_URLURL to kafka connect rest endpoint

For Strimzi-managed OAUTHBEARER listeners, the slim and full images now also ship the Strimzi OAuth client classes, including io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler. That enables JAAS configs using oauth.access.token.location with projected or pre-obtained JWT token files.

For Keycloak federated client authentication, the slim and full images also ship io.spoud.oauth.KeycloakFederatedLoginCallbackHandler from kafka-oauth-extensions, so Kubernetes workloads can exchange a projected ServiceAccount token for a Keycloak access token without a static client secret.

⁠Local image builds

The toolbox image downloads kafka-oauth-extensions from the published GitHub release artifacts during the Docker build. The default version is 1.5-SNAPSHOT, which contains the federated Keycloak callback handler.

⁠Run on kubernetes

You can run the toolbox on kubernetes cluster as a pod. Here is an example manifest file:

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-toolbox-deployment
spec:
  selector:
    matchLabels:
      app: kafka-toolbox
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka-toolbox
    spec:
      securityContext:
        fsGroup: 1000660000   # choose an allowed GID from your cluster policy
      volumes:
      - name: home
        emptyDir:
          sizeLimit: "200Mi"
      - name: tmp
        emptyDir:
          sizeLimit: "200Mi"
      containers:
      - name: kafka-toolbox
        image: spoud/kafka-toolbox:latest
        command: ["/bin/bash"]
        args: ["-c", "mkdir -p /tmp /home/kafka && chmod 1777 /tmp || true && cp -R /etc/skel/. /home/kafka/ && chown -R $(id -u):$(id -g) /home/kafka/ || true; sleep infinity"]
        volumeMounts:
        - name: home
          mountPath: /home/kafka
        - name: tmp
          mountPath: /tmp
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          privileged: false
          runAsNonRoot: true
          capabilities:
            drop:
            - ALL
          seccompProfile:
            type: RuntimeDefault
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
            ephemeral-storage: "200Mi"
          limits:
            memory: "1Gi"
            cpu: "2000m"
            ephemeral-storage: "1Gi"
EOF

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-toolbox-slim-deployment
spec:
  selector:
    matchLabels:
      app: kafka-toolbox-slim
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka-toolbox-slim
    spec:
      securityContext:
        fsGroup: 1000660000   # choose an allowed GID from your cluster policy
      volumes:
      - name: home
        emptyDir:
          sizeLimit: "200Mi"
      - name: tmp
        emptyDir:
          sizeLimit: "200Mi"
      containers:
      - name: kafka-toolbox-slim
        image: spoud/kafka-toolbox:slim
        command: ["/bin/bash"]
        args: ["-c", "mkdir -p /tmp /home/kafka && chmod 1777 /tmp || true && cp -R /etc/skel/. /home/kafka/ && chown -R $(id -u):$(id -g) /home/kafka/ || true; sleep infinity"]
        volumeMounts:
        - name: home
          mountPath: /home/kafka
        - name: tmp
          mountPath: /tmp
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          privileged: false
          runAsNonRoot: true
          capabilities:
            drop:
            - ALL
          seccompProfile:
            type: RuntimeDefault
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
            ephemeral-storage: "200Mi"
          limits:
            memory: "1Gi"
            cpu: "2000m"
            ephemeral-storage: "1Gi"
EOF

⁠How to datagen

You can use the ksqlDB datagen tool to generate sample data into your Kafka topics. Here is an example command to generate data:

ksql-datagen quickstart=orders format=avro topic=orders maxInterval=1000 iterations=1000

Tag summary

Content type

Image

Digest

sha256:b6b6a4933…

Size

3.1 GB

Last updated

about 1 month ago

docker pull spoud/kafka-toolbox