Part V — Deployment & Operations

Table of contents

• What You’ll Have at the End
• The Chapters
  • Build It
  • Share and Change It
  • Extend and Replicate It
  • Put It Together
• The Payments Question
• Practice

Parts I to IV use the lab as make all builds it. This Part is about building and running a cluster yourself: installing the Strimzi operator, Kafka and Kates, and then securing, sharing, upgrading and extending what you’ve built.

You don’t read this Part straight through, and none of it is required for Parts II to IV. The first four chapters are for the day you stand up a cluster; the others are for the day their job comes up.

What You’ll Have at the End

It depends on the chapters you pick. Read the build chapters and you’ll have the operator, a kafka-cluster release and the Kates stack installed and verified, sized for your environment. Read the others and you’ll have the procedures for day-2 work: onboarding a tenant, upgrading each component with a way back, and running Kafka Connect and MirrorMaker 2.

The Chapters

The chapters fall into four groups. Read the first group in install order, and reach for the others when their trigger comes up.

Build It

Read these when you stand up a cluster, in the order you install things: the operator, then Kafka, then Kates. Installing Kafka is the step-by-step walkthrough, Kafka Deployment Engineering gives the reasons behind its choices, and the Deployment Guide is about Kates rather than Kafka.

• Deploying the Strimzi Operator: how do you install, migrate and upgrade the operator that every Kafka cluster here depends on? About 30 minutes.
• Installing Kafka with the kafka-cluster Helm Chart: how do you deploy and verify a Kafka cluster with the chart, step by step? About 70 minutes.
• Kafka Deployment Engineering: why is the krafter Kafka cluster built the way it is, from node pools to listeners and certificates? About 25 minutes.
• Deployment Guide: how do you size and deploy the Kates stack, on Kind or on EKS, GKE or AKS? About 30 minutes.

Share and Change It

Read these when other teams join the cluster, or when a new version is out. The Upgrade Playbook uses Kates to check its own work: record a baseline, upgrade, run the same tests again and compare.

• Security & Compliance: who can reach the cluster, with which rights, and how do you audit that? About 30 minutes.
• Multi-Tenancy: how do several teams share krafter without getting in each other’s way? About 10 minutes.
• Upgrade Playbook: in what order do you upgrade each component, and how do you know you can still roll back? About 15 minutes.

Extend and Replicate It

Read these when you move data into or out of Kafka, or between clusters. MirrorMaker 2 runs as a Kafka Connect cluster, so the Connect chapters come first. If you run neither Connect nor MirrorMaker 2, skip all four.

• Kafka Connect & CDC Pipelines: how do you build a change-data-capture pipeline from PostgreSQL with Kafka Connect and Debezium? About 25 minutes.
• Operating Kafka Connect: how do you keep a Connect cluster healthy through scaling, tuning, credential rotation, upgrades and recovery? About 20 minutes.
• Cross-Cluster Replication and Migration: how do you copy records and consumer offsets from one Kafka cluster to another with MirrorMaker 2? About 10 minutes.
• Migrating a Legacy Kafka Source: what changes when the cluster you migrate from runs an old Kafka version? About 15 minutes.

Put It Together

The last chapter closes this Part but draws more on Parts II and III: it combines their commands into end-to-end procedures.

• Recipes & Patterns: how do you validate an upgrade, run a nightly regression suite or certify resilience, from start to finish? About 5 minutes.

The Payments Question

This Part moves the payments question from the lab to a cluster you run. In Multi-Tenancy a tenant is one block of kafka-cluster chart values in tenants.yaml, applied with one helm upgrade, and the payments workload becomes the payments tenant: a payments-events topic and a KafkaUser with prefix ACLs on payments. The Upgrade Playbook’s re-test becomes payments-scenarios.yaml, run before and after an upgrade, and kates report compare <pre-id>,<post-id> compares the two runs. On that cluster, the answer comes from the same gates, grades and verdicts as in Parts II and III.

Practice

For the build chapters, Tutorial 8 — Deploy, Detect & Clean checks a cluster with kates detect, deploys the stack with the interactive kates deploy -i, and removes it with kates clean. Tutorial 7: Kyverno & Security goes with Security & Compliance: it manages the Kyverno admission policies and runs a security audit.

For Connect, Tutorial 9: Kafka Connect Working Examples (CDC + JDBC) builds a Debezium CDC pipeline and JDBC connectors, and the Kafka Connect Source/Sink Quick Runbook follows one row from PostgreSQL through Kafka to a replica table. For MirrorMaker 2, start with Tutorial 10: Installing and Setting Up MirrorMaker 2. Then rehearse a migration on Kind with Tutorial 11: Migrating Kafka 2.x to 4.x with MirrorMaker 2 or Tutorial 12: Migrating Kafka 3.x to 4.x with MirrorMaker 2.

Tutorial 6: CI/CD Integration goes with Recipes & Patterns: it turns performance, integrity and chaos checks into pipeline gates, and schedules nightly runs.