AQA Kafka Test Framework (Streams + Avro + Docker E2E)

A production-style Kafka automation framework built on Java + Gradle + JUnit 5 that demonstrates end-to-end and integration testing approaches for Kafka-based systems:

• Kafka Streams topology testing (stateless + stateful, high-level + low-level APIs)
• Embedded Kafka cluster for integration tests (ZooKeeper + Kafka Broker + Confluent Schema Registry)
• Avro + Schema Registry (generation + serialization)
• HTTP → Kafka → HTTP materialization demo pipeline (Dropwizard + WireMock)
• E2E tests in Docker via Testcontainers + Docker Compose

---

Contents

• Project Highlights
• Tech Stack
• Modules Overview
• Project Structure
• Prerequisites
• Quick Start
• How to Run Tests
• Avro Code Generation
• Data Pipeline (Local Run)
• CI (GitHub Actions)
• Troubleshooting

---

Project Highlights

✅ Kafka Streams testing

• Unit tests for topologies using TopologyTestDriver (fast, deterministic, no real broker)
• Examples for:
  • Stateless processing (e.g., mapping/uppercase)
  • Stateful processing with state stores (e.g., counting/grouping)
  • Avro SerDes (Confluent SerDe / Schema Registry)
  • Low-level Processor API (Transformer + StateStore) for advanced scenarios (e.g., deduplication)

✅ Embedded Kafka Cluster (integration tests)

• Local in-process cluster to run “real” integration tests:
  • ZooKeeper (Curator TestingServer)
  • Kafka Broker
  • Confluent Schema Registry (RestApp)
• Programmatic topic creation/deletion, bootstrap discovery, Schema Registry URL access.

✅ Data Pipeline demo (microservices-style)

• HTTP Producer (Dropwizard) → writes Avro events to Kafka
• HTTP Materializer → consumes Kafka, transforms, and writes to a database REST service
• Integration tests with WireMock + Awaitility (retries/timeouts for async pipelines)

✅ Docker E2E

• E2E test suite using:
  • Testcontainers (Kafka container / GenericContainer)
  • DockerComposeContainer (compose-based environment)
  • Rest Assured for system verification
• Both patterns are shown:
  • “Bring up everything from docker-compose”
  • “Manually wire containers into a shared network”

---

Tech Stack

Build & Test

• Gradle (Wrapper), multi-module setup
• JUnit 5 (Jupiter)
• Awaitility (async assertions)
• Rest Assured (HTTP assertions)

Kafka & Streaming

• Apache Kafka Streams (+ streams test utils)
• Kafka Clients

Avro & Schema Registry

• Apache Avro
• Confluent Schema Registry
• Confluent Avro Serializer / Streams Avro SerDe
• Avro code generation via com.github.davidmc24.gradle.plugin.avro

Infra / E2E

• Testcontainers (Kafka + JUnit integration)
• Docker Compose (compose-based E2E)
• WireMock (service stubbing)

Utilities

• SLF4J + Log4j
• Typesafe Config
• Lombok

---

Modules Overview

Module	Purpose	Key tech
src (root module)	Kafka Streams topology tests + producer/consumer integration examples	TopologyTestDriver, Kafka Streams, Avro
kafka-cluster	Embedded cluster for integration tests	ZooKeeper, Kafka Broker, Schema Registry
data-pipeline:http-producer	HTTP service producing Avro messages into Kafka	Dropwizard, Confluent Avro
data-pipeline:http-materializer	Consumer/materializer that writes to REST DB service	Kafka Streams/Clients, WireMock, Awaitility
e2e-docker	End-to-end tests in Docker	Testcontainers, DockerComposeContainer, Rest Assured

---

Project Structure

.
├─ readme.md
├─ readme.png
├─ build.gradle
├─ settings.gradle
├─ kafka-cluster/
│  └─ src/main/java/kafka/...
├─ src/
│  ├─ main/java/
│  │  ├─ topology/...
│  │  ├─ avroModels/...
│  │  └─ jsonModels/...
│  └─ test/java/
│     ├─ streams/...
│     └─ producer_consumer/...
├─ data-pipeline/
│  ├─ docker-compose.yml
│  ├─ http-producer/
│  └─ http-materializer/
└─ e2e-docker/
   └─ src/test/...

---

Prerequisites

• Java 11+ (recommended: Temurin)
• Docker (required for e2e-docker tests and Testcontainers)
• Linux/macOS/Windows (GitHub Actions runner uses Linux)

---

Quick Start

# run all tests (all modules)
./gradlew clean test

Run a specific module:

./gradlew :kafka-cluster:test
./gradlew :e2e-docker:test
./gradlew :data-pipeline:http-producer:test
./gradlew :data-pipeline:http-materializer:test

---

How to Run Tests

Run everything

./gradlew clean test

Run a single test class

./gradlew test --tests "streams.StreamProcessingTest"

Run Integration tests (*IT)

Gradle’s test task may not include *IT naming by default in some setups.

Option A — run by pattern

./gradlew test --tests "*IT"

Option B — include *IT in Gradle config (recommended) Add this to your root build.gradle:

test {
    useJUnitPlatform()
    include '**/*Test.class', '**/*IT.class'
}

---

Avro Code Generation

This project uses the Gradle Avro plugin to generate Java models from .avsc.

Root module schemas

./gradlew generateAvro

Data pipeline schemas

./gradlew :data-pipeline:http-materializer:generateAvro
./gradlew :data-pipeline:http-producer:generateAvro

Tip: After generation, reimport Gradle in IDE so generated sources are indexed.

---

Data Pipeline (Local Run)

The data-pipeline module includes a sample docker-compose.yml and a flow diagram.

Useful commands:

# list topics
docker-compose exec kafka kafka-topics --list --zookeeper zookeeper:2181

# create a topic
docker-compose exec kafka kafka-topics --create --zookeeper zookeeper:2181   --replication-factor 1 --partitions 1 --topic events-message-v1

---

CI (GitHub Actions)

A typical CI pipeline for this repository:

• checks out code
• sets up Java
• enables Gradle cache
• runs ./gradlew clean test

The CI badge at the top uses a relative GitHub Actions badge URL that works when this README is placed in the repository root and the workflow is located at .github/workflows/tests.yml.

---

Troubleshooting

Testcontainers can’t start Docker

• Ensure Docker is running locally
• On Linux runners: Docker is available by default
• On Windows/macOS: make sure Docker Desktop is started

*IT tests not running

Use:

./gradlew test --tests "*IT"

Or add include '**/*IT.class' into test { ... } as shown above.

Avro models not found / compilation errors

Run generation tasks:

./gradlew generateAvro
./gradlew :data-pipeline:http-producer:generateAvro
./gradlew :data-pipeline:http-materializer:generateAvro