CP Testcontainers

Table of Contents

• 1. Features
• 2. Quick Start
  • 2.1. Basic Usage
  • 2.2. Kafka Connect with Custom Connectors
  • 2.3. RBAC-Enabled Setup
• 3. Supported Components
• 4. Installation
  • 4.1. Gradle
  • 4.2. Maven
• 5. Example Use Cases
  • 5.1. S3 Integration Testing
  • 5.2. Multi-Version Testing
• 6. Troubleshooting
  • 6.1. Common Issues
    • 6.1.1. No Available IPv4 Addresses
    • 6.1.2. Out of Memory Errors
• 7. Contributing
• 8. License

A collection of Testcontainers modules for Confluent Platform components, designed to simplify integration testing of Kafka-based applications.

1. Features

• Supports all major Confluent Platform components

• Easy setup of RBAC and MDS-enabled environments

• Integration with Confluent Hub connectors

• Compatible with Confluent Platform 7.8.0

• Requires Java 17 or later

2. Quick Start

2.1. Basic Usage

Start Kafka and Schema Registry:

final var factory = new CPTestContainerFactory();

// Create and start Kafka and Schema Registry
final KafkaContainer kafka = factory.createKafka();
final SchemaRegistryContainer schemaRegistry = factory.createSchemaRegistry(kafka);
schemaRegistry.start(); // implicitly starts kafka container

// Configure client properties
Properties properties = new Properties();
properties.put("bootstrap.servers", kafka.getBootstrapServers());
properties.put("schema.registry.url", schemaRegistry.getBaseUrl());

2.2. Kafka Connect with Custom Connectors

Start a single-node Kafka Connect cluster with connectors from Confluent Hub:

final var connect = factory.createCustomConnector(
    Set.of(
        "confluentinc/kafka-connect-s3:latest",
        "confluentinc/kafka-connect-datagen:0.4.0"
    ),
    kafka
);
connect.start();

2.3. RBAC-Enabled Setup

Configure CP-Server and Schema Registry with RBAC:

final var factory = new CPTestContainerFactory(network);
final var ldap = factory.createLdap();

final var confluentServer = factory.createConfluentServer()
    .enableRbac();

final var sr = factory.createSchemaRegistry(confluentServer)
    .enableRbac();
sr.start();

3. Supported Components

Component	RBAC Support	Description
Confluent Server	✅	Core Kafka component with enterprise features
Schema Registry	✅	Schema management and compatibility
Kafka Connect	✅ (without secret registry)	Data integration framework
ksqlDB	✅	Stream processing engine
REST Proxy	✅	RESTful interface to Kafka
Replicator	❌ (investigating OOM issues)	Cross-cluster data replication

4. Installation

4.1. Gradle

repositories {
    maven { url 'https://jitpack.io' }
}

dependencies {
    testImplementation 'com.github.testcontainers-all-things-kafka:cp-testcontainers:Tag'
}

4.2. Maven

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>

<dependency>
    <groupId>com.github.testcontainers-all-things-kafka</groupId>
    <artifactId>cp-testcontainers</artifactId>
    <version>Tag</version>
    <scope>test</scope>
</dependency>

5. Example Use Cases

5.1. S3 Integration Testing

See LocalStackIntTest for an example of setting up an S3 sink connector with LocalStack.

5.2. Multi-Version Testing

See MultiVersionTest for examples of testing against different Confluent Platform versions.

6. Troubleshooting

6.1. Common Issues

6.1.1. No Available IPv4 Addresses

Status 404: {"message":"could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network"}

Solution: Clean up unused Docker networks:

docker network prune

6.1.2. Out of Memory Errors

If you encounter OOM errors with Kafka Connect, try increasing the heap size:

final var connect = factory.createKafkaConnect(kafka)
    .withEnv("KAFKA_HEAP_OPTS", "-Xmx2G -Xms1G");

7. Contributing

1. Fork the repository

2. Create a feature branch

3. Submit a Pull Request

8. License

See License