🔐 Client-Side Field Level Encryption (CSFLE) & Client-Side Payload Encryption (CSPE)

This repository provides demos for implementing Client-Side Field Level Encryption (CSFLE) and Client-Side Payload Encryption (CSPE) with Confluent Cloud and Confluent Platform.

Encrypt sensitive data at the source before it ever reaches Kafka, ensuring end-to-end data protection and compliance with privacy regulations like GDPR, HIPAA, and CCPA.

This repository is intended for demonstration purposes only. If you plan to use it in production, you must review and strengthen the security practices accordingly.

⚙️ Generator

Want to scaffold a new CSFLE client without copy-pasting? Run the interactive generator from generator/:

Supported clients: Python, Java, Javascript, Go, .NET
Supported KMS: AWS, Azure, GCP, Hashicorp Vault
Supports Confluent Cloud and Confluent Platform

cd generator
uv run csfle-gen new

It asks a series of questions about your KMS provider and Confluent target, then writes a ready-to-run client under generated/<project_name>/.

The producer will generate 20 messages with the following AVRO schema. The birthday field will be encrypted.

{
  "name": "PersonalData",
  "type": "record",
  "namespace": "com.csfleExample",
  "fields": [
    {
      "name": "id",
      "type": "string"
    },
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "birthday",
      "type": "string",
      "confluent:tags": [
        "PII"
      ]
    },
    {
      "name": "timestamp",
      "type": [
        "string",
        "null"
      ]
    }
  ]
}

☁️ Confluent Cloud Examples

Step-by-step guidelines for different KMS integrations with Confluent Cloud:

Scenario	Client	Key Vault	Encryption Type	KEK Shared
AWS	Kotlin	AWS Key Management Service	CSFLE	❌
AWS	Python	AWS Key Management Service	CSFLE	❌
AWS Shared KEK	Kotlin	AWS Key Management Service	CSFLE	✅
Azure	Kotlin	Azure Key Vault	CSFLE	❌
Azure	Python	Azure Key Vault	CSFLE	❌
Azure CSPE	Kotlin	Azure Key Vault	CSPE	❌
Azure SM Connect	Kafka Connect	Azure Key Vault	CSFLE	❌
HashiCorp	Kotlin	HashiCorp Vault	CSFLE	❌
GCP	Kotlin	GCP Key Management Service	CSFLE	❌

🖥️ Confluent Platform Examples

Step-by-step guidelines for different KMS integrations with Confluent Platform:

Scenario	Client	Key Vault	Encryption Type	KEK Shared
AWS	Java	AWS Key Management Service	CSFLE	❌
AWS	Python	AWS Key Management Service	CSFLE	❌
AWS	.NET	AWS Key Management Service	CSFLE	❌
AWS	Go	AWS Key Management Service	CSFLE	❌
AWS	JavaScript	AWS Key Management Service	CSFLE	❌
Azure	Java	Azure Key Vault	CSFLE	❌
Azure	Python	Azure Key Vault	CSFLE	❌
Azure	.NET	Azure Key Vault	CSFLE	❌
Azure	Go	Azure Key Vault	CSFLE	❌
Azure	JavaScript	Azure Key Vault	CSFLE	❌

📋 Prerequisites

✅ Confluent Cloud cluster with Advanced Stream Governance package
✅ To use CSFLE with Confluent Platform in a production cluster, you must use Confluent Platform 8.0 or later
- ⚠️ Confluent Platform 7.9 introduces CSFLE already but as an Early Access feature and is not supported for production workloads
✅ For clients, Confluent Platform 7.4.2 or 7.5.1 are required

🎯 Goal

This demo shows how to protect sensitive personal data by encrypting specific fields before they're sent to Kafka.

Example Scenario

We produce personal data to Confluent Cloud/Confluent Platform in the following form:

{
  "id": "0",
  "name": "Anna",
  "birthday": "1993-08-01",
  "timestamp": "2023-10-07T19:54:21.884Z"
}

The birthday field is automatically encrypted using CSFLE before being sent to Kafka. When a consumer reads the data with the proper decryption configuration, the field is seamlessly decrypted.

Implementation

To demonstrate a realistic use case, we build complete producer and consumer applications (not just CLI commands) using modern programming languages like Kotlin, Python, and Java.

📚 Official Documentation & Resources