Crabka is a Rust implementation of Apache Kafka infrastructure. It speaks the Kafka wire protocol, stores records in Kafka-compatible log segments, runs metadata on KRaft, and is tested against the official JVM clients and command-line tools.
Use Crabka when you want Kafka-compatible streaming infrastructure without a JVM runtime: memory-safe Rust, async I/O, no ZooKeeper mode, no GC pauses, and a workspace that includes the broker, Rust clients, Schema Registry, gateways, operators, rebalancing, replication, and observability services.
Crabka is beta, pre-1.0 software. The workspace version is defined in Cargo.toml.
The project is still greenfield infrastructure: there are no production users, and on-disk compatibility across Crabka versions is not promised yet. Use it for evaluation, development, interoperability testing, and non-critical workloads while the project hardens.
Kafka compatibility is the primary constraint. The repository validates protocol encoding, record formats, storage behavior, KRaft metadata, and JVM tool interoperability against Apache Kafka behavior where those surfaces are in scope.
- Kafka wire compatibility: protocol codecs are generated from Apache Kafka
message schemas and checked byte-for-byte against
kafka-clients. - JVM tooling works: acceptance tests drive tools such as
kafka-topics.sh,kafka-configs.sh,kafka-acls.sh,kafka-consumer-groups.sh,kafka-leader-election.sh, andkafka-reassign-partitions.shagainst Crabka. - Rust runtime: Crabka uses
tokio, forbids unsafe code across the workspace, and avoids JVM heap tuning and garbage-collection behavior. - KRaft-native: metadata is stored in a native KRaft quorum; ZooKeeper mode and ZooKeeper-to-KRaft migration are out of scope.
- Operations included: the workspace includes a Kubernetes operator, Prometheus metrics, OTLP tracing, Helm charts, OCI images, and a Cruise-Control-style partition rebalancer.
- Rust clients included: producer, consumer, admin, streams, schema-serde, gateway, connector, and replication crates live in the same repository.
Crabka targets Kafka's wire, storage, and operational semantics. JVM implementation internals are not compatibility goals.
| Area | Status |
|---|---|
| Wire protocol and API version negotiation | Implemented |
| Kafka-compatible record batches, compression, and log segments | Implemented |
| KRaft metadata quorum and controller records | Implemented |
| Replication, ISR maintenance, leader election, and reassignment | Implemented |
| Idempotent and transactional produce / consume | Implemented |
| Classic and next-generation consumer groups | Implemented |
| Share groups / queues | Implemented |
| Tiered storage | Implemented; segment-data JVM interop is still being validated |
| TLS, SASL, delegation tokens, ACLs, and quotas | Implemented |
| Schema Registry-compatible REST service | Implemented |
| Kubernetes operator | Implemented; external listener surfaces are still maturing |
| Rust Streams client | Partial versus the full JVM Kafka Streams library |
| Kafka Connect-equivalent runtime | Partial; connector SPI exists and continues to evolve |
| ZooKeeper mode and ZooKeeper-to-KRaft migration | Out of scope |
For the detailed per-KIP breakdown, see docs/KIP_MATRIX.md.
Crabka is a Rust workspace. The pinned toolchain is in rust-toolchain.toml.
git clone https://github.com/robot-head/crabka.git
cd crabka
cargo build --workspaceInstall the local broker and CLI binaries from a checkout:
cargo install --path crates/cli
cargo install --path crates/brokerRust client crates are published independently. For example:
cargo add crabka-client-producer
cargo add crabka-client-consumer
cargo add crabka-client-adminContainer images are published to GHCR and Docker Hub:
docker pull ghcr.io/robot-head/crabka-broker:latest
docker pull mirror.gcr.io/robothead/crabka-broker:latestImage build, signing, SBOM, and attestation details are in packaging/README.md. Helm chart usage is documented in charts/README.md.
Start a single local broker from the source tree:
export CRABKA_CLUSTER_ID=00000000-0000-0000-0000-000000000001
rm -rf target/crabka-data
cargo run -p crabka-cli --bin crabka -- format \
--log-dir target/crabka-data \
--cluster-id "$CRABKA_CLUSTER_ID" \
--standalone \
--node-id 1 \
--controller-listener 127.0.0.1:9093
cargo run -p crabka-broker --bin crabka-broker -- \
--log-dir target/crabka-data \
--cluster-id "$CRABKA_CLUSTER_ID" \
--broker-id 1 \
--listen-addr 127.0.0.1:9092In another shell, use normal Kafka tooling against the broker:
kafka-topics.sh \
--bootstrap-server 127.0.0.1:9092 \
--create \
--topic demo \
--partitions 1 \
--replication-factor 1
kafka-console-producer.sh \
--bootstrap-server 127.0.0.1:9092 \
--topic demo
kafka-console-consumer.sh \
--bootstrap-server 127.0.0.1:9092 \
--topic demo \
--from-beginningcrabka format initializes an empty log directory. To start over locally, stop
the broker and remove target/crabka-data.
- KIP implementation matrix
- Contributing guide
- Container image docs
- Helm chart docs
- Benchmark harness
- Style guides
- docs.rs package documentation
- Project website
Crabka is organized as a Cargo workspace. The main runtime path is:
flowchart LR
clients[Kafka and Crabka clients] --> broker[crabka-broker]
broker --> log[Kafka-compatible log]
broker --> kraft[KRaft metadata quorum]
broker --> remote[Tiered storage]
broker --> telemetry[Metrics / logs / traces]
operator[crabka-operator] --> broker
registry[crabka-schema-registry] --> broker
gateway[crabka-grpc-gateway] --> broker
rebalancer[crabka-rebalancer] --> broker
replicator[crabka-replicator] --> broker
| Layer | Key crates |
|---|---|
| Broker runtime | crabka-broker, crabka-cli, crabka-authz, crabka-security, crabka-telemetry |
| Protocol, records, and storage | crabka-protocol, crabka-log, crabka-raft, crabka-metadata, crabka-remote-storage |
| Rust clients | crabka-client-core, crabka-client-producer, crabka-client-consumer, crabka-client-admin, crabka-client-streams |
| Services and integration | crabka-schema-registry, crabka-grpc-gateway, crabka-connect, crabka-connect-postgres, crabka-replicator |
| Operations and observability | crabka-operator, crabka-rebalancer, crabka-bench-driver, crabka-blockstore, crabka-metrics, crabka-observability |
| Postgres-compatible engine (Chapter Gres) | crabka-gres, crabka-gres-control, crabka-gres-balancer, crabka-pgexec, crabka-pgwire, crabka-pgtypes, crabka-pgparser, crabka-pgkv, crabka-pgmvcc, crabka-pgcatalog, crabka-gres-fdw |
Crate READMEs and rustdoc contain API-level usage details.
Prerequisites:
- Rust toolchain from rust-toolchain.toml
- JDK 17 for JVM differential tests
- Docker or a compatible container runtime for integration tests that use Kafka containers
Common checks:
cargo build --workspace
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspaceRun JVM-backed differential and acceptance tests:
(cd tools/oracle && ./gradlew installDist)
cargo test --workspace -- --include-ignoredRegenerate protocol code after editing Kafka schemas:
./tools/regenerate.sh
git diff crates/protocol/generatedMore contributor workflow details are in CONTRIBUTING.md.
Near-term work focuses on production hardening and compatibility depth:
- More JVM interop coverage for edge-case protocol and storage behavior.
- Continued Kubernetes operator maturity.
- More complete Connect runtime and connector surfaces.
- Better deployment, security, and operations documentation.
- Compatibility and upgrade testing as the project approaches 1.0.
Detailed implementation status lives in docs/KIP_MATRIX.md and the design notes under docs/superpowers/specs.
Contributions are welcome. Start with CONTRIBUTING.md, open an issue for substantial design or compatibility changes, and keep Kafka wire and behavior compatibility as the primary constraint.
Run cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings,
and the relevant tests before opening a pull request. Conventional commits are
used by release-plz for automated versioning and changelog generation.
Crabka includes authentication, authorization, TLS, mTLS, delegation-token, and OPA integration work, but it is still beta infrastructure. Do not use it as the sole security boundary for critical production systems yet.
If you believe you have found a security vulnerability, avoid posting exploit details in a public issue. Use GitHub private vulnerability reporting if it is enabled for the repository, or contact the maintainers privately through the repository owner.
Crabka is licensed under the Apache License, Version 2.0. See LICENSE and NOTICE.
Crabka is a derivative, compatibility-focused implementation of Apache Kafka protocols, record formats, and operational semantics. The project depends on the Apache Kafka schema corpus and JVM client/tool behavior as its compatibility oracle.