PythonWebhookRelay

Production-style webhook relay system built with FastAPI, Async SQLAlchemy, Kafka, Redis, Celery, Docker, and Kubernetes.

Project Goal

PythonWebhookRelay accepts incoming webhooks, stores them safely, queues them through Kafka, and delivers them asynchronously with retry and dead-letter handling.

This project demonstrates real backend reliability patterns.

Infrastructure Setup (Important)

Kafka, Redis, and MySQL are running on the physical host machine (Ubuntu), not inside Docker/Kubernetes.

That means host-level configs were adjusted for pod connectivity:

• Kafka listeners and advertised.listeners
• Redis bind
• MySQL bind-address and user host grants

Tech Stack

• Python 3.12
• FastAPI (async)
• Pydantic v2
• SQLAlchemy 2.0 (Async ORM) + asyncmy
• MySQL 8 (external)
• Redis (external)
• Kafka (external)
• Celery
• Docker
• Kubernetes (Minikube)

Core Features

• Unified API response format for success/failure/validation/exception
• Idempotency support via header/body key
• Async persistence to MySQL
• Kafka producer with key-based publishing (event_id)
• Multi-consumer architecture using Kafka consumer groups
• Celery worker delivery pipeline with retry/backoff
• Dead-letter flow after max retries
• Health and readiness endpoints
• Kubernetes deployment with API, worker, and consumer components

Unified API Response Contract

Every API response follows:

{
  "status": 200,
  "messages": ["success"],
  "data": {}
}

API Endpoints

• GET /v1/healthz
• GET /v1/readyz
• POST /v1/webhooks/incoming
• GET /v1/events/{event_id}
• GET /v1/events/{event_id}/deliveries

High-Level Flow

1. Client sends webhook to ingest API.
2. API validates payload and idempotency key.
3. Event is stored in MySQL.
4. Event is published to Kafka topic webhook.events using event_id as key.
5. Kafka consumer receives message and enqueues Celery task.
6. Celery worker calls target URL.
7. Delivery attempts are stored.
8. On repeated failure, event moves to dead-letter table and DLQ topic.

Data Tables

• webhook_events
• delivery_attempts
• dead_letters

Design note: No foreign-key constraints are used by project rule. Integrity is enforced at service/application layer.

Environment Variable Convention

Environment variables are grouped by prefix:

• APP_*
• UVICORN_*
• MYSQL_*
• REDIS_*
• KAFKA_*
• CELERY_*

Prerequisites

• Ubuntu 24
• Python 3.12
• Docker
• Minikube (--nodes=3 --driver=docker)
• kubectl
• Running host services: MySQL, Redis, Kafka

Host Service Connectivity Checklist

MySQL

• bind-address = 0.0.0.0
• DB user grant for Minikube subnet (example: 192.168.49.%)
• Port 3306 reachable from Minikube nodes

Redis

• Proper bind config for Minikube access
• Port 6379 reachable

Kafka

• Correct listeners and advertised.listeners
• Broker reachable from pods
• Topics created:
  • webhook.events (multiple partitions)
  • webhook.events.dlq

Local Run (without Kubernetes)

source venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload

Worker:

celery -A app.workers.celery_app.celery_app worker --loglevel=info --concurrency=2

Kafka consumer:

python -m app.workers.run_kafka_consumer

Docker Build and Push

docker build -f docker/Dockerfile.api -t <dockerhub_user>/pythonwebhookrelay-api:1.0.1 .
docker build -f docker/Dockerfile.worker -t <dockerhub_user>/pythonwebhookrelay-worker:1.0.1 .

docker push <dockerhub_user>/pythonwebhookrelay-api:1.0.1
docker push <dockerhub_user>/pythonwebhookrelay-worker:1.0.1

Kubernetes Deploy

kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/secret.yaml
kubectl apply -f k8s/api-deployment.yaml
kubectl apply -f k8s/api-service.yaml
kubectl apply -f k8s/worker-deployment.yaml
kubectl apply -f k8s/kafka-consumer-deployment.yaml

Verify:

kubectl get deployments -n python-webhook-relay
kubectl get pods -n python-webhook-relay
kubectl get svc -n python-webhook-relay

Access API (NodePort)

Your service config:

• Service name: python-webhook-relay-service
• Type: NodePort
• NodePort: 30007

Get Minikube IP:

minikube ip

Use:

http://<minikube-ip>:30007

Example:

curl -s http://<minikube-ip>:30007/v1/healthz
curl -s http://<minikube-ip>:30007/v1/readyz

Quick Smoke Test

curl -s -X POST http://<minikube-ip>:30007/v1/webhooks/incoming \
  -H "Content-Type: application/json" \
  -H "X-Idempotency-Key: demo-001" \
  -d '{
    "event_type":"payment.succeeded",
    "source":"billing-service",
    "target_url":"http://<minikube-ip>:30007/v1/test/receiver",
    "payload":{"amount":100,"currency":"USD"}
  }'

Then:

curl -s http://<minikube-ip>:30007/v1/events/<event_id>
curl -s http://<minikube-ip>:30007/v1/events/<event_id>/deliveries

Production Patterns Demonstrated

• Stateless API suitable for multiple uvicorn workers and replicas
• At-least-once delivery semantics
• Service + repository layering
• Externalized config using env vars
• Liveness/readiness probes
• Rolling deployment support

Common Troubleshooting

• no route to host: Minikube context/network issue
• connect refused MySQL/Redis/Kafka: host service bind/listener/firewall issue
• Access denied user@ip: MySQL host grant/user mismatch
• cryptography package required: add cryptography dependency for MySQL auth plugin
• probe failures: inspect API deployment logs first