A production-inspired distributed build orchestration platform built in Go.
Engineering at Scale demonstrates the architecture, tooling, and engineering practices behind modern developer platforms — distributed systems, cloud-native infrastructure, CI/CD automation, observability, and developer experience.
Modern software engineering extends far beyond writing application code.
Platform teams build the infrastructure that enables hundreds of developers to ship software safely and efficiently. This project explores the design and implementation of production-inspired platform engineering:
| Domain | What this project covers |
|---|---|
| Distributed Systems | Job queuing, worker pools, concurrent execution |
| Container Orchestration | Docker, Kubernetes, Helm, horizontal scaling |
| CI/CD Automation | GitHub Actions, build pipelines, artifact lifecycle |
| Build Infrastructure | Job scheduling, metadata, artifact storage |
| Observability | Prometheus metrics, Grafana dashboards, OpenTelemetry tracing |
| Developer Experience | Go CLI, REST API, documentation-first development |
| Cloud Native | Service discovery, secrets management, configuration management |
✅ REST API Job submission, status, cancellation
✅ Go CLI easctl — developer-facing command line interface
✅ Docker Compose Local development environment, single command startup
✅ GitHub Actions CI pipeline — lint, test, build on every push
✅ Project Structure Production-grade layout following Go standards
⬜ Redis Job Queue Priority queues, retry logic, dead letter handling
⬜ Worker Pool Concurrent job execution with graceful shutdown
⬜ PostgreSQL Build metadata, job history, artifact lineage
⬜ Prometheus Metrics Latency, throughput, queue depth, worker utilization
⬜ Grafana Dashboards Operational visibility out of the box
⬜ OpenTelemetry Distributed tracing across services
⬜ Kubernetes Horizontal scaling, service discovery, Helm charts
⬜ Artifact Storage Build outputs with versioning and retention policies
⬜ Web Dashboard Real-time build status and metrics UI
⬜ Authentication RBAC, API keys, multi-tenancy
# Clone
git clone https://github.com/hydrangeas20/engineering-at-scale.git
cd engineering-at-scale
# Start local environment (API + dependencies)
make dev
# Or with Docker Compose directly
docker compose up
# Submit a job via CLI
./easctl job submit --name "my-build" --image "ubuntu:22.04" --cmd "echo hello"
# Check job status
./easctl job status <job-id>
# View API docs
open http://localhost:8080/docsengineering-at-scale/
├── .github/
│ └── workflows/
│ └── ci.yml # GitHub Actions CI pipeline
├── assets/
│ ├── architecture.png # System architecture diagram
│ ├── project-progress-card.png # Current release / progress card
│ └── roadmap.png # Version roadmap graphic
├── docker/
│ └── Dockerfile # Multi-stage production build
├── docs/
│ └── adr/
│ ├── 001-go-as-primary-language.md
│ ├── 002-redis-for-job-queue.md
│ ├── 003-postgresql-for-metadata.md
│ └── 004-prometheus-for-metrics.md
├── .gitignore
├── docker-compose.yml # Local development stack
├── go.mod # Go module definition
├── Makefile # Developer commands
└── README.md # Project overview
This project is built with production engineering practices from day one.
Every feature is designed with:
- Simplicity over cleverness — readable, maintainable code over clever optimisations
- Observability by default — metrics, logs, and traces are not afterthoughts
- Incremental delivery — every merge leaves the system in a working state
- Documentation-first — ADRs capture why decisions were made, not just what
- Testability — interfaces and dependency injection make components testable in isolation
- Reproducibility — Docker and Makefile targets ensure consistent local and CI environments
- Fail safely — graceful shutdown, retries, and dead letter queues over silent failures
Week 1 ──────────────────────────────────────────────────
Repository structure · Architecture docs
Go CLI (easctl) · REST API · Docker Compose · CI
Week 2 ──────────────────────────────────────────────────
Redis job queue · Worker pool
PostgreSQL metadata · Job scheduling
Week 3 ──────────────────────────────────────────────────
Prometheus metrics · Structured logging
Grafana dashboards · OpenTelemetry tracing
Week 4 ──────────────────────────────────────────────────
Kubernetes manifests · Helm chart
Horizontal scaling · Service discovery
Week 5–6 ──────────────────────────────────────────────────
v1.0 Production MVP · End-to-end demo
Performance benchmarks · Full documentation
| Layer | Technology | Why |
|---|---|---|
| Language | Go 1.26 | Performance, concurrency, strong ecosystem for platform tooling |
| API | Chi router + Go net/http | Lightweight, idiomatic, no heavy framework overhead |
| Queue | Redis Streams | Reliable delivery, consumer groups, built-in retry |
| Database | PostgreSQL | ACID guarantees for build metadata and artifact lineage |
| Containers | Docker + Kubernetes | Industry standard for isolated job execution and scaling |
| Metrics | Prometheus + Grafana | Pull-based metrics, rich dashboards, operator familiarity |
| Tracing | OpenTelemetry | Vendor-neutral distributed tracing across services |
| CI/CD | GitHub Actions | Native VCS integration, matrix builds, artifact publishing |
| Config | Viper | 12-factor app configuration, env + file + flags |
| Logging | zerolog | Structured JSON logging, zero-allocation hot path |
Major design decisions are documented as ADRs in docs/adr/.
| ADR | Decision | Status |
|---|---|---|
| 001 | Go as primary language | ✅ Accepted |
| 002 | Redis Streams for job queue | ✅ Accepted |
| 003 | PostgreSQL for build metadata | ✅ Accepted |
| 004 | Prometheus for metrics | ✅ Accepted |
# Install dependencies
make deps
# Run tests
make test
# Run with hot reload
make watch
# Lint
make lint
# Build binary
make build
# Build Docker image
make docker-build
# Tear down
make downEngineering at Scale is a long-term platform engineering project exploring how modern cloud-native infrastructure is designed, built, deployed, and operated.
The project follows an incremental release model where each version introduces production-inspired platform capabilities.