████████╗ █████╗ ███████╗██╗ ██╗███████╗██╗██████╗ ███████╗
╚══██╔══╝██╔══██╗██╔════╝██║ ██╔╝██╔════╝██║██╔══██╗██╔════╝
██║ ███████║███████╗█████╔╝ █████╗ ██║██████╔╝█████╗
██║ ██╔══██║╚════██║██╔═██╗ ██╔══╝ ██║██╔══██╗██╔══╝
██║ ██║ ██║███████║██║ ██╗██║ ██║██║ ██║███████╗
╚═╝ ╚═╝ ╚═╝╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚══════╝
A production-grade distributed background job engine with a priority queue, DAG dependency resolver, and real-time dashboard.
Getting Started · Architecture · API Reference · How It Works · Contributing
Taskfire is a self-hosted background job processing system built for engineers who need more than a simple task queue. Jobs are submitted through a typed REST API, persisted in PostgreSQL, and brokered through a Redis sorted-set priority queue to a pool of dynamically-scaling Go goroutines. The worker engine enforces DAG-based job dependencies — a job with unmet prerequisites stays blocked until every ancestor completes — and retries failures with per-type exponential backoff. A React 18 dashboard connects over WebSocket and renders live queue depth, per-minute throughput, worker utilization, and a dead-letter queue browser — all without a page refresh. Prometheus scrapes the worker's /metrics endpoint every 10 seconds, giving you 15 days of TSDB retention and a ready-made target for Grafana.
┌─────────────────────────────────────────────────────────────────────────┐
│ Browser │
│ Dashboard :3000 ──────────────────────────────────► API :8080 │
│ (React 18 · Vite · Tailwind) REST + WebSocket (Fastify · Zod) │
└──────────────────────────────────────────────────────────┬──────────────┘
│
┌──────────────┴──────────────┐
│ │
┌─────────▼──────────┐ ┌─────────────▼────────────────────────┐
│ PostgreSQL :5432 │ │ Redis :6379 │
│ jobs │ │ taskfire:queue:high (sorted set) │
│ job_dependencies │ │ taskfire:queue:medium (sorted set) │
│ job_logs │ │ taskfire:queue:low (sorted set) │
│ job_metrics │ │ taskfire:delayed (sorted set) │
│ cron_jobs │ │ taskfire:processing (hash map) │
└─────────┬───────────┘ │ taskfire:dlq (list) │
│ │ taskfire:lock:<id> (string TTL) │
│ └──────────────┬────────────────────────┘
│ │ BZPOPMIN (blocking pop)
│ ▼
│ ┌──────────────────────────────────────┐
│ │ Go Worker Pool │
│ │ ┌────────┐ ┌────────┐ ┌────────┐ │
│ │ │ Worker │ │ Worker │ │ Worker │ │ MinWorkers–MaxWorkers
│ │ │ #1 │ │ #2 │ │ #N │ │ scale on queue depth
│ │ └────────┘ └────────┘ └────────┘ │
│ │ │
│ │ ┌───────────────────────────────┐ │
│ │ │ DAG Dependency Engine │ │
│ │ │ Kahn's BFS · cycle detection │ │
│ │ └───────────────────────────────┘ │
│ │ │
│ │ ┌───────────────────────────────┐ │
│ │ │ Exponential Backoff Retry │ │
│ │ │ per-type config · jitter │ │
│ │ └───────────────────────────────┘ │
│ │ │
│ │ ┌───────────────────────────────┐ │
│ │ │ Cron Scheduler │ │
│ │ │ robfig/cron · delayed poller │ │
│ │ └───────────────────────────────┘ │
└───────────────│ job status writes · log appends │
└──────────────────┬────────────────────┘
│ /metrics
▼
┌──────────────────────────────────────┐
│ Prometheus :9090 │
│ 10 s scrape · 15-day TSDB retention │
└──────────────────────────────────────┘
| Component | Technology | Why |
|---|---|---|
| Worker engine | Go 1.22 | Goroutines make it trivial to run hundreds of concurrent workers with minimal overhead; the runtime scheduler handles preemption without OS threads |
| Message broker | Redis 7 sorted sets | BZPOPMIN gives atomic, blocking priority dequeue in O(log N); sorted sets make scheduling by score (timestamp or priority weight) a first-class operation |
| Distributed locking | Redis SET NX PX |
Single-command compare-and-set with TTL prevents duplicate processing across restarts without a separate coordination service |
| Persistence | PostgreSQL 15 | JSONB payloads for schema-free job data, composite indexes for queue-claim queries, job_dependencies adjacency list for DAG traversal |
| API server | Fastify + TypeScript | Fastest Node.js HTTP framework by raw throughput; Zod schemas give compile-time and runtime type safety on every request body |
| Real-time push | WebSocket (@fastify/websocket) |
Sub-second latency to the dashboard without polling; Redis pub/sub fans out job-state events to every connected browser |
| Dashboard | React 18 + Vite + Tailwind | Concurrent rendering for smooth live updates; Vite's ESM dev server with HMR for instant feedback; Tailwind utility classes eliminate stylesheet overhead |
| Data fetching | TanStack Query v5 | Stale-while-revalidate caching, automatic background refetch, and devtools built in — no custom fetch layer needed |
| Charts | Recharts | Composable SVG chart primitives that compose naturally with React's rendering model |
| Metrics | Prometheus client (Go) | Idiomatic instrumentation with counters, histograms, and gauges; Prometheus scrape model works without inbound access to the worker |
| Containers | Docker + Compose | Multi-stage Dockerfiles produce minimal images (~15 MB for the Go binary, ~50 MB for Node); Compose wires health checks and dependency ordering |
Jobs are assigned high, medium, or low priority at submission time. Each maps to a Redis sorted set (taskfire:queue:high/medium/low) scored by enqueue timestamp, giving strict FIFO ordering within a lane. The dequeue Lua script checks high → medium → low in a single round-trip, ensuring high-priority work is never delayed by a backlog of lower-priority jobs.
Any job can declare dependencies: [uuid, ...] at creation time. Before a dequeued job begins execution, the Go worker queries the job_dependencies table and performs a BFS traversal up to 50 levels deep to verify every ancestor has status = completed. If any dependency is still pending or failed, the job is re-enqueued rather than executed. Cycle detection runs at submission time using Kahn's topological sort — circular dependency graphs are rejected with a 422 before they ever reach the queue.
The pool starts WORKER_MIN goroutines and scales up to WORKER_MAX based on queue depth. A background scaler goroutine checks depth every 5 seconds: if depth exceeds the high watermark (50 jobs) and headroom exists, it spawns new workers; if depth drops below the low watermark (5 jobs), idle workers are signaled to exit. Workers track per-goroutine statistics (jobs processed, errors, last job start time) using atomic int64 counters — zero lock contention on the hot path.
Each job type can register a custom retry configuration with BaseDelay, MaxDelay, MaxRetries, and Multiplier. The default profile uses 500 ms base, 30 s cap, 5 retries, and a 2× multiplier. Each retry interval is jittered ±10% to prevent thundering herd on a failing downstream. After MaxRetries exhausted, the job transitions to dead status and moves to the dead-letter queue, where it remains visible and replayable from the dashboard.
The scheduler runs inside the Go worker process using robfig/cron with second-granularity (six-field) expressions. Scheduled jobs land in a Redis sorted set (taskfire:delayed) scored by their Unix timestamp. A 1-second tick polls the sorted set and atomically moves any job whose score has passed into its priority lane — no separate scheduler process required.
The React dashboard opens a WebSocket to the API on load and receives a full state snapshot immediately, then incremental metrics pushes every ~2 seconds. It renders: live queue depth by priority lane, a 60-minute throughput time series (jobs/minute), failure rate with an alert badge above 5%, per-worker utilization, and a paginated dead-letter queue browser with one-click retry.
Failed jobs that exhaust all retry attempts are archived to the DLQ with their last error message, retry count, and timestamp. The dashboard's DLQ panel lists them with full payload inspection and a Retry button that re-enqueues the job at its original priority, resetting the retry counter.
The worker exposes a /metrics endpoint with:
| Metric | Type | Labels |
|---|---|---|
taskfire_jobs_processed_total |
Counter | job_type, priority |
taskfire_jobs_failed_total |
Counter | job_type, failure_reason |
taskfire_retry_attempts_total |
Counter | job_type |
taskfire_job_processing_duration_seconds |
Histogram | job_type |
taskfire_queue_depth_gauge |
Gauge | priority |
taskfire_worker_utilization_gauge |
Gauge | worker_id |
taskfire_dead_letter_queue_size |
Gauge | — |
taskfire_active_workers_total |
Gauge | — |
| Tool | Minimum version |
|---|---|
| Docker | 24.x |
| Docker Compose | v2.x (plugin) |
| Go | 1.22 (local dev only) |
| Node.js | 20.x (local dev only) |
git clone https://github.com/aaravshah/taskfire.git
cd taskfire
docker compose up --buildAll services start with health checks and dependency ordering. Once everything is healthy:
| Service | URL |
|---|---|
| Dashboard | http://localhost:3000 |
| API | http://localhost:8080 |
| Prometheus | http://localhost:9090 |
| Redis | localhost:6379 |
| PostgreSQL | localhost:5432 |
curl -s -X POST http://localhost:8080/api/jobs \
-H 'Content-Type: application/json' \
-d '{
"type": "send-email",
"payload": { "to": "[email protected]", "subject": "Hello from Taskfire" },
"priority": "high",
"max_retries": 3
}' | jq .Start only infrastructure (Redis + Postgres):
docker compose up redis postgresThen in separate terminals:
# Install Node.js deps (first time only)
make install
# Go worker
make worker-dev
# Fastify API
make api-dev
# React dashboard (Vite dev server with HMR on :5173)
make dashboard-devmake test # all suites
make test-worker # Go: testify + miniredis
make test-api # Jest + supertest
make test-dashboard # Vitest + React Testing LibraryAll endpoints are prefixed with /api. Errors return { "error": "<message>" } with an appropriate HTTP status code.
| Method | Path | Description |
|---|---|---|
POST |
/api/jobs |
Enqueue a new job |
GET |
/api/jobs |
List jobs with filters and pagination |
GET |
/api/jobs/:id |
Fetch a single job by UUID |
DELETE |
/api/jobs/:id |
Cancel a pending job |
POST |
/api/jobs/:id/retry |
Re-queue a failed or dead-letter job |
GET |
/api/jobs/:id/logs |
Fetch structured execution logs |
curl -X POST http://localhost:8080/api/jobs \
-H 'Content-Type: application/json' \
-d '{
"type": "process-video",
"payload": { "video_id": "abc123", "resolution": "1080p" },
"priority": "high",
"max_retries": 5,
"scheduled_at": "2025-06-01T12:00:00Z",
"dependencies": ["d290f1ee-6c54-4b01-90e6-d701748f0851"]
}'| Field | Type | Required | Description |
|---|---|---|---|
type |
string | ✓ | Identifies the handler to invoke |
payload |
object | ✓ | Arbitrary JSON passed to the handler |
priority |
"high" | "medium" | "low" |
— | Defaults to "medium" |
max_retries |
integer | — | Defaults to 3 |
scheduled_at |
ISO 8601 | — | Enqueue immediately if omitted |
dependencies |
UUID[] | — | Job will not run until all listed jobs are completed |
curl 'http://localhost:8080/api/jobs?status=failed&priority=high&page=1&limit=25'| Parameter | Type | Description |
|---|---|---|
status |
string | pending | active | completed | failed | dead |
priority |
string | high | medium | low |
type |
string | Filter by job type |
page |
integer | 1-based page number |
limit |
integer | Results per page (max 100) |
curl http://localhost:8080/api/jobs/d290f1ee-6c54-4b01-90e6-d701748f0851Cancels a pending job. Returns 409 if the job is already active or completed.
curl -X DELETE http://localhost:8080/api/jobs/d290f1ee-6c54-4b01-90e6-d701748f0851Re-queues a failed or dead job at its original priority, resetting the retry counter to 0.
curl -X POST http://localhost:8080/api/jobs/d290f1ee-6c54-4b01-90e6-d701748f0851/retrycurl http://localhost:8080/api/jobs/d290f1ee-6c54-4b01-90e6-d701748f0851/logsReturns an array of { level, message, metadata, timestamp } log entries written by the handler during execution.
| Method | Path | Description |
|---|---|---|
GET |
/api/metrics/summary |
Queue depth by lane, status counts, failure rate, avg processing time |
GET |
/api/metrics/throughput |
Jobs completed/failed per minute for the last 60 minutes |
GET |
/api/metrics/workers |
Active worker count, in-flight job list, Redis discrepancy |
GET |
/api/metrics/dead-letter |
Paginated DLQ with limit and offset query params |
curl http://localhost:8080/api/metrics/summary{
"queue_depth": { "high": 4, "medium": 12, "low": 1, "delayed": 3, "total": 20 },
"dlq_depth": 2,
"counts": { "pending": 20, "active": 3, "completed": 8941, "failed": 47, "dead": 2, "total_processed": 8988 },
"failure_rate": 0.0052,
"avg_processing_ms": 142
}Returns 60 data points, one per minute, zero-filled for minutes with no activity.
[
{ "time": "2025-01-01T10:00:00Z", "completed": 14, "failed": 1 },
{ "time": "2025-01-01T10:01:00Z", "completed": 22, "failed": 0 }
]{
"active_workers": 8,
"in_flight_redis": 8,
"discrepancy": 0,
"active_jobs": [
{ "job_id": "abc", "job_type": "send-email", "started_at": "...", "running_for_ms": 340 }
],
"processing_ids_redis": ["abc", "def", "..."]
}curl 'http://localhost:8080/api/metrics/dead-letter?limit=25&offset=0'Connect to ws://localhost:8080/ws.
Server → Client events:
Client → Server:
{ "type": "ping" }| Variable | Default | Description |
|---|---|---|
REDIS_URL |
redis://redis:6379 |
Redis connection URL (uses Docker service name internally) |
DATABASE_URL |
postgresql://taskfire:taskfire@postgres:5432/taskfire |
PostgreSQL connection URL |
POSTGRES_USER |
taskfire |
Postgres user (docker compose init) |
POSTGRES_PASSWORD |
taskfire |
Postgres password |
POSTGRES_DB |
taskfire |
Postgres database name |
API_PORT |
3000 |
Port the Fastify server binds to inside the container |
CORS_ORIGIN |
http://localhost:3000 |
Allowed CORS origin(s) |
JWT_SECRET |
— | ≥ 32-char secret for signing tokens |
NODE_ENV |
production |
development | production | test |
WORKER_MIN |
4 |
Minimum goroutines in the worker pool |
WORKER_MAX |
32 |
Maximum goroutines in the worker pool |
METRICS_PORT |
9090 |
Worker Prometheus /metrics port |
LOG_LEVEL |
info |
debug | info | warn | error |
VITE_API_URL |
http://localhost:8080 |
API base URL baked into the dashboard bundle |
VITE_WS_URL |
ws://localhost:8080 |
WebSocket origin baked into the bundle |
The queue is built on three Redis sorted sets — taskfire:queue:high, taskfire:queue:medium, and taskfire:queue:low. When a job is enqueued, it is ZADDed to the appropriate set with a score equal to its Unix nanosecond timestamp, establishing strict FIFO ordering within each lane. Dequeue is handled by a Lua script executed atomically on the Redis server:
1. BZPOPMIN taskfire:queue:high (blocking, 1 s timeout)
2. If empty → BZPOPMIN taskfire:queue:medium
3. If empty → BZPOPMIN taskfire:queue:low
4. On hit:
a. SET taskfire:lock:<job_id> 1 NX PX 30000 (30 s distributed lock)
b. HSET taskfire:processing <job_id> <payload>
c. Return payload to caller
The lock prevents a re-enqueued retry from being claimed by two workers simultaneously if the original worker stalls before acknowledging. Ack removes the hash entry and deletes the lock. Nack removes the hash entry, deletes the lock, and re-adds the job to its priority lane for another worker to claim.
Delayed jobs are stored in taskfire:delayed scored by their scheduled Unix timestamp. The cron scheduler's 1-second poller calls ZRANGEBYSCORE taskfire:delayed 0 <now>, atomically removes matching members with ZREM, and ZADDs them into their priority lane.
Job dependencies are modeled as a directed acyclic graph stored in the job_dependencies table (an adjacency list of (job_id, depends_on_job_id) edges). When a job is submitted with dependencies, the engine:
- Validates the proposed graph by loading the full transitive closure of the new job's ancestors and running Kahn's algorithm. If
|sorted| < |nodes|, a cycle was detected and the submission is rejected with422 Unprocessable Entity. - Gates execution at dequeue time: the worker loads all direct and transitive predecessors up to 50 BFS levels deep, queries their statuses in a single
SELECT, and checks that every ancestor hasstatus = 'completed'. If any ancestor ispending,active, orfailed, the job is re-enqueued with a short delay rather than executed immediately.
This means dependency checks are enforced by the worker, not the API, so they survive process restarts — a job will keep re-checking its ancestors until all are complete.
Each job type can be registered with a custom retry.Config:
type Config struct {
BaseDelay time.Duration // initial wait (default: 500 ms)
MaxDelay time.Duration // ceiling (default: 30 s)
MaxRetries int // attempts before DLQ (default: 5)
Multiplier float64 // growth factor (default: 2.0)
}The delay for attempt n is:
delay = min(BaseDelay × Multiplier^n, MaxDelay) × jitter
jitter ∈ [0.9, 1.1] (uniform random ±10%)
Jitter prevents multiple failing jobs of the same type from retrying in lockstep and hammering a recovering downstream service. After MaxRetries attempts, the job's status is set to dead, it is written to the Redis DLQ list, and its final error is stored in Postgres for inspection.
taskfire/
├── worker/ Go worker engine
│ ├── main.go Startup, signal handling, graceful shutdown
│ ├── pool/ Dynamic goroutine pool with watermark autoscaling
│ ├── queue/ Redis priority queue (Lua atomic scripts)
│ ├── processor/ Job executor: handler registry, retry, DLQ routing
│ ├── scheduler/ robfig/cron + delayed-job 1 s poller
│ ├── dag/ Kahn's BFS topological sort + dependency gating
│ ├── retry/ Per-type exponential backoff with jitter
│ ├── metrics/ Prometheus registry: counters, histograms, gauges
│ └── Dockerfile Multi-stage: golang:1.22 builder → alpine runtime
├── api/ Node.js Fastify API
│ ├── src/
│ │ ├── index.ts Server bootstrap, plugin registration, shutdown
│ │ ├── routes/ jobs.ts · metrics.ts
│ │ ├── services/ redis.ts · postgres.ts (singleton clients)
│ │ ├── websocket/ handler.ts (snapshot, heartbeat, pub/sub fan-out)
│ │ └── types/ job.ts (Zod schemas + TypeScript interfaces)
│ └── Dockerfile Multi-stage: node:20 builder → slim runtime
├── dashboard/ React 18 SPA
│ ├── src/
│ │ ├── App.tsx Shell: sidebar navigation + dark mode toggle
│ │ ├── components/ MetricCard · JobTable · ThroughputChart ·
│ │ │ FailureRateChart · QueueDepthChart ·
│ │ │ WorkerUtilization · DeadLetterPanel
│ │ ├── hooks/ useWebSocket · useJobs
│ │ ├── api/ client.ts (Axios + axios-retry + ApiError)
│ │ └── types/ job.ts
│ └── Dockerfile Multi-stage: node:20 Vite build → nginx static
├── postgres/
│ └── init.sql Schema, enums, triggers, covering indexes, views
├── nginx/
│ └── nginx.conf Rate limiting, gzip, security headers, SPA fallback
├── prometheus/
│ └── prometheus.yml 10 s scrape interval, worker target
├── docker-compose.yml Six-service orchestration with health checks
├── Makefile Dev, build, test, lint targets
├── .env Local development environment variables
└── .env.example Annotated environment variable template
make dev Start all services (docker compose up --build -d)
make infra Start Redis + Postgres only
make build Build all Docker images
make migrate Apply postgres/init.sql schema
make logs Tail logs from all services
make stop Stop all running containers
make clean Remove containers, volumes, and built binaries
make ps Show running container status
make test Run all test suites
make test-worker Go worker tests (testify + miniredis)
make test-api Node.js API tests (Jest + supertest)
make test-dashboard React component tests (Vitest + RTL)
make lint go vet + eslint across all packages
make install npm ci for api/ and dashboard/
make worker-dev Run Go worker locally (requires local Go)
make api-dev Run Fastify API locally
make dashboard-dev Run Vite dev server on :5173 with HMR
Contributions are welcome. Here's how to get oriented:
# Fork and clone
git clone https://github.com/<your-fork>/taskfire.git
cd taskfire
# Start infrastructure
docker compose up redis postgres -d
# Install deps and run tests
make install
make test
# Make your changes, then verify
make test
make lintConventions:
- Go code:
gofmtformatted, packages named after their directory, errors wrapped with context usingfmt.Errorf("...: %w", err) - TypeScript: strict mode enabled, Zod schemas for all external data, no
any - Commit messages: imperative mood, present tense (
add retry jitter, notadded) - Tests: new behaviour should come with a test; prefer table-driven tests in Go
Open an issue first for significant changes so the approach can be discussed before implementation.
MIT — free to use, modify, and distribute. See the LICENSE file for the full text.