🧠 FLOW – Distributed Workflow Orchestration Engine

A distributed, event-driven workflow orchestration engine designed to execute DAG-based workflows with support for triggers, conditional branching, retries, pause/resume functionality, and async execution.

📋 Table of Contents

Overview
Key Features
Architecture
Tech Stack
Project Structure
Prerequisites
Installation & Setup
Running the Application
API Documentation
Database Documentation
Project Documentation
Development Roadmap
Contributing
License

Overview

FLOW is a production-ready workflow orchestration platform that enables organizations to:

Define complex workflows as Directed Acyclic Graphs (DAGs)
Execute asynchronously with horizontal scaling via message queues
Monitor execution with comprehensive logging and audit trails
Control workflows with pause, resume, and stop capabilities
Trigger workflows via webhooks, cron schedules, or manual invocation
Retry intelligently with exponential backoff and partial recovery
Integrate seamlessly with external APIs, services, and webhooks

Use Cases

📊 Data Pipeline Management - ETL workflows, data syncing, batch processing
📧 Communication Automation - Email campaigns, notifications, alerts
🔄 Integration Workflows - API orchestration, multi-service coordination
👤 User Journey Automation - Onboarding flows, welcome sequences
📈 Business Process Automation - Approval workflows, report generation

Key Features

✅ Core Functionality

DAG-Based Workflows - Define complex workflows as directed acyclic graphs
Event-Driven Execution - Async execution via message queues (Redis)
Multiple Trigger Types - Webhooks, Cron schedules, Manual invocation
Conditional Branching - Execute different paths based on conditions
Retry Strategy - Exponential backoff with configurable limits
Logging & Monitoring - Structured logs at node and execution levels
Audit Trail - Complete action history for compliance

⭐ Advanced Features

Pause/Resume - Pause workflow execution, resume from paused state
Stop/Kill - Terminate running workflows gracefully
Partial Recovery - Resume from specific nodes after failures
Context Passing - Share data between nodes via execution context
Webhook Callbacks - External webhooks for event notifications
Rate Limiting - API rate limiting for resource protection
JWT Authentication - Secure API access with JWT tokens

Architecture

High-Level System Design

                ┌──────────────────────┐
                │   API Gateway        │
                │ (Auth + Rate Limit)  │
                └─────────┬────────────┘
                          │
        ┌─────────────────┼──────────────────┐
        ▼                 ▼                  ▼
┌──────────────┐  ┌──────────────┐   ┌──────────────┐
│ Workflow API │  │ Trigger Svc  │   │ Execution API│
└──────┬───────┘  └──────┬───────┘   └──────┬───────┘
       │                  │                  │
       ▼                  ▼                  ▼
┌────────────────────────────────────────────────────┐
│                Workflow Store (DB)                 │
│  - workflows (DAG JSON)                           │
│  - executions                                     │
│  - node_logs                                      │
└────────────────────────────────────────────────────┘
                          │
                          ▼
                ┌──────────────────┐
                │   Message Queue  │  (Redis)
                └─────────┬────────┘
                          │
        ┌─────────────────┼──────────────────┐
        ▼                 ▼                  ▼
┌──────────────┐  ┌──────────────┐   ┌──────────────┐
│ Worker Node  │  │ Worker Node  │   │ Worker Node  │
│ (Executor)   │  │ (Executor)   │   │ (Executor)   │
└──────┬───────┘  └──────┬───────┘   └──────┬───────┘
       │                  │                  │
       ▼                  ▼                  ▼
┌────────────────────────────────────────────────────┐
│         External Services / APIs / Webhooks       │
└────────────────────────────────────────────────────┘

Execution State Machine

CREATED
   │
   ▼
RUNNING
   │
   ├──> PAUSED ──> RESUMED ──> RUNNING (cycle allowed)
   │
   ├──> STOPPED (user requested kill)
   │
   ├──> FAILED ──> RETRYING ──> RUNNING (if retries < max)
   │
   ▼
SUCCESS

Tech Stack

Backend

Framework: Spring Boot 3.x (Java 21)
API: REST with JWT Authentication
ORM: Spring Data JPA / Hibernate
Database: PostgreSQL 14+
Message Queue: Redis with connection pooling
Task Scheduler: Spring Scheduler / Quartz
Validation: Spring Validation, Bean Validation
Logging: SLF4J with Logback
Testing: JUnit 5, Mockito, TestContainers

Frontend

Framework: React / Vue / Angular (TBD)
State Management: Redux / Pinia / NgRx
HTTP Client: Axios / Fetch API
UI Framework: Material UI / Bootstrap / TailwindCSS
Real-time: WebSocket for live updates
Build Tool: Vite / Webpack

Infrastructure

Database: PostgreSQL 14+
Cache: Redis
Message Broker: Redis Streams or Kafka (future)
Container: Docker (optional)
Orchestration: Kubernetes-ready (future)

Project Structure

Flow/
├── README.md                           # This file
├── LICENSE                            # MIT License
├── .gitignore                         # Git ignore rules
│
├── assets/                            # Documentation & Configuration
│   ├── SYSTEM_CONTEXT.md             # Architecture & system design
│   ├── API_DESIGN_CONTRACT.md        # Complete REST API specification
│   └── Flow One Pager.pdf            # Original one-pager
│
├── db/                               # Database & Migrations
│   ├── DATABASE_DESIGN.md            # Complete database schema design
│   ├── README.md                     # Database setup guide
│   ├── migrations/
│   │   ├── 001_initial_schema.sql   # Initial schema creation
│   │   └── rollback_001.sql         # Rollback script
│   └── seeds/
│       └── initial_data.sql         # Optional seed data
│
├── flow-backend/                     # Backend (Spring Boot)
│   ├── src/
│   │   ├── main/
│   │   │   ├── java/
│   │   │   │   └── com/flow/
│   │   │   │       ├── api/              # REST API controllers
│   │   │   │       ├── service/          # Business logic
│   │   │   │       ├── executor/         # Workflow execution engine
│   │   │   │       ├── queue/            # Message queue management
│   │   │   │       ├── model/            # Data models & entities
│   │   │   │       ├── repository/       # Data access layer
│   │   │   │       ├── config/           # Application configuration
│   │   │   │       ├── security/         # JWT & auth
│   │   │   │       ├── exception/        # Custom exceptions
│   │   │   │       └── FlowApplication.java
│   │   │   └── resources/
│   │   │       ├── application.properties
│   │   │       └── application-dev.properties
│   │   └── test/                    # Integration & unit tests
│   ├── pom.xml                      # Maven dependencies
│   └── README.md                    # Backend-specific setup
│
└── flow-frontend/                   # Frontend (React/Vue/Angular)
    ├── src/
    │   ├── components/              # Reusable React/Vue components
    │   ├── pages/                   # Page components
    │   ├── services/                # API service layer
    │   ├── store/                   # State management
    │   ├── hooks/                   # Custom React hooks
    │   ├── utils/                   # Utility functions
    │   └── App.tsx/App.vue         # Root component
    ├── package.json                # NPM dependencies
    ├── vite.config.ts              # Vite/Webpack config
    └── README.md                   # Frontend-specific setup

Prerequisites

System Requirements

Java: JDK 21+ (for Spring Boot backend)
Node.js: 18+ LTS (for frontend)
PostgreSQL: 14+
Redis: 7.0+
Git: Latest version
IDE: IntelliJ IDEA or VS Code (recommended)

Development Tools

# Java/Maven
java -version  # >= 21
mvn -version   # >= 3.8

# Node/NPM
node -v        # >= 18
npm -v         # >= 8

# Database
psql --version # >= 14
redis-cli --version  # >= 7.0

Installation & Setup

1. Clone the Repository

git clone https://github.com/yourusername/flow.git
cd flow

2. Backend Setup

cd flow-backend

# Install dependencies
mvn clean install

# Copy and configure environment
cp src/main/resources/application.properties.example src/main/resources/application.properties
# Edit application.properties with your database credentials

3. Database Setup

# Create PostgreSQL database
createdb -U postgres flow_db

# Run migrations
psql -U postgres -d flow_db -f ../db/migrations/001_initial_schema.sql

# (Optional) Seed test data
psql -U postgres -d flow_db -f ../db/seeds/initial_data.sql

See db/README.md for detailed database setup instructions.

4. Redis Setup

# Start Redis (macOS with Homebrew)
redis-server

# Or Docker
docker run -d -p 6379:6379 redis:7-alpine

5. Frontend Setup

cd ../flow-frontend

# Install dependencies
npm install

# Copy and configure environment
cp .env.example .env
# Edit .env with API endpoint and other configs

Running the Application

Backend (Spring Boot)

cd flow-backend

# Development mode with hot reload
mvn spring-boot:run

# Or with Maven Wrapper
./mvnw spring-boot:run

# Production build
mvn clean package
java -jar target/flow-backend-1.0.0.jar

Backend URL: http://localhost:8080
API Docs: http://localhost:8080/swagger-ui.html

Frontend

cd flow-frontend

# Development mode with HMR
npm run dev

# Production build
npm run build
npm run preview

Frontend URL: http://localhost:5173 (Vite default)

Docker (Optional)

# Build images
docker-compose build

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down

API Documentation

Complete API Reference

All REST API endpoints are fully documented in assets/API_DESIGN_CONTRACT.md

API Endpoints Overview

Authentication

POST /api/v1/auth/signup - User registration
POST /api/v1/auth/login - User login
POST /api/v1/auth/logout - User logout
POST /api/v1/auth/refresh - Refresh JWT token
GET /api/v1/auth/verify - Verify token validity

Workflows

POST /api/v1/workflows - Create workflow
GET /api/v1/workflows - List workflows
GET /api/v1/workflows/{id} - Get workflow details
PUT /api/v1/workflows/{id} - Update workflow
DELETE /api/v1/workflows/{id} - Delete workflow
POST /api/v1/workflows/validate - Validate DAG

Executions

POST /api/v1/workflows/{id}/executions - Trigger execution
GET /api/v1/workflows/{id}/executions - List executions
GET /api/v1/executions/{id} - Get execution details
PATCH /api/v1/executions/{id}/pause - Pause execution ⭐
PATCH /api/v1/executions/{id}/resume - Resume execution ⭐
PATCH /api/v1/executions/{id}/stop - Stop execution ⭐
POST /api/v1/executions/{id}/retry - Retry execution

Monitoring

GET /api/v1/executions/{id}/logs - Get execution logs
GET /api/v1/executions/{id}/metrics - Get execution metrics
GET /api/v1/workflows/{id}/stats - Get workflow statistics
GET /api/v1/health - System health check

Triggers

POST /api/v1/workflows/{id}/triggers - Create trigger
GET /api/v1/workflows/{id}/triggers - List triggers
PATCH /api/v1/workflows/{id}/triggers/{id} - Update trigger
DELETE /api/v1/workflows/{id}/triggers/{id} - Delete trigger
POST /api/v1/workflows/{id}/triggers/{id}/test - Test trigger

Example: Create & Execute Workflow

# 1. Create workflow
curl -X POST http://localhost:8080/api/v1/workflows \
  -H "Authorization: Bearer <jwt_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Daily Report",
    "dag": {
      "nodes": [...],
      "edges": [...]
    }
  }'

# 2. Create webhook trigger
curl -X POST http://localhost:8080/api/v1/workflows/<id>/triggers \
  -H "Authorization: Bearer <jwt_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "webhook",
    "config": {"path": "/daily-report"}
  }'

# 3. Trigger execution via webhook
curl -X POST http://localhost:8080/api/webhooks/<trigger_id> \
  -H "Content-Type: application/json" \
  -d '{"report_date": "2026-05-09"}'

# 4. Monitor execution
curl -X GET http://localhost:8080/api/v1/executions/<exec_id> \
  -H "Authorization: Bearer <jwt_token>"

# 5. Pause execution if needed
curl -X PATCH http://localhost:8080/api/v1/executions/<exec_id>/pause \
  -H "Authorization: Bearer <jwt_token>"

# 6. Resume execution
curl -X PATCH http://localhost:8080/api/v1/executions/<exec_id>/resume \
  -H "Authorization: Bearer <jwt_token>"

Database Documentation

Schema Overview

FLOW uses PostgreSQL with 10 tables:

Table	Purpose
`users`	User accounts & authentication
`refresh_tokens`	Session token management
`workflows`	Workflow definitions (DAG storage)
`executions`	Workflow execution instances
`node_executions`	Individual node execution tracking
`execution_events`	Event audit trail
`node_logs`	Detailed execution logs
`triggers`	Webhook & cron configurations
`audit_logs`	User action audit trail
`failed_login_logs`	Security logging

Database Diagram

See db/DATABASE_DESIGN.md for:

Complete schema design
ER diagram
Index strategy
Constraints & relationships
Backup & recovery procedures

Quick Database Commands

# Connect to database
psql -U postgres -d flow_db

# List tables
\dt

# Check table structure
\d executions

# Run migrations
psql -U postgres -d flow_db -f db/migrations/001_initial_schema.sql

# Create backup
pg_dump -U postgres flow_db > flow_db_backup.sql

# Restore backup
psql -U postgres flow_db < flow_db_backup.sql

Project Documentation

Complete Documentation Files

assets/SYSTEM_CONTEXT.md
- System architecture & design
- State machines
- Scalability model
- Fault tolerance
assets/API_DESIGN_CONTRACT.md
- REST API specification
- All 40+ endpoints detailed
- Request/response examples
- Error handling & rate limiting
db/DATABASE_DESIGN.md
- Complete schema design
- Table specifications with all fields
- Index strategy & performance
- Migration & backup procedures
db/README.md
- Database setup guide
- Connection configuration
- Maintenance procedures
- Troubleshooting guide

Development Roadmap

Phase 1: Core Backend (Current)

Phase 2: Execution Engine

Phase 3: Queue & Workers

Phase 4: Frontend

Phase 5: Advanced Features

Phase 6: Production & Deployment

Contributing

Getting Started

Fork the repository
Create a feature branch: git checkout -b feature/amazing-feature
Commit changes: git commit -m 'Add amazing feature'
Push to branch: git push origin feature/amazing-feature
Open a Pull Request

Development Guidelines

Follow Spring Boot & Clean Code principles
Write unit tests for all features (target: >80% coverage)
Document API changes in comments
Update relevant documentation files
Use meaningful commit messages
Format code with project formatter

Code Style

// Backend: Follow Google Java Style Guide
// Frontend: Follow Airbnb JavaScript/TypeScript Style Guide

Running Tests

# Backend
cd flow-backend
mvn test
mvn test -DskipIntegrationTests=false

# Frontend
cd flow-frontend
npm test
npm run test:coverage

Support & Resources

Issues: GitHub Issues
Discussions: GitHub Discussions
Documentation: See assets/ and db/ directories
API Docs: /swagger-ui.html (when running backend)

License

This project is licensed under the MIT License - see LICENSE file for details.

MIT License

Copyright (c) 2026 FLOW Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

Acknowledgments

Team members who contributed to architecture & design
Open source community for incredible libraries
Users providing feedback and feature requests

Quick Links

Resource	Link
System Architecture	SYSTEM_CONTEXT.md
API Documentation	API_DESIGN_CONTRACT.md
Database Design	DATABASE_DESIGN.md
Database Setup	db/README.md
Backend Repo	flow-backend/README.md
Frontend Repo	flow-frontend/README.md

Project Status: 🟡 Development Phase
Version: 1.0.0
Last Updated: May 9, 2026

For the latest updates, visit GitHub Repository

Name		Name	Last commit message	Last commit date
Latest commit History 50 Commits
assets		assets
db		db
flow-backend		flow-backend
flow-frontend		flow-frontend
.editorconfig		.editorconfig
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation

🧠 FLOW – Distributed Workflow Orchestration Engine

📋 Table of Contents

Overview

Use Cases

Key Features

✅ Core Functionality

⭐ Advanced Features

Architecture

High-Level System Design

Execution State Machine

Tech Stack

Backend

Frontend

Infrastructure

Project Structure

Prerequisites

System Requirements

Development Tools

Installation & Setup

1. Clone the Repository

2. Backend Setup

3. Database Setup

4. Redis Setup

5. Frontend Setup

Running the Application

Backend (Spring Boot)

Frontend

Docker (Optional)

API Documentation

Complete API Reference

API Endpoints Overview

Authentication

Workflows

Executions

Monitoring

Triggers

Example: Create & Execute Workflow

Database Documentation

Schema Overview

Database Diagram

Quick Database Commands

Project Documentation

Complete Documentation Files

Development Roadmap

Phase 1: Core Backend (Current)

Phase 2: Execution Engine

Phase 3: Queue & Workers

Phase 4: Frontend

Phase 5: Advanced Features

Phase 6: Production & Deployment

Contributing

Getting Started

Development Guidelines

Code Style

Running Tests

Support & Resources

License

Acknowledgments

Quick Links

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Contributors

Uh oh!

Languages