Skip to content
BranchesTags

Repository files navigation

Status Rust Go Kafka ClickHouse Docker React Platform

God's Eye — Next-Generation IDS & SIEM Platform 👁️

God's Eye is a next-generation Intrusion Detection System (IDS) and Security Information & Event Management (SIEM) platform built from the ground up. Designed for enterprise SOC teams and government/defense organizations, it replaces legacy systems like Splunk Enterprise Security and Security Onion with a lightweight, adaptive, and forensically complete security platform.

🏛️ Eight Core Architectural Pillars

  1. Hybrid ML Detection Engine — XGBoost, Autoencoders, LSTM, combined through a calibrated Bayesian ensemble.
  2. Active IPS Response — Automated blocking, firewall rule injection, VLAN quarantine, BGP blackholing, and SOAR playbooks.
  3. God's Eye Query Language (GEQL) — SQL-compatible search with temporal security operators.
  4. Full MSSP Multi-Tenancy from v1 — Tenant-isolated data, RBAC, baselines, and billing.
  5. Three-Tier Storage — ClickHouse (hot), TimescaleDB (warm), S3 Parquet (cold) with Merkle-chained SHA-256 integrity.
  6. Selective PCAP — Alert-triggered packet capture indexed by 5-tuple for sub-second retrieval.
  7. Behavioral Analytics — Entity trust scoring, CUSUM drift detection, and Apache AGE graph correlation.
  8. Native Deception Layer — Honeytokens and honeyports as a near-zero-false-positive feed.

🏗️ System Architecture (10 Layers)

┌──────────────────────────────────────────────────────────────────────────────┐
│                          God's Eye SIEM Platform                             │
├──────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  L1  Collection       ge-sensor (Rust) + ge-agent (Go)                       │
│  L2  Ingestion        Apache Kafka (RF=3, acks=all, Schema Registry)         │
│  L3  Parsing          Custom parser engine (Go) + GES Schema validation      │
│  L4  Enrichment       GeoIP, TI feeds, MITRE ATT&CK, CMDB (Go)             │
│  L5  Storage          ClickHouse (hot) / TimescaleDB (warm) / S3 (cold)     │
│  L6  Detection        XGBoost + Autoencoder + LSTM + Sigma (ensemble)       │
│  L7  Correlation      Apache AGE entity graph + GEQL engine                  │
│  L8  Response         IPS module + SOAR playbook runner                      │
│  L9  Investigation    GEQL search, PCAP retrieval, case manager              │
│  L10 Platform         Kubernetes, Cilium mTLS, multi-tenant RBAC             │
│                                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐    │
│  │ Web Dashboard│  │  Mobile App  │  │  ge-desktop  │  │  ge CLI      │    │
│  │ React + Vite │  │ Expo SDK 55  │  │ Tauri + React│  │  `ge` cmd    │    │
│  └──────────────┘  └──────────────┘  └──────────────┘  └──────────────┘    │
└──────────────────────────────────────────────────────────────────────────────┘

graph TD
    subgraph L1_L4 [Layer 1-4: Collection & Processing]
        S[ge-sensor<br/>Rust] -->|Raw packets, Flows| K
        A[ge-agent<br/>Go] -->|Host & Cloud Logs| K
        K[Apache Kafka<br/>Ingestion Bus] --> P[Parser Engine<br/>Auto-Detect]
        P --> E[Enrichment<br/>GeoIP, IOC, AD, MITRE]
    end

    subgraph L5 [Layer 5: Storage Tier]
        E --> CH[(ClickHouse<br/>Hot - 30d)]
        E --> TS[(TimescaleDB<br/>Warm - 90d)]
        E --> S3[(S3 Parquet<br/>Cold - 7y)]
    end

    subgraph L6_L8 [Layer 6-8: Detection & Response]
        E --> ML[ML Ensemble<br/>XGBoost + Autoencoder + LSTM]
        E --> SIGMA[Sigma Rules]
        ML --> COR[Correlation<br/>Apache AGE Entity Graph & GEQL]
        SIGMA --> COR
        COR --> IPS[Response Module<br/>IPS & SOAR Playbooks]
    end

    subgraph UI [Layer 9-10: Interfaces & Platform]
        WEB[Web Dashboard<br/>React 18 + Vite]
        MOB[Mobile App<br/>Expo SDK 55]
        DESK[ge-desktop<br/>Tauri]
        CLI[ge CLI Terminal]
    end

    COR -.-> UI
    UI -.-> CH
Loading

🛡️ Agent & Collection Layer Deep Dive

God's Eye relies on a powerful three-component collection and management layer at Layer 1. (Structure Reference: gods_eye_agents.html)

1. ge-sensor (Network Packet Capture & IDS/IPS)

The ge-sensor is a high-performance network daemon written in Rust. It performs deep packet inspection, bidirectional flow tracking, and inline/passive IPS from a single lightweight binary.

  • Language & Backends: Rust (libpcap, AF_PACKET, AF_XDP planned)
  • Deployment: Network TAPs, SPAN Ports, or Inline execution
  • Key Features:
    • Raw Packet Capture: Highly optimized; requires <20MB RAM in passive mode. Supports alert-triggered selective PCAP capture to conserve storage.
    • Protocol Dissection: Real-time decoding of TCP, UDP, DNS, HTTP, TLS (including JA3/JA4 fingerprinting), NTP, DHCP, and critical OT protocols like Modbus, DNP3, BACnet, and EtherNet/IP.
    • Flow Tracking & ML: Tracks up to 100K concurrent 5-tuple bidirectional flows with LRU eviction. Extracts 14 specific ML features per flow (e.g., conn_rate, dns_entropy, traffic_asymmetry) for the upstream ML ensemble.
    • IPS Engine: Operates in Tap (passive) or Inline (active blocking) modes with built-in rules (e.g., SYN scan, XMAS scan, unauthorized Modbus writes).
    • Embedded Dashboard: Features an embedded web UI at :9090 for live stats, Wireshark-style packet inspection, and config editing.
  • Read More

2. ge-agent (Host Telemetry & Log Collection)

The ge-agent is a Go-based host telemetry agent and local ge-sensor supervisor. It is responsible for gathering disparate logs and normalizing them into God's Eye Event Schema (GES) JSON events before hitting Kafka.

  • Language: Go (Goroutine Concurrency)
  • Deployment: Windows, Linux, and macOS Servers, Endpoints, and Cloud VMs
  • Key Features:
    • Broad Input Support: Windows Event Log/Sysmon, Syslog UDP input, file tailing (polling) for configured paths, and HTTP webhook reception.
    • Cloud & SaaS Integrations: Modules for AWS CloudTrail, Azure AD, GCP Audit, Okta, GitHub, and Slack logs.
    • Container & DB Audit: Natively collects Docker/Kubernetes pod logs and tails PostgreSQL/MySQL audit logs.
    • Bounded Queues & Resiliency: Features an internal bounded queue (flushing on 200 events or 5s) and a local RocksDB-backed Dead Letter Queue (DLQ) for 7-day spooling to prevent data loss during Kafka outages.
    • Sensor Management: Can scrape managed ge-sensor endpoints to merge network feeds into the main collection queue seamlessly.
  • Read More

3. ge-desktop (SOC Analyst Workstation App)

The ge-desktop app is the tactical control surface for SOC analysts. It is not a data collector, but rather a localized management interface.

  • Language: Tauri (Rust backend) + React (Frontend)
  • Deployment: SOC Analyst Desktops
  • Key Features:
    • Pipeline Visualization: Visualizes the health of the entire ingestion and ML pipeline.
    • Local Agent Management: Provides abilities to start, stop, and restart local ge-sensor and ge-agent processes.
    • Log Exploration: Houses a GEQL quick search interface and recent event viewer.
    • Configuration: Local config editor with validation and diagnostic exports.

🔍 Log Analyzer & Pipeline Workflow

God's Eye features a universal log ingestion, parsing, and analysis pipeline designed to support over a dozen parsers automatically detecting input formats.

flowchart LR
    Sources((Log Sources)) --> Kafka[Kafka Ingestion\nRF=3, acks=all]
    Kafka --> Parser[Parser Engine\n14+ Auto-detect Parsers]
    Parser --> Normalizer[GES Normalizer]
    Normalizer --> Enrichment[Enrichment Pipeline\nGeoIP/IOC/CMDB]
    
    Enrichment --> Storage[(3-Tier Storage)]
    Enrichment --> ML[ML Ensemble]
    Enrichment --> Sigma[Sigma Rules]
    
    ML --> Correlation((Correlation Graph\nApache AGE))
    Sigma --> Correlation
    Correlation --> Alerts[Alerts & PCAP Trigger]
    Alerts --> SOAR[SOAR / IPS Module]
Loading

🧠 ML Detection Engine (4-Layer Ensemble)

God's Eye moves beyond traditional static rules by employing a 4-layer ML ensemble that ultimately unifies into a single calibrated Bayesian Risk Score.

  1. XGBoost (Supervised): Gradient-boosted trees utilizing 14 real-time flow features extracted by ge-sensor (e.g., conn_rate, dns_entropy, beaconing_score, traffic_asymmetry, off_hours_pct).
  2. Deep Autoencoder (Unsupervised): PyTorch-based anomaly detection learning normal traffic distributions, anchored against adversarial drift, with global priors for cold start.
  3. LSTM Sequence Detection: Bidirectional LSTM tracking 32-event sliding windows per entity to detect complex temporal attack patterns.
  4. Sigma Rules & Threat Intel: Deterministic rule matching combined with dynamic TI feeds (AbuseIPDB, OTX, MISP) and deception (honeytokens/ports).

Entity Trust Bands

Trust Level Action
85–100 TRUSTED Normal monitoring
65–84 LOW Elevated monitoring
40–64 MEDIUM Alert + review
20–39 HIGH Isolate + investigate
0–19 CRITICAL Quarantine + trigger SOAR playbook

🔎 God's Eye Query Language (GEQL)

The God's Eye Query Language (GEQL) provides SQL-compatible search alongside powerful security-specific temporal operators for advanced threat hunting.

-- Find failed logins followed by privilege escalation within 5 minutes
SEQUENCE OF (event_action = 'login_failed') FOLLOWED BY (event_action = 'privilege_escalation')
WITHIN 5m
CORRELATE BY user.id

-- Baseline deviation query to spot anomaly spikes
SELECT * FROM ge_events
WHERE BASELINE_DEVIATION(conn_rate) > 3.0
AND timestamp > NOW() - INTERVAL '1 hour'

💻 Platforms & Applications

Web Dashboard

  • Tech Stack: React 18, TypeScript, Vite 6, TailwindCSS 4, Recharts.
  • Scale: 17 comprehensive screens, including live topology graphs using Apache AGE, compliance posturing, and an interactive CLI terminal.

Mobile App

  • Tech Stack: Expo SDK 55, React Native 0.83.
  • Scale: 16 screens equipped with secure push notifications for critical alerts, remote device isolation, and incident case management on the go.

ge Command Line Interface

  • Provides 14 distinct command categories accessible across Web, Mobile, and standalone shells. Includes live log tailing (ge logs tail), threat intel lookups (ge ti check), and system health status.

🏆 Competitive Differentiators

Feature Splunk ES Security Onion Microsoft Sentinel God's Eye
Hybrid ML Partial Partial Cloud-only Native 4-layer ensemble
Active IPS Via SOAR add-on Suricata only Via Logic Apps Native IPS + SOAR
Search SPL (proprietary) OpenSearch DSL KQL GEQL (SQL + security)
Multi-Tenancy Enterprise only No Azure AD Full MSSP v1
PCAP No Always-on No Selective (alert-triggered)
Log Integrity Add-on No Azure WORM Native Merkle chain
Deception No No No Native honeytokens v1

🚀 Current Status & Target Metrics

God's Eye MVP frontend components are currently completed and fully designed (mock layer), with the ML baseline trained. The backend services (Go microservices, Rust daemon, Kafka bus, Storage tiers) are in active development.

Evaluation Targets

  • Precision: ≥ 88% (Phase 3 Target: ≥ 92%)
  • Recall: ≥ 85% (Phase 3 Target: ≥ 90%)
  • False Positive Rate: ≤ 12% (Phase 3 Target: ≤ 5%)
  • Time to Detection: ≤ 60s (Phase 3 Target: ≤ 30s)

📂 Project Structure & References

For a deep dive into the platform's vision, data flow, and components, please refer to the following architectural documents:

Active Components

God's Eye sees everything. It correlates everything. It responds to everything. No blind spots, no licensing walls, no black boxes.

About

Enterprise-scale AI-powered IDS & SIEM platform combining behavioral analytics, real-time threat detection, and multi-agent security intelligence

userdashboardui.vercel.app/

Topics

apache-kafkadockerllmmachine-learningreinforcement-learningxgboost

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Contributors

Languages