This document captures the full architectural vision for Singine as a multi-layer platform. All terms below are canonical references for development.
Key Concepts: Admin document processing, Paperwork digitisation, OCR pipeline, Intermediary format indexing, RDF triples as interoperability layer, SPARQL + natural language querying, Cosine similarity, Disjoint sets (Union-Find) for entity equality, Self-executing lambdas for late execution, Middle Graphs (Collibra metamodel), Model Context Protocol (MCP) as meta-programming protocol, URI/URL generation for resource identification, Edge site message brokering, Logseq graph projection to SQLite, Kafka event streaming, Quantum-compatible abstractions (qbit environments).
Singine is a multi-layer platform built around a core engine that processes admin documents (paperwork), indexes their content, and exposes it through a knowledge graph queryable via both SPARQL and natural language.
The platform originated as a Unix-idiomatic CLI tool for Logseq todo management and is evolving into a full document processing and data governance engine with Collibra metamodel compatibility.
The project has an existing Python codebase (singine/) with:
- Logseq markdown parser, temporal algebra, SQL-like query language
- RDF/OWL semantic layer (DCAT, PROV, ODRL, TIME, FOAF, SCHEMA)
- Collibra Operating Model abstraction, Lens framework
- Contract model with scenario analysis, FIBO integration
- Knowledge graph with multi-source loading
A polyglot platform with Clojure at the centre, orchestrating document ingestion, event processing, and knowledge graph queries across distributed edge nodes.
Each layer is an independent project. All report to Electron as the presentation shell.
+------------------------------------------------------------------+
| ELECTRON SHELL |
| (Desktop app, renders all layers, single unified interface) |
+------------------------------------------------------------------+
| | | | |
+--------+--+ +------+-----+ +--+-------+ +-+--------+ +--+-------+
| FRONTEND | | CLI LAYER | | SDK | | MIDDLEWARE| | BACKEND |
| (UI/UX) | | (singine | | (Public | | (Middle | | (Storage,|
| | | commands) | | API for | | Graph, | | Events, |
| | | | | 3rd | | Routing,| | Kafka, |
| | | | | party) | | Certs) | | Docker) |
+------------+ +-------------+ +---------+ +----------+ +----------+
| | | | |
+------------------------------------------------------------------+
| CORE LAYER |
| (Clojure: Document processing, OCR, indexing, RDF, SPARQL, |
| temporal algebra, query engine, Union-Find, embeddings) |
+------------------------------------------------------------------+
|
+------------------------------------------------------------------+
| JVM ABSTRACTION LAYER |
| (Java, Python via Jython/GraalPy, NodeJS via GraalJS) |
+------------------------------------------------------------------+
| Layer | Language | Responsibility |
|---|---|---|
| Core | Clojure | Document processing, OCR pipeline, RDF triple generation, SPARQL engine, temporal algebra, query language, Union-Find entity resolution, embeddings, cosine similarity |
| CLI | Clojure/Python | singine command-line interface (existing Python CLI continues as a bridge) |
| SDK | Clojure | Public API for third-party integration, URI/URL generation for MCP resources |
| Middleware | Clojure | Middle Graph routing, certificate management, event dispatch, Collibra Edge brokering |
| Backend | Clojure + Docker | Kafka consumers/producers, SQLite projection, email ingestion (/var/mail), HTTPS API |
| Frontend | ClojureScript | Electron-hosted UI, Logseq graph visualisation |
Goal: Route emails sent to a local address (e.g., a@localhost) into the Singine processing pipeline.
[External Email / sendmail] --> /var/mail/<user>
|
v
[Singine Mail Watcher] (inotify/polling on /var/mail/)
|
v
[Kafka Producer] --> topic: singine.inbound.email
|
v
[Kafka Consumer: Document Processor]
|
v
[Apache Tika: Extract text + metadata]
|
v
[Intermediary Format]
+-- [Lucene Index] (full-text search)
+-- [RDF Triples] (Apache Jena / Aristotle)
+-- [SQLite] (Logseq graph projection)
- Configure local MTA (Postfix/Exim) to accept mail for the local domain
- Set up
/var/mail/<user>as the mailbox directory - Create a Clojure file watcher on
/var/mail/usingjava.nio.file.WatchService - Parse email content (MIME) using Apache Tika
- Publish parsed content to Kafka topic
singine.inbound.email - Consumer processes message through the OCR/indexing pipeline
The primary focus is converting paperwork (scanned images, PDFs, email attachments) into an intermediary indexed format.
| Library | Purpose | License | Clojure Wrapper |
|---|---|---|---|
| Apache Tika | Universal document extraction (PDF, DOCX, email, images, 1000+ formats) | Apache 2.0 | Java interop |
| Tess4J | OCR engine (JNA wrapper for Tesseract) | Apache 2.0 | clj-ocr |
| Apache PDFBox | PDF text extraction + PDF-to-image rendering | Apache 2.0 | Java interop |
| JavaCV/OpenCV | Image pre-processing (deskew, denoise, binarise) | Apache 2.0 / BSD | Java interop |
[Scanned Image / PDF / Email]
|
v
[Pre-processing: JavaCV] -- deskew, denoise, binarise, rescale to 300 DPI
|
v
[OCR: Tess4J / Tesseract] -- image to text
|
v
[Extraction: Apache Tika] -- text + Dublin Core metadata
|
v
[Intermediary Format] -- indexed, queryable content
The extracted text and metadata are stored in two complementary formats:
| Format | Library | Purpose |
|---|---|---|
| Full-text index | Apache Lucene (via Clucy or lucene-clj) | Token-based search, fuzzy matching |
| RDF triple store | Apache Jena TDB2 (via Aristotle) | Structured knowledge graph, SPARQL queries |
| SQLite projection | SQLite (via JDBC) | Logseq graph projection, relational queries |
| Vector embeddings | DJL + ONNX Runtime (via clj-djl) | Cosine similarity, semantic search |
Two query strategies, selectable based on query type:
Strategy A: SPARQL (Structured Queries)
- For known attributes: date, sender, document type, amount
- Uses Apache Jena ARQ engine
- Clojure data structures as SPARQL queries (via Aristotle)
Strategy B: Cosine Similarity (Semantic Queries)
- For content-based questions: "What does clause 3 say about termination?"
- Document chunks embedded via sentence-transformers (DJL + ONNX Runtime)
- Query embedded the same way, cosine similarity finds relevant chunks
Strategy C: Natural Language to SPARQL (LLM Translation)
- User asks in natural language
- LLM translates to SPARQL query
- Executed against Jena triple store
Activities in Singine are modelled as self-executing lambdas to enable late execution (deferred evaluation).
;; Activity as a self-executing lambda with late binding
(defn make-activity [name f]
{:name name
:execute (fn [ctx] (f ctx))
:status (atom :pending)})
;; Activities compose into pipelines
(defn pipeline [& activities]
(fn [ctx]
(reduce (fn [ctx' act]
(reset! (:status act) :running)
(let [result ((:execute act) ctx')]
(reset! (:status act) :completed)
result))
ctx activities)))Entity resolution uses Union-Find to determine when two extracted entities refer to the same real-world object:
(defn make-union-find []
(let [parent (atom {})
find (fn find [x]
(let [p (get @parent x x)]
(if (= p x) x
(let [root (find p)]
(swap! parent assoc x root)
root))))
union (fn [x y]
(let [rx (find x) ry (find y)]
(when (not= rx ry)
(swap! parent assoc rx ry))))]
{:find find :union union :equal? (fn [x y] (= (find x) (find y)))}))| Prefix | Vocabulary | Usage |
|---|---|---|
rdf: |
RDF | Core triple model |
rdfs: |
RDF Schema | Class/property hierarchy |
owl: |
OWL | Ontology definitions |
dcat: |
Data Catalog Vocabulary | Dataset descriptions |
prov: |
Provenance Ontology | Activity/agent tracking |
odrl: |
Open Digital Rights Language | Intent-based authorisation |
time: |
Time Ontology | Temporal constraints |
foaf: |
Friend of a Friend | People and relationships |
schema: |
Schema.org | General entity descriptions |
skos: |
SKOS | Concept hierarchies |
dcterms: |
Dublin Core Terms | Document metadata (Tika outputs this natively) |
fibo: |
Financial Industry Business Ontology | Financial entities |
singine: |
Singine Custom Namespace | Project-specific terms |
The platform generates URIs and URLs interchangeably to identify any resource, compatible with MCP (Model Context Protocol) as a meta-programming protocol:
singine:<type>/<id> -- URI form (abstract identifier)
https://singine.local/<type>/<id> -- URL form (resolvable address)
Users may provide custom URIs/URLs. The system treats URI and URL as interchangeable resource identifiers.
Middle graphs are the integration pattern from the Collibra metamodel: they connect any source to any target through a typed, governed intermediary layer.
[Source A] ---> [Middle Graph] ---> [Target X]
[Source B] -/ \--> [Target Y]
| Collibra Concept | Singine Equivalent |
|---|---|
| Asset Type | Document Type (Invoice, Contract, Letter, Form) |
| Domain Type | Processing Domain (Inbound Email, Scanned Doc, API Upload) |
| Relation Type | RDF predicate linking entities |
| Status Workflow | Activity state machine (pending → running → completed) |
| Community | Edge Site / Node |
| Responsibility | Certificate-bound agent identity |
Singine replicates the Collibra Edge architecture for distributed message brokering:
+------------------+ +------------------+ +------------------+
| Edge Site A | | Central Broker | | Edge Site B |
| (Docker node) |<--->| (Kafka cluster) |<--->| (Docker node) |
| - Local ingestion| | - Topic routing | | - Local ingestion|
| - Local index | | - Schema registry| | - Local index |
| - Local RDF store| | - Central graph | | - Local RDF store|
+------------------+ +------------------+ +------------------+
Each edge site:
- Runs in its own Docker container
- Has a local Lucene index and Jena TDB2 triple store
- Publishes events to Kafka topics
- Synchronises with the central broker
- Identified by a single TLS certificate (mapped to Unix inodes)
environments/
├── dev/ # Development: local volumes, debug logging
├── prod/ # Production: persistent storage, monitoring
└── x/ # Placeholder environment (qbit: undetermined state)
The x environment represents a quantum-inspired placeholder -- an environment whose assignment is undetermined until observation (deployment decision). In quantum computing terms, it exists in superposition until collapsed to a specific configuration.
;; Environment as a qbit: exists in superposition until observed
(defprotocol QBit
(observe [this] "Collapse to a definite state")
(superpose [this states] "Set possible states"))
(defrecord Environment [name state possible-states]
QBit
(observe [this]
(if (= state :undetermined)
(assoc this :state (first possible-states))
this))
(superpose [this states]
(assoc this :possible-states states :state :undetermined)))
;; The x environment
(def x-env (->Environment "x" :undetermined [:dev :staging :prod :edge]))A single TLS certificate is used across all nodes to create a trustworthy network:
[Root CA Certificate]
|
+-- [Node Certificate A] --> mapped to inode on host A
+-- [Node Certificate B] --> mapped to inode on host B
+-- [Node Certificate C] --> mapped to inode on host C
Each certificate maps to a Unix inode, binding cryptographic identity to filesystem identity.
Topics:
singine.inbound.email -- Raw email content from /var/mail
singine.inbound.api -- Documents uploaded via HTTPS API
singine.processed.text -- Extracted text + metadata
singine.processed.triples -- Generated RDF triples
singine.events.activity -- Activity lifecycle events
singine.edge.sync -- Edge site synchronisation
Clojure is the central language. It handles:
- File system communication
- Events triggered by email or HTTPS API
- Coordination between all layers
- Interaction with Logseq as the abstract graph layer
Through the JVM, the platform has access to:
| Language | Access Method | Usage |
|---|---|---|
| Java | Native JVM | Apache Tika, PDFBox, Jena, Lucene, Kafka clients |
| Python | GraalPy / subprocess | ML models, NLP, existing singine Python code |
| NodeJS | GraalJS / subprocess | Electron integration, npm ecosystem |
| Clojure | Native | Core platform logic |
| Library | Purpose |
|---|---|
| Aristotle | Data-oriented RDF/SPARQL/OWL (wraps Apache Jena) |
| Clucy | Clojure interface to Lucene full-text search |
| clj-ocr | Tesseract OCR wrapper |
| clj-djl | Deep Java Library for embeddings + ML inference |
| clj-pdf | PDF generation |
| clj-kafka | Kafka producer/consumer |
| Mount / Component | Lifecycle management for services |
The current Python implementation continues as a bridge and will be incrementally migrated to Clojure.
| Module | Purpose | Migration Priority |
|---|---|---|
cli.py |
Command-line interface | Bridge (keep as Python entry point) |
logseq.py |
Logseq markdown parser | Core (migrate to Clojure) |
temporal.py |
Temporal algebra (Pendulum) | Core (migrate to Clojure) |
query.py |
WHERE clause parser | Core (migrate to Clojure) |
rdf_ontology.py |
RDF/OWL semantic layer | Core (replace with Aristotle) |
operating_model.py |
Collibra Operating Model | Middleware (migrate) |
knowledge_graph.py |
Multi-source knowledge graph | Core (migrate) |
lens/ |
Polymorphic metamodel views | Middleware (migrate) |
contract_model.py |
Contract abstraction | Core (migrate) |
fibo_integration.py |
FIBO + game theory | SDK (migrate) |
config.py |
Configuration management | Backend (migrate) |
cd /Users/skh/ws/singine
pip install -e .
# Configuration
# Create ~/.singine/backend.config:
# [logseq]
# graph_path = /path/to/your/logseq/graph
singine ls tasks
singine ls tasks -where 'Last Updated Date >= pastDay#"3 months"'
singine ls tasks -where 'Priority = A'- Chunk extracted document text (by paragraph or fixed token window)
- Generate embeddings using sentence-transformers via DJL/ONNX Runtime
- Store embeddings alongside Lucene index
- At query time: embed the question, find top-k chunks by cosine similarity
- Optionally feed chunks to an LLM for answer synthesis
- Extract entities and relationships from documents using NLP
- Store as RDF triples in Jena TDB2
- At query time: translate natural language to SPARQL (via LLM or rule-based)
- Execute SPARQL, return structured results
- Combine with embedding-based results for hybrid answers
When processing multiple documents, the same entity may appear under different names. Union-Find resolves these:
- Extract entity mentions from each document
- Compare mentions pairwise using cosine similarity of embeddings
- If similarity > threshold, union the two mentions
- All mentions in the same set are treated as the same entity
- The equality function:
(fn [a b] (= (find a) (find b)))
singine/
├── core/ # Clojure: document processing, RDF, SPARQL, query engine
│ ├── project.clj
│ └── src/singine/core/
├── cli/ # Clojure + Python bridge: command-line interface
├── sdk/ # Clojure: public API, URI generation, MCP
├── middleware/ # Clojure: Middle Graph, routing, certificates, Edge
├── backend/ # Clojure + Docker: Kafka, SQLite, email ingestion
├── frontend/ # ClojureScript: Electron UI
├── singine/ # Python: existing codebase (bridge, gradual migration)
├── environments/
│ ├── dev/
│ ├── prod/
│ └── x/
├── docker/
│ ├── Dockerfile
│ └── docker-compose.yml
├── certs/ # Single root CA + node certificates
├── docs/
└── CLAUDE.md # This file
# Existing Python CLI
pip install -e .
singine ls tasks
# Future Clojure core
cd core && lein repl
# Docker environment
docker-compose -f docker/docker-compose.yml up -d
# Kafka
docker exec -it kafka kafka-topics --list --bootstrap-server localhost:9092| Term | Definition |
|---|---|
| Middle Graph | Integration pattern from Collibra metamodel; a typed intermediary graph connecting any source to any target |
| Edge Site | A distributed node (Docker container) that ingests documents locally and syncs with the central broker |
| Self-Executing Lambda | An activity modelled as a closure that captures its execution context and defers evaluation until invoked |
| Disjoint Set (Union-Find) | Data structure for tracking entity equivalence classes; the equality function (= (find a) (find b)) |
| Cosine Similarity | Measure of angular distance between embedding vectors; used for semantic document search |
| Intermediary Format | The indexed representation of a document after OCR/extraction: Lucene index + RDF triples + SQLite |
| qbit Environment | An environment in superposition (undetermined assignment) until deployment collapses it to a specific configuration |
| MCP | Model Context Protocol; a meta-programming protocol for resource identification via URIs/URLs |
| Temporal Algebra | Singine's natural language date expression system: pastDay#"3 months", day#"last Monday" |
| Lens | A polymorphic view that transforms heterogeneous data sources into a target metamodel |
| Logseq Graph Projection | Mapping a Logseq markdown graph onto a SQLite database for relational queries |
- Configure local MTA for mail delivery to
/var/mail/ - Implement Clojure file watcher on mailbox
- Set up Kafka with
singine.inbound.emailtopic - Parse email MIME content via Apache Tika
- Store extracted text in Lucene index
- Integrate Tess4J for OCR on scanned attachments
- Implement RDF triple generation via Aristotle/Jena
- Set up Jena TDB2 persistent triple store
- Build SPARQL query interface
- Implement Middle Graph routing
- Set up certificate infrastructure (single root CA)
- Docker containerisation of edge sites
- Kafka-based edge synchronisation
- Public Clojure SDK with URI/URL generation
- MCP-compatible resource identification
- Electron shell with ClojureScript frontend
- Logseq graph visualisation
Located at core/java/singine/activity/. Compiled automatically by
ant compile (the compile target uses find ... -name '*.java').
| Class/Interface | Role |
|---|---|
Activity |
Template interface: defines what is to be done |
Policy |
Execution algorithm: governs how an action runs |
Action |
Runtime instance: follows a Policy against an Activity |
Outcome |
Measurement: usage value + business value − platform cost |
Taxonomy |
Classification node: domain → category → subcategory |
ActivityStatus |
Enum: PENDING, RUNNING, COMPLETED, FAILED, CANCELLED, DEFERRED |
OutcomeType |
Enum: SUCCESS, FAILURE, PARTIAL, SKIPPED, DEFERRED |
ActivityTemplate |
Abstract base: default XML/EDN serialisation |
BaseAction |
Default Action with atomic status and policy application |
BaseOutcome |
Immutable Outcome with ultimate-metric components |
usage_cost_benefit_score = usage_value + business_value - platform_cost
core/resources/singine/activity/taxonomy.edn
This file is the single source of truth for generating: XML, LaTeX, SVG, HTML (SilkPage), GraphQL, SPARQL, Logseq queries.
make activity-edn # print the taxonomy
singine activity list # list all activities
singine activity inspect ID # inspect a single activity (--xml, --edn, --json)
singine activity run ID # run an activity and get an OutcomeThe data-production taxonomy domain documents the meta-concept:
the activity taxonomy is itself a data product governed by an activity.
Logo: sina.khakbaz.com/img/dpdp-logo.svg
man singine-activity — see man/singine-activity.1.
sina.khakbaz.com/docs/activity/ — SilkPage pages documenting the model.
The singine collibra command family provides live Collibra REST operations
plus full lifecycle management for the Collibra edge stack, contract IDs,
data contracts, the ORM/SBVR pipeline, and the id-gen HTTP server.
singine collibra <component> <command> [options]
- Edge stack (
singine collibra edge): delegates tosingine/edge.py(add_edge_parseris reused withname="edge"under the collibra subparser). - id/contract/server: dynamically imported from the
singine_collibraPython package located in the collibra repo (~/ws/git/github/sindoc/collibraor$COLLIBRA_DIR).
Adds singine_collibra package root to sys.path at runtime, then calls
singine_collibra.command.add_collibra_subcommands(collibra_sub).
If the package is missing, a clear error is printed and exit code 1 is
returned — the rest of the singine CLI is unaffected.
# Core REST (always available)
singine collibra env
singine collibra fetch community|domain|asset-type|view|workflow
singine collibra search <query>
# Edge stack (requires singine edge stack in collibra repo)
singine collibra edge build [--target base|collibra-edge|cdn|all]
singine collibra edge up [--detach]
singine collibra edge down
singine collibra edge status [--json]
singine collibra edge logs [--service cdn] [--follow]
singine collibra edge agent run --task "..."
# Contract ID generation (requires collibra repo id-gen/)
singine collibra id gen [--ns c|a|b] [--project P] [--kind K]
singine collibra id gen-topic [--project P]
singine collibra id import --uuid UUID --kind KIND
singine collibra id tags
singine collibra id push-tags
singine collibra id detect-conflicts
singine collibra id resolve-conflicts [--strategy COLLIBRA_WINS]
# Data contract lifecycle
singine collibra contract new [--project P] [--kind DataContract]
singine collibra contract list
singine collibra contract status <id> <STATUS>
singine collibra contract pipeline [--id ID]
singine collibra contract step N [--id ID]
singine collibra contract advance [--id ID]
singine collibra contract progress [--all] [--json]
# id-gen HTTP server
singine collibra server start [--port 7331] [--mode net|dmz]
singine collibra server stop
singine collibra server status
singine collibra server dmz [--port 7331]| Variable | Default | Purpose |
|---|---|---|
COLLIBRA_DIR |
~/ws/git/github/sindoc/collibra |
Collibra repo root |
COLLIBRA_EDGE_DIR |
$COLLIBRA_DIR/edge |
Edge stack directory |
Last Updated: 2026-03-15 Project Status: Alpha (v0.2.0) -- Transitioning from Python to Clojure Primary Developer: skh