Feature: Document-Graph — Structured Data ETL Complement to Lexical-Graph

[evanerwee]

Package

other

Problem statement

GitHub Issue Draft — awslabs/graphrag-toolkit

Title: Feature: Document-Graph — Structured Data ETL Complement to Lexical-Graph

---

Summary

Document-graph is a structured data ETL layer that complements lexical-graph by handling CSV, Excel, JSON, and Parquet data in the same Neptune database. Where lexical-graph extracts knowledge from unstructured text using LLMs, document-graph provides deterministic, schema-driven transformation of structured data into typed graph nodes — no LLM required.

The two coexist in the same Neptune cluster with isolated label namespaces, and document-graph data can be indexed into lexical-graph for hybrid semantic search.

Problem Statement

graphrag-toolkit excels at extracting knowledge graphs from unstructured text (PDFs, web pages, documents). However, many real-world applications also need to incorporate structured data (user records, transactions, events, configuration) into the same knowledge graph. Currently there's no standard path for:

1. Deterministic ETL of structured data into Neptune alongside lexical-graph content
2. Schema-driven graph construction without LLM calls (fast, cheap, predictable)
3. Hybrid queries that combine typed node lookups with semantic search
4. Multi-tenant isolation across both structured and unstructured data

What Document-Graph Provides

Capability	Description
Schema Providers	Auto-discover or define ETL schemas from CSV, JSON, S3, Glue
Schema Discovery	Infer graph schema from data files (CSV, Excel, JSON, Parquet, XML)
20+ Transformers	Normalizers, field transformers, document transformers, filters, graph constructors
Graph Build	Cypher generation (node_to_cypher, edge_to_cypher) with tenant-scoped labels
Query Engine	Typed node queries with tenant isolation
Multi-Tenancy	Same TenantId model as lexical-graph — __Type__tenant_id__ label scoping
Hybrid Search	Index document-graph content into lexical-graph for semantic retrieval

Architecture & Integration

graphrag-toolkit (upstream dependency)
├── TenantId, VersioningConfig        ← document-graph uses these directly
├── GraphStore, GraphStoreFactory     ← document-graph writes through these
├── VectorStore, VectorStoreFactory   ← used for hybrid search
├── LexicalGraphIndex                 ← indexes document-graph data
└── LexicalGraphQueryEngine           ← semantic search across both

document-graph (this contribution)
├── schema/           → ETL schema model + providers (CSV/JSON/S3/Static/Glue)
├── transform/        → 20+ transformers (normalizers, field, document, filter, graph)
├── graph_build/      → Cypher generation with tenant-scoped labels
├── query/            → DocumentGraphQueryEngine (typed node queries)
├── pipeline/         → Extract providers (CSV, Excel, JSON, Parquet)
├── ingest/           → Column selectors, row filters, renamers
└── storage/drivers/  → Complementary backends (GraphJSON, GraphCode, FalkorDB)

Tight Alignment with graphrag-toolkit

• Uses TenantId and to_tenant_id() directly from graphrag_toolkit.lexical_graph
• Uses VersioningConfig and VersioningMode from graphrag_toolkit.lexical_graph
• Writes to Neptune via GraphStoreFactory.for_graph_store()
• Adds complementary storage drivers (GraphJSON for local dev, FalkorDB) that register with GraphStoreFactory.register()
• Adds ReadOnlyGraphStore wrapper and for_reader()/for_writer() factory convenience methods

Coexistence in Neptune

	Lexical Graph	Document Graph
Input	Unstructured text (PDF, web)	Structured data (CSV, Excel, JSON)
Extraction	LLM-based (Claude)	Deterministic ETL (pandas)
Graph Model	Source → Chunk → Topic → Statement → Entity	Row → Typed Node + Edges
Labels	__Source__, __Chunk__, __Topic__	__User__tenant__, __Account__tenant__
Isolation	Same Neptune cluster, different label namespaces	No collision

Hybrid Search Flow

1. Document-graph writes typed nodes to Neptune (__User__acme__)
2. Same data is converted to LlamaIndex Documents with lineage headers
3. LexicalGraphIndex.extract_and_build() indexes them into semantic search
4. LexicalGraphQueryEngine.retrieve() returns results with embedded lineage
5. Lineage headers ([User | u1 | acme]) enable correlation back to original nodes

Example Usage

from graphrag_toolkit.lexical_graph.storage import GraphStoreFactory
from document_graph.graph_build import node_to_cypher
from document_graph import Node

# Write structured data to Neptune
gs = GraphStoreFactory.for_graph_store('neptune-db://endpoint:8182').__enter__()
node = Node(id='u1', labels=['User'], properties={'name': 'Alice', 'role': 'admin'})
cypher, params = node_to_cypher(node, tenant_id='my_app')
gs.execute_query(cypher, params)

# Query typed nodes
from document_graph.query import DocumentGraphQueryEngine
engine = DocumentGraphQueryEngine(gs, tenant_id='my_app')
admins = engine.find_by_property('User', 'role', 'admin')

What We're Proposing

We'd like to contribute document-graph as a companion package within graphrag-toolkit (similar to how lexical-graph is structured), providing:

1. A standard path for structured data → Neptune alongside lexical-graph
2. Shared TenantId, versioning, and storage infrastructure
3. Hybrid search patterns that combine both graph types
4. Local development drivers (GraphJSON, FalkorDB) that benefit both packages

Happy to discuss the integration approach, packaging structure, or scope adjustments.

Proposed solution

No response

Alternatives considered

No response

[noel-improv]

Thanks for the detailed write-up @evanerwee this is a clean proposal and the lexical-graph alignment is already most of the way there.

On the question of merging or keeping separate: one solution, two packages. document-graph is a second ingestion route into the same Neptune store, not a competitor to lexical-graph. It already declares lexical as an optional dependency, and lexical already ships a DocumentGraphReaderProvider seam, so half the bridge exists. I'd keep them as separate installs the same way byokg-rag works rather than folding them into one package or shipping two standalone products.

Here are the proposed steps:

1. Namespace: move the code under graphrag_toolkit/document_graph and name the dist graphrag-toolkit-document-graph, matching byokg-rag. Right now it lives at top-level document_graph, outside the shared namespace.
2. Tenant model: write tenant labels through lexical's TenantId.format_label() instead of hand-building {label}{tenant_id}__ in cypher_builder.py. The two formats don't match today (lexical emits backtick-quoted Label__tenant__, document-graph emits Labeltenant__), so coexistence and hybrid search only work once both sides go through the same code.
3. Dependency direction: keep lexical optional so document-graph still runs standalone for pure ETL, but have the hybrid path import the real TenantId and VersioningConfig instead of re-implementing them.
4. Repo hygiene: drop the executed notebooks and committed sample/output data from the PR. Strip notebook outputs and keep inputs minimal, in line with the other packages.

A question on ownership: If we take this path to clean it up (+ subsequently merging and shipping the hybrid path) is your team planning to use and maintain document-graph going forward? That changes how much we build into the shared seam versus treating this as a one-time contribution.

[evanerwee]

WIll do so today, and re submit.

[evanerwee]

@noel-improv — update on progress and a commitment on ownership:

Status of requested changes:

1. ✅ Namespace: Code is structured under src/graphrag_toolkit/document_graph on the feature/document-graph branch. Still need to update pyproject.toml dist name from document-graph → graphrag-toolkit-document-graph and move the hatch build target.
2. 🔧 Tenant model: Currently cypher_builder.py hand-builds labels as __Label__tenant__. Will refactor to use TenantId.format_label() so both packages produce identical `Label{tenant}__` format. This is the key fix for hybrid search coexistence.
3. ✅ Dependency direction: lexical is already optional — document-graph runs standalone for pure ETL, imports TenantId and VersioningConfig from lexical only when available.
4. ✅ Repo hygiene: Notebooks are stripped of outputs. Will trim sample data to minimal inputs only.

On ownership:

Yes — we are actively using document-graph for a client engagement and will maintain it going forward. Specifically:

• We will keep document-graph in lockstep with lexical-graph (API changes, versioning, TenantId evolution)
• We will fix bugs and respond to issues
• We will contribute improvements back as we build on it

This is not a one-time contribution. It is a package we depend on in production and will steward accordingly.

Will submit the cleaned-up PR with the namespace rename and tenant format fix shortly.

[evanerwee]

@noel-improv — all four items now addressed in PR #365 (commit 8333bbb):

Item	Fix
1. Namespace	✅ graphrag-toolkit-document-graph under src/graphrag_toolkit/document_graph
2. Tenant model	✅ cypher_builder.py now uses TenantId.format_label() — produces `Label{tenant}__` matching lexical-graph exactly
3. Dependency direction	✅ Lexical is optional — standalone ETL works, hybrid path imports real TenantId
4. Repo hygiene	✅ All 24 notebooks stripped of outputs

PR #365 description updated with the full breakdown. Ready for re-review when you have a moment.