Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs-site/astro.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ export default defineConfig({
{ label: 'Neptune Database', slug: 'lexical-graph/graph-store-neptune-db' },
{ label: 'Neo4j', slug: 'lexical-graph/graph-store-neo4j' },
{ label: 'FalkorDB', slug: 'lexical-graph/graph-store-falkor-db' },
{ label: 'RDF / SPARQL Stores', slug: 'lexical-graph/graph-store-sparql' },
],
},
{
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import { Card, CardGrid, Code } from '@astrojs/starlight/components';
Bring your own knowledge graph. Plug an existing graph into a multi-strategy KGQA pipeline without re-extracting anything.
</Card>
<Card title="Pluggable storage" icon="document">
Graph stores: Amazon Neptune (DB and Analytics), Neo4j, FalkorDB. Vector stores: Neptune, OpenSearch, Postgres, S3 Vectors.
Graph stores: Amazon Neptune (DB and Analytics), Neo4j, FalkorDB, RDF/SPARQL stores. Vector stores: Neptune, OpenSearch, Postgres, S3 Vectors.
</Card>
<Card title="Open source" icon="github">
Apache 2.0, developed in the open by AWS Labs.
Expand Down
152 changes: 152 additions & 0 deletions docs-site/src/content/docs/lexical-graph/graph-store-sparql.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
---
title: RDF / SPARQL stores
---

The RDF / SPARQL contributor package stores the lexical graph in an existing
SPARQL 1.1 query/update endpoint. Generic standards-compatible endpoints are the
default; Amazon Neptune IAM authentication is an optional transport.

The adapter executes native SPARQL. Builders and retrievers provide a
backend-neutral operation identifier and structured parameters. Property-graph
stores execute their native queries, while the RDF store selects the
corresponding native SPARQL operation.

### Install and register

Install the contributor package from a repository checkout:

```bash
pip install ./lexical-graph-contrib/sparql
```

Register its factory before creating the graph store:

```python
from graphrag_toolkit.lexical_graph.storage import GraphStoreFactory
from graphrag_toolkit_contrib.lexical_graph.storage.graph.sparql import (
SPARQLGraphStoreFactory,
)

GraphStoreFactory.register(SPARQLGraphStoreFactory)
```

Connect the graph store to an existing repository or dataset.

### Generic endpoint

Provide the query URL and, if different, the update URL:

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+https://rdf.example.com/query',
update_endpoint='https://rdf.example.com/update',
)
```

Supported schemes are:

| Scheme | Endpoint transport |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all these bespoke scheme extensions really nessecary? It measn that developers will have to do some string concatination any time they want to pass in a SPARQL endpoint. I would prefer we found another way to specify specific headers for authentication.

|---|---|
| `sparql://host/path` | HTTP |
| `sparql+http://host/path` | HTTP |
| `sparql+s://host/path` | HTTPS |
| `sparql+https://host/path` | HTTPS |
| `sparql+neptune://host:8182` | HTTPS with Neptune IAM SigV4 |

If `update_endpoint` is omitted, the query URL is used for both operations.

HTTP Basic credentials can be passed in the URL, through `username` and
`password`, or through the `SPARQL_USER` and `SPARQL_PASSWORD` environment
variables. Custom headers support bearer tokens and endpoint-specific headers:

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+https://rdf.example.com/query',
headers={'Authorization': 'Bearer token'},
)
```

### Amazon Neptune IAM

Install the optional botocore dependency:

```bash
pip install './lexical-graph-contrib/sparql[neptune]'
```

Then use the Neptune scheme and AWS region:

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+neptune://my-cluster.cluster-abcdefghijkl.eu-central-1.neptune.amazonaws.com:8182',
region_name='eu-central-1',
)
```

Each request is signed for the `neptune-db` service using credentials from
botocore's standard provider chain. Credentials are resolved on every request,
allowing botocore to refresh temporary role, web-identity, IAM Identity Center,
ECS, and EC2 credentials. IAM transport requires HTTPS.

The same transport is available as a plain RDFLib graph:

```python
from graphrag_toolkit_contrib.lexical_graph.storage.graph.sparql.neptune_iam import (
neptune_iam_graph,
)

graph = neptune_iam_graph(
'https://my-cluster.cluster-abcdefghijkl.eu-central-1.neptune.amazonaws.com:8182',
region_name='eu-central-1',
)
try:
rows = graph.query('SELECT * WHERE { ?s ?p ?o } LIMIT 10')
finally:
graph.close()
```

### Namespaces and tenants

The default schema namespace is
`https://awslabs.github.io/graphrag-toolkit/lexical#`; the default instance
namespace is `https://awslabs.github.io/graphrag-toolkit/lexical/`.

Customize them only when creating a new dataset:

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+https://rdf.example.com/query',
update_endpoint='https://rdf.example.com/update',
lexical_prefix='gt',
lexical_schema_namespace='https://example.com/graph/schema#',
lexical_instance_namespace='https://example.com/graph/data/',
)
```

Every tenant, including the default tenant, uses a deterministic named graph.
Writes target that graph explicitly, and reads select it with the standard
SPARQL Protocol `default-graph-uri` parameter.

### RDF model

Sources, chunks, topics, statements, facts, and entities retain their existing
lexical-graph identities. A node becomes a deterministic IRI with an RDF class
and `lg:id`; node properties become literal-valued predicates. Simple edges
become RDF predicates.

Extracted facts record their subject, predicate, and object directly:

```turtle
<fact/f1> a lg:Fact ;
lg:subject <entity/amazon> ;
lg:predicate <relation/produces> ;
lg:object <entity/ec2> ;
lg:supports <statement/s1> ;
lg:value "Amazon PRODUCES EC2" .

<relation/produces> a lg:Relation ;
lg:value "PRODUCES" .
```

This is the toolkit's fact model, with the fact resource carrying its subject,
predicate, object, and supporting statement.
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/lexical-graph/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The graphrag-toolkit [lexical-graph](https://github.com/awslabs/graphrag-toolkit
Source → chunk → topic → statement → fact → entity, all linked. Retrieval can hop between any of these levels.
</Card>
<Card title="Pluggable storage" icon="document">
Graph: Amazon Neptune (DB and Analytics), Neo4j, FalkorDB. Vectors: Neptune, OpenSearch, Postgres, S3 Vectors.
Graph: Amazon Neptune (DB and Analytics), Neo4j, FalkorDB, RDF/SPARQL stores. Vectors: Neptune, OpenSearch, Postgres, S3 Vectors.
</Card>
<Card title="Two-stage indexing" icon="seti:config">
Extract and build run as separate micro-batched pipelines so ingest is continuous and resumable.
Expand Down
3 changes: 2 additions & 1 deletion docs-site/src/content/docs/lexical-graph/storage-model.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Graph stores and vector stores provide connectivity to an *existing* storage ins

### Graph store

Graph stores must support the [openCypher](https://opencypher.org/) property graph query language. Graph construction queries typically use an `UNWIND ... MERGE` idiom to create or update the graph for a [batch of inputs](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/best-practices-content.html#best-practices-content-14). The Neptune graph store implementations override the `GraphStore.node_id()` method to ensure that node ids in the code (e.g. `chunkId`) are mapped to Neptune's `~id` reserved property. Alternative graph store implementations can leave the base implementation of `node_id()` as-is. This will result in node ids being mapped to a property of the same name (i.e. a reference to `chunkId` in the code will be mapped to a `chunkId` property of a node).
The built-in property-graph stores use [openCypher](https://opencypher.org/). Graph construction queries typically use an `UNWIND ... MERGE` idiom to create or update the graph for a [batch of inputs](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/best-practices-content.html#best-practices-content-14). Graph builders and retrievers also identify their operation explicitly, allowing stores such as the RDF / SPARQL contributor to execute a native implementation. The Neptune graph store implementations override the `GraphStore.node_id()` method to ensure that node ids in the code (e.g. `chunkId`) are mapped to Neptune's `~id` reserved property. Alternative graph store implementations can leave the base implementation of `node_id()` as-is. This will result in node ids being mapped to a property of the same name (i.e. a reference to `chunkId` in the code will be mapped to a `chunkId` property of a node).

You use the `GraphStoreFactory.for_graph_store()` static factory method to create a graph store.

Expand All @@ -30,6 +30,7 @@ The lexical-graph supports the following graph databases:
- [Amazon Neptune](/graphrag-toolkit/lexical-graph/graph-store-neptune-db/)
- [Amazon Neptune Analytics](/graphrag-toolkit/lexical-graph/graph-store-neptune-analytics/)
- [Neo4j](/graphrag-toolkit/lexical-graph/graph-store-neo4j/)
- [RDF / SPARQL Stores](/graphrag-toolkit/lexical-graph/graph-store-sparql/)

#### Logging graph queries

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: local-dev
name: graphrag-toolkit-rdf-dev
services:
neo4j-local:
image: neo4j:5.25-community
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ RUN pip install --no-cache-dir \
plotly

# LlamaIndex readers (hard imports in lexical-graph source)
USER root
RUN pip install --no-cache-dir \
llama-index-readers-web \
llama-index-readers-file \
Expand All @@ -42,6 +43,8 @@ RUN pip install --no-cache-dir \
pymupdf

# Build tools for packages requiring C compilation (e.g. lru-dict)
USER root
RUN apt-get update && apt-get install -y --no-install-recommends build-essential && rm -rf /var/lib/apt/lists/*
RUN apt-get update && apt-get install -y --no-install-recommends build-essential && \
rm -rf /var/lib/apt/lists/* && \
fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}"
USER jovyan
170 changes: 170 additions & 0 deletions lexical-graph-contrib/sparql/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
# RDF / SPARQL graph store

This contributor package stores the AWS GraphRAG Toolkit lexical graph in an
existing SPARQL 1.1 query/update endpoint. It is endpoint-neutral by default and
has an optional Amazon Neptune IAM transport.

Core graph builders and retrievers identify a backend-neutral `GraphOperation`.
Property-graph stores execute their native queries, while this package provides
the native SPARQL implementation of each operation.

The update renderer batches related statements into a single request. Calling
RDFLib `Graph.add()` and `Graph.remove()` for every remote triple would be much
less efficient and would make multi-statement counters non-atomic.

## Install

From a repository checkout:

```bash
pip install ./lexical-graph-contrib/sparql
```

Include the optional botocore dependency only when Neptune IAM authentication is
required:

```bash
pip install './lexical-graph-contrib/sparql[neptune]'
```

## Generic SPARQL endpoint

Register the contributor factory once before resolving a graph store:

```python
from graphrag_toolkit.lexical_graph.storage import GraphStoreFactory
from graphrag_toolkit_contrib.lexical_graph.storage.graph.sparql import (
SPARQLGraphStoreFactory,
)

GraphStoreFactory.register(SPARQLGraphStoreFactory)

graph_store = GraphStoreFactory.for_graph_store(
'sparql+https://rdf.example.com/query',
update_endpoint='https://rdf.example.com/update',
)
```

Connect the graph store to an existing repository or dataset.

Supported connection schemes are:

| Scheme | HTTP endpoint |
|---|---|
| `sparql://host/path` | `http://host/path` |
| `sparql+http://host/path` | `http://host/path` |
| `sparql+s://host/path` | `https://host/path` |
| `sparql+https://host/path` | `https://host/path` |
| `sparql+neptune://host:8182` | `https://host:8182/sparql`, signed with IAM |

If `update_endpoint` is omitted, the query endpoint is used for both operations.
It can also be supplied as an encoded `update_endpoint` query parameter.

### Generic authentication

HTTP Basic credentials can be supplied in the connection URL, as keyword
arguments, or through `SPARQL_USER` and `SPARQL_PASSWORD`:

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+https://rdf.example.com/query',
username='service-user',
password='secret',
headers={'X-Request-Origin': 'graphrag-toolkit'},
)
```

Use `headers={'Authorization': 'Bearer ...'}` for bearer-token endpoints. Avoid
putting credentials directly in a URL when the URL may be logged by surrounding
application code.

## Amazon Neptune with IAM

The Neptune scheme changes only the transport. The RDF model, operations, and
SPARQL implementation remain the same as for every other endpoint.

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+neptune://my-cluster.cluster-abcdefghijkl.eu-central-1.neptune.amazonaws.com:8182',
region_name='eu-central-1',
)
```

`NeptuneIAMAuth` uses botocore's standard credential provider chain and obtains
credentials for every request. This allows botocore to refresh temporary role,
web-identity, IAM Identity Center, ECS, and EC2 credentials. Requests are signed
for the `neptune-db` service with SigV4 and require HTTPS.

The transport can also be used as a plain RDFLib graph:

```python
from graphrag_toolkit_contrib.lexical_graph.storage.graph.sparql.neptune_iam import (
neptune_iam_graph,
)

graph = neptune_iam_graph(
'https://my-cluster.cluster-abcdefghijkl.eu-central-1.neptune.amazonaws.com:8182',
region_name='eu-central-1',
)
try:
rows = graph.query('SELECT * WHERE { ?s ?p ?o } LIMIT 10')
finally:
graph.close()
```

## Namespaces and tenant isolation

The default schema namespace is
`https://awslabs.github.io/graphrag-toolkit/lexical#`; the default instance
namespace is `https://awslabs.github.io/graphrag-toolkit/lexical/`.

They can be changed when creating the store:

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+https://rdf.example.com/query',
update_endpoint='https://rdf.example.com/update',
lexical_prefix='gt',
lexical_schema_namespace='https://example.com/graph/schema#',
lexical_instance_namespace='https://example.com/graph/data/',
)
```

Changing namespaces changes the IRIs written for new data. Use one configuration
consistently for a repository.

Every tenant, including the default tenant, uses a deterministic named graph
below the instance namespace. Writes use `GRAPH` and reads pass the tenant graph
as the SPARQL Protocol `default-graph-uri`, keeping tenant selection independent
of endpoint-specific union-default-graph behavior.

## RDF model

Lexical nodes receive deterministic IRIs, an RDF class, and the original id as
`lg:id`. Simple edges become RDF predicates. Predicates whose property-graph
name is ambiguous are specialized by domain, for example
`lg:statementMentionedIn` versus `lg:topicMentionedIn`.

Extracted facts are represented as statement resources:

```turtle
<fact/f1> a lg:Fact ;
lg:subject <entity/amazon> ;
lg:predicate <relation/produces> ;
lg:object <entity/ec2> ;
lg:supports <statement/s1> ;
lg:value "Amazon PRODUCES EC2" .

<relation/produces> a lg:Relation ;
lg:value "PRODUCES" .
```

This is the toolkit's fact model, with the fact resource carrying its subject,
predicate, object, and supporting statement.

## Tests

```bash
pytest lexical-graph-contrib/sparql/tests
pytest lexical-graph/tests
```
Loading