Skip to content

[FEATURE] Add RDF/SPARQL graph store with optional Neptune IAM authentication#405

Open
KilianTrunk wants to merge 3 commits into
awslabs:mainfrom
Ortecha:feat/native-sparql-rdflib-iam
Open

[FEATURE] Add RDF/SPARQL graph store with optional Neptune IAM authentication#405
KilianTrunk wants to merge 3 commits into
awslabs:mainfrom
Ortecha:feat/native-sparql-rdflib-iam

Conversation

@KilianTrunk

Copy link
Copy Markdown

Description

This PR adds a new lexical-graph-contrib/sparql package for storing and retrieving the lexical graph through SPARQL 1.1 query and update endpoints.

The package uses RDFLib and supports generic HTTP endpoints. It also provides optional IAM authentication for the Amazon Neptune SPARQL endpoint.

Lexical graph builders and retrievers now pass a GraphOperation value together with the existing graph query. The SPARQL graph store uses this value to select its native SPARQL implementation. Existing graph stores continue to execute their existing queries.

Changes

  • Add the graphrag-toolkit-lexical-graph-sparql contributor package.
  • Add RDFLib and Requests as package dependencies.
  • Add Botocore as an optional dependency for Neptune IAM authentication.
  • Add a graph store factory for sparql, sparql+http, sparql+https, and sparql+neptune connection schemes.
  • Support separate SPARQL query and update endpoints.
  • Support HTTP Basic authentication, custom HTTP headers, and request timeouts for generic endpoints.
  • Add native SPARQL implementations for all 32 graph operations introduced by this PR: 17 update operations and 15 query operations.
  • Represent lexical graph nodes with deterministic RDF IRIs and configurable namespaces.
  • Represent facts as RDF resources connected to their subject, predicate, object, and supporting statement.
  • Store each tenant in a deterministic named graph.
  • Sign Neptune SPARQL requests with AWS Signature Version 4 for the neptune-db service.
  • Resolve Neptune credentials through Botocore for every request so temporary credentials can be refreshed.
  • Pass GraphOperation through retries, batched writes, query trees, and multi-tenant graph stores.
  • Keep the existing query execution as the default for graph stores that do not provide an operation-specific implementation.
  • Add SPARQL unit tests and extend core tests for operation forwarding and graph store factory selection.
  • Add a contributor README and an RDF/SPARQL page to the documentation site.
  • Rename the local development Compose project from local-dev to graphrag-toolkit-rdf-dev.
  • Run the Jupyter reader dependency installation as root, apply fix-permissions to the Conda and notebook-user directories, and then return to the jovyan user.

Problem

The toolkit did not provide a GraphStore implementation for SPARQL query and update endpoints.

Because of this, the lexical graph storage interface could not be used with an RDF store through SPARQL. The toolkit also did not provide an IAM-signed transport for requests to the Amazon Neptune SPARQL endpoint.

This PR adds the missing SPARQL graph store as a contributor package while keeping the current graph stores and query execution paths.

Related issue (if any): N/A

Testing

  • Unit tests added/updated
  • Integration tests added (as appropriate)
  • Existing tests pass (pytest)
  • Tested manually (describe below)

Test results:

  • pytest lexical-graph-contrib/sparql/tests: 108 passed
  • pytest lexical-graph/tests: 1,756 passed and 1 skipped
  • git diff --check upstream/main...HEAD: no errors

Endpoint behavior was tested manually against RDFox, GraphDB, and Amazon Neptune.

Checklist

  • Code follows existing style and conventions
  • License headers present on new files
  • Documentation updated (if applicable)
  • No breaking changes

The new GraphOperation argument is optional. Graph stores without an operation-specific implementation continue to execute their existing query.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.


Supported schemes are:

| Scheme | Endpoint transport |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all these bespoke scheme extensions really nessecary? It measn that developers will have to do some string concatination any time they want to pass in a SPARQL endpoint. I would prefer we found another way to specify specific headers for authentication.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants