[FEATURE] Add OpenSearch (opensearch://) vector store support alongside AOSS by noel-improv · Pull Request #397 · awslabs/graphrag-toolkit

noel-improv · 2026-07-14T21:48:43Z

Description

Adds an opensearch:// connection string for an OpenSearch endpoint that isn't Amazon OpenSearch Serverless (AOSS) — a local Docker or Finch container, an EC2 instance, or any other OpenSearch deployment — alongside the existing aoss:// one. Today the vector store factory only recognizes AOSS endpoints, and the client creation code hardcodes AWS SigV4 signing, so there's no way to point the toolkit at a plain OpenSearch instance without AWS credentials.

Changes

OpenSearchVectorIndexFactory recognizes an opensearch://<endpoint> prefix and routes it through a new is_local flag on OpenSearchIndex, instead of AOSS detection.
create_os_client/create_os_async_client skip AWS SigV4 on this path. They authenticate with HTTP basic auth from GraphRAGConfig.opensearch_username/opensearch_password (backed by OPENSEARCH_USERNAME/OPENSEARCH_PASSWORD), or with no auth if neither is set, and log a warning if only one of the two is set.
index_exists() now accepts client_kwargs, threaded from OpenSearchIndex.client_kwargs, so a plain-HTTP or self-signed endpoint can override use_ssl/verify_certs (this path previously always forced use_ssl=True, which the AOSS-only code never needed to override).
is_local also selects is_aoss=False on the underlying vector client, so bulk ingest calls refresh() after writes, matching OpenSearch's own behavior instead of AOSS's.
Reuses the existing Classic knn_vector mapping (engine: faiss/nmslib) unchanged — no new mapping logic needed.
README and docs-site updated with the new connection string and its auth behavior.

Problem

VectorStoreFactory.for_vector_store() has no way to target an OpenSearch endpoint that isn't Amazon OpenSearch Serverless; every code path assumes AWS SigV4 and an AOSS endpoint.

Related issue (if any): #

Testing

Unit tests added/updated
Integration tests added (as appropriate)
Existing tests pass (pytest)
Tested manually (describe below)

Unit tests cover the factory's opensearch:// detection, the basic-auth/no-auth/partial-credential client paths, and client_kwargs threading through index_exists(). Also ran a live smoke test against two Finch-hosted OpenSearch containers: an unsecured plain-HTTP instance (index creation, bulk ingest, query, confirming refresh() fires) and a basic-auth instance with a real admin password (confirming both successful auth and rejection of a wrong password).

Checklist

Code follows existing style and conventions
License headers present on new files
Documentation updated (if applicable)
No breaking changes (or clearly documented)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Add an opensearch:// connection string for self-managed (non-AWS) OpenSearch instances, alongside the existing aoss:// one. The local path skips AWS SigV4 entirely: it authenticates with HTTP basic auth from OPENSEARCH_USERNAME/OPENSEARCH_PASSWORD (env vars or GraphRAGConfig), or with no auth if neither is set. index_exists() now threads client_kwargs through to the OpenSearch client, so callers can override use_ssl/verify_certs for a plain-HTTP or self-signed local endpoint. The existing Classic knn_vector mapping is reused unchanged.

…gs last Docs and docstrings previously called the new non-AOSS path "self-managed" or "local" OpenSearch, which understates it -- OpenSearch runs anywhere (local, EC2, on-prem), and "local" only covers one case. Switch to plain "OpenSearch" for the open source path and "Amazon OpenSearch Serverless (AOSS)" for the managed one, matching the existing env var naming convention. Also make client_kwargs consistently the last parameter across create_opensearch_vector_client, OpenSearchIndex.for_index, and the OpenSearchIndex model fields -- it was sometimes before is_local, sometimes after.

mykola-pereyma

Few suggestions:

Could we rename is_local to is_serverless (inverted, default True)? The flag means "use AOSS SigV4 auth" vs "use basic auth or no auth" — not whether the instance is local. An EC2-hosted OpenSearch cluster is not local but would use this path. is_serverless matches AWS terminology and reads naturally at call sites.
The docs page and README should mention how to connect over plain HTTP or with self-signed certs — local Docker users will hit SSL errors with the https default. Something like: "For plain HTTP, use opensearch://http://localhost:9200. For self-signed TLS, pass client_kwargs={"verify_certs": False} to the factory."
Worth adding a note that opensearch:// is for instances without AWS authentication (basic auth or no auth). A user with a managed OpenSearch Service domain (non-serverless but still on AWS) might try this prefix and get auth failures — a sentence clarifying that managed OpenSearch Service with IAM auth is not yet supported would save confusion.

…SS detection The flag distinguishes Amazon OpenSearch Serverless (AOSS, SigV4 auth) from an OpenSearch endpoint that isn't AOSS (basic auth or no auth), which is what the auth path actually keys off -- not whether the instance is local. An EC2-hosted OpenSearch cluster isn't local but takes the non-serverless path. is_serverless matches AWS terminology and reads at call sites; it defaults to True (AOSS), so aoss:// stays the zero-config default and opensearch:// sets it False. - Invert is_local -> is_serverless across create_os_client/async, index_exists, create_opensearch_vector_client, OpenSearchIndex, and the factory. - Rename the OPENSEARCH_LOCAL constant to OPENSEARCH and _local_http_auth to _basic_http_auth. - Addresses review feedback on awslabs#397.

…S-auth scope for opensearch:// Note that opensearch:// targets instances without AWS authentication (basic auth or no auth) and that a managed Amazon OpenSearch Service domain using IAM auth is not supported through this prefix yet. Document plain-HTTP endpoints (opensearch://http://...) and disabling certificate verification for self-signed TLS via client_kwargs={'verify_certs': False}. Addresses review feedback on awslabs#397.

acarbonetto

looks good - I agree that we need to move away from Open Source == Local. OpenSearch can be deployed in more ways than just a localhost. IF we only want to support localhost (for now) and raise a follow-up ticket to support more options later, that's fine. But the language in the code should be updated now.

Follows review on awslabs#397: the flag selects AWS SigV4 auth vs basic/no auth, so is_sigv4_auth names the mechanism directly at call sites. Same truth values as before (True for AOSS), no behavior change.

…view Rename the section to "Connecting to an OpenSearch vector store", link the opensearch-py client, note OpenSearch's SSL-on default, and mark managed Amazon OpenSearch Service (IAM/SigV4) as not yet supported, tracked in awslabs#407. Addresses review on awslabs#397.

…-vector-store-AN-3253 # Conflicts: # lexical-graph/src/graphrag_toolkit/lexical_graph/config.py # lexical-graph/src/graphrag_toolkit/lexical_graph/storage/vector/opensearch_vector_indexes.py # lexical-graph/tests/unit/test_config.py

mykola-pereyma

LGTM

acarbonetto · 2026-07-17T22:14:51Z

+
+### Connecting to an OpenSearch vector store
+
+You can also use OpenSearch directly as a vector store, instead of Amazon OpenSearch Serverless — in a Docker or Finch container, on an EC2 instance, or on your own infrastructure. graphrag-toolkit connects to it with the [opensearch-py](https://pypi.org/project/opensearch-py/) client. Supply a connection string that begins `opensearch://`, followed by the endpoint:


Suggested change

You can also use OpenSearch directly as a vector store, instead of Amazon OpenSearch Serverless — in a Docker or Finch container, on an EC2 instance, or on your own infrastructure. graphrag-toolkit connects to it with the [opensearch-py](https://pypi.org/project/opensearch-py/) client. Supply a connection string that begins `opensearch://`, followed by the endpoint:

You can use [OpenSearch](opensearch.org) directly as a vector store — in a Docker container, on an EC2 instance, or on your own infrastructure. graphrag-toolkit connects to it with the [opensearch-py](https://pypi.org/project/opensearch-py/) client. Supply a connection string that begins with `opensearch://`, followed by the endpoint:

Question: why do we require the "opensearch://" string? Doesn't this get removed before being sent to opensearch-py?

You are right, the prefix is stripped before opensearch-py sees anything; fixing

acarbonetto · 2026-07-17T22:19:19Z

+
+The `opensearch://` prefix targets OpenSearch instances that don't use AWS authentication (basic auth or no auth). A managed Amazon OpenSearch Service domain that authenticates with IAM is not supported through this prefix yet ([tracked in #407](https://github.com/awslabs/graphrag-toolkit/issues/407)).
+
+For a local Docker or Finch container over plain HTTP, use `opensearch://http://localhost:9200`. For an endpoint with a self-signed TLS certificate, disable certificate verification by passing `client_kwargs`:


A lot of the above is supplied by the opensearch-py documentation, since we are mostly just forwarding certs and auth to opensearch-py there's no need to duplicate documentation.

I would suggest removing a lot of the above documentation, and remove any reference to Amazon (opensearch is not defined by not being an Amazon product). We can include a "limitations" section to discuss how we don't yet support https:// endpoints or any auth other than basic auth (please reference #407 or other new issues to fix these limitations).

Actually, we do support https:// with certs, by passing client_kwargs. Good. We can document this is it differs from the opensearch-py documentation. However, if its the same documentation & examples, just reference opensearch-py documentation directly.

acarbonetto · 2026-07-17T22:21:43Z

            endpoint = vector_index_info[len(OPENSEARCH_SERVERLESS):]
            if not endpoint.startswith('https://') and not endpoint.startswith('http://'):
                endpoint = f'https://{endpoint}'
+        elif vector_index_info.startswith(OPENSEARCH):


consider remove this constraint on the endpoint. If it starts with aoss:// - it's an amazon opensearch serverless endpoint. Otherwise, it's for opensearch endpoint.

acarbonetto · 2026-07-17T22:23:05Z

-    region = GraphRAGConfig.aws_region
-    credentials = session.get_credentials()
-    service = 'aoss'
+    if not is_sigv4_auth:


add a comment here to #407 since this will need to be cleaned up for more use cases. We should support more auth options than just sigv4 and basic auth.

acarbonetto · 2026-07-17T22:23:18Z

-    region = GraphRAGConfig.aws_region
-    credentials = session.get_credentials()
-    service = 'aoss'
+    if not is_sigv4_auth:


add a comment here to #407

acarbonetto · 2026-07-17T22:26:08Z


        if not self._client:
-            if index_exists(self.endpoint, self.underlying_index_name(), self.dimensions, self.writeable):
+            if index_exists(self.endpoint, self.underlying_index_name(), self.dimensions, self.writeable, is_sigv4_auth=self.is_sigv4_auth, client_kwargs=self.client_kwargs):


this line is more readable if you save the result to a variable.

does_index_exist = index_exists(self.endpoint, self.underlying_index_name(), self.dimensions, self.writeable, is_sigv4_auth=self.is_sigv4_auth, client_kwargs=self.client_kwargs) if does_index_exist: self._client = create_opensearch_vector_client( ...

Pass opensearch:// endpoints through without forcing an https:// scheme. Derive the clients' use_ssl default from the endpoint scheme, since opensearch-py only honors the scheme for https:// and a plain-HTTP endpoint would otherwise still attempt TLS; an explicit use_ssl in client_kwargs still wins. Reject an empty endpoint instead of silently falling through to the dummy vector store. Trim docs that duplicated opensearch-py documentation, link it directly, and add a limitations section referencing awslabs#407. Add TODO(awslabs#407) markers at the auth branches and extract does_index_exist for readability.

noel-improv added 2 commits July 14, 2026 15:46

docs(lexical-graph): document the opensearch:// local vector store

3e06dc0

noel-improv marked this pull request as ready for review July 15, 2026 22:45

noel-improv changed the title ~~[FEATURE] Add self-managed OpenSearch (opensearch://) vector store support~~ [FEATURE] Add OpenSearch (opensearch://) vector store support alongside AOSS Jul 15, 2026

noel-improv requested a review from oussamahansal July 16, 2026 16:57

mykola-pereyma reviewed Jul 16, 2026

View reviewed changes

noel-improv added 2 commits July 16, 2026 15:49

acarbonetto reviewed Jul 16, 2026

View reviewed changes

noel-improv mentioned this pull request Jul 17, 2026

[FEATURE] Support Amazon OpenSearch Service (managed, non-serverless) with SigV4/IAM auth #407

Open

noel-improv added 3 commits July 17, 2026 09:36

refactor(lexical-graph): name the AOSS auth flag is_sigv4_auth

9c80c3c

Follows review on awslabs#397: the flag selects AWS SigV4 auth vs basic/no auth, so is_sigv4_auth names the mechanism directly at call sites. Same truth values as before (True for AOSS), no behavior change.

mykola-pereyma approved these changes Jul 17, 2026

View reviewed changes

acarbonetto reviewed Jul 17, 2026

View reviewed changes


		### Connecting to an OpenSearch vector store

		You can also use OpenSearch directly as a vector store, instead of Amazon OpenSearch Serverless — in a Docker or Finch container, on an EC2 instance, or on your own infrastructure. graphrag-toolkit connects to it with the [opensearch-py](https://pypi.org/project/opensearch-py/) client. Supply a connection string that begins `opensearch://`, followed by the endpoint:

	You can also use OpenSearch directly as a vector store, instead of Amazon OpenSearch Serverless — in a Docker or Finch container, on an EC2 instance, or on your own infrastructure. graphrag-toolkit connects to it with the [opensearch-py](https://pypi.org/project/opensearch-py/) client. Supply a connection string that begins `opensearch://`, followed by the endpoint:
	You can use [OpenSearch](opensearch.org) directly as a vector store — in a Docker container, on an EC2 instance, or on your own infrastructure. graphrag-toolkit connects to it with the [opensearch-py](https://pypi.org/project/opensearch-py/) client. Supply a connection string that begins with `opensearch://`, followed by the endpoint:


		The `opensearch://` prefix targets OpenSearch instances that don't use AWS authentication (basic auth or no auth). A managed Amazon OpenSearch Service domain that authenticates with IAM is not supported through this prefix yet ([tracked in #407](https://github.com/awslabs/graphrag-toolkit/issues/407)).

		For a local Docker or Finch container over plain HTTP, use `opensearch://http://localhost:9200`. For an endpoint with a self-signed TLS certificate, disable certificate verification by passing `client_kwargs`:

Uh oh!

Conversation

noel-improv commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Problem

Testing

Checklist

Uh oh!

mykola-pereyma left a comment

Choose a reason for hiding this comment

Uh oh!

acarbonetto left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mykola-pereyma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

noel-improv commented Jul 14, 2026 •

edited

Loading