[FEATURE] Add support for AOSS NextGen vector search collections

[acarbonetto]

Package

lexical-graph

Problem statement

The existing OpenSearch Serverless (AOSS) vector store integration targets Classic collections. When pointed at an AOSS NextGen vector search collection endpoint, the integration fails at index creation with:

illegal_argument_exception: "Field parameter 'engine' is not supported"

This is because AOSS NextGen (OpenSearch 3.x) removed the engine and mode parameters from knn_vector field mappings — the system now determines the optimal configuration internally. The existing code unconditionally includes "engine": "nmslib" or "engine": "faiss" in the method block, which is rejected by NextGen endpoints.

NextGen is AWS's recommended collection type for new deployments and offers significant advantages: scale-to-zero pricing, GPU-accelerated index builds, 32x compression by default, and simplified index management. Users who create new collections today will likely be using NextGen, making this a growing compatibility issue.

Proposed solution

Add a GraphRAGConfig flag to opt in to NextGen-compatible behaviour. The flag approach is intentional — if NextGen supersedes Classic entirely in future, removing the flag and its conditional branches is straightforward.

1. Two new GraphRAGConfig properties

opensearch_serverless_nextgen: bool

• Default: False
• Env var: OPENSEARCH_SERVERLESS_NEXTGEN ("true" to enable)
• Controls whether NextGen-compatible index mappings and settings are used

opensearch_serverless_nextgen_compression: Optional[str]

• Default: None (uses NextGen's built-in default of 32x)
• Env var: OPENSEARCH_SERVERLESS_NEXTGEN_COMPRESSION
• Valid values: "1x", "2x", "8x", "16x", "32x"
• When set, adds compression_level to the field mapping

Both follow the existing opensearch_engine pattern (private backing field, property with env var fallback, setter).

2. NextGen index mapping in opensearch_vector_indexes.py

When GraphRAGConfig.opensearch_serverless_nextgen is True, index_exists() builds a NextGen-compatible mapping:

Classic mapping (existing, unchanged):

{
    "settings": {"index": {"knn": True, "knn.algo_param.ef_search": 100}},
    "mappings": {
        "date_detection": False,
        "properties": {
            "embedding": {
                "type": "knn_vector",
                "dimension": <dimensions>,
                "method": {
                    "name": "hnsw",
                    "space_type": "l2",
                    "engine": "nmslib",
                    "parameters": {"ef_construction": 256, "m": 48}
                }
            }
        }
    }
}

NextGen mapping (new):

{
    "settings": {"index": {"knn": True}},
    "mappings": {
        "date_detection": False,
        "properties": {
            "embedding": {
                "type": "knn_vector",
                "dimension": <dimensions>,
                "space_type": "l2",
                # "compression_level": "32x"  # optional, only if configured
            }
        }
    }
}

Key differences from Classic:

• No method block (no engine, no parameters)
• space_type moves to the field level (not inside method)
• No knn.algo_param.ef_search index setting
• Optional compression_level field (1x–32x, default 32x built-in)

3. Bulk ingest — no changes required

AOSS NextGen supports custom document IDs, so the existing "id" field in bulk requests works correctly for both Classic and NextGen. No changes to _bulk_ingest_embeddings.

4. Usage

# Via GraphRAGConfig
GraphRAGConfig.opensearch_serverless_nextgen = True
GraphRAGConfig.opensearch_serverless_nextgen_compression = "8x"  # optional

# Or via environment variables
# OPENSEARCH_SERVERLESS_NEXTGEN=true
# OPENSEARCH_SERVERLESS_NEXTGEN_COMPRESSION=8x

# Connection string is unchanged — same aoss:// prefix
VectorStoreFactory.for_vector_store('aoss://https://<collection-id>.<region>.aoss.amazonaws.com')

5. Integration tests

New test: integration-tests/test-scripts/graphrag_toolkit_tests/aoss_nextgen.py

• AossNextgenExtractAndBuild — sets GraphRAGConfig.opensearch_serverless_nextgen = True, runs a real extract_and_build() against a NextGen AOSS endpoint, asserts source node count matches input document count and that vector index contains embeddings.

New playbook: integration-tests/lexical.nextgen

• Dedicated playbook for NextGen-specific tests, run against an environment where VECTOR_STORE points at a NextGen collection.
• Classic playbooks (lexical.short, lexical.long) are unchanged and continue to run against Classic collections.

Alternatives considered

Runtime auto-detection: Probe the cluster version or attempt a test mapping on first connection and fall back automatically. Rejected — adds cold-start latency, fragile, and hides configuration intent from the user.

New aoss-nextgen:// connection string prefix: Cleaner routing but aoss-nextgen:// is not an AWS-defined prefix and could confuse users. The flag approach is easier to remove when NextGen becomes the only supported type.