Skip to content

feat(mcp): add governed table imports and Milvus semantic tools - #7192

Open
xingranya wants to merge 10 commits into
t8y2:mainfrom
xingranya:feat/enterprise-import-vector-mcp
Open

feat(mcp): add governed table imports and Milvus semantic tools#7192
xingranya wants to merge 10 commits into
t8y2:mainfrom
xingranya:feat/enterprise-import-vector-mcp

Conversation

@xingranya

@xingranya xingranya commented Aug 25, 2026

Copy link
Copy Markdown

Summary

This PR adds governed local table-import and Milvus semantic-search capabilities to the DBX MCP server while reusing dbx-core parsing and database drivers.

  • Add dbx_preview_import_file, dbx_prepare_table_import, dbx_start_table_import, dbx_get_import_status, and dbx_cancel_import.
  • Add dedicated dbx_vector_search and dbx_vector_upsert_file; keep v1 batch deletion registered but fail-closed as VECTOR_DELETE_DISABLED_V1 until DBX has an authoritative publication-state source.
  • Classify vector REST entity operations so dbx_execute_query cannot bypass the dedicated tools.
  • Stream governed XLSX/XLSM imports into a server-generated, unique staging table whose source and lineage columns are all TEXT.
  • Preserve duplicate headers by 1-based source position and stable canonical names.
  • Require exact semantic_version, approval_status=approved, and a YYYY-MM-DD activity window for every semantic search.
  • Correct Milvus single-query response flattening, including data: [[]] as zero rows.
  • Make connection management opt-in for an unscoped maintenance process, and correctly resolve every named connection in a multi-ID runtime scope.

Safety model

  • Local import tools fail closed unless DBX_MCP_IMPORT_ROOTS is configured; web mode returns IMPORT_UNSUPPORTED_IN_WEB_MODE_V1.
  • Absolute paths, canonical allowed roots, regular files, O_NOFOLLOW, file-descriptor identity checks, SHA-256 revalidation, single-use 30-minute plans, and task-private snapshots protect the preview/prepare/start boundary.
  • Inspection/import concurrency, timeouts, ZIP entry and decompression budgets, row/cell/string/output limits, disk-space checks, bounded XLSX channels, cancellation, and RAII cleanup bound resource use.
  • Imports finish their governed snapshot before opening the database and can write only a new server-generated staging relation.
  • Semantic JSONL is fully validated against the Milvus collection contract before the first network write, including UTF-8 byte limits, checksums, version consistency, embedding shape, and card ownership.
  • Generic vector entity search/query/upsert/delete calls are blocked; collection diagnostics remain available where already allowed.
  • dbx_add_connection, dbx_duplicate_connection, and dbx_remove_connection are hidden by default. They require DBX_MCP_ENABLE_CONNECTION_MANAGEMENT=1 and no runtime scope; method-level guards remain in place.

v1 boundaries

  • Governed imports support XLSX/XLSM and explicitly UTF-8 CSV/TSV.
  • Legacy .xls, JSON table imports, GBK, UTF-16, and automatic encoding detection remain bounded preview-only and fail before database access during prepare.
  • Preview row counts and used ranges expose exactness flags; a bounded preview is not reported as a full-file count.
  • Semantic deletion remains disabled rather than trusting a caller-provided publication flag.

Validation

  • cargo test -p dbx-mcp --no-default-features: 93 passed, 4 ignored, 7 filtered out.
  • cargo test -p dbx-core --lib: 5,447 passed, 60 ignored.
  • cargo check -p dbx-mcp: passed.
  • cargo check -p dbx-core --example data_transfer_bench --locked --no-default-features: passed; covers the all-target benchmark initializer added for source row lineage.
  • cargo clippy -p dbx-mcp --no-default-features: 0 errors; two pre-existing nonminimal_bool warnings outside this change.
  • pnpm --filter @dbx-app/mcp-server test: 1 test file, 2 tests passed.
  • An opt-in local XLSX regression fixture verified a 145-column sheet, repeated headers, stale used-range metadata, 81 all-empty columns, absolute source row numbers, complete row hashes, and all-TEXT governed mappings without opening a database or logging business cell values. The fixture itself is not included.
  • Protocol tests cover structured MCP output, path/scope rejection, immutable plans, staging isolation, connection-management hiding, multi-connection routing, vector permission boundaries, and semantic-version filtering.

No production database, Milvus collection, credentials, or private fixture data is included in this PR.

@github-actions github-actions Bot added area/core Shared DBX core runtime area/docs Documentation site or repository documentation area/mcp MCP server or packages dependencies/backend Adds a backend dependency enhancement New feature or request labels Aug 25, 2026
@xingranya
xingranya force-pushed the feat/enterprise-import-vector-mcp branch from 2b5a69f to 5747853 Compare August 27, 2026 14:10
@xingranya

Copy link
Copy Markdown
Author

CI follow-up: the original rust-test and rust-fmt-clippy failures had the same compile cause: ParsedImportFile gained source_row_numbers, while crates/dbx-core/examples/data_transfer_bench.rs still used the old initializer. This is fixed, and the branch has been cleanly rebased onto the current main (no workflow-file changes in the PR diff). The latest compatibility commit also preserves 1-based source record numbers for the new SQL import path on current main. Local merged-base validation passed: cargo fmt --check; 19 SQL-import tests; cargo check -p dbx-mcp --locked; cargo check -p dbx-core --example data_transfer_bench --locked --no-default-features --features sqlite-bundled; and 93 DBX MCP tests. The new CI run 33080745104 is action_required with zero jobs. Could a maintainer approve this Actions run?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/core Shared DBX core runtime area/docs Documentation site or repository documentation area/mcp MCP server or packages dependencies/backend Adds a backend dependency enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant