-
Notifications
You must be signed in to change notification settings - Fork 165
Add vector_search_indexes resource (direct engine) #5123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
959b0cb
c07e291
71d3a37
60e86ed
de9d236
69f5267
e0e3ac3
4c73247
aa9e2a9
9eb36d0
713aa25
d552b36
52ed4bc
e3607f1
3c39f20
8492ed1
0e5a274
ef94326
25f3b7e
011739e
2c5d654
55d564e
2dab74b
80baa25
9599b21
647e6a0
7739af7
c3078ea
d765db4
89d82fe
edeceb2
dcd3110
44ade3f
86375c6
5fe3dea
20fa7bc
18e63e9
a191d5e
0c33818
c4076df
3d51621
ec7351f
093b974
41fadce
8f24901
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| bundle: | ||
| name: test-bundle-$UNIQUE_NAME | ||
|
|
||
| sync: | ||
| paths: [] | ||
|
|
||
| resources: | ||
| vector_search_indexes: | ||
| index1: | ||
| name: $INDEX_NAME | ||
| endpoint_name: $ENDPOINT_NAME | ||
| primary_key: id | ||
| index_type: DIRECT_ACCESS | ||
| direct_access_index_spec: | ||
| schema_json: '{"columns":[{"name":"id","type":"integer"}]}' |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
|
|
||
| >>> [CLI] vector-search-endpoints create-endpoint test-vs-endpoint-[UNIQUE_NAME] STANDARD | ||
| { | ||
| "name": "test-vs-endpoint-[UNIQUE_NAME]", | ||
| "endpoint_type": "STANDARD" | ||
| } | ||
|
|
||
| >>> [CLI] vector-search-indexes create-index --json {"name":"main.default.test_vs_index_[UNIQUE_NAME]","endpoint_name":"test-vs-endpoint-[UNIQUE_NAME]","primary_key":"id","index_type":"DIRECT_ACCESS","direct_access_index_spec":{"schema_json":"{\"columns\":[{\"name\":\"id\",\"type\":\"integer\"}]}"}} | ||
| { | ||
| "name": "main.default.test_vs_index_[UNIQUE_NAME]", | ||
| "endpoint_name": "test-vs-endpoint-[UNIQUE_NAME]", | ||
| "index_type": "DIRECT_ACCESS", | ||
| "primary_key": "id" | ||
| } | ||
|
|
||
| >>> [CLI] bundle deployment bind index1 main.default.test_vs_index_[UNIQUE_NAME] --auto-approve | ||
| Updating deployment state... | ||
| Successfully bound vector_search_index with an id 'main.default.test_vs_index_[UNIQUE_NAME]' | ||
| Run 'bundle deploy' to deploy changes to your workspace | ||
|
|
||
| >>> [CLI] bundle deploy --auto-approve | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> [CLI] vector-search-indexes get-index main.default.test_vs_index_[UNIQUE_NAME] | ||
| { | ||
| "name": "main.default.test_vs_index_[UNIQUE_NAME]", | ||
| "endpoint_name": "test-vs-endpoint-[UNIQUE_NAME]", | ||
| "index_type": "DIRECT_ACCESS", | ||
| "primary_key": "id" | ||
| } | ||
|
|
||
| >>> [CLI] bundle deployment unbind index1 | ||
| Updating deployment state... | ||
|
|
||
| >>> [CLI] bundle destroy --auto-approve | ||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default | ||
|
|
||
| Deleting files... | ||
| Destroy complete! | ||
|
|
||
| >>> [CLI] vector-search-indexes get-index main.default.test_vs_index_[UNIQUE_NAME] | ||
| { | ||
| "name": "main.default.test_vs_index_[UNIQUE_NAME]", | ||
| "endpoint_name": "test-vs-endpoint-[UNIQUE_NAME]", | ||
| "index_type": "DIRECT_ACCESS", | ||
| "primary_key": "id" | ||
| } | ||
|
|
||
| >>> [CLI] vector-search-indexes delete-index main.default.test_vs_index_[UNIQUE_NAME] | ||
|
|
||
| >>> [CLI] vector-search-endpoints delete-endpoint test-vs-endpoint-[UNIQUE_NAME] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| ENDPOINT_NAME="test-vs-endpoint-$UNIQUE_NAME" | ||
| INDEX_NAME="main.default.test_vs_index_$UNIQUE_NAME" | ||
| export ENDPOINT_NAME INDEX_NAME | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
|
|
||
| cleanup() { | ||
| trace $CLI vector-search-indexes delete-index "${INDEX_NAME}" | ||
| trace $CLI vector-search-endpoints delete-endpoint "${ENDPOINT_NAME}" | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| trace $CLI vector-search-endpoints create-endpoint "${ENDPOINT_NAME}" STANDARD | jq '{name, endpoint_type}' | ||
|
|
||
| trace $CLI vector-search-indexes create-index --json "{\"name\":\"${INDEX_NAME}\",\"endpoint_name\":\"${ENDPOINT_NAME}\",\"primary_key\":\"id\",\"index_type\":\"DIRECT_ACCESS\",\"direct_access_index_spec\":{\"schema_json\":\"{\\\"columns\\\":[{\\\"name\\\":\\\"id\\\",\\\"type\\\":\\\"integer\\\"}]}\"}}" | jq '{name, endpoint_name, index_type, primary_key}' | ||
|
|
||
| trace $CLI bundle deployment bind index1 "${INDEX_NAME}" --auto-approve | ||
|
|
||
| trace $CLI bundle deploy --auto-approve | ||
|
|
||
| trace $CLI vector-search-indexes get-index "${INDEX_NAME}" | jq '{name, endpoint_name, index_type, primary_key}' | ||
|
|
||
| trace $CLI bundle deployment unbind index1 | ||
|
|
||
| trace $CLI bundle destroy --auto-approve | ||
|
|
||
| # Read the pre-defined index again (expecting it still exists and is not deleted): | ||
| trace $CLI vector-search-indexes get-index "${INDEX_NAME}" | jq '{name, endpoint_name, index_type, primary_key}' |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| Local = true | ||
| Cloud = true | ||
| RequiresUnityCatalog = true | ||
|
|
||
| Ignore = [ | ||
| ".databricks", | ||
| "databricks.yml", | ||
| ] | ||
|
|
||
| [EnvMatrix] | ||
| DATABRICKS_BUNDLE_ENGINE = ["direct"] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| bundle: | ||
| name: test-bundle-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| vector_search_endpoints: | ||
| bar: | ||
| name: test-endpoint-$UNIQUE_NAME | ||
| endpoint_type: STANDARD | ||
| vector_search_indexes: | ||
| foo: | ||
| name: test-index-$UNIQUE_NAME | ||
| endpoint_name: ${resources.vector_search_endpoints.bar.name} | ||
| primary_key: id | ||
| index_type: DELTA_SYNC | ||
| delta_sync_index_spec: | ||
| source_table: main.default.source_$UNIQUE_NAME | ||
| pipeline_type: TRIGGERED |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| bundle: | ||
| name: deploy-vs-index-$UNIQUE_NAME | ||
|
|
||
| sync: | ||
| paths: [] | ||
|
|
||
| resources: | ||
| vector_search_indexes: | ||
| my_index: | ||
| name: vs-index-$UNIQUE_NAME | ||
| endpoint_name: vs-endpoint-$UNIQUE_NAME | ||
| primary_key: id | ||
| index_type: DELTA_SYNC | ||
| delta_sync_index_spec: | ||
| source_table: main.default.source_$UNIQUE_NAME | ||
| pipeline_type: TRIGGERED |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.0/vector-search/indexes", | ||
| "body": { | ||
| "delta_sync_index_spec": { | ||
| "pipeline_type": "TRIGGERED", | ||
| "source_table": "main.default.source_[UNIQUE_NAME]" | ||
| }, | ||
| "endpoint_name": "vs-endpoint-[UNIQUE_NAME]", | ||
| "index_type": "DELTA_SYNC", | ||
| "name": "vs-index-[UNIQUE_NAME]", | ||
| "primary_key": "id" | ||
| } | ||
| } |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
|
|
||
| >>> [CLI] bundle validate | ||
| Name: deploy-vs-index-[UNIQUE_NAME] | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/deploy-vs-index-[UNIQUE_NAME]/default | ||
|
|
||
| Validation OK! | ||
|
|
||
| >>> [CLI] bundle summary | ||
| Name: deploy-vs-index-[UNIQUE_NAME] | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/deploy-vs-index-[UNIQUE_NAME]/default | ||
| Resources: | ||
| Vector Search Indexes: | ||
| my_index: | ||
| Name: vs-index-[UNIQUE_NAME] | ||
| URL: (not deployed) | ||
|
|
||
| >>> [CLI] vector-search-endpoints create-endpoint vs-endpoint-[UNIQUE_NAME] STANDARD | ||
| { | ||
| "id": "[UUID]", | ||
| "name": "vs-endpoint-[UNIQUE_NAME]", | ||
| "endpoint_type": "STANDARD" | ||
| } | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/deploy-vs-index-[UNIQUE_NAME]/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> [CLI] vector-search-indexes get-index vs-index-[UNIQUE_NAME] | ||
| { | ||
| "name": "vs-index-[UNIQUE_NAME]", | ||
| "endpoint_name": "vs-endpoint-[UNIQUE_NAME]", | ||
| "index_type": "DELTA_SYNC", | ||
| "primary_key": "id" | ||
| } | ||
|
|
||
| >>> [CLI] bundle summary | ||
| Name: deploy-vs-index-[UNIQUE_NAME] | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/deploy-vs-index-[UNIQUE_NAME]/default | ||
| Resources: | ||
| Vector Search Indexes: | ||
| my_index: | ||
| Name: vs-index-[UNIQUE_NAME] | ||
| URL: (not deployed) | ||
|
|
||
| >>> print_requests.py //vector-search/indexes | ||
|
|
||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.vector_search_indexes.my_index | ||
|
|
||
| This action will result in the deletion of the following Vector Search indexes. | ||
| For Delta Sync indexes, the source Delta table is preserved but the embedding pipeline is removed. | ||
| For Direct Access indexes, all upserted vectors are permanently lost: | ||
| delete resources.vector_search_indexes.my_index | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/deploy-vs-index-[UNIQUE_NAME]/default | ||
|
|
||
| Deleting files... | ||
| Destroy complete! | ||
|
|
||
| >>> [CLI] vector-search-endpoints delete-endpoint vs-endpoint-[UNIQUE_NAME] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
| endpoint_name="vs-endpoint-${UNIQUE_NAME}" | ||
|
|
||
| cleanup() { | ||
| trace $CLI bundle destroy --auto-approve | ||
| trace $CLI vector-search-endpoints delete-endpoint "${endpoint_name}" | ||
| rm -f out.requests.txt | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| trace $CLI bundle validate | ||
|
|
||
| trace $CLI bundle summary | ||
|
|
||
| trace $CLI vector-search-endpoints create-endpoint "${endpoint_name}" STANDARD | jq '{id, name, endpoint_type}' | ||
|
|
||
| rm -f out.requests.txt | ||
| trace $CLI bundle deploy | ||
|
|
||
| # Get index details | ||
| index_name="vs-index-${UNIQUE_NAME}" | ||
| trace $CLI vector-search-indexes get-index "${index_name}" | jq '{name, endpoint_name, index_type, primary_key}' | ||
|
|
||
| trace $CLI bundle summary | ||
|
|
||
| trace print_requests.py //vector-search/indexes > out.requests.$DATABRICKS_BUNDLE_ENGINE.json | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| # All configuration inherited from parent test.toml |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| bundle: | ||
| name: drift-vs-index-columns-$UNIQUE_NAME | ||
|
|
||
| sync: | ||
| paths: [] | ||
|
|
||
| resources: | ||
| vector_search_indexes: | ||
| my_index: | ||
| name: vs-index-$UNIQUE_NAME | ||
| endpoint_name: vs-endpoint-$UNIQUE_NAME | ||
| primary_key: id | ||
| index_type: DELTA_SYNC | ||
| delta_sync_index_spec: | ||
| source_table: main.default.source_$UNIQUE_NAME | ||
| pipeline_type: TRIGGERED | ||
| columns_to_sync: | ||
| - id | ||
| - text |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you create it as part of the same bundle and refer to it in the index?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no strong feeling either way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we are strict about it, we have already resources combining both (see catalogs or database_catalogs for example).
Having them together helps us to test the actual use case users will have + cleaning up is a bit easier too