Merge existing policies on CREATE OR REPLACE (RTAS)

[mkuchenbecker]

Summary

CREATE OR REPLACE ... AS SELECT (RTAS) silently dropped the table's policies. The replace path rebuilt the policies table property purely from the incoming request, so a replace that omitted policies wiped the existing retention, sharing, PII column tags, replication, and history, even though ordinary user table properties survived.

Fix

Policies are table metadata that a replace must not silently drop. Before the replace properties are built, the existing table's policies are merged with the request's policies. The merge is based on the existing policies, so any plane that the request does not explicitly provide is carried forward from the existing table, and each plane that the request does provide overrides the existing value.

Merge behavior

The table below describes how each policy plane behaves during a replace.

Plane	When the request provides it	When the request omits it
retention	The request value is applied.	The existing value is carried forward.
replication	The request value is applied.	The existing value is carried forward.
history	The request value is applied.	The existing value is carried forward.
lockState	The request value is applied.	The existing value is carried forward.
columnTags	The request map replaces the existing map in full. This is an overwrite, not a per-key merge.	The existing map is carried forward. An omitted field and an empty map are treated the same way.
sharingEnabled	Not applicable, because this is a primitive boolean and a provided value cannot be distinguished from an omitted one.	The existing value is carried forward.

Two consequences follow from this behavior. First, because column tags use overwrite semantics and an empty map is treated the same as an omitted field, a replace cannot clear all column tags. Clearing tags is done with ALTER TABLE ... MODIFY COLUMN ... UNSET TAG. Second, because sharingEnabled is a primitive boolean with no unset state, its value is always preserved across a replace. Sharing is changed with ALTER TABLE ... SET POLICY (SHARING=...).

Spark RTAS has no policy clause, so it always sends a request with no policies, and the entire existing policies object is carried forward unchanged. A partial policy payload can only arrive from a client that calls the REST API directly.

This behavior is consistent with the intent of RTAS, which should preserve table properties so that a replace does not require re-granting access to the same entity.

Testing Done

The REST level partial payload path is exercised through RepositoryTest, which is the layer that can send a partial Policies object. Spark cannot reach this path because it always sends a request with no policies.

• testReplaceMergesExistingPolicies replaces a table without policies and asserts that the retention policy survives.
• testReplaceAppliesRequestedPolicies asserts that a retention policy provided on the request is applied.
• testReplaceWithPartialPoliciesPreservesSharing sends a partial payload containing only retention and asserts that sharingEnabled stays true while the new retention is applied.
• testReplaceWithPartialPoliciesPreservesOmittedPlanes overrides only retention and asserts that the omitted history plane is carried forward.
• testReplaceWithPartialPoliciesPreservesColumnTags sends a payload that provides retention but omits column tags, and asserts that the existing column tag is carried forward.
• testReplaceOverwritesColumnTags sends a new column tag map and asserts that it replaces the existing map in full, dropping the previous tag.

Black box coverage is exercised through RtasPolicyPreservationTest against an embedded OpenHouse server driven by Spark SQL. It asserts that retention, sharing, the PII column tag, and history all survive a REPLACE TABLE ... AS SELECT.

The existing SnapshotsControllerTest.testPutSnapshotsReplaceCommit still passes, which confirms that a replace on a table that never had policies still yields none. ./gradlew :services:tables:test and :integrations:spark:spark-3.1:openhouse-spark-itest:catalogTest pass on JDK 17, and Spotless is clean on the module.

[mkuchenbecker]

PTAL @cbb330 , we can add a tripwire to make sure this is updated if we want to guard against silent loss - #659

No re-review within a week; requested change is staged as a separate PR for review.

[kamanavishnu]

LGTM!