Add partition-weight guard rail for addWagedResource

[sjainit]

Summary

Adds a concrete guard rail rule on top of the now-merged guard rail framework (#213): PartitionWeightCapacityGuardrailRule, wired into the addWagedResource REST endpoint. It rejects — before the write reaches ZooKeeper — a WAGED resource whose per-partition weight is larger than any single instance could ever host, which would otherwise be accepted and only surface later as a WAGED rebalance failure.

This is the first rule added after the framework landed, so it also serves as a worked example of the framework's extension points (author a rule, add an additive context field, guard another endpoint).

The rule: PartitionWeightCapacityGuardrailRule

What it checks: WAGED places each partition replica on exactly one instance, so a partition can only ever be placed if — for every capacity dimension d — some instance has capacity_d >= weight_d. This rule rejects addWagedResource when a partition's effective weight in any dimension exceeds the largest single instance's capacity in that dimension, because such a partition is permanently unplaceable no matter how the cluster is arranged.

Gap it closes: today addWagedResource only validates that the required weight keys are present (WagedValidationUtil.validateAndGetPartitionCapacity checks key coverage, never magnitude). So a resource whose weight is larger than any instance can hold is accepted, the write succeeds, and it only surfaces later as a WAGED rebalance failure. This rule compares magnitude against real instance capacity on the write path, so the impossible resource is rejected before it reaches ZooKeeper.

It is deliberately a necessary (not sufficient) condition — it compares each dimension independently against the best instance in that dimension, so it never blocks a resource that could plausibly be placed; it only fails the provably-impossible cases. It is a no-op for non-WAGED clusters (no capacity keys), honors cluster-level default instance/partition weights (merged in the same way the rebalancer does), and evaluates the DEFAULT partition so resources relying purely on default weights are still checked.

How it uses the framework's extension points

1. Author a rule — implement GuardrailRule: null-guard → read what it needs from the context (through the read-only ReadOnlyDataAccessor) → reuse the same weight/capacity primitives WAGED uses → return a ValidationResult (feasible, or an infeasible verdict with an actionable message). Mirrors the framework's LiveInstanceGuardrailRule.
2. Add an additive context field — the rule needs the object the mutation would write, not just current cluster state, so GuardrailContext.proposedResourceConfig is added via the Builder. Existing rules/endpoints are untouched — this is the intended extensibility seam.
3. Guard another endpoint — wired into ResourceAccessor.addResource (the addWagedResource command) with the same force / dryRun preflight the instance-delete endpoint already uses. No registry, no dispatch — the endpoint just builds new GuardrailPipeline(new PartitionWeightCapacityGuardrailRule()).

Behavior

PUT /clusters/{cluster}/resources/{resource}?command=addWagedResource (body = map of IdealState + ResourceConfig ZNRecords):

• enforce (no query param): a partition weight exceeding max instance capacity → 400 + JSON verdict, resource not created; within-capacity → 200, created.
• dry-run (dryRun=true): always 200 + verdict, resource never created (validation-probe semantics, consistent with the framework).
• force (force=true): proceeds even for an over-weight resource → 200, created, with the overridden verdict logged.

Tests

• Rule unit tests (TestPartitionWeightCapacityGuardrailRule, 8): null proposed config, null cluster config, no capacity keys (non-WAGED), no instance capacity, weight within capacity, weight exceeding capacity (DEFAULT → unscoped), per-partition override exceeding, and max-across-instances used. Mocked accessor + real ClusterConfig / InstanceConfig / ResourceConfig.
• REST integration test (TestResourceAccessor#testAddWagedResourceWeightGuardrail): full Jersey + embedded ZK, exercising enforce / dryRun / force / within-capacity with status codes, the JSON verdict, and actual resource presence/absence. Saves and restores the cluster + instance capacity configuration in try/finally so it does not disturb sibling tests.

Validated on JDK 11: rule unit tests 8/8 and the full TestResourceAccessor suite 18/18 pass.

Not in scope

• The stricter exact check ("does any single instance fit the whole weight vector"); this PR implements the per-dimension necessary condition by design.
• Additional rules from the design catalog.

Co-authored-by: Copilot [email protected]

[sjainit]

@LZD-PratyushBhatt thanks for the thorough review — all eight comments are addressed and each thread has an inline reply with details. Summary:

Must-fix

• Reject force/dryRun for non-addWagedResource commands (6990e089d)
• Ghost-partition false positive — only evaluate the IdealState's real partitions (0e47b04bf)
• Skip (not zero-fill) capacity dimensions missing from the weight map (6990e089d)
• Fold capacity only over assignable instances; documented fail-closed getChildValues (6990e089d)

Tier 2 / polish (1249ec570)

• Dropped the force=true hint and documented the cluster-wide CAPACITY_DEFICIT blast radius
• Aggregate all over-capacity violations in a deterministic order
• Run cheap structural checks (name match, non-negative weights) before the guard rail so dry-run reflects them and invalid requests never hit ZK
• Pinned the guard rail integration test with dependsOnMethods; added multi-violation + structural-check tests

Validation: rule unit tests 13/13, TestResourceAccessor 20/20. Resolving the threads; happy to reopen any you want to revisit.

[LZD-PratyushBhatt]

This justification isn't right. validateAndGetInstanceCapacity is only reached from validateInstancesForWagedRebalance, which is a separate admin call nobody makes here, and from AssignableNode / WagedInstanceCapacity inside the rebalancer. It is not on the addResourceWithWeight path, so nothing reports this at add time.

I checked on a live cluster, instances advertising only FOO with cluster keys still [FOO, BAR], and a resource carrying a BAR weight gets a 200 and never places.

To be fair to the skip, in that state every WAGED resource is already unplaceable and not just this one, so skipping isn't making a healthy cluster worse, and I agree we shouldn't tell the author to lower a weight they can't lower. My problem is the comment. Someone will read this later and believe the case is covered somewhere. Can we either reword it to say the gap is deliberate and uncovered, or fail the add here with a message pointing at the instances?

[sjainit]

Good catch, the comment was wrong. Fixed in 3512d34 by rewording rather than changing behavior. It no longer claims validateAndGetInstanceCapacity covers this at add time; it now states plainly that the gap is deliberate and uncovered here — nothing on the addResourceWithWeight path validates instance-side capacity coverage (that method only runs in the rebalancer and in the separate validateInstancesForWagedRebalance admin call), so such a resource is accepted and only surfaces later as a WAGED placement failure. Kept the skip because, as you note, in that state every WAGED resource is already unplaceable and we should not tell the author to lower a weight that cannot go below 0. Happy to switch to failing the add with an instance-pointed message instead if you would prefer that over the documented skip.

[LZD-PratyushBhatt]

LGTM, Thanks for addressing all my comments!