[controller] Throttle parent-controller version-creation attempts

[pthirun]

Problem Statement

Rapid retries with different push IDs can repeatedly reach version creation after an earlier attempt fails before any durable Version metadata or resources exist. Venice needs a best-effort abuse/throttling guard without adding durable store state, schema changes, ZK nodes, or admin protocol fields.

Solution

• Add the cluster-level controller.push.retry.cooldown.ms config with a default of 600000 ms (10 minutes); 0 disables the mechanism.
• Enforce only in the leader parent-controller version-creation path. The state is intentionally process-local, so parent-controller restart or leadership handoff starts with a fresh cooldown map.
• Apply only to user-store BATCH and STREAM_REPROCESSING requests that are about to create a new version. Venice system stores, STREAM, INCREMENTAL, explicit-version child/admin processing, and migration replication are excluded.
• Preserve request ordering: ongoing-push handling and existing-version same-push-ID idempotency run first, followed by degraded-mode and target-region validation. Invalid or rejected requests do not enter the map.
• Atomically check and record a ConcurrentHashMap entry keyed by cluster/store immediately before addVersionAndTopicOnly(), before lifecycle hooks, version metadata, topics, push monitoring, or Helix resources.
• Record the first admitted attempt; allow the same push ID without sliding its timestamp; reject a different push ID inside the window with HTTP 429 and remaining time; replace expired entries. Rejections do not slide the window, while admitted attempts remain recorded if later creation fails.
• Remove an entry after successful parent-level store deletion. Other stale entries expire lazily.
• Record a status-neutral cooldown rejection counter by push type.

Code changes

• Added new code behind a config. Config: controller.push.retry.cooldown.ms; default: 600000 ms.
• Introduced new log lines.
  • Confirmed if logs need to be rate limited to avoid excessive logging. No new log lines were added.

Concurrency-Specific Checks

Both reviewer and PR author to verify

• Code has no race conditions or thread safety issues. Admission for each cluster/store key is serialized with ConcurrentHashMap.compute.
• Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed. No additional lock is required beyond the atomic map operation.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
• Local code review completed.

Commands run successfully:

• ./gradlew :services:venice-controller:test --tests com.linkedin.venice.controller.TestVeniceParentHelixAdmin --tests com.linkedin.venice.controller.TestVeniceControllerClusterConfig --tests com.linkedin.venice.controller.stats.VeniceAdminOtelMetricEntityTest --tests com.linkedin.venice.controller.stats.VeniceAdminStatsOtelTest --tests com.linkedin.venice.controller.stats.VeniceAdminTehutiMetricNameEnumTest --no-daemon
• ./gradlew :internal:venice-test-common:integrationTest --tests com.linkedin.venice.controller.VeniceParentHelixAdminTest.testPushRetryCooldownThroughControllerApi --no-daemon
• ./gradlew :internal:venice-test-common:integrationTest --tests com.linkedin.venice.controller.VeniceParentHelixAdminTest.testAddVersion --no-daemon
• ./gradlew spotlessCheck --no-daemon

The focused integration test uses a deterministic failing pre-version lifecycle hook, verifies the first admitted attempt creates no Version, version topic, or child Helix resource, verifies a different push ID receives HTTP 429 before resources, then removes the failure and confirms the original push ID succeeds.

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.

When enabled, a new push ID for a user-store BATCH or STREAM_REPROCESSING version-creation attempt can receive HTTP 429 during the configured process-local cooldown. Same-push-ID retries remain idempotently admissible. The guard does not survive parent-controller restart or leadership handoff by design, and no durable metadata or schemas change.

🤖 Generated with GitHub Copilot CLI