feat: cron-based MonoVertex autoscaling spec by devsarvesh92 · Pull Request #3558 · numaproj/numaflow

devsarvesh92 · 2026-07-23T14:48:58Z

What this PR does / why we need it

Adds cron-window-based autoscaling bounds for MonoVertex.

Users can configure recurring windows with minimum and maximum replica
counts. While a window is active:

Replicas outside the configured bounds move gradually toward the nearest
bound.
Existing scale-up and scale-down cooldowns remain respected.
Reactive metric-based autoscaling continues within the active bounds.
Cron bounds are applied before daemon metrics are queried, allowing
scheduled scale-up from zero.

When cron is omitted, existing autoscaling behavior is unchanged.

Example:

scale:
  min: 0
  max: 50
  cron:
    timezone: America/Los_Angeles
    schedules:
      - start: "0 2 * * * *"
        end: "0 3 * * * *"
        min: 1
        max: 5

Related issues

Fixes #3546

Parent issue: #3544

Testing

make test
make pre-push -B
go vet ./pkg/apis/numaflow/v1alpha1 ./pkg/reconciler/monovertex/scaling ./pkg/reconciler/validator
git diff --check
Local kind integration:
- Verified active cron scale-up patches spec.replicas before metrics.
- Verified gradual replica changes honor replicasPerScaleUp.
- Verified the controller reconciles patched replicas and creates worker pods.
- Verified required timezone is rejected by the CRD schema.
- Verified incompatible start/end calendar fields are rejected by the webhook.

Special notes for reviewers

Cron configuration is optional.
When cron is configured, timezone, schedules, min, and max are required.
Schedule bounds must remain within the parent scale bounds.
Overlapping schedule detection is deferred; the first active schedule wins.
Scale is shared, so generated Pipeline and Vertex CRDs also contain the
cron schema. Pipeline runtime support is outside this sub-issue.
CRD, OpenAPI, protobuf, install manifest, and API documentation changes are
generated.

Contributors

Define scheduled scale bounds and reject invalid cron windows before resources are reconciled. Co-authored-by: Sarvesh Sawant <[email protected]> Signed-off-by: Shantanu Vashishtha <[email protected]> Signed-off-by: Sarvesh Sawant <[email protected]>

Apply scheduled replica bounds before daemon metrics are available while preserving gradual, metric-driven scaling within active windows. Co-authored-by: Shantanu Vashishtha <[email protected]> Signed-off-by: Sarvesh Sawant <[email protected]> Signed-off-by: Shantanu Vashishtha <[email protected]>

vaibhavtiwari33 · 2026-07-23T15:23:21Z

+	Timezone  string         `json:"timezone" protobuf:"bytes,1,opt,name=timezone"`
+	Schedules []CronSchedule `json:"schedules" protobuf:"bytes,2,rep,name=schedules"`


~~nit: Could we document the cron format being used. Standard vs extended~~

I see it documented below

vaibhavtiwari33 · 2026-07-23T15:27:14Z

+	// Start of the cron window. Linux cron format (Minute Hour Dom Month Dow).
+	Start string `json:"start" protobuf:"bytes,1,opt,name=start"`


nit: For sideinputs we're using extended format (6 fields). This might create confusion between users about which one to use.
ref: #3054

vaibhavtiwari33 · 2026-07-23T15:35:37Z

+
+// IsActiveAt reports whether time t falls within this cron window.
+func (cs CronSchedule) IsActiveAt(t time.Time) bool {
+	parser := cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow)


vaibhavtiwari33 · 2026-07-23T15:40:25Z

+// CronScheduling defines cron-based autoscaling overrides.
+type CronScheduling struct {
+	// Timezone for interpreting cron expressions. IANA Time Zone Database format.
+	Timezone  string         `json:"timezone" protobuf:"bytes,1,opt,name=timezone"`


nit: Can this be made optional with default being UTC?

vaibhavtiwari33 · 2026-07-23T15:43:14Z

+	for i := range s.Cron.Schedules {
+		if s.Cron.Schedules[i].IsActiveAt(at) {
+			return &s.Cron.Schedules[i], true
+		}


We should document that we honor the first matching schedule in case when there exist multiple overlapping schedules.

vaibhavtiwari33 · 2026-07-23T15:51:07Z

Could we also please update the autoscaling doc (specifically the mvtx section)

vigith · 2026-07-23T16:10:58Z

+
+// IsActiveAt reports whether time t falls within this cron window.
+func (cs CronSchedule) IsActiveAt(t time.Time) bool {
+	parser := cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow)


parsed schedules should be pre-computed, it is computationally heavy and performance will be impacted since it is in hot code path of reconciliation.

vigith · 2026-07-23T16:12:03Z

+func (s Scale) GetEffectiveScaleBounds() (int32, int32, bool) {
+	minR := s.GetMinReplicas()
+	maxR := s.GetMaxReplicas()
+	sched, active := s.GetActiveCronSchedule(time.Now())


this can make it tough to unit-test, can we pass time.Now() as an arg.

vigith · 2026-07-23T16:21:15Z

227 + if v.Scale.Cron != nil { 228 + return fmt.Errorf("vertex %q: cron autoscaling is not supported for pipeline vertices", v.Name) 229 + }

we need to add this pipeline else it will be silently ignored.

vigith · 2026-07-23T16:24:31Z

+	if scale.Min != nil && *scale.Min < 0 {
+		return fmt.Errorf("scale.min must not be negative")
+	}
+	if scale.Max != nil && *scale.Max < 0 {
+		return fmt.Errorf("scale.max must not be negative")
+	}


The cronMin >= parentMin and cronMax <= parentMax is concerning. This blocks the common pattern where parentMin=1 (always keep at least one replica running) but during a quiet window you want the cron to scale down to 0. If the intent of cron overrides is to override the parent bounds, this constraint should be relaxed.

vigith · 2026-07-23T16:25:28Z

+	if cronActive {
+		if current < minReplicas {
+			return s.scaleUp(ctx, monoVtx, current, minReplicas, secondsSinceLastScale, scaleUpCooldown)
+		}
+		if current > maxReplicas {
+			return s.scaleDown(ctx, monoVtx, current, maxReplicas, secondsSinceLastScale, scaleDownCooldown)
+		}


log.Infof("Cron window active [min=%d, max=%d], current=%d is outside bounds; scaling.", minReplicas, maxReplicas, current)

Precompute cron schedules in the autoscaler and align validation with six-field expressions and UTC defaults. Reject cron configuration on unsupported Pipeline vertices and document schedule precedence. Co-authored-by: Shantanu Vashishtha <[email protected]> Signed-off-by: Sarvesh Sawant <[email protected]> Signed-off-by: Shantanu Vashishtha <[email protected]>

learncoder4848 · 2026-07-24T16:44:38Z

Thanks for the thorough reviews, @vaibhavtiwari33 and @vigith! The latest commit addresses all the comments. Summary below.

@vigith

Pre-compute parsed cron schedules (hot path perf): Schedules are now parsed once via CronScheduling.Compile() and cached in the scaler (cronScheduleCache, keyed by UID/Generation), so no cron parsing happens on the reconciliation hot path. The cache invalidates automatically on spec changes (Generation) and object recreation (UID).
Pass time.Now() for testability: Bound evaluation now flows through effectiveScaleBoundsAt(scale, parsed, at) with the time passed in explicitly, so it's unit-testable without wall-clock dependence.
Reject cron on pipeline vertices (silent-ignore): Added validation in validateVertex — pipeline vertices with scale.cron are now rejected with cron autoscaling is not supported for pipeline vertices instead of being silently ignored.
Relax cronMin >= parentMin / cronMax <= parentMax: Removed these containment checks so cron windows can override the parent bounds (e.g. parentMin=1 scaling down to 0 during a quiet window).
Log when cron forces scaling outside bounds: Added the info log when an active cron window scales because current replicas are outside [min, max].

@vaibhavtiwari33

Cron format documentation / consistency with side inputs: Switched to the six-field extended format (second minute hour dom month dow), matching the side-input convention, and documented it. (ref Sideinput not working in v1.6: ERROR numaflow: Schedule("*/1 * * * *\n ^\n") #3054)
Timezone optional, default UTC: timezone is now optional and defaults to UTC via CronScheduling.GetTimezone(); validation and CRDs updated accordingly.
Document first-match on overlapping schedules: Documented that schedules are evaluated in order and the first active schedule takes precedence, both in the API godoc and the autoscaling reference doc.
Update autoscaling docs (mvtx section): Updated docs/user-guide/reference/autoscaling.md with the cron field, examples, format, timezone default, and precedence behavior.

codecov · 2026-07-24T17:19:23Z

Codecov Report

❌ Patch coverage is 79.02098% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.57%. Comparing base (7b02ed9) to head (9108a87).

Files with missing lines	Patch %	Lines
pkg/reconciler/monovertex/scaling/scaling.go	71.42%	15 Missing and 9 partials ⚠️
pkg/reconciler/validator/mvtx_validate.go	88.67%	3 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3558      +/-   ##
==========================================
- Coverage   83.58%   83.57%   -0.01%     
==========================================
  Files         311      311              
  Lines       83097    83220     +123     
==========================================
+ Hits        69455    69550      +95     
- Misses      13046    13055       +9     
- Partials      596      615      +19

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

vigith

thank you, LGTM

learncoder4848 · 2026-07-25T16:27:22Z

thank you, LGTM

@vigith Can you approve the awaiting workflows so that we can merge the changes ?

vaibhavtiwari33 · 2026-07-24T17:09:24Z

    targetProcessingSeconds: 20 # Optional, defaults to 20.
    replicasPerScaleUp: 2 # Optional, defaults to 2.
    replicasPerScaleDown: 2 # Optional, defaults to 2.
+    cron: # Optional; supported only for MonoVertex.


nit:

Suggested change

cron: # Optional; supported only for MonoVertex.

cron: # Optional; currently only supported for MonoVertex.

Since, I believe there exists an issue to add this functionality for pipeline as well

learncoder4848 · 2026-07-25T17:04:49Z

@vigith can you re-trigger/approve this [test / E2E Tests (jetstream, e2e) (pull_request)] , don't know what is this

vigith · 2026-07-25T17:38:43Z

@vigith can you re-trigger/approve this [test / E2E Tests (jetstream, e2e) (pull_request)] , don't know what is this

it is successful now, Github Runners are not very stable

learncoder4848 and others added 2 commits July 23, 2026 19:49

devsarvesh92 marked this pull request as ready for review July 23, 2026 14:52

devsarvesh92 requested review from vigith, whynowy and yhl25 as code owners July 23, 2026 14:52

vigith changed the title ~~feat: support cron-based MonoVertex autoscaling. Fixes #3546~~ feat: cron-based MonoVertex autoscaling spec Jul 23, 2026

vaibhavtiwari33 reviewed Jul 23, 2026

View reviewed changes

vigith reviewed Jul 23, 2026

View reviewed changes

learncoder4848 force-pushed the feat/cron-monovertex-autoscaling branch from d527eba to e2a0a35 Compare July 24, 2026 16:39

Merge branch 'main' into feat/cron-monovertex-autoscaling

467fcc3

vigith approved these changes Jul 24, 2026

View reviewed changes

Merge branch 'main' into feat/cron-monovertex-autoscaling

0d1d5aa

Merge branch 'main' into feat/cron-monovertex-autoscaling

9108a87

vigith enabled auto-merge (squash) July 25, 2026 16:29

vaibhavtiwari33 approved these changes Jul 25, 2026

View reviewed changes

vigith merged commit 05660f7 into numaproj:main Jul 25, 2026
41 of 42 checks passed

		Timezone string `json:"timezone" protobuf:"bytes,1,opt,name=timezone"`
		Schedules []CronSchedule `json:"schedules" protobuf:"bytes,2,rep,name=schedules"`

		// Start of the cron window. Linux cron format (Minute Hour Dom Month Dow).
		Start string `json:"start" protobuf:"bytes,1,opt,name=start"`

	parser := cron.NewParser(cron.Minute \| cron.Hour \| cron.Dom \| cron.Month \| cron.Dow)
	parser := cron.NewParser(cron.Second \| cron.Minute \| cron.Hour \| cron.Dom \| cron.Month \| cron.Dow)

	cron: # Optional; supported only for MonoVertex.
	cron: # Optional; currently only supported for MonoVertex.

Uh oh!

Conversation

devsarvesh92 commented Jul 23, 2026 • edited by vigith Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it

Related issues

Testing

Special notes for reviewers

Contributors

Uh oh!

vaibhavtiwari33 Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vaibhavtiwari33 commented Jul 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

learncoder4848 commented Jul 24, 2026

Uh oh!

codecov Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vigith left a comment

Choose a reason for hiding this comment

Uh oh!

learncoder4848 commented Jul 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

learncoder4848 commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vigith commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

devsarvesh92 commented Jul 23, 2026 •

edited by vigith

Loading

vaibhavtiwari33 Jul 23, 2026 •

edited

Loading

codecov Bot commented Jul 24, 2026 •

edited

Loading

learncoder4848 commented Jul 25, 2026 •

edited

Loading