-
Notifications
You must be signed in to change notification settings - Fork 125
docs: add volcano plugin guides #499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
hemantch01
wants to merge
2
commits into
volcano-sh:master
Choose a base branch
from
hemantch01:docs/add-volcano-plugin-guides
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| +++ | ||
| title = "Binpack Plugin User Guide" | ||
| date = 2024-05-10 | ||
| type = "docs" | ||
| weight = 50 | ||
| url = "/en/docs/user-guide/how_to_use_binpack_plugin/" | ||
| [menu.docs] | ||
| parent = "user-guide" | ||
| +++ | ||
|
|
||
| ## Introduction | ||
|
|
||
| The Binpack plugin attempts to pack as many pods as possible onto a single node before moving on to the next. This minimizes the fragmentation of cluster resources and leaves larger, empty nodes available for future workloads that may require massive amounts of contiguous resources. By heavily utilizing a subset of nodes, it is also beneficial for cluster autoscalers to downscale completely empty nodes. | ||
|
|
||
| ## Environment setup | ||
|
|
||
| ### Update scheduler configmap | ||
|
|
||
| Ensure the `binpack` plugin is enabled and configured with weights for different resources in the `volcano-scheduler-configmap`. | ||
|
|
||
| ```yaml | ||
| kind: ConfigMap | ||
| apiVersion: v1 | ||
| metadata: | ||
| name: volcano-scheduler-configmap | ||
| namespace: volcano-system | ||
| data: | ||
| volcano-scheduler.conf: | | ||
| actions: "enqueue, allocate, backfill" | ||
| tiers: | ||
| - plugins: | ||
| - name: priority | ||
| - name: binpack | ||
| arguments: | ||
| binpack.weight: 10 | ||
| binpack.cpu: 5 | ||
| binpack.memory: 5 | ||
| ``` | ||
|
|
||
| The arguments dictate the relative importance of binpacking by CPU vs Memory. | ||
|
|
||
| ## How to use Binpack Plugin | ||
|
|
||
| When you submit a workload to Volcano, the Binpack plugin automatically influences the node scoring phase. | ||
|
|
||
| ### Example YAML | ||
|
|
||
| Create a file named `binpack-job.yaml`: | ||
|
|
||
| ```yaml | ||
| apiVersion: batch.volcano.sh/v1alpha1 | ||
| kind: Job | ||
| metadata: | ||
| name: binpack-job | ||
| spec: | ||
| minAvailable: 1 | ||
| schedulerName: volcano | ||
| queue: default | ||
| tasks: | ||
| - replicas: 10 | ||
| name: worker | ||
| template: | ||
| spec: | ||
| containers: | ||
| - image: nginx | ||
| name: nginx | ||
| resources: | ||
| requests: | ||
| cpu: "500m" | ||
| memory: "500Mi" | ||
| restartPolicy: OnFailure | ||
| ``` | ||
|
|
||
| ## Verification | ||
|
|
||
| 1. Apply the job to your cluster: | ||
| ```bash | ||
| kubectl apply -f binpack-job.yaml | ||
| ``` | ||
|
|
||
| 2. Check the node assignment of the pods: | ||
| ```bash | ||
| kubectl get pods -o wide | grep binpack-job | ||
| ``` | ||
|
|
||
| 3. **Expected Result**: Without Binpack (using standard Kubernetes spreading), the scheduler usually distributes pods evenly across all available nodes. With the Binpack plugin, you will observe that Volcano schedules pods densely onto `Node A` until it is completely full, and only then starts placing pods on `Node B`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| +++ | ||
| title = "DRF Plugin User Guide" | ||
| date = 2024-05-10 | ||
| type = "docs" | ||
| weight = 50 | ||
| url = "/en/docs/user-guide/how_to_use_drf_plugin/" | ||
| [menu.docs] | ||
| parent = "user-guide" | ||
| +++ | ||
|
|
||
| ## Introduction | ||
|
|
||
| The DRF (Dominant Resource Fairness) scheduling plugin allocates resources based on the dominant resource requested by a job. The dominant resource is the resource type (e.g., CPU or Memory) that takes up the largest percentage of the total cluster capacity. The DRF algorithm calculates the dominant resource share for each job and prioritizes jobs with the lowest share, ensuring fairness across multiple resource types. | ||
|
|
||
| This is highly effective in mixed workloads where some jobs are CPU-intensive and others are memory-intensive. | ||
|
|
||
| ## Environment setup | ||
|
|
||
| ### Update scheduler configmap | ||
|
|
||
| Ensure the `drf` plugin is enabled in the `volcano-scheduler-configmap` under the `tiers` section. | ||
|
|
||
| ```yaml | ||
| kind: ConfigMap | ||
| apiVersion: v1 | ||
| metadata: | ||
| name: volcano-scheduler-configmap | ||
| namespace: volcano-system | ||
| data: | ||
| volcano-scheduler.conf: | | ||
| actions: "enqueue, allocate, backfill" | ||
| tiers: | ||
| - plugins: | ||
| - name: priority | ||
| - name: gang | ||
| - name: drf | ||
| ``` | ||
|
|
||
| ## How to use DRF Plugin | ||
|
|
||
| DRF works automatically out of the box when enabled. To see DRF in action, we can create two jobs in the same queue: one that is CPU-intensive and another that is Memory-intensive. | ||
|
|
||
| ### Example YAML | ||
|
|
||
| Create a file named `drf-jobs.yaml`: | ||
|
|
||
| ```yaml | ||
| apiVersion: batch.volcano.sh/v1alpha1 | ||
| kind: Job | ||
| metadata: | ||
| name: cpu-heavy-job | ||
| spec: | ||
| minAvailable: 1 | ||
| schedulerName: volcano | ||
| queue: default | ||
| tasks: | ||
| - replicas: 5 | ||
| name: worker | ||
| template: | ||
| spec: | ||
| containers: | ||
| - image: nginx | ||
| name: nginx | ||
| resources: | ||
| requests: | ||
| cpu: "2" | ||
| memory: "500Mi" | ||
| restartPolicy: OnFailure | ||
| --- | ||
| apiVersion: batch.volcano.sh/v1alpha1 | ||
| kind: Job | ||
| metadata: | ||
| name: mem-heavy-job | ||
| spec: | ||
| minAvailable: 1 | ||
| schedulerName: volcano | ||
| queue: default | ||
| tasks: | ||
| - replicas: 5 | ||
| name: worker | ||
| template: | ||
| spec: | ||
| containers: | ||
| - image: nginx | ||
| name: nginx | ||
| resources: | ||
| requests: | ||
| cpu: "500m" | ||
| memory: "4Gi" | ||
| restartPolicy: OnFailure | ||
| ``` | ||
|
|
||
| ### How DRF Balances Resources | ||
|
|
||
| If the total cluster capacity is 10 CPUs and 20Gi Memory: | ||
| - `cpu-heavy-job` requests 2 CPUs (20% of cluster CPU) and 500Mi Memory (2.5% of cluster Memory) per pod. Its dominant resource is CPU. | ||
| - `mem-heavy-job` requests 500m CPUs (5% of cluster CPU) and 4Gi Memory (20% of cluster Memory) per pod. Its dominant resource is Memory. | ||
|
|
||
| As Volcano schedules pods, the DRF plugin constantly recalculates the dominant resource share for both jobs. If `cpu-heavy-job` has 2 pods running (40% CPU share) and `mem-heavy-job` has 1 pod running (20% Memory share), DRF will prioritize the next pod from `mem-heavy-job` because its dominant resource share is currently lower. | ||
|
|
||
| ## Verification | ||
|
|
||
| 1. Submit both jobs simultaneously: | ||
| ```bash | ||
| kubectl apply -f drf-jobs.yaml | ||
| ``` | ||
|
|
||
| 2. Monitor the scheduling order: | ||
| ```bash | ||
| kubectl get pods -w | ||
| ``` | ||
|
|
||
| 3. You will observe that Volcano interleaves the scheduling of pods from both jobs to keep their dominant resource shares roughly equal, rather than starving one job entirely while the other finishes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| +++ | ||
| title = "Gang Plugin User Guide" | ||
| date = 2024-05-10 | ||
| type = "docs" | ||
| weight = 50 | ||
| url = "/en/docs/user-guide/how_to_use_gang_plugin/" | ||
| [menu.docs] | ||
| parent = "user-guide" | ||
| +++ | ||
|
|
||
| ## Introduction | ||
|
|
||
| The Gang scheduling algorithm is one of the core scheduling plugins in Volcano. It meets the "All or nothing" scheduling requirements. This means that a group of pods (usually within a Job) will only be scheduled if the minimum required number of pods can be scheduled together. If the cluster does not have enough resources to satisfy the minimum number of running pods, none of them will be scheduled. This prevents resource deadlock where multiple jobs hold partial resources but none can complete. | ||
|
|
||
| ## Environment setup | ||
|
|
||
| ### Update scheduler configmap | ||
|
|
||
| To use the Gang plugin, ensure that the `gang` plugin is enabled in your Volcano Scheduler configuration (`volcano-scheduler-configmap`). | ||
|
|
||
| ```yaml | ||
| kind: ConfigMap | ||
| apiVersion: v1 | ||
| metadata: | ||
| name: volcano-scheduler-configmap | ||
| namespace: volcano-system | ||
| data: | ||
| volcano-scheduler.conf: | | ||
| actions: "enqueue, allocate, backfill" | ||
| tiers: | ||
| - plugins: | ||
| - name: priority | ||
| - name: gang | ||
| enablePreemptable: false | ||
| - name: conformance | ||
| ``` | ||
|
|
||
| ## How to use Gang Plugin | ||
|
|
||
| When you submit a `vcjob` (Volcano Job) or use a `PodGroup` with regular Pods, you can define the `minAvailable` field. The Gang plugin will check if the cluster has enough resources to satisfy this number of pods. | ||
|
|
||
| ### Example 1: Using Volcano Job (vcjob) | ||
|
|
||
| Create a file named `gang-job.yaml` with the following content: | ||
|
|
||
| ```yaml | ||
| apiVersion: batch.volcano.sh/v1alpha1 | ||
| kind: Job | ||
| metadata: | ||
| name: gang-job | ||
| spec: | ||
| minAvailable: 3 | ||
| schedulerName: volcano | ||
| policies: | ||
| - event: PodEvicted | ||
| action: RestartJob | ||
| tasks: | ||
| - replicas: 3 | ||
| name: worker | ||
| template: | ||
| spec: | ||
| containers: | ||
| - image: nginx | ||
| name: nginx | ||
| resources: | ||
| requests: | ||
| cpu: "1" | ||
| memory: "1Gi" | ||
| restartPolicy: OnFailure | ||
| ``` | ||
|
|
||
| In this example, `minAvailable` is set to 3. The Gang plugin ensures that either all 3 workers are scheduled simultaneously, or none are. | ||
|
|
||
| ### Example 2: Using PodGroup for generic Pods | ||
|
|
||
| You can also use the Gang plugin with native Kubernetes Pods by creating a `PodGroup`. | ||
|
|
||
| Create a file named `gang-podgroup.yaml`: | ||
|
|
||
| ```yaml | ||
| apiVersion: scheduling.volcano.sh/v1beta1 | ||
| kind: PodGroup | ||
| metadata: | ||
| name: gang-pg | ||
| spec: | ||
| minMember: 3 | ||
| queue: default | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: gang-pod-1 | ||
| labels: | ||
| scheduling.volcano.sh/pod-group-name: gang-pg | ||
| spec: | ||
| schedulerName: volcano | ||
| containers: | ||
| - name: nginx | ||
| image: nginx | ||
| resources: | ||
| requests: | ||
| cpu: "1" | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: gang-pod-2 | ||
| labels: | ||
| scheduling.volcano.sh/pod-group-name: gang-pg | ||
| spec: | ||
| schedulerName: volcano | ||
| containers: | ||
| - name: nginx | ||
| image: nginx | ||
| resources: | ||
| requests: | ||
| cpu: "1" | ||
| ``` | ||
|
|
||
| In this example, only 2 pods are submitted for a `PodGroup` that requires `minMember: 3`. The pods will remain in the `Pending` state until a 3rd pod belonging to `gang-pg` is created and enough resources are available. | ||
|
|
||
| ## Verification | ||
|
|
||
| 1. Apply the job to your cluster: | ||
| ```bash | ||
| kubectl apply -f gang-job.yaml | ||
| ``` | ||
|
|
||
| 2. Check the status of the pods: | ||
| ```bash | ||
| kubectl get pods | ||
| ``` | ||
| If the cluster has at least 3 CPUs and 3Gi of memory available, all 3 pods will transition to `Running`. If not, they will all remain in `Pending` due to the Gang scheduling constraint. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The configuration for the
gangplugin is missing theargumentskey. In Volcano, plugin-specific configurations must be nested underarguments. Additionally, the standard argument name for controlling preemption in the gang plugin isgang.preemptablerather thanenablePreemptable.