docs(customizer): update documentation and notebooks for customizer #414
Merged
Conversation
Signed-off-by: Sam Oluwalana <[email protected]>
…ssue with embedding model Signed-off-by: Sam Oluwalana <[email protected]>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughMigrates customization job creation from ChangesAutomodel API Migration and Tutorial Expansion
Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 10
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
docs/customizer/tutorials/lora-customization-job.ipynb (1)
54-87: 🩺 Stability & Availability | 🔴 CriticalMissing
import osbreaks the first cell.
import osis absent from the imports block, butos.environ.getis called at line 83 andos.getenvat line 230. This causesNameError: name 'os' is not definedon execution.Proposed fix
- "import json\n", + "import json\n", + "import os\n", "import re\n",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/customizer/tutorials/lora-customization-job.ipynb` around lines 54 - 87, The imports block at the beginning of the cell is missing the `import os` statement, but the code later calls `os.environ.get()` at line 83, which will cause a NameError. Add `import os` to the imports section along with the other standard library imports like `json`, `re`, `time`, and `uuid`, placing it before the imports from nemo_platform modules.docs/customizer/tutorials/embedding-customization-job.ipynb (2)
697-700: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winRaise on failed training jobs before deployment.
The loop exits on failure states, then the next cells still retrieve and deploy
OUTPUT_NAME.Proposed fix
- if status.status in ("completed", "failed", "cancelled", "error"): + if status.status == "completed": print(f"\nJob finished with status: {status.status}") break + + if status.status in ("failed", "cancelled", "error"): + raise RuntimeError(f"Job finished with status: {status.status}")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/customizer/tutorials/embedding-customization-job.ipynb` around lines 697 - 700, The current logic in the status check exits the loop for both successful completion and failure states, then continues to retrieve and deploy the output even when the job has failed. Modify the condition to only break when status is "completed", and raise an appropriate exception with a descriptive error message when the status is in the failure states "failed", "cancelled", or "error". This will prevent deployment from proceeding after a failed training job.
761-767: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winPass the NGC pull secret for the fine-tuned deployment.
The baseline deployment needs
image_pull_secret=NGC_SECRET_NAMEfor the same nvcr.io image source; this deployment omits it.Proposed fix
nim_deployment=NIMDeploymentParam( image_name=NIM_IMAGE, image_tag=NIM_TAG, gpu=1, + image_pull_secret=NGC_SECRET_NAME, model_name=OUTPUT_NAME, model_namespace="default", )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/customizer/tutorials/embedding-customization-job.ipynb` around lines 761 - 767, The NIMDeploymentParam constructor for the fine-tuned model deployment is missing the image_pull_secret parameter that is required to pull the NGC image. Add the image_pull_secret parameter set to NGC_SECRET_NAME to the NIMDeploymentParam initialization, matching the configuration used in the baseline deployment. This parameter is needed because the deployment uses the same nvcr.io image source which requires proper authentication via the NGC secret.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/customizer/tutorials/distillation-customization-job.ipynb`:
- Around line 882-884: The variable OUTPUT_NAME is not defined in the notebook
and will cause an error during deployment troubleshooting. Replace OUTPUT_NAME
with DISTILLED_STUDENT_NAME in the client.models.retrieve call, as
DISTILLED_STUDENT_NAME is the variable that actually stores the distilled model
output in this notebook. This ensures the verification command references the
correct, defined variable.
- Around line 268-270: The code is using model_dump_json() to dump the entire
hf_secret object to the notebook output, which risks exposing the actual token
value in persistent notebook outputs. Replace the third print statement that
calls hf_secret.model_dump_json(indent=2) with a print statement that outputs
only the secret name instead, such as printing hf_secret.name or a similar
property that contains just the identifier without the sensitive credential
value.
- Around line 439-441: The current code returns all terminal statuses
(completed, failed, cancelled, error) without raising an exception for failures,
which allows subsequent code to proceed with missing or invalid models. Modify
the logic in the status check conditional block to raise an exception when the
status is failed, cancelled, or error, and only return the status object when
the status equals "completed". This ensures failed training jobs are caught
early and prevent downstream processing with invalid models.
- Around line 514-522: The deployment status checking logic returns the
deployment object even when it's in a failed or timed-out state, allowing
downstream code to attempt using an unavailable deployment. Instead of returning
the deployment when the status is in the failed states (FAILED, ERROR,
TERMINATED, LOST) or when the timeout is exceeded, raise an exception to fail
fast. This ensures the function properly signals that the deployment is not
ready and prevents subsequent cells from attempting to interact with an
unavailable deployment.
In `@docs/customizer/tutorials/format-training-dataset.mdx`:
- Line 221: Replace the relative file path link in the "Tool Calling tutorial"
reference with a canonical documentation URL using the `/documentation/...`
format. Instead of linking to the relative `.ipynb` file path
`../../example-applications/tool-calling.ipynb`, determine the correct canonical
documentation URL for the tool calling tutorial and use that absolute path
format to ensure the link remains valid in published documentation.
- Around line 224-230: The Python code snippet uses os.environ.get in the
NeMoPlatform constructor initialization but the os module is not imported,
causing a NameError at runtime. Add an import statement for the os module at the
beginning of the code snippet, before the imports from nemo_automodel_plugin and
nemo_platform.
In `@docs/customizer/tutorials/optimize-throughput.ipynb`:
- Line 788: The issue is that `status_details['val_loss']` is being accessed
directly without verifying that `status_details` exists or contains the
`val_loss` key. Since `wait_for_job` treats `val_loss` as optional, a completed
job may have `status_details=None` or be missing the `val_loss` key entirely,
causing TypeError or KeyError exceptions. Add conditional guards before all
direct accesses to `job_with_sequence_packing_status.status_details['val_loss']`
(at lines 548, 590, 622-623, and 788) to check if `status_details` is not None
and the `val_loss` key exists, then safely retrieve the value with appropriate
fallback handling or default display message if unavailable.
In `@docs/customizer/tutorials/sft-customization-job.ipynb`:
- Around line 877-879: The troubleshooting snippets in the notebook contain
outdated references that no longer match the current Automodel API. Update the
troubleshooting text at the indicated locations to reflect the correct API
usage: change the job reference from `job.name` to `job.job.name` in the
get_status call, ensure any model or dataset references use the `workspace/name`
format instead of just the name, and verify the NIM tag is updated to version
`1.15.5`. Apply these same corrections to both the primary location (lines
877-879) and the secondary location (line 910) mentioned in the review.
- Around line 569-572: The current status check loop breaks silently when a job
fails, cancelled, or errors, which allows subsequent cells to continue executing
and attempt to retrieve/deploy OUTPUT_NAME with an invalid or missing result.
Instead of just breaking when the status is "failed", "cancelled", or "error",
raise an exception with a descriptive error message that includes the job
status. This will prevent downstream cells from executing and ensure the
notebook stops with a clear error indication when the training job does not
complete successfully.
- Around line 362-365: The code is printing the entire hf_secret object using
model_dump_json() which could expose sensitive credential values in the notebook
output. Instead of calling model_dump_json(indent=2) on the hf_secret object,
modify the print statement to only output the secret name (e.g., accessing the
name property of the secret object returned by create_or_get_secret). This
ensures credentials are not persisted in the notebook output while still
confirming the secret was created successfully.
---
Outside diff comments:
In `@docs/customizer/tutorials/embedding-customization-job.ipynb`:
- Around line 697-700: The current logic in the status check exits the loop for
both successful completion and failure states, then continues to retrieve and
deploy the output even when the job has failed. Modify the condition to only
break when status is "completed", and raise an appropriate exception with a
descriptive error message when the status is in the failure states "failed",
"cancelled", or "error". This will prevent deployment from proceeding after a
failed training job.
- Around line 761-767: The NIMDeploymentParam constructor for the fine-tuned
model deployment is missing the image_pull_secret parameter that is required to
pull the NGC image. Add the image_pull_secret parameter set to NGC_SECRET_NAME
to the NIMDeploymentParam initialization, matching the configuration used in the
baseline deployment. This parameter is needed because the deployment uses the
same nvcr.io image source which requires proper authentication via the NGC
secret.
In `@docs/customizer/tutorials/lora-customization-job.ipynb`:
- Around line 54-87: The imports block at the beginning of the cell is missing
the `import os` statement, but the code later calls `os.environ.get()` at line
83, which will cause a NameError. Add `import os` to the imports section along
with the other standard library imports like `json`, `re`, `time`, and `uuid`,
placing it before the imports from nemo_platform modules.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f841e4ed-6dbe-48b9-94bc-41fd03072ca4
📒 Files selected for processing (32)
docs/customizer/manage-customization-jobs/get-job-status.mdxdocs/customizer/tutorials/distillation-customization-job.ipynbdocs/customizer/tutorials/dpo-customization-job.ipynbdocs/customizer/tutorials/embedding-customization-job.ipynbdocs/customizer/tutorials/format-training-dataset.mdxdocs/customizer/tutorials/import-hf-model.mdxdocs/customizer/tutorials/lora-customization-job.ipynbdocs/customizer/tutorials/metrics.mdxdocs/customizer/tutorials/optimize-throughput.ipynbdocs/customizer/tutorials/sft-customization-job.ipynbdocs/customizer/tutorials/understand-configurations-and-models.mdxdocs/example-applications/tool-calling.ipynbdocs/fern/components/notebooks/distillation-customization-job.jsondocs/fern/components/notebooks/distillation-customization-job.tsdocs/fern/components/notebooks/dpo-customization-job.jsondocs/fern/components/notebooks/dpo-customization-job.tsdocs/fern/components/notebooks/embedding-customization-job.jsondocs/fern/components/notebooks/embedding-customization-job.tsdocs/fern/components/notebooks/lora-customization-job.jsondocs/fern/components/notebooks/lora-customization-job.tsdocs/fern/components/notebooks/optimize-throughput.jsondocs/fern/components/notebooks/optimize-throughput.tsdocs/fern/components/notebooks/sft-customization-job.jsondocs/fern/components/notebooks/sft-customization-job.tsdocs/fern/components/notebooks/tool-calling.jsondocs/fern/components/notebooks/tool-calling.tsdocs/get-started/concepts/filtering.mdxdocs/troubleshooting/customizer.mdxpackages/nmp_testing/src/nmp/testing/e2e/customizer.pyplugins/nemo-automodel/src/nemo_automodel_plugin/transform.pyservices/automodel/src/nmp/automodel/tasks/training/backends/checkpoints.pyservices/core/models/src/nmp/core/models/api/v2/models.py
💤 Files with no reviewable changes (2)
- plugins/nemo-automodel/src/nemo_automodel_plugin/transform.py
- docs/customizer/tutorials/dpo-customization-job.ipynb
Contributor
|
🌿 Preview your docs: https://nvidia-preview-solu-fix-docs.docs.buildwithfern.com/nemo-platform |
Signed-off-by: Sam Oluwalana <[email protected]>
Contributor
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/customizer/tutorials/distillation-customization-job.mdx`:
- Around line 692-693: The troubleshooting step at lines 692-693 incorrectly
references `batch_size` as the parameter to reduce, but this tutorial's job
configuration uses `batch.global_batch_size` and `batch.micro_batch_size`
instead. Update the troubleshooting text in step 3 to reference the correct
configuration keys that match the actual job spec shown in this tutorial,
replacing the generic `batch_size` mention with the specific keys used
throughout this distillation customization job configuration.
- Around line 505-512: Replace the fixed time.sleep(60) call with a
polling-based approach that checks the deployment status. After calling
client.inference.deployments.delete(name=BASELINE_DEPLOYMENT_NAME,
workspace="default"), add a loop that repeatedly queries the deployment status
(using client.inference.deployments.get or similar) with a small sleep between
attempts until the deployment returns a deleted/not-found status. Only then
proceed to delete the deployment config using
client.inference.deployment_configs.delete(name=BASELINE_DEPLOYMENT_CONFIG,
workspace="default"). This ensures the deployment is actually deleted before
attempting config deletion, eliminating the race condition that can cause
walkthrough failures.
- Around line 70-380: The tutorial currently presents all operational steps
using only Python SDK code blocks, but it should include CLI alternatives for
consistency. For each major operational step that involves API calls (such as
file uploads in the dataset preparation, secret creation, model fileset
creation, and job submission), add corresponding CLI command examples alongside
the Python SDK code using MDX tab sets that toggle between "Python SDK" and
"CLI" variants. This applies to sections like "Prepare Dataset", "Secrets
Setup", "Create Model FileSets and Model Entities", and "Train Teacher with Full
SFT" to ensure users can complete the tutorial using either the Python SDK or
command-line tools.
- Around line 708-709: In the troubleshooting section where it verifies the
output model exists, replace the undefined variable `OUTPUT_NAME` with the
correct variable name `DISTILLED_STUDENT_NAME` that was actually defined earlier
in the tutorial. This appears in the client.models.retrieve command that checks
if the output model exists.
In `@docs/customizer/tutorials/embedding-customization-job.mdx`:
- Around line 411-412: Remove the stale constraint documentation that requires
MODEL_NAME or HF_REPO_ID to contain the substring "embed" for embedding models.
This note in the embedding-customization-job.mdx file is outdated based on the
backend changes in this PR that removed the embedding-model SFT guard
requirement. Delete the entire line starting with "*Note*: Either `MODEL_NAME`
or `HF_REPO_ID`..." to eliminate this false blocker from the documentation.
In `@docs/customizer/tutorials/lora-customization-job.mdx`:
- Around line 35-66: The code uses os.environ.get() to retrieve the NMP_BASE_URL
environment variable, but the os module is not imported in the imports section
at the top of the cell. Add import os to the import statements along with json,
re, time, and uuid to resolve the NameError that occurs when the NMP_BASE_URL
variable is accessed.
In `@docs/customizer/tutorials/optimize-throughput.mdx`:
- Around line 682-683: The code directly accesses `status_details['val_loss']`
without checking if the key exists, which will raise a KeyError if val_loss is
missing after training. Replace the direct dictionary access with the `.get()`
method to provide a default value (such as None or a placeholder message) in
case the key is absent. Apply this fix to all instances where
`status_details['val_loss']` is accessed directly throughout the tutorial at the
lines mentioned, ensuring safe access to potentially missing status details
keys.
In `@docs/customizer/tutorials/sft-customization-job.mdx`:
- Around line 697-734: The troubleshooting section contains outdated and
inconsistent guidance that does not align with the actual configuration values
and API usage shown earlier in the tutorial. Review the actual implementation
details shown above in the page (the model submission method, batch_size and
micro_batch_size values used in the example configuration, and the NIM image tag
used in the deployment example), then update the troubleshooting section to
reference the correct batch values in the OOM error section, remove or correct
the model_uri/fileset:// reference to match how models are actually submitted in
this tutorial, and update the NIM image tag reference from 1.13.1 to match the
image tag actually used in the primary deployment example in this page.
In `@docs/fern/scripts/ipynb-to-mdx.py`:
- Line 24: The script imports NotebookConverter from nemo_nb.converter but the
nemo-nb package is not declared in the requirements.txt file for the
docs/fern/scripts directory. Add nemo-nb as a dependency entry to
docs/fern/scripts/requirements.txt to ensure the required package is available
when the ipynb-to-mdx.py script runs in clean environments.
In `@docs/fern/scripts/README.md`:
- Line 47: The heading "### Run" at line 47 is a duplicate of an earlier section
heading in the README file, which violates the MD024 rule and creates ambiguous
anchor links. Rename this heading to something more specific and descriptive
that reflects the content of that section, ensuring it does not match any other
heading in the document. For example, consider using a more detailed heading
like "### Run the Scripts" or similar that better describes what that particular
section covers.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 20906397-01ac-440a-90c6-c9608d6e1fcb
📒 Files selected for processing (7)
docs/customizer/tutorials/distillation-customization-job.mdxdocs/customizer/tutorials/embedding-customization-job.mdxdocs/customizer/tutorials/lora-customization-job.mdxdocs/customizer/tutorials/optimize-throughput.mdxdocs/customizer/tutorials/sft-customization-job.mdxdocs/fern/scripts/README.mddocs/fern/scripts/ipynb-to-mdx.py
Signed-off-by: Sam Oluwalana <[email protected]>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/customizer/tutorials/lora-customization-job.ipynb (1)
54-59: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick winRestore the missing
osimport.Line 83 calls
os.environ, so the setup cell now fails immediately withNameError.Proposed fix
import json +import os import re import time🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/customizer/tutorials/lora-customization-job.ipynb` around lines 54 - 59, The `os` module is not imported in the setup cell, but `os.environ` is called later at line 83, causing a NameError. Add `import os` to the imports section alongside the other import statements (json, re, time, uuid, Path, NeMoPlatform, ConflictError) to ensure the module is available when referenced.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/customizer/tutorials/lora-customization-job.ipynb`:
- Around line 427-448: The deployment is being created before confirming that
the LoRA adapter (OUTPUT_NAME) is properly attached to the model (MODEL_NAME).
This can cause the inference call to target an adapter that the server never
loaded. Move or add the verification step that confirms OUTPUT_NAME is attached
to MODEL_NAME before calling client.inference.deployments.create() in the
deployment creation block.
In `@docs/customizer/tutorials/sft-customization-job.ipynb`:
- Around line 673-692: The vLLM deployment section contains mixed guidance by
including NIM-specific parallelism configuration instructions that do not apply
to vLLM deployments. Remove all references to NIM-specific environment variables
(specifically NIM_TENSOR_PARALLEL_SIZE and NIM_PIPELINE_PARALLEL_SIZE) from the
vLLM deployment example that uses engine="vllm" and vllm-openai image. Keep only
the vLLM-appropriate guidance about configuring the gpu count in
executor_config, which vLLM uses to automatically compute tensor parallelism
based on model architecture.
---
Outside diff comments:
In `@docs/customizer/tutorials/lora-customization-job.ipynb`:
- Around line 54-59: The `os` module is not imported in the setup cell, but
`os.environ` is called later at line 83, causing a NameError. Add `import os` to
the imports section alongside the other import statements (json, re, time, uuid,
Path, NeMoPlatform, ConflictError) to ensure the module is available when
referenced.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3502fbb3-d380-4d87-a242-fa221a2ca8a5
📒 Files selected for processing (12)
docs/customizer/tutorials/distillation-customization-job.ipynbdocs/customizer/tutorials/distillation-customization-job.mdxdocs/customizer/tutorials/format-training-dataset.mdxdocs/customizer/tutorials/lora-customization-job.ipynbdocs/customizer/tutorials/lora-customization-job.mdxdocs/customizer/tutorials/optimize-throughput.ipynbdocs/customizer/tutorials/optimize-throughput.mdxdocs/customizer/tutorials/sft-customization-job.ipynbdocs/customizer/tutorials/sft-customization-job.mdxdocs/example-applications/tool-calling.ipynbdocs/fern/components/notebooks/tool-calling.jsondocs/fern/components/notebooks/tool-calling.ts
🚧 Files skipped from review as they are similar to previous changes (7)
- docs/customizer/tutorials/sft-customization-job.mdx
- docs/customizer/tutorials/distillation-customization-job.ipynb
- docs/customizer/tutorials/format-training-dataset.mdx
- docs/customizer/tutorials/optimize-throughput.ipynb
- docs/customizer/tutorials/lora-customization-job.mdx
- docs/customizer/tutorials/distillation-customization-job.mdx
- docs/customizer/tutorials/optimize-throughput.mdx
Signed-off-by: Sam Oluwalana <[email protected]>
Signed-off-by: Sam Oluwalana <[email protected]>
Signed-off-by: Sam Oluwalana <[email protected]>
Signed-off-by: Sam Oluwalana <[email protected]>
Signed-off-by: Sam Oluwalana <[email protected]>
Signed-off-by: Sam Oluwalana <[email protected]>
gabwow
approved these changes
Jun 23, 2026
gabwow
left a comment
gabwow
left a comment
Contributor
There was a problem hiding this comment.
added to questions but lgtm
Signed-off-by: Sam Oluwalana <[email protected]>
Signed-off-by: Sam Oluwalana <[email protected]>
Contributor
Author
|
Verified using the fern api locally, looks good. Merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
Release Notes
Documentation
training) and refreshed REST/CLI examples and response samples.Improvements
Documentation Tooling
ipynb-to-mdx.pyconversion utility and updated Fern script run instructions.