feat(customizer): add docs and skills for customizer rl backend

plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/batch-sizing.md

@@ -0,0 +1,150 @@
+# Batch sizing
+
+VRAM tables, multi-GPU rules, and throughput tuning for **automodel** and **unsloth** on ≥48 GB GPUs. Field glossary and full JSON templates live in `hyperparameters.md`; the skill workflow lives in `SKILL.md`.
+
+## Batch sizing — automodel (≥48 GB VRAM)
+
+Tables, multi-GPU rules, and the tuning loop below are **automodel-specific** (fields `global_batch_size` / `micro_batch_size` / `tensor_parallel_size` / `num_gpus_per_node`). For unsloth see **Batch sizing — unsloth** further down.
+
+Assume **one GPU with at least 48 GB** (e.g. RTX 5880 / A6000 / L40), `parallelism` = 1 node × 1 GPU, `tensor_parallel_size` 1, bf16, `training_type` `sft`, LoRA **rank 16** unless the user asks otherwise.
+
+**How to size**
+
+1. Read **model size** from the entity (`nemo models get`) or HF card (parameter count).
+2. Pick **`finetuning_type`**: `lora` (adapter only, default) vs `all_weights` (full SFT — much heavier).
+3. Set **`max_seq_length`** (2048 is the skill default; shorter seq → more batch headroom).
+4. Set **`micro_batch_size`** first (drives peak VRAM), then **`global_batch_size`** as a multiple of `micro_batch_size` (gradient accumulation when GBS > micro).
+
+**Constraint:** `global_batch_size` must be divisible by `micro_batch_size × data_parallel_size`, where `data_parallel_size = (num_nodes × num_gpus_per_node) / (tensor_parallel_size × pipeline_parallel_size × context_parallel_size)` (1 for a single-GPU job).
+
+### LoRA (`finetuning_type: lora`) — `max_seq_length` 2048
+
+**VRAM does not scale linearly with `micro_batch_size`.** LoRA loads the full base weights once; activation memory grows slowly. On 48 GB, **`micro_batch_size` must decrease as model size grows** (smaller models always ≥ larger models in the table). Use **`global_batch_size` ≈ 4 × `micro_batch_size`**.
+
+**Default batch** — start here for a reliable full epoch. **High utilization** — optional; double from default (or ramp in steps) to reach **~35–40 GiB**. Halve both if OOM (exit **137**) or training crashes (exit **1**).
+
+| Model params | Default `micro` | Default GBS | `learning_rate` | High-util `micro` | High-util GBS |
+|--------------|------------------:|------------:|----------------:|------------------:|--------------:|
+| ≤4B | 32 | 128 | `1e-4` | 64 | 256 |
+| 4B–8B | 24 | 96 | `8e-5` | 48 | 192 |
+| 8B–14B | 16 | 64 | `8e-5` | 24 | 96 |
+| >14B | 8 | 32 | `5e-5` | 16 | 64 |
+
+Validated (`commonsense_qa` @ 2048, 48 GB, one job per GPU): **Qwen3-1.7B** — `micro` 16 / GBS 64 ~8 min; defaults above leave headroom to ramp. **Qwen3-8B** — `micro` 2–4 ≈16–18.5 GiB (under-filled); **`micro` 16 / GBS 64** stable default (~153 steps/epoch); high-util **`micro` 24 / GBS 96** (32 / 128 hit ~40 GiB but failed mid-epoch with exit 1).
+
+### Multi-GPU (same node)
+
+Pick the path by whether the **base model fits in ~48 GB on one GPU** (LoRA or full SFT):
+
+| Situation | `tensor_parallel_size` | Goal |
+|-----------|------------------------:|------|
+| Model **fits** on one ≥48 GB GPU | **1** | **Data parallel** — more GPUs = faster training; keep `micro` per GPU, scale `global_batch_size` |
+| Model **does not fit** on one ≥48 GB GPU | **> 1** (e.g. 2 on a 2-GPU node) | **Tensor parallel** — shard layers across GPUs so the model fits; lower `micro` / GBS vs single-GPU tables |
+
+**Data parallel (TP = 1)** — default for Qwen3-8B LoRA and similar on 48 GB cards:
+
+| Rule | Detail |
+|------|--------|
+| `micro_batch_size` | **Per GPU** — same as a stable single-GPU run |
+| `global_batch_size` | ≈ **single-GPU GBS × `num_gpus_per_node`**; step count ≈ `samples / GBS` |
+| Divisibility | `global_batch_size` ÷ **`micro_batch_size × num_gpus_per_node`** must be an integer |
+| Scheduling | **One job** owns all GPUs; no overlapping 1-GPU and multi-GPU jobs |
+
+```json
+"parallelism": { "num_nodes": 1, "num_gpus_per_node": 2, "tensor_parallel_size": 1 },
+"batch": { "global_batch_size": 128, "micro_batch_size": 16 }
+```
+
+**Tensor parallel (TP > 1)** — when weights + activations OOM on a single ≥48 GB GPU (large full SFT, very long `max_seq_length`, or models above the LoRA sizing table without fitting):
+
+- Set **`num_gpus_per_node`** and **`tensor_parallel_size`** so **`num_gpus_per_node` is divisible by `tensor_parallel_size`** (e.g. 2 GPUs → `tensor_parallel_size: 2`, or 4 GPUs → TP 2 or 4).
+- **`data_parallel_size`** = `(num_nodes × num_gpus_per_node) / (tensor_parallel_size × pipeline_parallel_size × context_parallel_size)` — use this in the GBS divisibility rule instead of raw GPU count.
+- Start with **lower `micro_batch_size`** than the single-GPU table; increase only if VRAM allows. MoE models: if `expert_parallel_size > 1`, **`tensor_parallel_size` must be 1**.
+
+```json
+"parallelism": { "num_nodes": 1, "num_gpus_per_node": 2, "tensor_parallel_size": 2 },
+"batch": { "global_batch_size": 8, "micro_batch_size": 1 }
+```
+
+`execution_profile` is usually still **`"gpu"`** — confirm with `nemo jobs list-execution-profiles -f json`.
+
+**Example — Qwen3-8B LoRA, 2× 48 GB (fits one GPU):** single-GPU **micro 16 / GBS 64** → 2-GPU data parallel **micro 16 / GBS 128**, `learning_rate` `8e-5`.
+
+### Full-weight SFT (`finetuning_type: all_weights`) — `max_seq_length` 2048
+
+| Model params | `micro_batch_size` | `global_batch_size` | `learning_rate` |
+|--------------|-------------------:|--------------------:|----------------:|
+| ≤2B | 2 | 8 | `2e-5` |
+| 2B–4B | 1 | 4 | `1e-5` |
+| 4B–8B | 1 | 2 | `5e-6` |
+| >8B | 1 | 1 | lower LR or use TP / shorter seq |
+
+Output type is **model** (full checkpoint), not adapter. Expect much longer runs than LoRA at the same batch. **Inference:** deploy `default/<output.name>` as a new model entity — full SFT does not hot-reload onto the base model's LoRA deployment.
+
+### `max_seq_length` scaling
+
+Scale **`micro_batch_size`** from the 2048 tables (round down, minimum 1):
+
+| `max_seq_length` | Multiply `micro_batch_size` by |
+|------------------|-------------------------------:|
+| 512 | 4× |
+| 1024 | 2× |
+| 2048 | 1× (tables above) |
+| 4096 | 0.5× |
+
+Then set `global_batch_size` to a multiple of the new `micro_batch_size` (often keep the same ratio as the table, e.g. GBS = 4 × micro for LoRA).
+
+### LoRA rank
+
+Higher rank uses more VRAM. If OOM at rank 16, drop to rank 8 before lowering batch; if headroom remains, rank 32 is fine for training (deploy rank ≤32 on default NIM/vLLM).
+
+### Tuning loop
+
+| Symptom | Action |
+|---------|--------|
+| CUDA OOM | Halve `micro_batch_size`, then `global_batch_size`, then `max_seq_length` |
+| Slow / low GPU memory use | Step up toward the **high-util** column (or double default `micro`+GBS); stop at ~35–40 GiB or when training fails, then use **default** for the retry |
+| User wants max throughput | Raise `micro_batch_size` first; keep GBS ≈ 4× micro — avoid `micro_batch_size` 1 with huge GBS |
+
+Field glossary, distillation/KD, and schema pointers: `references/hyperparameters.md` (batch/multi-GPU → **this file**, not hyperparameters).
+
+## Batch sizing — unsloth (single GPU)
+
+Unsloth is single-GPU by design. The effective batch is the **product** of two fields, not a global/micro split:
+
+```text
+effective_batch = batch.per_device_train_batch_size × batch.gradient_accumulation_steps
+```
+
+There is no `parallelism` block, no TP / PP / DP, no GBS divisibility math. Multi-GPU sharding → switch to automodel.
+
+**Field mapping from the automodel tables above:**
+
+| Automodel field | Unsloth analogue | Notes |
+|-----------------|------------------|-------|
+| `micro_batch_size` | `batch.per_device_train_batch_size` | Drives peak VRAM. |
+| `global_batch_size` | `batch.per_device_train_batch_size × batch.gradient_accumulation_steps` | Set `gradient_accumulation_steps` so the product matches the GBS you'd pick on automodel. |
+| `parallelism.num_gpus_per_node` | n/a — single GPU | Use `hardware.gpus: "0"` to pin to one GPU. |
+| `tensor_parallel_size` | n/a | If the model doesn't fit on one GPU → use automodel. |
+
+**Starting points (LoRA, `max_seq_length` 2048, one ≥48 GB GPU):**
+
+| Model params | `per_device_train_batch_size` | `gradient_accumulation_steps` | Effective batch | `learning_rate` |
+|--------------|------------------------------:|------------------------------:|----------------:|----------------:|
+| ≤4B | 8 | 16 | 128 | `1e-4` |
+| 4B–8B | 4 | 24 | 96 | `8e-5` |
+| 8B–14B | 2 | 32 | 64 | `8e-5` |
+| >14B | 1 | 32 | 32 | `5e-5` |
+
+`load_in_4bit: true` (default) keeps base weights in 4-bit, which is what makes the "smaller per-device batch on bigger models" rule milder than vanilla HF. If you raise `per_device_train_batch_size` and hit OOM (exit 137) or training crashes (exit 1), halve `per_device_train_batch_size` first and double `gradient_accumulation_steps` to keep the effective batch the same.
+
+**Save method.** Default `output.save_method: "lora"` (adapter only — small, fast, hot-reloads on LoRA-enabled deployments). Use `"merged_16bit"` if the user wants a full-weight checkpoint to deploy as a standalone model entity; `"merged_4bit"` only when storage is tight (lossy). Merged methods require `training.finetuning_type: "lora"`. Merged and full SFT outputs must be **deployed for inference** — they do not hot-reload onto the base adapter deployment.
+
+**Tuning loop (unsloth):**
+
+| Symptom | Action |
+|---------|--------|
+| CUDA OOM | Halve `per_device_train_batch_size` (keep effective batch via `gradient_accumulation_steps`); then lower `model.max_seq_length`; then drop `lora.rank` to 8 |
+| Missing `nmp-unsloth-training` image | Build/pull the Unsloth container image — see `references/troubleshooting.md` and `docker/unsloth/README.md` |
+| `Unsloth training requires platform.runtime: docker` (platform not using the Docker executor) | Start platform with `platform.runtime: docker` and a GPU execution profile; training runs in containers on that host's Docker daemon |
+| Loss not moving | Raise `learning_rate` one step (e.g. `5e-5` → `1e-4`); confirm `apply_chat_template` matches the data shape; check the LoRA `target_modules` covers the right layers (defaults are Unsloth's 7-module set) |

plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.md

@@ -1,8 +1,11 @@
 # Dataset formats
 
-Both backends read JSONL from a platform fileset, but the **row shape and the job-JSON dataset block differ**. Pick the section that matches your plugin.
+All three backends read JSONL from a platform fileset, but the **row shape and the job-JSON dataset block differ**. Pick the section that matches your plugin (automodel, unsloth, or rl/DPO).
 
-Upload `train.jsonl` and optional `validation.jsonl` at the **fileset root**. For automodel use the same fileset for `dataset.training` and `dataset.validation`. For unsloth use `dataset.path` (and `dataset.validation_path`).
+Upload the JSONL files at the **fileset root**, then reference the fileset from the job JSON `dataset` block. The filenames differ by backend, so keep the two contracts separate:
+
+- **SFT (automodel, unsloth):** upload `train.jsonl` and optional `validation.jsonl`. Automodel points `dataset.training` / `dataset.validation` at the fileset; unsloth uses `dataset.path` (and `dataset.validation_path`).
+- **rl (DPO):** upload both `training.jsonl` **and** `validation.jsonl` to a single fileset, referenced by one `dataset` string (no separate validation ref) — see § NeMo-RL.
 
 ## Automodel
 
@@ -63,6 +66,53 @@ Eval rows must use the **same CHAT `messages` shape** as training. Do not flatte
 |----------------|--------------|------------------------|------------------|
 | `messages` (single- or multi-turn) | Same fileset split (`validation.jsonl`) | `messages[:-1]` — exclude final assistant label — see `post-training-eval.md` | `{{ item.messages[-1].content }}` |
 
-LoRA inference and eval use the **provider** gateway on the **base** entity (`/provider/<name>/-/v1`, `model: default--<adapter>`). Base model uses the model-entity path. Full SFT / merged checkpoints use the **output** model entity's model-entity URL — deploy first. See `post-training-eval.md` and the **Using the adapter** / **Using the fine-tuned model** sections in `SKILL.md`.
+LoRA inference and eval use the **provider** gateway on the **base** entity (`/provider/<name>/-/v1`, `model: default--<adapter>`). Base model uses the model-entity path. Full SFT / merged checkpoints use the **output** model entity's model-entity URL — deploy first. See `post-training-eval.md` and the **Using the adapter** / **Using the fine-tuned model** sections in `reporting.md`.
 
 Shared helpers and compare CLI: `references/eval_helpers.py`. Full workflow: `references/post-training-eval.md`.
+## NeMo-RL (DPO) — preference data
+
+DPO trains on **preference pairs**, not prompt→completion examples. The `rl` backend takes a **single** dataset fileset that must contain **both** `training.jsonl` **and** `validation.jsonl` at the fileset root (unlike automodel/unsloth, the dataset block in the job JSON is a single ref — there is no separate validation ref).
+
+The dataset-preparation step **auto-detects the row schema from the first line** and selects the matching NeMo-RL loader. **Three preference formats are supported** (platform schemas `BinaryPreferenceDatasetItemSchema` / `HelpSteer3DatasetItemSchema` / `Tulu3PreferenceDatasetItemSchema`):
+
+### Binary preference (`BinaryPreferenceDataset`)
+
+Simple `prompt` / `chosen` / `rejected` — the `prompt` may be a plain string **or** a list of chat messages:
+
+```json
+{"prompt": "What is the capital of France?", "chosen": "The capital of France is Paris.", "rejected": "I'm not sure."}
+```
+
+| Key | Meaning |
+|-----|---------|
+| `prompt` | The input/context shown to the model (string or list of chat messages). |
+| `chosen` | The preferred (higher-reward) response. |
+| `rejected` | The dispreferred response. |
+
+### HelpSteer3 (`HelpSteer3`)
+
+A conversation `context` (string or chat messages), two candidate responses, and a signed `overall_preference` in **-3..3** — **negative** means `response1` is preferred, **positive** means `response2`, **0** is a tie. This is the **raw** schema of `nvidia/HelpSteer3` (the `preference` subset), so no conversion is needed:
+
+```json
+{"context": [{"role": "user", "content": "Explain how to use git rebase"}], "response1": "...", "response2": "...", "overall_preference": -2}
+```
+
+### Tulu3 preference (`Tulu3Preference`)
+
+Full chat conversations for both branches — `chosen` and `rejected` are each a **list of messages** ending with the assistant turn:
+
+```json
+{"chosen": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "preferred"}], "rejected": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "dispreferred"}]}
+```
+
+Whichever format you upload, the job JSON dataset block is just the single fileset ref:
+
+```json
+"dataset": "default/<preference-fileset>"
+```
+
+**Notes**
+- Upload both files to the **same** fileset (`--remote-path training.jsonl` and `--remote-path validation.jsonl`).
+- `prompt` may be a plain string; the model's chat template is applied at training time (override with `training.chat_template` only when needed).
+- DPO is **full-weight** — there is no LoRA/adapter dataset variant. The output is a full model checkpoint.
+

plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.md

@@ -55,7 +55,7 @@ Then upload (see main skill). Validate with `nemo files list <DATASET_NAME> --wo
 
 ## Mapping to job JSON
 
-The same converted JSONL works for both backends, but the **dataset block in job JSON is shaped per backend**.
+Only the **chat-template / `messages` (`to_chat`) output is backend-agnostic** — that JSONL feeds both SFT backends (automodel + unsloth). The `to_sft` (`prompt` / `completion`) shape is **automodel-only**; unsloth needs a `to_text` rendering instead (see below). Either way, the **dataset block in job JSON is shaped per backend**. (rl/DPO uses preference data, not this SFT output — see `dataset-formats.md` § NeMo-RL.)
 
 | Backend | Row format used | Dataset block in job JSON |
 |---------|----------------|---------------------------|
@@ -73,4 +73,4 @@ def to_text(ex):
     return {"text": f"{user}\n{assistant}"}
 ```
 
-For the chat path (`has_chat` True), the `to_chat` JSONL works unchanged across both backends — only the job-JSON dataset block differs.
+For the chat path (`has_chat` True), the `to_chat` JSONL works unchanged across both SFT backends (automodel + unsloth) — only the job-JSON dataset block differs.