Question about Alpamayo SFT CoC

[moonzerokevin]

Hi,

From the paper I expected Chain-of-Causation (CoC) reasoning to be part of supervised fine-tuning, not only trajectory prediction. The default configs in this repo look trajectory-focused: vla_processor.yaml uses label_components: ["traj_future"] and does not add a cot block in components_order.

load_physical_aiavdataset (used by PAIDataset) loads images and ego trajectories but does not expose ground-truth CoC text, so it’s unclear how one would train CoC supervision with the stock data path.

In TrainableReasoningVLA.forward there is a losses["others"] term, but with the default masks it isn’t obvious that this is meant for CoC.

Could you clarify: Is the released recipe intentionally trajectory-centric, while full CoC-supervised SFT used internal data and a different pipeline? If CoC SFT is supported here, what config + data fields should users set?

Thanks.

[super-anova]

Hi, thanks for your question.

In our released SFT code, we focused on trajectory to give an example of SFT with Alpamayo 1 model. The SFT code supports other labels like CoC. You would need to modify https://github.com/NVlabs/alpamayo/blob/main/finetune/sft/configs/vla_processor.yaml to add a CoC component just like how it is done for trajectory. The model already supports next token loss like you said (losses["others"]): https://github.com/NVlabs/alpamayo/blob/main/finetune/sft/models/sft_base_model.py

We have also released some CoC label in Physical AI AV dataset. You can check it out here: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles#reasoning-labels

[HuangChiEn]

Hi, thanks for your question.

In our released SFT code, we focused on trajectory to give an example of SFT with Alpamayo 1 model. The SFT code supports other labels like CoC. You would need to modify https://github.com/NVlabs/alpamayo/blob/main/finetune/sft/configs/vla_processor.yaml to add a CoC component just like how it is done for trajectory. The model already supports next token loss like you said (losses["others"]): https://github.com/NVlabs/alpamayo/blob/main/finetune/sft/models/sft_base_model.py

We have also released some CoC label in Physical AI AV dataset. You can check it out here: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles#reasoning-labels

TKS for your reply, i was also finding the missing part of CoC SFT training.
Seems it can be reached out by adjust the label type ~

[zhang-wen]

@super-anova

We have also released some CoC label in Physical AI AV dataset. You can check it out here: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles#reasoning-labels

hi anova, when extracting information from the ood_reasoning.parquet file, the clip UUID was not retrieved. Where can I find the clip UUID information corresponding to each CoC label?