is combined VLM and VLA training supported?

[rohun-tripathi]

Thanks for open sourcing this code.
Do you support combined VLM and VLA training as done in works such as pi0.5?

If yes, can you point me to the code entry point for it?

If not, do you have plans to add it?

[jmercat]

Hi, thanks for the interest!
My understanding is that you mean co-training the VLM and VLA with both Image/text VLM data source and robotics data source with a sum of losses for token prediction with the VLM and flow matching loss for the VLA.
We plan on integrating that functionality soon.
Currently the dataloader does support alternating batches from different sources but we did not use co-training from this codebase yet and did not co-train a VLA with different losses. It is on our todo list already.