Performance gap: Pre-trained LIBERO-Spatial-Pro achieves lower results than reported

[HuadongTang]

Hi, I am trying to replicate the evaluation results using the provided pre-trained weights for LIBERO-Spatial-Pro. However, I am seeing a performance drop. Do you know the reason?

[WangYH-BUPT]

Hi, I am trying to replicate the evaluation results using the provided pre-trained weights for LIBERO-Spatial-Pro. However, I am seeing a performance drop. Do you know the reason?

I suspect you're not using the H100. We use the H100 for both training and inference. Therefore, if the inference stage doesn't use the H100, the different card precision and architecture will cause performance degradation. This isn't unique to us; OpenVLA-OFT and even other large multimodal models experience this issue. You can try training on your own machine. Some students have achieved higher performance than our paper using a 4-card 4090 for training and inference.

[HuadongTang]

Hi, I am trying to replicate the evaluation results using the provided pre-trained weights for LIBERO-Spatial-Pro. However, I am seeing a performance drop. Do you know the reason?

I suspect you're not using the H100. We use the H100 for both training and inference. Therefore, if the inference stage doesn't use the H100, the different card precision and architecture will cause performance degradation. This isn't unique to us; OpenVLA-OFT and even other large multimodal models experience this issue. You can try training on your own machine. Some students have achieved higher performance than our paper using a 4-card 4090 for training and inference.

Thank you so much for your detailed reply and explanation! Your work is truly excellent and very inspiring to my research. I will follow your advice and try fine-tuning on my local hardware to address the alignment issue.

[12hasdief]

Hi authors, I currently use H100, but also can not reproduce 99.7% by your checkpoints, do you have any suggestion towards this problem? Thank you so much!

[WangYH-BUPT]

Hi authors, I currently use H100, but also can not reproduce 99.7% by your checkpoints, do you have any suggestion towards this problem? Thank you so much!

This is indeed an interesting question. I recently ran it with other H cards, and it certainly didn't reach the performance of the original chkpt, but this isn't due to randomness or inaccuracy. You can follow our settings and run it with four H cards for 5,000 steps; the spatial performance will generally be above 98%, and around 20,000 steps, it will reach above 99%. Therefore, you can treat chkpt as a quick and easy tool. Fine-tuning it on your own machine will yield even better results!

[12hasdief]

Thank you so much for your fast response! Really appreciate it!