Latency discrepancy on H100

[irenaby]

Thanks for open sourcing your work!
I'm having a problem reproducing the reported latency. I followed the instructions in README to set up the environment and to run evaluation on a pretrained checkpoint and I measure the latency around get_action() call in run_libero_eval.py over multiple calls. I consistently get ~0.09s on H100, while the reported latency was 0.036s.
I actually get a slightly better latency for OpenVLA-OFT compared to VLA-Adapter. When I measure only the vision backbone, which should be identical for both models, I already get ~0.04s (verified separately on both models), with the 0.5B LLM and the action head adding another ~0.02 and 0.015s respectively (for the non-Pro version).
I'm not sure where the discrepancy could come from. I don't observe any degradation for the OFT that could point to an issue in the vision backbone acceleration. Am I missing anything? Any help would be appreciated.

[WangYH-BUPT]

Thanks for open sourcing your work! I'm having a problem reproducing the reported latency. I followed the instructions in README to set up the environment and to run evaluation on a pretrained checkpoint and I measure the latency around get_action() call in run_libero_eval.py over multiple calls. I consistently get ~0.09s on H100, while the reported latency was 0.036s. I actually get a slightly better latency for OpenVLA-OFT compared to VLA-Adapter. When I measure only the vision backbone, which should be identical for both models, I already get ~0.04s (verified separately on both models), with the 0.5B LLM and the action head adding another ~0.02 and 0.015s respectively (for the non-Pro version). I'm not sure where the discrepancy could come from. I don't observe any degradation for the OFT that could point to an issue in the vision backbone acceleration. Am I missing anything? Any help would be appreciated.

First, is your h100 a PCIe interface? This might affect the speed to some extent. Also, are you running other experiments? OpenVLA-OFT and VLA-Adapter do use the same visual backbone. OpenVLA-OFT uses a 7-bit language model, while VLA-Adapter uses a 0.5-bit model, theoretically making it faster. When we test the speed, we add time before and after generating the action; you can also check if the addition is done correctly.