[Question] llava1.5-openclip vs llava1.5-openai-clip benchmarking

[sarvghotra]

Were you able to reproduce llava1.5 (with openai CLIP) results (on some benchmarks used in the llava1.5 paper) with this repo's implementation (llava1.5 with openclip)?

[sarvghotra]

On MME benchmark I got the following results which is lower than llava with openai CLIP:

LLaVA-OpenCLIP:
Perception: 1373.39
Cognition: 266.42

LLaVA (original code with openai CLIP):
Perception: 1517.37
Cognition: 292.85

I got similar results to this repo, Bliss-e-V/LLaVA-OpenCLIP, with my implementation of LLaVA-openCLIP, so I think it's OpenCLIP (ViT-L-14-CLIPA-336) not at par with OpenAI's CLIP in terms of generalization even though the benchmark numbers show otherwise. If you could comment on it based on your results, it would be great. TIA!