Estimated Cost of Evaluating with GPT-4o

[XinbeiCai]

Thanks for the great work! I'm interested in using GPT-4o for evaluation as described in the repo, but I'm trying to estimate the associated cost. Could you share a rough estimate of how much it typically costs to run the full evaluation suite using GPT-4o?

For context, I'm mainly curious about:

• Number of input/output tokens involved (if known)
• Approximate total cost per evaluation run
  Any ballpark figures or guidance would be very helpful for planning purposes. Thanks in advance!

[Purshow]

Hello! That’s a very practical question.

Unfortunately, I haven't kept a record of the precise token counts, and I currently cannot access the official GPT API to run a new test.

However, based on community estimates, a full evaluation run (e.g., 1,000 images) was reported to cost roughly $7 to $8 USD.

A specific community breakdown for 1,000 images suggested:

• Input Tokens per Image: $\approx 1,415$
• Output Tokens per Image: $\approx 15$

Using estimated GPT-4o-2024-05-13 pricing (using the $5/\text{M token input}$ and $15/\text{M token output}$ rates):

Component	Calculation (per 1,000 images)	Cost (USD)
Input Cost	$(1,000 * 1,415) * (5 / 1,000,000)$ tokens	~$7.075
Output Cost	$(1,000 * 15) * (15 / 1,000,000)$ tokens	~$0.225
Total Estimated Cost	(Input + Output)	~$7.30

This is a community-reported ballpark figure, and actual costs may vary. I hope this helps with your planning!