[Question] High variance in evaluation results for the same model on identical route (id="2091")

[NinoNeumann]

Describe the question

Hi team,

I am encountering a significant inconsistency when evaluating my model. I ran the exact same model twice on the same case, but the results are completely different.

Test Case Details:

• Route ID: 2091
• Town: Town12

Evaluation Results:

• Run 1: score_composed = 21.14 (Failed)
• Run 2: score_composed = 100.0 (Completed)

Questions

1. Is this level of variance normal for this evaluation framework?
2. Is the randomness in the environment (e.g., NPC behaviors, traffic manager, spawning) expected to cause such a huge gap between two runs?
3. Are there any specific configurations or random seeds I should fix to ensure reproducible evaluation results?

Thanks in advance for your help!

[PDillis]

Hello. I am not one of the authors, but I would note that this is typical behavior in CARLA. Take a very simple benchmark, such as the NoCrash Regular in Town02:

For the first 5 routes, we try 3 different seeds and display the DS for each. Note that, on Seed 0 Route 1, the policy gets a measly DS of 8.8. However, it perfectly completes the route on the other two seeds. What is typically reported as the standard deviation of the Driving Scores is actually the standard deviation of the average Driving Scores per seed, and this is a much lower statistic: $70.3 \pm 15.2$. The real standard deviation of all the routes and all the seeds is in the bottom right corner, so we get: $70.3 \pm 35.3$, and that is not so nice to see when publishing papers, but is exactly what you are observing in your own experiments. Still, note that the model is much more stable within routes (rows), the true "instability" comes from agglomerating the statistics between seeds (columns).