Possible False Positives in Stage-Based Evaluation for Tasks 10 and 20–24

[wechatwang]

Hi, thanks for releasing this benchmark. We found two possible issues in the stage-based evaluation:

1. Task 10 and counting-related tasks: After checking the evaluation criterion for Task 10, it appears to only check whether the robot joint/end-effector is tilted beyond a certain angle, without verifying whether the bottle is actually grasped or lifted. As a result, the process stage can be marked as successful even when the bottle is not successfully picked up. In addition, for some other counting-related tasks, executions with more actions than the required number can still be reported as successful. We believe the criteria may be too permissive and may not enforce the intended exact action count.

2. Microwave tasks: In Tasks 20, 21, 23, and 24, our policy did not complete the first step of opening the microwave, but the CSR was consistently reported as 25%. This suggests that the evaluator may be treating 01_Open_Microwave as completed even when the door is not actually opened. Task 22 uses the same microwave-opening predicate at Stage 4. Could you please help verify this issue?

Thanks!

[wpyaoxueros]

1. Regarding question 2, the reason for the problem is that the old evaluation logic did not correctly read the actual joint of the microwave oven door; instead, it fell back to using the absolute x-coordinate of the handle to determine whether the door was opened. Since the x-coordinate of the handle in the reset initial state already met the threshold, Task20/21/23/24 might mistakenly be judged as having completed the first stage of Open_Microwave when the microwave oven door was not open; the reused microwave oven opening judgment in Task22 also has the same risk.
   After the fix, the evaluation logic will prioritize reading the actual hinge joint: microwave_1_microjoint. Only when this joint cannot be read will the fallback use the displacement relative to the initial position of the handle instead of the absolute x-coordinate.
   I also conducted no-op verification: Task20/21/22/23/24 did not execute any valid actions for 100 consecutive steps after reset. The old logic would falsely trigger the open stage in the 0 frame, while the new logic would not trigger it. This indicates that this false positive has been eliminated.
   The fix has been pushed to the main branch of the remote OpenHelix-Team/RoboMemArena, and the commit is: b175708
   2.Regarding Question 1, we will quickly formulate stricter evaluation criteria and conduct verification before committing. We will reply to you again under this question at that time.