Questions about task 15

[wechatwang]

Hi, thank you for maintaining RoboMemArena.

I am evaluating SmolVLA on RoboMemArena using the latest evaluator (aeb985c), and I found a possible issue in Task 15.

According to the evaluator, Task 15 achieves:

Stage success: 45/50 (90%)
TSR: 45/50 (90%)
Goal success: 45/50 (90%)

However, after checking the rollout videos, I found that in many episodes the robot arm does not touch, grasp, or move the milk at all. It only places the butter into the frypan, but stages 2–4 are still marked as successful:

01_Place_Butter_Frypan = Y
02_Pour_One = Y
03_Pour_Two = Y
04_Place_Milk_Table = Y

The stage timestamps also look suspicious. In 38 of the 45 successful episodes, Pour Two is triggered exactly 10 steps after Pour One, which is the same as the current min_steps=10 setting. Place_Milk_Table is then usually triggered one step later.

For example:

seed100:
Place Butter Frypan: t=583
Pour One: t=684
Pour Two: t=694
Place Milk Table: t=695

This suggests that the current tilt-based pour checker may classify small residual orientation changes as two pours, even when the milk is never actually manipulated.

Could you please help check whether Task 15 needs additional constraints, such as requiring the milk to be grasped/lifted/displaced before counting a pour, and requiring it to leave the initial table region before counting Place_Milk_Table?

I can provide representative rollout videos if helpful. Thank you again for your careful maintenance of the benchmark.

[wpyaoxueros]

Hi, thank you for the feedback.

To clarify, in commit aeb985c3a77e2f80e96f983406e1d475c0842b5e, the Task 15 evaluator already includes the logic you mentioned: the milk must first satisfy the Lift_Milk stage, meaning its object height must rise above its initial position by a threshold, before the later Pour_One and Pour_Two stages can be counted.

For the pour counting part, our current logic is based on the object pose / tilt change, and we previously validated this logic on training rollouts. However, if you still observe inaccurate counting under this exact commit, we will check it further.

If convenient, please feel free to share more detailed information, such as the exact commit, logs, stage timestamps, or representative videos. That would help us locate the issue more precisely. Thank you.