Hi, thanks for the benchmark. While evaluating Task 6 ("pour tomato sauce over cookies twice, then place into bowl drainer") we found the success criteria can mark a rollout as fully successful even when the policy pours only once. The criteria of TSR is whether the sauce is put in the drainer and the CSR is the fractrion of sub-stages completed. Both pour stages use the exact same check (_pour_stage(0.30, 10)) The down-swing and the up-swing of the same pour are counted as two separate pours...
Hi, thanks for the benchmark. While evaluating Task 6 ("pour tomato sauce over cookies twice, then place into bowl drainer") we found the success criteria can mark a rollout as fully successful even when the policy pours only once. The criteria of TSR is whether the sauce is put in the drainer and the CSR is the fractrion of sub-stages completed. Both pour stages use the exact same check (_pour_stage(0.30, 10)) The down-swing and the up-swing of the same pour are counted as two separate pours...