Summary
run_valgrind_tests.sh only fails when valgrind exits 100 (--error-exitcode=100):
if [[ $valgrind_exit_code -eq 100 ]]; then
echo "Valgrind detected memory issues in $f"
exit 1
fi
Any other non-zero — missing binary, ASan startup SIGSEGV (typically 139), interpreter crash — is treated as success and the loop continues. That is how #186 stayed hidden: valgrind never ran the program (0 allocs, 0 frees) and the script still exited 0.
Why this matters
Even after #202 (make valgrind builds a non-ASan brainrot-valgrind), a mis-invoked script (./run_valgrind_tests.sh with no args / ASan ./brainrot) or a hard crash still reports green.
Expected Brainrot parse/semantic errors must keep passing: those fixtures exit non-zero on purpose, and valgrind then forwards that exit code when there are no leaks. Do not fail on every non-zero.
Suggested direction
Fail (or at least abort the suite) on unexpected valgrind/child deaths without treating application-level error exits as leaks. For example:
- treat signal exits (128+N, e.g. 139 SIGSEGV / 134 SIGABRT) as failure
- treat valgrind usage errors / missing binary as failure
- keep allowing the interpreter's own non-zero exits when valgrind reports no errors
Spotted while reviewing #202.
Summary
run_valgrind_tests.shonly fails when valgrind exits100(--error-exitcode=100):Any other non-zero — missing binary, ASan startup SIGSEGV (typically 139), interpreter crash — is treated as success and the loop continues. That is how #186 stayed hidden: valgrind never ran the program (
0 allocs, 0 frees) and the script still exited 0.Why this matters
Even after #202 (
make valgrindbuilds a non-ASanbrainrot-valgrind), a mis-invoked script (./run_valgrind_tests.shwith no args / ASan./brainrot) or a hard crash still reports green.Expected Brainrot parse/semantic errors must keep passing: those fixtures exit non-zero on purpose, and valgrind then forwards that exit code when there are no leaks. Do not fail on every non-zero.
Suggested direction
Fail (or at least abort the suite) on unexpected valgrind/child deaths without treating application-level error exits as leaks. For example:
Spotted while reviewing #202.