Skip to content

run_valgrind_tests.sh treats only exit 100 as failure, so crashes look green #203

Description

@leo-aa88

Summary

run_valgrind_tests.sh only fails when valgrind exits 100 (--error-exitcode=100):

if [[ $valgrind_exit_code -eq 100 ]]; then
    echo "Valgrind detected memory issues in $f"
    exit 1
fi

Any other non-zero — missing binary, ASan startup SIGSEGV (typically 139), interpreter crash — is treated as success and the loop continues. That is how #186 stayed hidden: valgrind never ran the program (0 allocs, 0 frees) and the script still exited 0.

Why this matters

Even after #202 (make valgrind builds a non-ASan brainrot-valgrind), a mis-invoked script (./run_valgrind_tests.sh with no args / ASan ./brainrot) or a hard crash still reports green.

Expected Brainrot parse/semantic errors must keep passing: those fixtures exit non-zero on purpose, and valgrind then forwards that exit code when there are no leaks. Do not fail on every non-zero.

Suggested direction

Fail (or at least abort the suite) on unexpected valgrind/child deaths without treating application-level error exits as leaks. For example:

  • treat signal exits (128+N, e.g. 139 SIGSEGV / 134 SIGABRT) as failure
  • treat valgrind usage errors / missing binary as failure
  • keep allowing the interpreter's own non-zero exits when valgrind reports no errors

Spotted while reviewing #202.

Metadata

Metadata

Assignees

No one assigned

    Labels

    CIContinuous integrationbugSomething isn't workinghelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions