Missing functional-correctness test scripts (test_scripts/<id>.sh) for reproduction

[viviable]

Hi, thanks for releasing SecureVibeBench, really nice benchmark.

I'm reproducing the evaluation locally (Claude Code agent), and the agent run + dynamic security (PoV) evaluation work end-to-end. However, I can't reproduce the functional-correctness metric (IC / 1-IC), and I think a piece of data is missing from the public release.

The issue: evaluation/my_utils/patch_diff.py function run_test() reads a per-instance test script from ./test_scripts/{id}.sh (via TEST_SCRIPTS_DIR, default ./test_scripts). These scripts don't appear to be included anywhere in the public release:

• Not in the GitHub repo (no test_scripts/ directory, not tracked in git)
• Not in the Hugging Face dataset (iCSawyer/SecureVibeBench), the rows only contain localid, repo_url, vic, repo_cwd, description
• Not in data/full_dataset.zip, it only contains 105 *.json files, each with the same 5 metadata fields and no embedded test script

So run_test() always hits [WARN] Test script not found: ./test_scripts/.sh, skipping test phase, and the differential-testing step (which the paper describes for functional correctness) never runs. The id_map used in run_test() (the ARVO-id to script-id mapping) also seems to come from a non-public eval_test.py.

Could you share:

1. The test_scripts/ directory (the per-instance .sh differential-test scripts), and
2. The id_map / eval_test.py (or wherever the id mapping is defined)?

That would let us reproduce the IC / 1-IC numbers.

Thanks

[iCSawyer]

Hi @viviable, Thanks a lot for your acknowledgment!

We are inspecting this issue and will fix it asap. Once we are done, we will notify you in this thread.

We really appreciate your valuable feedback!

[iCSawyer]

@viviable Hi, we have added the functional tests to the repository. Please kindly check and thanks for your feedback!