Add regression test for concat string through println then shell by ATX24 · Pull Request #3726 · BoundaryML/baml

ATX24 · 2026-06-09T21:53:14Z

The error

Reported failing BAML snippet:

function main() -> string {
    let cmd = "echo " + "hello_world";
    baml.io.println("-> " + cmd);
    baml.sys.shell(cmd, null).stdout.to_string()
}

I could not reproduce the reported silent-empty failure on the current compiler2 implementation. Before changing repository files, I ran the snippet through the baml_tests VM harness from a disposable crate outside the repo. Its real output was:

-> echo hello_world
result: Ok(String("hello_world\n"))

Root cause

No current compiler/runtime defect was observed in baml_language/. The gap is test coverage: crates/baml_tests/tests/shell.rs covered dynamic shell command strings, but did not specifically lock down the sequence where a concat-built command is first consumed by baml.io.println and then passed to baml.sys.shell.

The fix

Added a regression test in baml_language/crates/baml_tests/tests/shell.rs that executes BAML snippets covering:

let cmd = "echo " + "hello_world"
baml.io.println("-> " + cmd) before baml.sys.shell(cmd, null)
println called twice before shell
multi-step concat through two let bindings
assigning the println result to _

Each case asserts that shell stdout is not empty and exactly equals "hello_world\n".

Verification

Initial reproduction before repository changes:

cargo +1.93.0 run  # from disposable /tmp/baml-concat-shell-repro
-> echo hello_world
result: Ok(String("hello_world\n"))

Targeted regression test:

cargo test --package baml_tests shell_concat_command_survives_println_before_shell -- --nocapture
-> echo hello_world
first -> echo hello_world
second -> echo hello_world
-> echo hello_world
-> echo hello_world
test shell_concat_command_survives_println_before_shell ... ok

Workspace/test gates run from baml_language/:

cargo test --quiet
# exit code 0

cargo test --workspace --lib --exclude bridge_nodejs
# ok; bridge_nodejs documents that forced Rust lib tests cannot link because Node provides napi_* symbols at addon-load time

cargo fmt --all
cargo clippy --all-targets -- -D warnings
# finished successfully

Additional setup/checks needed for the full local run:

/home/ubuntu/.local/bin/mise trust /workspace/baml_language
./sdk_tests/crates/typescript_node/setup.sh
cargo test -p sdk_test_python_pydantic2 --lib
# 14 passed
cargo test -p sdk_test_typescript_node --lib
# 15 passed

Snapshot status: no snapshot failures occurred during the full cargo test --quiet run, so no snapshots were regenerated.

CodeRabbit: not run because CODERABBIT_API_KEY is missing from the environment; the CLI could not be authenticated non-interactively.

Issue Reference

This PR fixes/closes #[issue number]

Changes

Added targeted compiler2 regression coverage for concat-built shell command strings after baml.io.println consumption.

Testing

Unit tests added/updated
Manual testing performed
Tested in local Linux cloud-agent environment

Screenshots

Not applicable.

PR Checklist

I have read and followed the contributing guidelines
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings

Additional Notes

No compiler/runtime fix was added because the reported failure did not reproduce on the current compiler2 code.

Summary by CodeRabbit

Tests
- Added a non-Windows async test verifying shell commands built via string concatenation still execute correctly even if println is called beforehand; asserts execution succeeds and stdout equals the expected trimmed output.

Co-authored-by: Dhilan Shah <[email protected]>

vercel · 2026-06-09T21:53:20Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
beps	Ready	Preview, Comment	Jun 10, 2026 12:36am
promptfiddle	Ready	Preview, Comment	Jun 10, 2026 12:36am
promptfiddle2	Ready	Preview, Comment	Jun 10, 2026 12:36am

coderabbitai · 2026-06-09T21:53:22Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: c9ae6f8e-3ddd-41dc-bc5c-8a5c2b1ba204

📥 Commits

Reviewing files that changed from the base of the PR and between 15689e1 and 07684ab.

📒 Files selected for processing (1)

baml_language/crates/baml_tests/tests/shell.rs

🚧 Files skipped from review as they are similar to previous changes (1)

baml_language/crates/baml_tests/tests/shell.rs

📝 Walkthrough

Walkthrough

Adds a non-Windows Tokio test shell_concat_command_survives_println_before_shell that runs several BAML programs which build a shell command via string concatenation, call baml.io.println before baml.sys.shell, and assert stdout equals hello_world after trimming.

Changes

Shell Test Addition

Layer / File(s)	Summary
Shell concatenation and println handling test `baml_language/crates/baml_tests/tests/shell.rs`	New test iterates several concatenation/println scenarios, executes `baml.sys.shell(cmd, null).stdout.to_string()`, and asserts successful execution with stdout trimmed to `hello_world`.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐰 I hopped through lines of test and cheer,
Concatenated echoes ringing clear,
A println whispers, then the shell replies,
"hello_world" returned beneath the skies,
A tiny test, a rabbit's joyful sigh.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding a regression test for a specific scenario involving string concatenation, println, and shell command execution.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch cursor/baml-concat-shell-regression-6fc4

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-09T21:55:02Z

⏭️ Performance benchmarks were skipped

Perf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to canary/main.

To run them on this PR, do any of the following, then push a commit (or re-run CI):

Add RUN_CODSPEED=1 to the PR description, or
Include run-perf or /perf in the PR title or any commit message.

github-actions · 2026-06-09T22:00:34Z

No description provided.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@baml_language/crates/baml_tests/tests/shell.rs`:
- Around line 50-51: The test shell_concat_command_survives_println_before_shell
is brittle on Windows because it asserts an exact "hello_world\n"; either gate
the test with #[cfg(not(target_os = "windows"))] to run only on non-Windows
platforms or normalize the captured stdout before asserting (trim trailing CR/LF
or replace "\r\n" with "\n") in the test function so the assertion is
platform‑agnostic.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: cf0a29f9-7fd0-4991-aaf7-35000e53f7f8

📥 Commits

Reviewing files that changed from the base of the PR and between e93d579 and 15689e1.

📒 Files selected for processing (1)

baml_language/crates/baml_tests/tests/shell.rs

Co-authored-by: Dhilan Shah <[email protected]>

Add concat println shell regression test

15689e1

Co-authored-by: Dhilan Shah <[email protected]>

vercel Bot deployed to Preview – beps June 9, 2026 21:53 View deployment

vercel Bot deployed to Preview – promptfiddle2 June 9, 2026 21:59 View deployment

vercel Bot deployed to Preview – promptfiddle June 9, 2026 22:11 View deployment

ATX24 marked this pull request as ready for review June 9, 2026 22:43

coderabbitai Bot requested changes Jun 9, 2026

View reviewed changes

Comment thread baml_language/crates/baml_tests/tests/shell.rs

Fix shell regression test line ending assertion

07684ab

Co-authored-by: Dhilan Shah <[email protected]>

vercel Bot deployed to Preview – beps June 10, 2026 00:18 View deployment

coderabbitai Bot approved these changes Jun 10, 2026

View reviewed changes

vercel Bot deployed to Preview – promptfiddle2 June 10, 2026 00:24 View deployment

vercel Bot deployed to Preview – promptfiddle June 10, 2026 00:36 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add regression test for concat string through println then shell#3726

Add regression test for concat string through println then shell#3726
ATX24 wants to merge 2 commits into
canaryfrom
cursor/baml-concat-shell-regression-6fc4

ATX24 commented Jun 9, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

vercel Bot commented Jun 9, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

github-actions Bot commented Jun 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ATX24 commented Jun 9, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The error

Root cause

The fix

Verification

Issue Reference

Changes

Testing

Screenshots

PR Checklist

Additional Notes

Summary by CodeRabbit

Uh oh!

vercel Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

github-actions Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⏭️ Performance benchmarks were skipped

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ATX24 commented Jun 9, 2026 •

edited by coderabbitai Bot

Loading

vercel Bot commented Jun 9, 2026 •

edited

Loading

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading

github-actions Bot commented Jun 9, 2026 •

edited

Loading