Skip to content

test(ext-agents): guard tier selectors against false-green #9115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
glharper wants to merge 2 commits into
base: main
Choose a base branch
from
+68 −2
Open

test(ext-agents): guard tier selectors against false-green #9115

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b6101a1
test(ext-agents): guard tier selectors against false-green
glharper Jul 13, 2026
a1f99f3
test(ext-agents): validate tier selectors before builds
glharper Jul 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 68 additions & 2 deletions .github/workflows/test-ext-azure-ai-agents.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,79 @@ permissions:
jobs:
test:
runs-on: ubuntu-latest
env:
TIER_0_TEST_PATTERN: '^Test_AIAgent_(Version|Help|Init_NoPrompt_MissingFlags|Doctor_Help)$'
TIER_1_TEST_PATTERN: '^Test_AIAgent_(Init_(NoPrompt_(Defer|WithProject)|NegativeControl_BadCassette)|SampleList_(Recorded|JSON_Recorded))$'
steps:
- uses: actions/checkout@v6

- uses: actions/setup-go@v6
with:
go-version-file: cli/azd/go.mod

- name: Validate tier test selectors

@jongio jongio Jul 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guard only needs Go and the test sources to compile. go test -list against ./test/functional/ai_agents/ runs fine with just checkout + setup-go (the package only imports test/azdcli and test/recording, no built extension is needed). As placed, it runs after the inspector and agents build/publish steps, so a bad selector won't surface until several minutes of extension builds have already run. Moving this step up to right after setup-go would fail fast on selector drift before the expensive build work. Non-blocking.

@glharper glharper Jul 13, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1f99f3. The selector validation now runs immediately after actions/setup-go, before building azd, the inspector dependency, or the agents extension, so selector drift fails fast. Re-ran the guard locally: it still validates the current 4 Tier 0 / 5 Tier 1 split.

working-directory: cli/azd
shell: bash
run: |
set -euo pipefail

list_tests() {
go test -tags=record -list "$1" ./test/functional/ai_agents/ |
awk '/^Test_AIAgent_/ { print }' |
tr -d '\r'
}

all_tests="$(list_tests '^Test_AIAgent_')"
tier_0_tests="$(list_tests "$TIER_0_TEST_PATTERN")"
tier_1_tests="$(list_tests "$TIER_1_TEST_PATTERN")"

if [[ -z "$all_tests" ]]; then
echo "::error::No Test_AIAgent_ tests were discovered."
exit 1
fi
if [[ -z "$tier_0_tests" ]]; then
echo "::error::The Tier 0 test selector matched no tests."
exit 1
fi
if [[ -z "$tier_1_tests" ]]; then
echo "::error::The Tier 1 test selector matched no tests."
exit 1
fi

duplicate_tests="$(
printf '%s\n%s\n' "$tier_0_tests" "$tier_1_tests" |
sort |
uniq -d |
tr -d '\r'
)"
if [[ -n "$duplicate_tests" ]]; then
echo "::error::Tests are assigned to multiple tiers:"
echo "$duplicate_tests"
exit 1
fi

all_tests_sorted="$(
printf '%s\n' "$all_tests" |
sort |
tr -d '\r'
)"
selected_tests_sorted="$(
printf '%s\n%s\n' "$tier_0_tests" "$tier_1_tests" |
sort |
tr -d '\r'
)"
if ! diff -u \
<(printf '%s\n' "$all_tests_sorted") \
<(printf '%s\n' "$selected_tests_sorted"); then
echo "::error::Tier selectors must cover every Test_AIAgent_ test exactly once."
exit 1
fi

echo "Tier 0 tests:"
echo "$tier_0_tests"
echo "Tier 1 tests:"
echo "$tier_1_tests"

- name: Build azd
working-directory: cli/azd
run: |
Expand Down Expand Up @@ -78,7 +144,7 @@ jobs:
env:
CLI_TEST_AZD_PATH: ${{ github.workspace }}/cli/azd/azd-record
run: |
go test -tags=record -run "Test_AIAgent_(Version|Help|Init_NoPrompt_MissingFlags|Doctor_Help)" \
go test -tags=record -run "$TIER_0_TEST_PATTERN" \
./test/functional/ai_agents/ -v -timeout 5m

- name: Swap extension to record binary for Tier 1
Expand All @@ -94,5 +160,5 @@ jobs:
AZURE_RECORD_MODE: playback
CLI_TEST_AZD_PATH: ${{ github.workspace }}/cli/azd/azd-record
run: |
go test -tags=record -run "Test_AIAgent_(Init_(NoPrompt_(Defer|WithProject)|NegativeControl_BadCassette)|SampleList_(Recorded|JSON_Recorded))" \
go test -tags=record -run "$TIER_1_TEST_PATTERN" \
./test/functional/ai_agents/ -v -timeout 5m
Loading