Skip to content

test(extensions): verify provider manifests (part 2/2)#9122

Draft
JeffreyCA wants to merge 3 commits into
Azure:mainfrom
JeffreyCA:jeffreyca/ext-provider-verification-ci
Draft

test(extensions): verify provider manifests (part 2/2)#9122
JeffreyCA wants to merge 3 commits into
Azure:mainfrom
JeffreyCA:jeffreyca/ext-provider-verification-ci

Conversation

@JeffreyCA

Copy link
Copy Markdown
Contributor

Fixes #8992

Summary

This PR rolls out provider-manifest verification across every provider-bearing Azure AI extension and adds CI enforcement so future registration drift is rejected before publication.

Issue

The extension SDK can verify runtime provider registrations against extension.yaml, but the Azure AI extensions must adopt that helper for the protection to cover the extensions affected by #8992. Without consistent tests and CI wiring, a manifest can still claim a provider that its extension does not register.

Azure AI extension coverage

The agents, connections, projects, routines, skills, and toolboxes extensions now consume azd SDK v1.27.2 and carry the canonical TestConfigureExtensionHostMatchesManifest test.

  • Each test invokes the extension's real configureExtensionHost callback.
  • Provider factories remain lazy, so verification does not connect to azd or external services.
  • Existing extension lint workflows invoke a shared provider-verification workflow.
  • Changes to the shared workflow trigger every caller so workflow regressions cannot bypass validation.

Central coverage guard

A central manifest scan requires every provider-bearing first-party extension to carry the canonical test. It parses the test source and requires require.NoError to assert the result of azdext.VerifyProvidersMatchManifest, rejecting comments, discarded errors, or calls placed in another test.

The reusable workflow also checks that the canonical test file and exact test name exist before executing it because go test -run otherwise succeeds when no tests match.

Behavior at a glance

Scenario Before After
Manifest declares a provider that code does not register Could pass extension CI Canonical verification test fails
Code registers a provider missing from the manifest Provider remains undiscoverable without detection Canonical verification test fails
Provider name casing differs between manifest and code Runtime lookup can fail after publication Canonical verification test fails
Provider-bearing extension omits the canonical test No central enforcement Coverage workflow fails
Shared verifier workflow changes Callers might not run Every Azure AI caller is triggered

Scope and follow-ups

This PR completes the focused first-party CI guard for #8992.

  • Registry provider shape, capability coherence, duplicate ownership, and complete registry-to-manifest equality checks remain deferred.
  • Build, pack, publish, runtime-registration, install-time collision, and non-Go SDK enforcement remain deferred to broader extension-contract work.
  • This PR does not change extension discovery, provider resolution, installation, or runtime registration behavior.

Testing

Testing covers the canonical provider-manifest check for the demo, agents, connections, projects, routines, skills, and toolboxes extensions. Coverage-guard tests verify that only an asserted helper call in the canonical test satisfies enforcement, including protection against discarded errors, comments, calls in another test, and assertions followed by later calls.

JeffreyCA and others added 3 commits July 13, 2026 22:43
Add registry and manifest checks for provider metadata, capabilities, and unique ownership.

Verify runtime registrations match extension.yaml and run the checks in extension CI.
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 967bafa7-69d3-4c13-b596-83ca90c56e7f
@github-actions github-actions Bot added area/extensions Extensions (general) ext-agents azure.ai.agents extension ext-connections azure.ai.connections extension ext-demo microsoft.azd.demo extension ext-projects azure.ai.projects extension ext-routines azure.ai.routines extension ext-skills azure.ai.skills extension ext-toolboxes azure.ai.toolboxes extension labels Jul 13, 2026
@JeffreyCA JeffreyCA changed the title test(extensions): verify provider manifests test(extensions): verify provider manifests (part 2/2) Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/extensions Extensions (general) ext-agents azure.ai.agents extension ext-connections azure.ai.connections extension ext-demo microsoft.azd.demo extension ext-projects azure.ai.projects extension ext-routines azure.ai.routines extension ext-skills azure.ai.skills extension ext-toolboxes azure.ai.toolboxes extension

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add tests or linting to ensure extensions register all the providers they claim in extensions.yaml

1 participant