Skip to content

Avoid listing models for explicit text model IDs#231

Open
chubes4 wants to merge 3 commits into
WordPress:trunkfrom
chubes4:explicit-model-no-list
Open

Avoid listing models for explicit text model IDs#231
chubes4 wants to merge 3 commits into
WordPress:trunkfrom
chubes4:explicit-model-no-list

Conversation

@chubes4

@chubes4 chubes4 commented May 2, 2026

Copy link
Copy Markdown

Summary

  • Adds a provider-overridable path for explicit model IDs to provide metadata without listing all provider models first.
  • Applies explicit metadata consistently across getModelMetadata(), hasModelMetadata(), and listed model metadata for matching listed IDs.
  • Keeps explicit metadata request-local and memoized per directory instance instead of storing synthetic metadata in the shared models cache.

Fixes #230.

Why

Explicit provider/model selection currently calls the provider model-list endpoint before the model can be instantiated. For providers that accept explicit/current model IDs, that means getProviderModel( 'openai', 'gpt-5.4' ) can fail on a transient GET /models issue before the actual generation endpoint has a chance to validate the explicitly requested model.

Testing

  • composer test -- --filter AbstractApiBasedModelMetadataDirectoryTest
  • composer lint

AI assistance

  • AI assistance: Yes
  • Tool(s): OpenCode (GPT-5.5)
  • Used for: Drafting the patch, tests, and PR description; Chris reviewed the behavior on a local Studio site and remains responsible for the change.

@github-actions

github-actions Bot commented May 2, 2026

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: chubes4 <[email protected]>
Co-authored-by: JasonTheAdams <[email protected]>
Co-authored-by: felixarntz <[email protected]>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@codecov

codecov Bot commented May 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.26%. Comparing base (6317042) to head (b5b0e68).
⚠️ Report is 7 commits behind head on trunk.

Additional details and impacted files
@@             Coverage Diff              @@
##              trunk     #231      +/-   ##
============================================
+ Coverage     88.12%   88.26%   +0.13%     
- Complexity     1213     1225      +12     
============================================
  Files            60       61       +1     
  Lines          3934     3962      +28     
============================================
+ Hits           3467     3497      +30     
+ Misses          467      465       -2     
Flag Coverage Δ
unit 88.26% <100.00%> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The OpenAI-compatible abstraction is about HTTP/JSON shape, not OpenAI's
model namespace. Drop the gpt-/o3/dall-e prefix gating and the synthetic
text-generation metadata override; the per-provider override belongs in
ai-provider-for-openai (and similar repos for other compatible providers).

The generic createModelMetadataForExplicitModelId() hook on
AbstractApiBasedModelMetadataDirectory is unchanged so providers can still
opt in.

@felixarntz felixarntz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chubes4 While this addresses the reported concern, it creates data inconsistencies when considering the API surface of this class holistically.

I think there's a path forward, but it requires some modifications in the approach.

Comment thread src/Providers/ApiBasedImplementation/AbstractApiBasedModelMetadataDirectory.php Outdated
Comment thread src/Providers/ApiBasedImplementation/AbstractApiBasedModelMetadataDirectory.php Outdated
@chubes4

chubes4 commented Jun 11, 2026

Copy link
Copy Markdown
Author

Updated this to address the consistency concern:

  • getModelMetadata() and hasModelMetadata() now check explicit metadata before the list/cache path, so provider-local metadata is authoritative when available.
  • listModelMetadata() still performs the required list request, then overlays explicit metadata for matching listed model IDs so listed data and direct lookup agree for the same ID.
  • Explicit metadata is memoized only for the directory instance, avoiding persistence/cache invalidation concerns for synthetic metadata.

Verified with:

  • composer test -- --filter AbstractApiBasedModelMetadataDirectoryTest
  • composer lint

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explicit model instantiation should not require live model-list metadata fetch

3 participants