[SDKs] Self-heal catalog cache on BYOM cache-miss

[bmehta001]

The SDKs cache the catalog in memory with a 6-hour TTL. When a user manually drops a model directory (BYOM) into the cache while the SDK is running, the SDK''s in-memory map is stale: Core returns the BYOM id from get_cached_models and get_model_list IPCs, but the SDK filters it out for getModel/getModelVariant because its modelIdToModelVariant map is still stale.

Self-heal on cache-miss

Force-refresh path in the four user-facing entry points:

• get_cached_models / get_loaded_models: when any id in the fresh list returned by Core does not resolve against the local map, force a catalog refresh and re-resolve only the unresolved ids.
• get_model(alias) / get_model_variant(id): on miss, force a refresh and retry the lookup once.

A UpdateModels(ct) warm-up call was added to the C# get_model / get_model_variant paths for consistency with the other SDKs (not strictly required for BYOM).

Catalog refresh is now incremental

Previously every catalog refresh was clear-and-rebuild, which orphaned every Model / ModelVariant wrapper a caller was holding. After this PR, refresh preserves wrapper identity for ids that survive:

• C# / JS / Python: Catalog.UpdateModels reuses existing ModelVariant wrappers and calls RefreshInfo(info) to surface fresh metadata in place. Model wrappers are reused per alias and RefreshVariants swaps the backing variant list by reference (no torn reads). Evicted ids are dropped from the maps.
• Rust: apply_model_list reuses the same Arc<Model> across refreshes when the (id, cached) fingerprint is unchanged, so the inner AtomicUsize selected-variant index survives.

Explicit SelectVariant() choices are preserved trivially because the wrapper identity is preserved.

Limitations

If the previously-selected variant is removed by a refresh, SelectedVariant keeps pointing at the dropped wrapper and callers must explicitly re-select - pre-existing issue that would take ~30 lines per SDK to fix, but unlikely to happen.

Tests

New CatalogSelfHealTests and CatalogIncrementalRefreshTests (+ JS / Python equivalents) cover: self-heal on cache miss, no refresh on empty/whitespace input, identity + info propagation across refreshes, selection survival, and add/remove transitions.

Validated against a real BYOM directory dropped into the cache at runtime.

=================

More detailed description:
_cachedModels (disk) and _cachedInfo (Azure metadata) are always populated at the start

FLCore Changes:

• Disk gets scanned in
• RefreshLocalCachedModelsAsync (thread-safe function bc lock) in
  AzureModelCatalog.cs —
  refreshes the in-memory _cachedModels dictionary from disk.
  • "We expect users will not manually delete an Azure-cataloged model,
    so we do not make a new trip to Azure Catalog or invalidate _cachedInfo here."
  • New BYOM ids get inserted with Model = null (we assume they do not match a model in Azure Catalog), so no Azure call.
• GetCurrentCachedModelsAndPaths (existing) to extract modelId.

In FLCore, RefreshLocalCachedModelsAsync is called by GetCachedModelsAsync, GetModelsAsync, and RemoveCachedModelAsync. It will be called by GetModelInfoAsync only if we have not created a BYOM model stub for the catalog, but it will be cached after that.

Core method	When RefreshLocalCachedModelsAsync is called
GetCachedModelsAsync	Always
GetModelsAsync	Always
RemoveCachedModelAsync	Always
GetModelInfoAsync	At most once each time we see a new BYOM model

GetModelInfoAsync ONLY calls RefreshLocalCachedModelsAsync if we looked up a BYOM model by calling (get_model_path, load_model, unload_model IPC handlers called by SDKs or OpenAI API and by GetChatClientAsync on every chat completion), and we have not created a BYOM model stub for the catalog. Once we have created a BYOM model stub, it is cached for the rest of the session, so we only validate Directory.Exists for the BYOM model (1 syscall is much quicker than scanning whole directory) after that to ensure it has not been deleted. (If it has been deleted, we return null for AzureFoundryLocalModel)

HTTP Method changes:

HTTP endpoint	Core method backing it
GET /v1/models	GetCachedModelsAsync() (always rescans disk after this PR) + then will only check if directory for BYOM model exists in GetModelInfoAsync
GET /openai/models	GetCachedModelsAsync() (always rescans disk after this PR)
POST /v1/chat/completions (and friends)	GetModelInfoAsync(id_or_alias) — validates Directory.Exists, falls through to refresh + synth if needed
GET /openai/load/{model}	same GetModelInfoAsync lookup
GET /openai/unload/{model}	same GetModelInfoAsync lookup

SDK changes:

SDK method	Scan disk trigger
catalog.GetModelAsync(alias)	Alias not in local map and if we exceed TTL of 6 hours*
catalog.GetModelVariantAsync(id)	Id not in local map and if we exceed TTL of 6 hours*
catalog.GetCachedModelsAsync()	Always **
catalog.GetLoadedModelsAsync()	If load-manager's list contains an id missing from local map (unlikely, but may be possible if multiple processes using SDK)
model.IsCachedAsync()	Always, bc this is fresher than model.Info.Cached

* we may end up scanning twice if we have a. once if we have hit the TTL and would have anyway and b. because of this PR, after scanning, if we still can't find the model (likely bc of a typo), we will scan again due to force refresh after cache miss

** - we will end up scanning twice - once to get the model name (string) for GetCachedModels, another time because we call UpdateModels(ct, force: true) to get the ModelInfo

Flow for GetModelAsync

Program.cs
  └─ catalog.GetModelAsync(byomAlias)
       │  sdk/cs/src/Catalog.cs:73 → GetModelImplAsync at line 157
       │   ├─ UpdateModels(ct, force: false)                # within 6h TTL → no-op
       │   ├─ _modelAliasToModel.TryGetValue(alias, …)→null # cache miss
       │   ├─ UpdateModels(ct, force: true)                 # SELF-HEAL
       │   │    └─ _coreInterop.ExecuteCommandAsync("get_model_list")
       │   │         │  IPC → foundry-local-service
       │   │         ▼
       │   │       Core AzureModelCatalog.GetModelsAsync(ct)
       │   │         ├─ base.GetModelsAsync(ct)                       # uses 4h Azure cache; no extra catalog calls
       │   │         ├─ RefreshLocalCachedModelsAsync(ct)             # ALWAYS re-scans disk
       │   │         │    └─ GetCurrentCachedModelsAndPaths()
       │   │         │         └─ Directory.GetDirectories(_cacheDirectory, *, AllDirectories)
       │   │         │              + for each dir: read inference_model.json → modelId
       │   │         ├─ drop stale Local stubs whose dir was deleted
       │   │         ├─ append cache-only stubs for new BYOMs
       │   │         └─ return combined list
       │   ▼
       └─ _modelAliasToModel.TryGetValue(alias, …) → Model

Flow for GetModelVariantAsync is similar; we just access a different map

Program.cs
  └─ catalog.GetModelVariantAsync(byomId)
       │  sdk/cs/src/Catalog.cs:81 → GetModelVariantImplAsync at line 187
       │   ├─ UpdateModels(ct, force: false)                # within 6h TTL → no-op
       │   ├─ _modelIdToModelVariant.TryGetValue(id, …) → null # cache miss
       │   ├─ UpdateModels(ct, force: true)                 # SELF-HEAL (same IPC + disk rescan as A1)
       │   ▼
       └─ _modelIdToModelVariant.TryGetValue(id, …) → Model 

Flow for GetCachedModelsAsync must scan the disk twice - once to return strings for model names from FLCore's GetCachedModelsAsync and once to fetch latest GetModelsAsync (which has Model information). GetLoadedModelsAsync is similar.

Program.cs
  └─ catalog.GetCachedModelsAsync(byomId)
       │  sdk/cs/src/Catalog.cs:73 → GetModelImplAsync at line 157
       │   ├─ UpdateModels(ct, force: false)                # within 6h TTL → no-op
       │   ├─ _modelAliasToModel.TryGetValue(alias, …)→null # cache miss
       │   ├─ UpdateModels(ct, force: true)                 # SELF-HEAL
       │   │    └─ _coreInterop.ExecuteCommandAsync("get_model_list")
       │   │         │  IPC → foundry-local-service
       │   │         ▼
       │   │       Core AzureModelCatalog.GetModelsAsync(ct)
       │   │         ├─ base.GetModelsAsync(ct)                       # uses 4h Azure cache; no extra catalog calls
       │   │         ├─ RefreshLocalCachedModelsAsync(ct)             # ALWAYS re-scans disk
       │   │         │    └─ GetCurrentCachedModelsAndPaths()
       │   │         │         └─ Directory.GetDirectories(_cacheDirectory, *, AllDirectories)
       │   │         │              + for each dir: read inference_model.json → modelId
       │   │         ├─ drop stale Local stubs whose dir was deleted
       │   │         ├─ append cache-only stubs for new BYOMs
       │   │         └─ return combined list
       │   ▼
       └─ _modelAliasToModel.TryGetValue(alias, …) → Model

Get Loaded models

Program.cs
  └─ model.LoadAsync(ct)                                    # sdk/cs/src/Detail/Model.cs:153
       └─ SelectedVariant.LoadAsync(ct)                     # sdk/cs/src/Detail/ModelVariant.cs:95
            └─ _modelLoadManager.LoadModelAsync(modelId, customPath=null, ct)  # ModelManager.cs:65
                ├─ _modelCatalog.GetModelInfoAsync(modelId)
                │    │  AzureModelCatalog.GetModelInfoAsync override
                │    ├─ base.GetModelInfoAsync(modelId)             # miss for new BYOM
                │    ├─ RefreshLocalCachedModelsAsync(ct)           # SELF-HEAL: rescan disk
                │    │    └─ GetCurrentCachedModelsAndPaths()       # finds the new BYOM dir
                │    └─ TrySynthesizeByomStubAsync(modelId)         # writes stub into _cachedInfo
                │         └─ returns stub                           # cached for next time
                ├─ modelInfo != null → proceed to load
                └─ _modelCatalog.GetModelPathAsync(modelId)         # re-uses cached stub
                        → InferenceModel constructed from dir
                        returns Status.Success  ✓ FIX_ACTIVE

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Jun 4, 2026 7:46am

[Copilot]

Pull request overview

Adds “self-healing” catalog refresh behavior across the Rust, Python, JS, and C# SDKs so that when the on-disk BYOM cache changes while the SDK is running (and Core returns new IDs), the SDK refreshes its in-memory catalog on cache-miss and retries resolution.

Changes:

• Add a forced refresh + retry path to get_model(alias) / get_model_variant(id) when the initial lookup misses.
• Add forced refresh-on-miss when resolving IDs returned by Core for get_cached_models / get_loaded_models, via new helper resolvers.
• Add force/force: true parameters to the various “update models” caching methods to bypass the 6-hour TTL when self-healing.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
sdk/rust/src/catalog.rs	Adds forced-refresh retry on alias/id miss and a helper to resolve cached/loaded model IDs.
sdk/python/src/catalog.py	Adds force refresh support and self-heal retry logic for lookups and cached/loaded model resolution.
sdk/js/src/catalog.ts	Adds force refresh support and self-heal retry logic plus ID resolution helper for cached/loaded APIs.
sdk/cs/src/Catalog.cs	Adds forced-refresh retry logic and shared ID-resolution helper; updates UpdateModels signature to support forced refresh.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

sdk/js/src/catalog.ts:71

• runRefreshExclusive() always overwrites this.updatePromise with a new promise. If multiple callers request updateModels(true) while a refresh is in-flight, each chained callback will call runRefreshExclusive() after the in-flight refresh completes, potentially starting multiple concurrent refreshes and letting earlier .finally() handlers clear updatePromise while a later refresh is still running. Make runRefreshExclusive() idempotent by returning the existing updatePromise when it’s already set.

    private runRefreshExclusive(): Promise<void> {
        this.updatePromise = this.fetchAndPopulateModels()
            .finally(() => { this.updatePromise = undefined; });
        return this.updatePromise;
    }