[provider-vllm] Continuation calls should retry retryable HTTP errors (524/502/503/529) with backoff

[danshapiro]

Problem

Module: amplifier-module-provider-vllm

When a response hits max_output_tokens and the provider enters its auto-continuation loop, any exception during a continuation call — including explicitly retryable infrastructure errors — causes the loop to immediately give up and return a partial (truncated) response.

The give-up handler in amplifier_module_provider_vllm/__init__.py (lines ~1486–1491, continuation loop ~1303–1491) is a bare except Exception that logs and breaks:

except Exception as e:
    logger.error(
        f"[PROVIDER] Continuation call {continuation_count} failed: {e}. "
        f"Returning partial response from {continuation_count} continuation(s)"
    )
    break  # Return what we have so far

There is no inspection of the error's retryability, no honoring of retry_after, and no backoff.

Observed behavior

This was observed in practice: a continuation call failed with an HTTP 524 whose error payload explicitly included "retryable": true and a retry_after value, yet the provider gave up after a single attempt and persisted a mid-sentence-truncated assistant message (vllm:status: "incomplete", vllm:incomplete_reason: "max_output_tokens").

Continuations are an especially safe/cheap retry case: the prompt is unchanged and largely KV-cached, so a retry costs little.

Proposed fix

In the continuation loop's exception handler:

1. Classify the error using the same retryable/retry_after error classification that already guards the primary request path (this exists in the sibling provider-openai code at __init__.py:~1700–1714, 1807, 1873–1898; the continuation loop bypasses it).
2. For retryable errors (524, 502, 503, 529, connection errors): retry with backoff, honoring retry_after when provided, up to N attempts (bounded — could reuse or parallel MAX_CONTINUATION_ATTEMPTS).
3. Only after retries are exhausted, fall back to the current behavior (log + return partial response).

Related

The identical pattern is duplicated in microsoft/amplifier-module-provider-openai (amplifier_module_provider_openai/__init__.py:~2140–2157), from which this module is derived — and likely in other Responses-API-lineage providers (provider-azure-openai, provider-chat-completions). This issue targets provider-vllm since that's where the failure was observed, but maintainers may want to apply the same fix across the family.

---

Note: The provider-vllm repository has issues disabled, so this issue is filed in the central Amplifier repo for coordination.