feat(osti): recover missing-PDF articles with polite backoff retry

[vedant1711]

Recovers fulltext PDFs for OSTI articles that previously failed to download, using polite retry within the normal spider run — no identity spoofing or WAF evasion.

Background

A prior OSTI crawl left ~6,800 articles without a stored PDF. Investigation showed most failures were HTTP 503 / dropped connections from OSTI's /servlets/purl/ endpoint rate-limiting bursts of downloads — not missing files. A sampled re-fetch after a pause recovered 75/75 of them as real PDFs served directly by OSTI. The remainder are genuine dead hosts (e.g. efw.bpa.gov no longer resolves) or deliberate WAF blocks (403), which are correctly skipped rather than retried.

Changes

1. BackoffRetryMiddleware (new, enabled only in OSTI custom_settings)

• Retries 503/429 responses and dropped connections with exponential backoff + jitter, instead of Scrapy's immediate retries that keep hitting the same rate limit.
• Gives up after a configurable number of attempts (default 5) so dead links are skipped and logged, never retried forever.
• 403 (WAF) and 404 pass straight through — no retries. DNS-dead hosts are left to the built-in RetryMiddleware so they fail fast.
• Non-blocking: the backoff wait happens after the download slot is freed, so other downloads continue.

2. Opt-in OPEN_IRE_SKIP_EXISTING_REQUIRES_FILES (enabled for OSTI only)

• With OPEN_IRE_SKIP_EXISTING, articles already in the DB that have a stored file are skipped, but file-less ones are re-processed so their PDF gets another attempt — without re-downloading the complete ones.

3. Fix: don't drop an article as its own DOI duplicate on re-crawl (DOIDuplicatesPipeline)

• On a re-crawl, an already-collected article's own DOI was in the dedup cache, so the pipeline dropped it as a duplicate of itself before it could reach the file-download step — which blocked missing-PDF recovery entirely.
• Now skips duplicate handling when the cached DOI owner is the same (repository, reference) as the incoming item. Cross-repository dedup is unchanged. This is a general correctness fix; first-time crawls and cross-repo duplicates behave exactly as before.

Scope / safety

• The middleware and the skip-existing modifier are wired only in OSTI's custom_settings; no other spider's behavior changes.
• The DOIDuplicatesPipeline fix touches a shared pipeline but only affects the previously-broken self-duplicate case; a second reviewer on that file is welcome.

Testing

• Deterministic end-to-end harness through the real media pipeline: 503-recovers-after-pause, perpetual-503 gives up and skips, 403 skipped immediately with zero retries, healthy PDF downloads first try.
• Live OSTI crawl (fresh DB): downloads PDFs, dead hosts quick-fail on DNS.
• Live OSTI crawl against a copy of the prior run's DB with --skip-existing: complete articles skipped, file-less articles pass through to re-attempt, dead-host links skipped.
• Full suite: 295 tests pass; ruff and mypy clean.

Usage

Recovery run (against the DB holding the prior articles):

ENVIRONMENT=production pixi run resume osti --skip-existing