fix: enhance error handling for document processing and add OCR requirement check for image files (backport of #1859)

[ricofurtado]

Backport of #1859 from release-cpd to main.

Merges two independent post-ingest verification improvements that had each evolved separately on release-cpd and main since they diverged:

• main's on_error="assume_exists" handling (don't fail a file on transient OpenSearch errors after Langflow already reported success) is combined with fix: enhance error handling for document processing and add OCR requirement check for image files #1859's wait_for_visibility retry (ride out OpenSearch's near-real-time refresh window).
• Keeps the real correctness fix from fix: enhance error handling for document processing and add OCR requirement check for image files #1859: the Langflow no-connector-id ingestion path now verifies success by filename via check_filename_exists, not by hash_id(item) — Langflow assigns its own document_id there, so the hash was never actually the stored id and the check was effectively a no-op.
• Also includes the OCR-required-image-type detection in task_service.py and a flaky-test fix in the TS SDK integration suite.

Test plan

• python3 -m py_compile on touched files
• Full unit suite for touched areas passes

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Improved document deletion and verification reliability so newly ingested files are recognized even with indexing delays.
  • Fixed cases where a document could appear missing right after ingestion, reducing false failures during upload and delete checks.
  • Updated test coverage to use unique sample files, helping prevent leftover data from affecting results.

[coderabbitai]

Walkthrough

Adds a wait_for_visibility retry option to TaskProcessor.check_document_exists and check_filename_exists to handle OpenSearch near-real-time refresh delays, wires this into Langflow/connector ingestion verification, and updates the SDK delete-document integration test to use a unique file to avoid interference from earlier ingests.

Changes

Visibility Retry and Ingestion Verification

Layer / File(s)	Summary
check_document_exists retry logic src/models/processors.py	Adds kw-only wait_for_visibility parameter and retry-with-backoff when no hits are found, with docstring updates.
check_filename_exists retry logic src/models/processors.py	Adds kw-only wait_for_visibility parameter; resets alias candidates and retries after delay when no hits found across all aliases.
Ingestion verification wiring src/models/processors.py	ConnectorFileProcessor.process_item passes wait_for_visibility=True; LangflowFileProcessor.process_item switches from hash-based check_document_exists to check_filename_exists(original_filename, wait_for_visibility=True).

SDK Integration Test

Layer / File(s)	Summary
Unique file per delete test sdks/typescript/tests/integration.test.ts	should delete a document now creates, ingests, and deletes a uniquely named temporary file instead of reusing the shared test file, keeping existing assertion logic.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LangflowFileProcessor
  participant TaskProcessor
  participant OpenSearch

  LangflowFileProcessor->>TaskProcessor: check_filename_exists(original_filename, wait_for_visibility=True)
  TaskProcessor->>OpenSearch: query for filename
  OpenSearch-->>TaskProcessor: no hits
  TaskProcessor->>TaskProcessor: sleep with backoff, retry
  TaskProcessor->>OpenSearch: query for filename (retry)
  OpenSearch-->>TaskProcessor: hits found
  TaskProcessor-->>LangflowFileProcessor: exists = True

Loading

Possibly related PRs

• langflow-ai/openrag#1695: Both PRs modify check_document_exists/OpenSearch existence check behavior in the same method.
• langflow-ai/openrag#1851: Both PRs update post-ingestion verification logic in src/models/processors.py for Langflow/connector flows.
• langflow-ai/openrag#1857: Both PRs address OpenSearch near-real-time visibility issues via retry-based existence checks in the same ingestion verification paths.

Suggested reviewers: edwinjosechittilappilly, lucaseduoli, mfortman11

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the main document-processing fixes and correctly frames the change as a backport.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cpd-backport-1859-ocr-error-handling

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/models/processors.py (1)

159-237: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add on_error handling to check_filename_exists
check_filename_exists still treats final OpenSearch errors as “missing”, so the post-ingest verification path can mark an already ingested file as FAILED on a transient outage. Mirror check_document_exists here, and pass on_error="assume_exists" at the verification call site; dedupe callers can keep the default assume_missing.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/models/processors.py` around lines 159 - 237, `check_filename_exists`
currently treats final OpenSearch failures as missing, which can incorrectly
fail post-ingest verification; update this helper to support an `on_error`
option like `check_document_exists`, and in the final exception path honor
`on_error="assume_exists"` instead of always returning false. Then update the
verification call site that uses `check_filename_exists` to pass
`on_error="assume_exists"`, while leaving dedupe/default callers on the existing
`assume_missing` behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/models/processors.py`:
- Around line 159-237: `check_filename_exists` currently treats final OpenSearch
failures as missing, which can incorrectly fail post-ingest verification; update
this helper to support an `on_error` option like `check_document_exists`, and in
the final exception path honor `on_error="assume_exists"` instead of always
returning false. Then update the verification call site that uses
`check_filename_exists` to pass `on_error="assume_exists"`, while leaving
dedupe/default callers on the existing `assume_missing` behavior.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7995aa04-520a-4e34-9a3a-64819cd6c5fd

📥 Commits

Reviewing files that changed from the base of the PR and between bdd873f and 027a6d0.

📒 Files selected for processing (2)

• sdks/typescript/tests/integration.test.ts
• src/models/processors.py