Skip to content

feat: add Context.dev integration - #3871

Open
aadithyanr wants to merge 5 commits into
deepset-ai:mainfrom
aadithyanr:feat/context-dev-integration
Open

feat: add Context.dev integration#3871
aadithyanr wants to merge 5 commits into
deepset-ai:mainfrom
aadithyanr:feat/context-dev-integration

Conversation

@aadithyanr

@aadithyanr aadithyanr commented Aug 29, 2026

Copy link
Copy Markdown

Related Issues

Proposed Changes

Adds context-dev-haystack, an official Context.dev integration with three Haystack components:

  • ContextWebSearch for relevance-ranked live web search
  • ContextFetcher for fetching known URLs as Markdown Documents
  • ContextCrawler for bounded website crawling

All components support sync and async execution, environment-backed secrets, Haystack serialization, retry handling, custom API parameters, and the production Context.dev API.

How did you test it?

  • hatch run fmt-check
  • hatch run test:types — 5 source files passed
  • hatch run test:unit — 25 passed
  • hatch run test:integration — 6 live production API tests passed across sync and async search, fetch, and crawl
  • lowest-direct dependency resolution — 25 unit tests passed with haystack-ai==2.24.1 and httpx==0.27.0
  • hatch build — sdist and wheel built successfully
  • installed the wheel in a clean virtual environment and verified imports plus Haystack serialization round-trips

Notes for the reviewer

  • The integration was scaffolded with scripts/create_new_integration.py.
  • The distribution is named context-dev-haystack because the unrelated Context.ai project already owns context-haystack on PyPI.
  • Crawling defaults to one page to avoid accidental credit consumption; callers can raise maxPages through crawl_params.
  • Context.dev search accepts at least 10 results, so ContextWebSearch requests 10 when top_k is lower and returns only the requested number.
  • The shared client normalizes boolean, list, and nested query parameters consistently for both Haystack sync and async HTTP paths.
  • The unreleased Answers API is intentionally not included.
  • The implementation was AI-assisted, then manually reviewed and validated against the live production API.

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I opened and linked the required new integration proposal
  • I added unit tests and updated the docstrings
  • The PR title uses a conventional commit type

@aadithyanr
aadithyanr requested a review from a team as a code owner August 29, 2026 20:27
@aadithyanr
aadithyanr requested review from sjrl and removed request for a team August 29, 2026 20:27
@CLAassistant

CLAassistant commented Aug 29, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions

Copy link
Copy Markdown
Contributor

Hi @aadithyanr, thanks a lot for contributing a new integration! 🙏

This PR looks like it adds a new integration (context), but we couldn't
find a linked issue labeled new integration.

Every integration we merge is maintained, tested nightly, and released by us, so before reviewing the code
we'd like to understand the adoption of the technology you're integrating with. Could you please
open a New Integration Proposal
and fill it out? The Adoption signals section (GitHub stars, PyPI downloads over the last 30 days,
release activity, community demand) is what helps us decide whether to add the integration to Haystack.

Then reference the issue in this PR's description (for example, by adding Related to #<issue-number>)
and this comment will disappear automatically.

@github-actions github-actions Bot added the type:documentation Improvements or additions to documentation label Aug 29, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @aadithyanr, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from sjrl August 29, 2026 21:51
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Aug 29, 2026
@HaystackBot
HaystackBot marked this pull request as draft August 29, 2026 21:51
@aadithyanr
aadithyanr marked this pull request as ready for review August 30, 2026 11:26
@HaystackBot
HaystackBot requested a review from sjrl August 30, 2026 11:51
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @aadithyanr! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:CI type:documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants