Skip to content

feat(test): Adding cross runtime tests suite for PocketPaw, LangChain, CrewAI#232

Open
Akshat-Raj wants to merge 8 commits into
qbtrix:mainfrom
Akshat-Raj:main
Open

feat(test): Adding cross runtime tests suite for PocketPaw, LangChain, CrewAI#232
Akshat-Raj wants to merge 8 commits into
qbtrix:mainfrom
Akshat-Raj:main

Conversation

@Akshat-Raj
Copy link
Copy Markdown

@Akshat-Raj Akshat-Raj commented May 5, 2026

What does this PR do?

This PR adds a test suite for cross runtime integration for PocketPaw, LangChain, CrewAI.

Fixes #228

How to test

uv run --python 3.12 --with pytest-asyncio pytest tests/cross_runtime/ -v

Checklist

  • Tests pass locally (uv run pytest tests/)
  • Lint passes (uv run ruff check . && uv run ruff format --check .)
  • No secrets or credentials in the diff
  • PR title follows Conventional Commits (feat:, fix:, docs:, etc.)

@Akshat-Raj Akshat-Raj marked this pull request as ready for review May 6, 2026 07:51
Copilot AI review requested due to automatic review settings May 6, 2026 07:51
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new tests/cross_runtime/ test suite intended to validate .soul export/import round-trips when integrating with other agent runtimes (PocketPaw, LangChain, CrewAI), aligned with Issue #228.

Changes:

  • Introduces three new async pytest tests for PocketPaw/LangChain/CrewAI integration scenarios.
  • Adds documentation for running the cross-runtime suite and a pytest marker registration for cross_runtime.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
tests/cross_runtime/test_pocketpaw.py Adds a round-trip export/awaken test and asserts core/semantic/episodic persistence.
tests/cross_runtime/test_langchain.py Adds a LangChain-skipped integration smoke test that wraps the system prompt in a SystemMessage.
tests/cross_runtime/test_crewai.py Adds a CrewAI-skipped integration smoke test that injects the system prompt into an Agent backstory.
tests/cross_runtime/readme.md Documents the new suite, optional dependencies, and Python version notes.
tests/cross_runtime/conftest.py Registers the cross_runtime pytest marker.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/cross_runtime/test_pocketpaw.py
Comment thread tests/cross_runtime/test_pocketpaw.py
Comment thread tests/cross_runtime/test_langchain.py Outdated
Comment thread tests/cross_runtime/test_langchain.py Outdated
Comment on lines +21 to +25
destination_file = str(tmp_path / "langchain_exported.soul")

await soul.export(destination_file)
recovered_soul = await Soul.awaken(destination_file)
assert "LangChain Agent" in recovered_soul.to_system_prompt() No newline at end of file
Comment thread tests/cross_runtime/test_crewai.py
Comment thread tests/cross_runtime/readme.md Outdated
Comment thread tests/cross_runtime/test_crewai.py Outdated
Comment thread tests/cross_runtime/test_pocketpaw.py Outdated
Comment thread tests/cross_runtime/test_pocketpaw.py Outdated
Comment thread tests/cross_runtime/test_crewai.py Outdated
Copy link
Copy Markdown
Contributor

@prakashUXtech prakashUXtech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking on the cross-runtime suite. The structure is right: marker registered, importorskip in place, three runtimes covered. Two things need fixing before this can merge, plus a handful of polish items.

Blockers

1. Retarget to dev, not main.

All v0.5.0 work goes through dev per release tracker #218 ("Sub-PRs target dev, not main directly"). Both of your PRs target main right now. For this PR: rebase the branch onto origin/dev and update the PR base. (I'll make sure this is louder in the contributing docs so future mentees don't trip on the same thing.)

2. The PocketPaw test doesn't actually test PocketPaw.

Issue #228 says "PocketPaw integration runs at the subprocess level." test_pocketpaw.py currently calls Soul.awaken and soul.export through the soul-protocol library directly. That tests soul-protocol's own round-trip, not whether PocketPaw can consume the file. The test would pass even if PocketPaw was completely broken.

To make it a true cross-runtime test, the PocketPaw side should either:

  • Subprocess to the paw CLI and verify it can load the soul, or
  • Import the pocketpaw Python runtime (separate package) and verify the persona / memory loads through PocketPaw's APIs

The LangChain and CrewAI tests have the right shape: they go through the target runtime's API. PocketPaw should follow the same pattern.

Smaller items

  • soul.remember() in test_pocketpaw.py:18 is being deprecated by #236. Switch to soul.note(...).
  • Inconsistent skip pattern. test_langchain.py uses pytest.importorskip, test_crewai.py uses try/except ImportError + pytest.skip(allow_module_level=True). Both work, but importorskip is the canonical pytest pattern. Make them consistent.
  • Private API access in test_pocketpaw.py:34: recovered_soul._memory._semantic.facts(). Tests that reach into _underscore members break on internal refactors. The cleaner shape is recovered_soul.recall(query=..., type=...) or whatever public API exists.
  • readme.mdREADME.md per convention (matches what the issue spec uses).
  • Missing newline at EOF in readme.md.
  • Memory round-trip assertion missing for LangChain and CrewAI. Issue acceptance: "LangChain and CrewAI round-trip passes for at least semantic memory." Your current tests only verify persona persistence in the system prompt string. Add at least one semantic-memory assertion per runtime.

What's working

  • Marker registration in conftest.py is exactly right
  • pytest.importorskip for the LangChain side is idiomatic
  • README explains the optional-dependency story clearly so future contributors know why some tests skip locally
  • The round-trip kernel test (export, re-awaken, assert) is the right shape

Once the base retarget + the PocketPaw subprocess test land, plus the small polish, this is solid. Good first pass on the cross-runtime story.

@Akshat-Raj
Copy link
Copy Markdown
Author

@prakashUXtech Thanks for the review, I'll fix the issues asap, also the with pocket paw I got confused because of our earlier conversation about the memory import and export apis not being available, I'll work to improve the test!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(test): cross-runtime test suite — PocketPaw, LangChain, CrewAI

3 participants