feat(test): Adding cross runtime tests suite for PocketPaw, LangChain, CrewAI#232
feat(test): Adding cross runtime tests suite for PocketPaw, LangChain, CrewAI#232Akshat-Raj wants to merge 8 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new tests/cross_runtime/ test suite intended to validate .soul export/import round-trips when integrating with other agent runtimes (PocketPaw, LangChain, CrewAI), aligned with Issue #228.
Changes:
- Introduces three new async pytest tests for PocketPaw/LangChain/CrewAI integration scenarios.
- Adds documentation for running the cross-runtime suite and a pytest marker registration for
cross_runtime.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
tests/cross_runtime/test_pocketpaw.py |
Adds a round-trip export/awaken test and asserts core/semantic/episodic persistence. |
tests/cross_runtime/test_langchain.py |
Adds a LangChain-skipped integration smoke test that wraps the system prompt in a SystemMessage. |
tests/cross_runtime/test_crewai.py |
Adds a CrewAI-skipped integration smoke test that injects the system prompt into an Agent backstory. |
tests/cross_runtime/readme.md |
Documents the new suite, optional dependencies, and Python version notes. |
tests/cross_runtime/conftest.py |
Registers the cross_runtime pytest marker. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| destination_file = str(tmp_path / "langchain_exported.soul") | ||
|
|
||
| await soul.export(destination_file) | ||
| recovered_soul = await Soul.awaken(destination_file) | ||
| assert "LangChain Agent" in recovered_soul.to_system_prompt() No newline at end of file |
prakashUXtech
left a comment
There was a problem hiding this comment.
Thanks for taking on the cross-runtime suite. The structure is right: marker registered, importorskip in place, three runtimes covered. Two things need fixing before this can merge, plus a handful of polish items.
Blockers
1. Retarget to dev, not main.
All v0.5.0 work goes through dev per release tracker #218 ("Sub-PRs target dev, not main directly"). Both of your PRs target main right now. For this PR: rebase the branch onto origin/dev and update the PR base. (I'll make sure this is louder in the contributing docs so future mentees don't trip on the same thing.)
2. The PocketPaw test doesn't actually test PocketPaw.
Issue #228 says "PocketPaw integration runs at the subprocess level." test_pocketpaw.py currently calls Soul.awaken and soul.export through the soul-protocol library directly. That tests soul-protocol's own round-trip, not whether PocketPaw can consume the file. The test would pass even if PocketPaw was completely broken.
To make it a true cross-runtime test, the PocketPaw side should either:
- Subprocess to the
pawCLI and verify it can load the soul, or - Import the pocketpaw Python runtime (separate package) and verify the persona / memory loads through PocketPaw's APIs
The LangChain and CrewAI tests have the right shape: they go through the target runtime's API. PocketPaw should follow the same pattern.
Smaller items
soul.remember()intest_pocketpaw.py:18is being deprecated by #236. Switch tosoul.note(...).- Inconsistent skip pattern.
test_langchain.pyusespytest.importorskip,test_crewai.pyusestry/except ImportError + pytest.skip(allow_module_level=True). Both work, butimportorskipis the canonical pytest pattern. Make them consistent. - Private API access in
test_pocketpaw.py:34:recovered_soul._memory._semantic.facts(). Tests that reach into_underscoremembers break on internal refactors. The cleaner shape isrecovered_soul.recall(query=..., type=...)or whatever public API exists. readme.md→README.mdper convention (matches what the issue spec uses).- Missing newline at EOF in
readme.md. - Memory round-trip assertion missing for LangChain and CrewAI. Issue acceptance: "LangChain and CrewAI round-trip passes for at least semantic memory." Your current tests only verify persona persistence in the system prompt string. Add at least one semantic-memory assertion per runtime.
What's working
- Marker registration in conftest.py is exactly right
pytest.importorskipfor the LangChain side is idiomatic- README explains the optional-dependency story clearly so future contributors know why some tests skip locally
- The round-trip kernel test (export, re-awaken, assert) is the right shape
Once the base retarget + the PocketPaw subprocess test land, plus the small polish, this is solid. Good first pass on the cross-runtime story.
|
@prakashUXtech Thanks for the review, I'll fix the issues asap, also the with pocket paw I got confused because of our earlier conversation about the memory import and export apis not being available, I'll work to improve the test! |
What does this PR do?
This PR adds a test suite for cross runtime integration for PocketPaw, LangChain, CrewAI.
Fixes #228
How to test
Checklist
uv run pytest tests/)uv run ruff check . && uv run ruff format --check .)feat:,fix:,docs:, etc.)