Rollout Models support recording experience data by pan-x-c · Pull Request #591 · agentscope-ai/Trinity-RFT

pan-x-c · 2026-06-24T12:12:44Z

Description

This PR introduces a new Experience generation and collection flow. The main goal is to let the Rollout Model record Experience data during inference, while updating the Workflow / Scheduler / WorkflowRunner interfaces to support this model-side recording mode.

The PR focuses on two major changes:

Rollout Models can record Experience data by themselves.
Workflow interfaces are updated to support the new Experience generation mode.

Key Changes

Rollout Model-side Experience Recording

Rollout Models can now convert finished generation requests into Experience objects and store them inside the model process.

Experiences are recorded by key in the format <batch_id>/<task_id>/<run_id>. They are later drained by the Scheduler / RolloutCoordinator instead of being returned directly from WorkflowRunner.

This PR adds the RecordStore interface and an in-memory MemoryStore implementation to support model-side Experience storage, update, overwrite, deletion, and prefix-based draining. Workflows can also patch reward / info back into recorded Experiences after task execution.

The recording path is integrated for vLLM, SGLang, and Tinker rollout models.

New Workflow Interface for Recording-based Workflows

This PR adds WorkflowBase and WorkflowWithRecording to support workflows that rely on rollout-model recording.

WorkflowWithRecording exposes:

base_url
api_key
model_name
update_reward(...)

A workflow can call the rollout model through an OpenAI-compatible API. The model records the generated Experience automatically, while the workflow only needs to run the task, compute reward, and write the reward back with update_reward.

The workflow execution result is normalized into Status, which reports completion state, metrics, and successful record keys.

The legacy Workflow interface is still kept for simple workflows that manually construct and return Experience objects.

Scheduler and WorkflowRunner Data Flow Update

WorkflowRunner no longer returns serialized Experience payloads directly. It now returns only execution status.

The Scheduler consumes successful record keys from Status and actively drains recorded Experiences from the corresponding Rollout Model actor.

This separates responsibilities more clearly:

WorkflowRunner: executes workflows and returns Status
Rollout Model: records and maintains Experiences
Scheduler: drains Experiences from rollout actors
RolloutCoordinator: handles batch finalization, cleanup, and pipeline submission

Checklist

Please check the following items before code is ready to be reviewed.

Code has passed all tests
Docstrings have been added/updated in Google Style
Documentation has been updated
Code is ready for review

…th legacy) Refactor experience production so heavy data (tokens/logprobs/routed_experts) no longer rides runner->scheduler->coordinator as serialized bytes. The vLLM recorder now captures it in-process into a MemoryStore keyed by task_id, and the coordinator pulls it at finalize time via /records/consume_task. Runners ship only a small reward map. Both paths coexist behind `explorer.use_recorded_experience` (default off = legacy). Recording module (trinity/common/models/vllm_patch/recording/): - store: drop SqlStore; MemoryStore.update_reward_by_task_id stamps reward/run/task on a whole task-id group, pops and returns it (the in-memory replacement for the SQL HistoryRecorder join). - recorder: track in-flight record tasks; add flush() (await pending + queue.join) so a consume sees a quiesced store; honor skip_recording_ctx. - models: build_experience emits one Experience per completion (n>1) with info["sample_index"]; eid.suffix=request_id kept for traceability. - context: add skip_recording_ctx; task_id already flows via api_key (RecordingIdentityMiddleware) and now also via VLLMModel.chat (Ray entry). - query: POST /records/consume_task (flush -> update_reward_by_task_id -> serialize_many); drop the SqlStore 503 branch. - config/server: remove RecordingConfig entirely; the logprob width is a recorder-internal constant (we store only the chosen token, which vLLM force-includes at logprobs=1). No static config threaded through launch. task_id propagation (Ray entry, same contextvar as the HTTP middleware): - vllm_model: chat/generate accept task_id_key, set task_id_ctx around _generate_internal; logprobs sets skip_recording_ctx (auxiliary forward). - model: ModelWrapper.chat/chat_async forward task_id_key; SGLang.chat accepts-and-ignores it (recording is vLLM-only). Coordinator + runner + workflow: - rollout_coordinator: _resolve_rank_urls (ray.get_actor per engine) and a recording-mode finalize that fans out /records/consume_task per engine, deserializes, and feeds objects to the pipeline (no re-serialization). - experience_pipeline: process_experiences(exps) public object entry. - workflow_runner: recording mode returns a pickled reward map keyed by the per-sample task_id_key the workflow stamped; legacy path unchanged. - workflow: SimpleWorkflow/AsyncSimpleWorkflow run a per-sample n=1 loop in recording mode (distinct task_id_key per sample == reward unit for GRPO), legacy n=repeat_times single-call path unchanged. - config: ExplorerConfig.use_recorded_experience flag. SQL path removal (MemoryStore only): - delete proxy/recorder.py (HistoryRecorder) and proxy_test.py; proxy service/app drop /feedback, /commit, record_feedback, submit_experiences, ready_experiences (keep allocate_model + weight sync); allocator no longer fills record_db_url; drop InferenceModelConfig.record_db_url and the dead ExplorerConfig.db_url field; RecordingConfig deleted. Serve-mode external reward reporting is intentionally left unimplemented this version (proxy /feedback//commit removed); the affected serve integration tests (TestServeWithTrainer, ServeTest) are skipped with a pointer to the recording refactor plan. convert_messages_to_experience redirect (multi-turn) is deferred with TODOs at its call sites. Co-Authored-By: Claude Fable 5 <[email protected]>

…-x-c/Trinity-RFT into feature/model_self_record_experience

chenyushuo · 2026-07-01T07:35:33Z

+    response_text: str = ""  # Text of the response
    prompt_text: Optional[str] = None  # Text of the prompt


Can response_text and prompt_text be the same default value

pan-x-c · 2026-07-01T13:42:28Z

/unittest-module-trainer

github-actions · 2026-07-01T15:00:39Z

unittest: Run #1803

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
29	25	0	4	0	0	0	1h 17m

🎉 All tests passed!

Github Test Reporter by CTRF 💚

pan-x-c · 2026-07-02T02:44:09Z

/unittest-all

github-actions · 2026-07-02T05:24:50Z

unittest: Run #1805

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
420	405	8	7	0	0	0	2h 39m

❌ Some tests failed!

Name	Failure Message
❌ tests/explorer/workflow_test.py::TestConcurrentWorkflowRunner::test_concurrent_workflow_runner	The test failed in the call phase due to an assertion error
❌ tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_local_0__workspace_tests_utils_plugins	The test failed in the call phase
❌ tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_local_1_tests_utils_plugins	The test failed in the call phase
❌ tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_remote_0__workspace_tests_utils_plugins	The test failed in the call phase due to an exception
❌ tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_remote_1_tests_utils_plugins	The test failed in the call phase due to an exception

See all failed tests here

Github Test Reporter by CTRF 💚

pan-x-c · 2026-07-02T05:56:39Z

/unittest-module-utils

pan-x-c · 2026-07-02T05:57:18Z

/unittest-pattern-TestConcurrentWorkflowRunner

github-actions · 2026-07-02T05:58:01Z

unittest: Run #1806

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
46	44	1	1	0	0	0	36.1s

❌ Some tests failed!

Name	Failure Message
❌ tests/utils/registry_test.py::TestRegistryWithRay::test_dynamic_import	The test failed in the call phase

Github Test Reporter by CTRF 💚

github-actions · 2026-07-02T06:00:50Z

unittest: Run #1807

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
1	1	0	0	0	0	0	1m 59s

🎉 All tests passed!

Github Test Reporter by CTRF 💚

pan-x-c · 2026-07-02T06:01:32Z

/unittest-pattern-test_dynamic_import

github-actions · 2026-07-02T06:02:36Z

unittest: Run #1808

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
1	1	0	0	0	0	0	21.0s

🎉 All tests passed!

Github Test Reporter by CTRF 💚

openai api server record experience data

776ae71

pan-x-c marked this pull request as draft June 24, 2026 12:26

pan-x-c and others added 28 commits June 25, 2026 11:15

simplify

5fe9053

add config

180b416

use api key as session id

f27988b

clean stale header

40b53e8

simplify code

4ae9392

unify vllm experience recording

4f0f0a7

add tests

8ce1f4b

add log

02eb4e4

fix middleware

db49262

update interface

e6ae7dc

fix prompt text

63a5404

fix streaming recorder

5ff21b2

add delta stream

6a0ad44

refactor history recording

bb2f22c

sglang self recording experiences

bf931de

add sglang tests

a274ed3

fix sglang tests

35952ae

remove enable recording

6d6ce69

Merge branch 'feature/model_self_record_experience' of github.com:pan…

034fb96

…-x-c/Trinity-RFT into feature/model_self_record_experience

fix models

82d75ba

add recording server

5cc3f05

fix sglang

936da53

remove redundant fields

2228d65

fix vllm test

3122b55

fix tests

579d189

add store

052562a

refactor store

d8f3de6

chenyushuo reviewed Jul 1, 2026

View reviewed changes

Comment thread trinity/common/workflows/workflow.py Outdated

chenyushuo reviewed Jul 1, 2026

View reviewed changes

Comment thread trinity/common/config.py

chenyushuo reviewed Jul 1, 2026

View reviewed changes

Comment thread trinity/common/config.py Outdated

chenyushuo reviewed Jul 1, 2026

View reviewed changes

pan-x-c added 6 commits July 1, 2026 19:57

update workflow doc

1146972

clean build experience

0ccf77c

clean build experience

815bd46

fix model version

0571edb

fix model version drift

14d5bdc

fix workflow reset

740805a

pan-x-c force-pushed the feature/model_self_record_experience branch from c85db9e to 740805a Compare July 1, 2026 13:37

fix comments

40a1dd6

pan-x-c changed the title ~~vLLM OpenAI API server supports recording experience data~~ Rollout Models support recording experience data Jul 2, 2026

pan-x-c added 2 commits July 2, 2026 13:48

fix workflow

a66afc8

fix plugin tests

2167e53

fix registry

fa7b269

pan-x-c added 4 commits July 2, 2026 14:13

finish harbor workflow

8347dd8

move recorder to a separate thread

f010e64

add harbor tests

722bf72

fix pre-commit

64b001d

		response_text: str = "" # Text of the response
		prompt_text: Optional[str] = None # Text of the prompt

Uh oh!

Conversation

pan-x-c commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes

Rollout Model-side Experience Recording

New Workflow Interface for Recording-based Workflows

Scheduler and WorkflowRunner Data Flow Update

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chenyushuo Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

pan-x-c commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

unittest: Run #1803

🎉 All tests passed!

Uh oh!

pan-x-c commented Jul 2, 2026

Uh oh!

github-actions Bot commented Jul 2, 2026

unittest: Run #1805

❌ Some tests failed!

Uh oh!

pan-x-c commented Jul 2, 2026

Uh oh!

pan-x-c commented Jul 2, 2026

Uh oh!

github-actions Bot commented Jul 2, 2026

unittest: Run #1806

❌ Some tests failed!

Uh oh!

github-actions Bot commented Jul 2, 2026

unittest: Run #1807

🎉 All tests passed!

Uh oh!

pan-x-c commented Jul 2, 2026

Uh oh!

github-actions Bot commented Jul 2, 2026

unittest: Run #1808

🎉 All tests passed!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pan-x-c commented Jun 24, 2026 •

edited

Loading