Skip to content

Rework evidence agent into structured assessor#1371

Draft
SachaProbo wants to merge 1 commit into
mainfrom
SachaProbo/evidence_agent_rework
Draft

Rework evidence agent into structured assessor#1371
SachaProbo wants to merge 1 commit into
mainfrom
SachaProbo/evidence_agent_rework

Conversation

@SachaProbo

@SachaProbo SachaProbo commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Replace the free-text evidence describer with an assessor that emits a structured assessment (status plus rationale) so downstream code can act on the verdict instead of parsing prose.

Rename the evidencedescriber package and worker to evidenceassessor and introduce a dedicated EvidenceAssessment coredata model backed by an assessment status enum. Split the schema changes into focused migrations and scope the FAILED transition to the status columns so unrelated evidence fields are left untouched.

Realign the surrounding configuration and wiring: rename the config to EvidenceAssessmentConfig, update probod aliases and builders, add a Thinking budget to the LLM agent config, and extract an OutputType.DecorateEnum helper. Bind jsonRawMessageOrNull through driver.Valuer so raw JSON assessments persist cleanly.


Summary by cubic

Replaced the free-text evidence describer with a structured assessor that outputs a JSON assessment and explicit status. Added an EvidenceAssessment model and updated the worker, services, and configs so downstream code can act on verdicts without parsing prose.

Coredata +336 -173

  • Added assessment JSONB and EvidenceAssessmentStatus; renamed description_* to assessment_* via 3 migrations.
  • New helpers: Evidence.SetAssessment, GetAssessment, and SetAssessmentFailed; added driver.Valuer for raw JSON nulls.
  • Renamed loaders/reset to assessment; switched to EvidenceFileID.

Service +303 -198

  • Introduced pkg/evidenceassessor and evidence-assessment worker; removed pkg/evidencedescriber.
  • Persist JSON assessment and derive Description from Summary; failure only touches status columns.
  • Added OutputType.DecorateEnum and enforced enum for confidence; added Thinking budget to LLMAgentConfig.
  • Updated wiring and configs to EvidenceAssessmentConfig; adjusted services to EvidenceFileID and new statuses.

Tests +145 -8

  • Added assessment round‑trip and status enum tests; schema decoration test for confidence.
  • Updated builder tests to the new assessor.

GraphQL API +2 -2

  • Switched console evidence type to use EvidenceFileID.

Other +58 -58

  • Bumped GitHub Actions pins (runner, Docker, Helm, cosign, upload/download‑artifact, setup‑node) with no functional changes.

Written for commit 75a131e. Summary will update on new commits.

Review in cubic

@SachaProbo SachaProbo force-pushed the SachaProbo/evidence_agent_rework branch from 823405a to 4fef960 Compare June 16, 2026 15:16
Replace the free-text evidence describer with an assessor that emits a
structured assessment (status plus rationale) so downstream code can act
on the verdict instead of parsing prose.

Rename the evidencedescriber package and worker to evidenceassessor and
introduce a dedicated EvidenceAssessment coredata model backed by an
assessment status enum. Split the schema changes into focused migrations
and scope the FAILED transition to the status columns so unrelated
evidence fields are left untouched.

Realign the surrounding configuration and wiring: rename the config to
EvidenceAssessmentConfig, update probod aliases and builders, add a
Thinking budget to the LLM agent config, and extract an
OutputType.DecorateEnum helper. Bind jsonRawMessageOrNull through
driver.Valuer so raw JSON assessments persist cleanly.

Signed-off-by: Sacha Al Himdani <[email protected]>
@SachaProbo SachaProbo force-pushed the SachaProbo/evidence_agent_rework branch from 4fef960 to 75a131e Compare June 16, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants