Describe evidence with structured assessment output

[aureliensibiril]

Summary

• Evidence describer now emits a typed EvidenceAssessment (system, setting, scope, captured_at, frameworks, issues, confidence, readable, rejection_reason, summary) validated by a JSON schema instead of free-text. The worker persists the record as JSONB in a new evidences.assessment column and derives Evidence.Description from Assessment.Summary, preserving the existing GraphQL Evidence.description: String contract.
• Renamed evidences.description_status / description_processing_started_at (+ the backing enum type) to assessment_status / assessment_processing_started_at — the worker now tracks the status of the whole assessment, not just a description. The four-state machine (PENDING → PROCESSING → COMPLETED | FAILED) and stale-recovery behavior are unchanged.
• Added an optional Thinking *int extended-thinking budget to LLMAgentConfig, propagated via ResolveAgent and wired into the describer. Providers that don't honour extended thinking ignore it.

The agent follows the same structured-output + enum-decoration pattern introduced for the vendor assessor in pkg/vetting; no orchestrator or sub-agent topology was ported since evidence describing has a single input artifact.

Out of scope (follow-up)

• Exposing new assessment fields in GraphQL / CLI / MCP. Fields are persisted but not yet surfaced.
• Backfill of existing COMPLETED rows. Operator may run UPDATE evidences SET assessment_status='PENDING', description=NULL WHERE assessment IS NULL AND type='FILE' post-deploy if desired.

Test plan

• make migrate-up and confirm psql -c "\\d evidences" shows assessment jsonb, assessment_status, assessment_processing_started_at
• go test ./pkg/coredata/... ./pkg/evidencedescriber/... green (new unit tests cover SetAssessment/GetAssessment round-trip and the confidence-enum schema decoration)
• Upload a compliance screenshot and verify: assessment_status = COMPLETED, description matches assessment->>'summary', assessment->>'confidence' IN ('HIGH','MEDIUM','LOW'), assessment->>'readable' = 'true'
• Upload a blank/corrupt file and verify: assessment_status = COMPLETED, assessment->>'readable' = 'false', rejection_reason populated, description equals the rejection reason
• Set llm.evidence-describer.thinking = 4000 and confirm the describer still produces valid output on supported models

Notes for reviewers

• Rolling-deploy note: the column rename is atomic. Workers poll with FOR UPDATE SKIP LOCKED, so an older binary on a stale pod would simply fail its next poll; acceptable given the async nature of the worker. Do not co-deploy the old binary.
• The enum injection on confidence is guarded by a unit test that asserts the generated JSON schema still contains the enum — if agent.NewOutputType ever reshapes the schema, that test fails loudly rather than silently shipping an un-constrained schema.

---

Summary by cubic

Switch evidence processing to a structured EvidenceAssessment stored in JSONB and replace the describer with a new assessor worker. Adds claim-safe state transitions, renames config/env to evidence-assessor, pins a vision-capable default model, and supports an optional thinking budget.

Tests +601 -14

• Add assessment state-machine tests with claim timestamp guards and stale recovery.
• Cover JSONB round-trips and enum decoration.
• Update bootstrap defaults for the assessor.

Coredata +427 -170

• Add evidences.assessment (JSONB) and rename status columns/enum to “assessment”.
• Evidence.SetAssessment / GetAssessment store SQL NULL for empty/typed-nil JSON; jsonRawMessageOrNull now implements driver.Valuer.
• Introduce EvidenceAssessmentStatus; guard PROCESSING → terminal transitions by assessment_processing_started_at; add Evidence.MarkAssessmentFailed to update status-only.
• Rename EvidenceFileId → EvidenceFileID and update CRUD/queries.

Service +939 -1041

• Replace pkg/evidencedescriber with pkg/evidenceassessor emitting typed EvidenceAssessment; mirror summary to Evidence.description.
• New evidence_assessment_worker with stale-claim-safe transitions; failure path updates only status columns.
• Add agent.OutputType.DecorateEnum; refresh pkg/llm registry; skip temperature when extended thinking is set.
• Rename config to evidence-assessor; pin default model to gpt-5.4-mini; add optional Thinking on LLMAgentConfig.
• Update services to use AssessmentStatus and EvidenceFileID; minor linter and context-order fixes to satisfy wsl_v5 and avoid lostcancel.

GraphQL API +2 -2

• Keep Evidence.description as-is; map file ID via EvidenceFileID.

Agents +1 -1

• Update docs to reference pkg/evidenceassessor.

Other +6 -2

• Rename env vars to AGENT_EVIDENCE_ASSESSOR_* / EVIDENCE_ASSESSOR_*; add interval/stale/concurrency and thinking examples.

^{Written for commit 7783c31. Summary will update on new commits.}

---

Note

Medium Risk
Introduces database migrations (new JSONB column and status/type renames) and changes the background worker flow that updates evidence rows, so deploy/migration ordering and data/state transitions need validation.

Overview
Switches evidence processing from a free-text “describer” to a structured EvidenceAssessment produced by a new pkg/evidenceassessor agent, storing the full payload in evidences.assessment (JSONB) while continuing to populate legacy evidences.description from assessment.summary.

Renames evidence processing state tracking from description_status/description_processing_started_at (and the underlying enum type) to assessment_status/assessment_processing_started_at, updates all evidence CRUD/queries accordingly, and adds safer failure handling via SetAssessmentFailed to avoid overwriting concurrent edits.

Adds OutputType.DecorateEnum for JSON-schema enum injection (used by both vendor vetting and the new assessor) and extends LLM agent config with optional Thinking tokens; also tweaks JSON binding for connector/evidence JSON fields to treat empty values as SQL NULL.

^{Reviewed by Cursor Bugbot for commit 478198b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cubic-dev-ai]

2 issues found across 15 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="pkg/probo/evidence_assessment_worker.go">

<violation number="1" location="pkg/probo/evidence_assessment_worker.go:168">
P2: `assessment` is set before the commit transaction, so the failure path can persist assessment data together with `FAILED` status.</violation>
</file>

<file name="pkg/evidencedescriber/prompt.txt">

<violation number="1" location="pkg/evidencedescriber/prompt.txt:33">
P2: The readable example contradicts the non-speculation rule by assigning SOC2/ISO27001 to a generic security screenshot, which can cause framework hallucinations in outputs.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

1 issue found across 9 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="pkg/coredata/evidence_assessment.go">

<violation number="1" location="pkg/coredata/evidence_assessment.go:28">
P2: `SetAssessment` misses typed-nil values and can persist JSON `null` instead of clearing the assessment.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

1 issue found across 10 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="pkg/probo/evidence_assessment_worker.go">

<violation number="1" location="pkg/probo/evidence_assessment_worker.go:202">
P1: `failEvidence` now writes the full `evidences` row from a stale in-memory snapshot, which can clobber concurrent updates. Update only failure-related columns (`assessment_status`, `assessment_processing_started_at`, `updated_at`) in this path.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 478198b. Configure here.}

[cursor]

You have used all of your free Bugbot PR reviews.

To receive reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

[SachaProbo]

I have opened a new PR based on this one: #1371