Skip to content

feat: validate EvaluationLog schema in CI using devcontainer #577

@sonupreetam

Description

@sonupreetam

Summary

Add a CI workflow that runs complyctl commands inside the devcontainer environment and validates the resulting EvaluationLog against the Gemara CUE schema. This gives PR authors and reviewers confidence that scan output conforms to the schema before merge. This will allow users to check the fields of the EvaluationLog and ensure their values are consistent with the expected testdata (e.g., EvaluationLog using mock-oci-registry test-opa-bp policy-id should include assessment-logs with the testdata gemara requirement ids check-run-as-nonroot and check-resource-limits).

Problem

The existing cross-repo integration test validates structural correctness but does not validate EvaluationLog conformance against the Gemara CUE schema. This gap allowed complytime/complytime-providers#63 to reach main — the OPA provider returned steps: [], violating the schema's minimum-one-step constraint.

Hat tip to @hbraswelrh for giving the notes for the issue.

Acceptance Criteria

  • CI workflow uses the devcontainer image from .devcontainer/
  • Runs complyctl get, generate, and scan for available test providers
  • Validates each EvaluationLog output with cue vet against the Gemara schema
  • Reports which schema constraint failed in CI logs
  • Supports workflow_dispatch for manual triggering

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestgithub_actionsPull requests that update GitHub Actions code

Type

No type
No fields configured for issues without a type.

Projects

Status
Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions