Skip to content

Conformance report is hardcoded to "pass" — wire to real tests or remove the badge #39

Description

@davidcrowe

Problem

packages/explicabl-core/src/reporting/saveReport.ts unconditionally writes all five conformance categories (pkce, jwt_verify, scope_enforcement, allowlist, expiry) as "pass" with a fresh timestamp — no test is consulted. The committed docs/conformance.json reflects this, and the README's "MCP/Auth Conformance" badge reads that file's version.

During a repo-visibility push this is the single highest-risk artifact: a skeptical evaluator finds saveReport.ts in one file-read, sees the hardcoded passes, and that discredits every other claim in the repo. There is currently zero mapping from any category to any test, and the root tests/smoke.test.ts is a 7-line "runner runs" assertion.

Fix (pick one, do before the star push)

  • Preferred: replace saveReport.ts with a script that derives each category's pass/fail from actual test results (one describe-block per category, real assertions), so the badge means something.
  • Or: remove the badge from the README and delete docs/conformance.json + saveReport.ts until real conformance tests exist. An absent badge is honest; a hardcoded one is not.

Acceptance

  • The conformance JSON is produced only from executed tests, OR the badge/file are removed.
  • No category can report pass without a corresponding assertion having run.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdocumentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions