Skip to content

task-observer: new docs/ai/experiment-diagnostics from 2026-07-01 #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
divad12 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

task-observer: new docs/ai/experiment-diagnostics from 2026-07-01 #73

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude/AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ Do not run full validation suites after every small edit by habit. Run the narro
| Editing AGENTS.md, .agents/skills/, docs/ai/ | writing-docs.md |
| Maintaining or troubleshooting ask-intern | ask-intern.md |
| Operating or changing the learning system | learning-system.md |
| Running probe loops, experiment diagnostics, or tuning sessions | experiment-diagnostics.md |
- Re-read the hard rules before implementation.

## Session Tools
Expand Down
14 changes: 14 additions & 0 deletions docs/ai/experiment-diagnostics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Experiment Diagnostics

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add the docs/ai load marker

When probe/tuning tasks hit the new routing row, this file is meant to be a canonical adaptive-docs reference, but it starts directly with the heading instead of the standard Context loaded: experiment-diagnostics.md probe. docs/ai/writing-docs.md says that top-of-file probe is how the harness verifies which docs/ai files actually loaded, so this new contract loses the observability that lets users and agents confirm the right guidance was read before experiment artifacts are promoted.

Useful? React with 👍 / 👎.


## Contract

- When a probe or experiment produces implausible user-facing guidance, pause the experiment queue before logging the artifact as evidence.
- Fix the diagnostic source first: write a behavior test capturing the bad output, fix the shared logic, and rerun the canonical probe.
- Only promote corrected artifacts to durable evidence or curated docs. An artifact that embeds known-bad guidance becomes a misleading fixture for future agents.
- Record both the discovered product risk and the corrected artifact path before resuming the probe queue.

## Notes

- Experiment artifacts are not only observations — they are validation fixtures for product guidance. When an artifact exposes bad guidance, fix the guidance and regenerate the artifact before promoting it to durable evidence.
- What counts as implausible: a recommendation that contradicts domain constraints, a metric far outside the expected range, or guidance a product owner would immediately reject.
- This applies to any long-running probe or tuning loop where the system being probed produces user-facing output: recommendation calibration, AI output evaluation, capacity diagnostics, or auto-populate tuning.