test(openrouter-images): add waza eval suite for the images skill

evals/openrouter-images/README.md

@@ -0,0 +1,44 @@
+# openrouter-images Eval Suite
+
+Waza evaluation suite for the [`openrouter-images`](../../skills/openrouter-images) skill, following the [waza eval spec guide](https://microsoft.github.io/waza/guides/eval-yaml/).
+
+## Structure
+
+```
+evals/openrouter-images/
+├── eval.yaml          # Capability suite (skill body injected)
+├── eval-trigger.yaml  # Trigger-precision suite (inject_skill_body: false)
+├── tasks/
+│   ├── discovery/     # discover.ts: list models, per-model params, passthrough options
+│   ├── generation/    # generate.ts: basic, aspect ratio, model override, output path,
+│   │                  #   multiple images, transparency, seeds
+│   ├── editing/       # edit.ts: basic edit, output path, capability check, generate→edit multi-turn
+│   ├── edge-cases/    # unsupported params/formats, missing files, invalid models, cost reporting
+│   └── trigger/       # should/should-not invoke the skill
+└── fixtures/          # small sample images copied into each task workspace
+```
+
+## What it measures
+
+- **Discovery-first behavior** — the skill instructs agents to check `supported_parameters` via `discover.ts` before sending flags; several tasks verify this instead of blind flag guessing.
+- **Correct script selection** — generate vs edit vs discover (enforced by the `correct_script_usage` LLM judge in `eval.yaml`).
+- **Option translation** — natural-language intent ("wide", "3 variations", "transparent", "reproducible") mapped to the right CLI flags.
+- **Error handling** — missing source files, unsupported formats/parameters, nonexistent models.
+- **Presentation** — reporting saved paths, model used, and generation cost per the skill's guidance.
+- **Trigger precision** — `eval-trigger.yaml` runs with `inject_skill_body: false` so positive tasks check the skill fires on image requests and negative tasks check it stays quiet for vision analysis, video, and text-only requests. Basic generate/edit tasks are marked `golden` for use with `waza gate`.
+
+## Running
+
+Requires `OPENROUTER_API_KEY` in the environment (generation/editing tasks make real API calls; discovery is keyless).
+
+```bash
+waza run evals/openrouter-images/eval.yaml --context-dir evals/openrouter-images/fixtures -o results.json
+waza run evals/openrouter-images/eval-trigger.yaml -o trigger-results.json
+
+# Filter subsets
+waza run evals/openrouter-images/eval.yaml --tags happy-path
+waza run evals/openrouter-images/eval.yaml --task "edge*"
+
+# Enforce golden tasks in CI
+waza gate results.json
+```

evals/openrouter-images/eval-trigger.yaml

@@ -0,0 +1,24 @@
+name: openrouter-images-trigger
+description: >
+  Trigger-precision suite for the openrouter-images skill. Measures whether the
+  agent invokes the skill when it should (image generation/editing requests) and
+  leaves it alone when it should not (text, video, audio, or vision-analysis requests).
+skill: openrouter-images
+schemaVersion: "1.0"
+version: "1.0"
+
+config:
+  trials_per_task: 1
+  timeout_seconds: 120
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+  inject_skill_body: false
+
+metrics:
+  - name: trigger_precision
+    weight: 1.0
+    threshold: 0.8
+
+tasks:
+  - "tasks/trigger/*.yaml"

evals/openrouter-images/eval.yaml

@@ -0,0 +1,47 @@
+name: openrouter-images-eval
+description: >
+  Evaluation suite for the openrouter-images skill. Tests capability discovery,
+  image generation, image editing, option handling, error recovery, and result
+  presentation using the dedicated Image API scripts (discover.ts, generate.ts, edit.ts).
+skill: openrouter-images
+schemaVersion: "1.0"
+version: "1.0"
+
+config:
+  trials_per_task: 1
+  timeout_seconds: 300
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+  judge_model: gpt-4o
+  inject_skill_body: true
+
+metrics:
+  - name: accuracy
+    weight: 1.0
+    threshold: 0.8
+
+graders:
+  - type: prompt
+    name: correct_script_usage
+    config:
+      prompt: |
+        You are grading an agent that has access to the openrouter-images skill,
+        which provides three scripts:
+          - discover.ts: lists image models and their supported parameters
+          - generate.ts "prompt" [flags]: generates an image from text
+          - edit.ts <path> "prompt" [flags]: edits an existing image
+
+        Call set_waza_grade_pass if the agent:
+          1. Picked the script that matches the user's request (discover vs generate vs edit),
+          2. Only passed flags that are plausibly supported by the target model
+             (or verified support with discover.ts first),
+          3. Did not hand-roll raw HTTP calls to the OpenRouter API when the
+             skill scripts could do the job.
+        Otherwise call set_waza_grade_fail with a short reason.
+
+tasks:
+  - "tasks/discovery/*.yaml"
+  - "tasks/generation/*.yaml"
+  - "tasks/editing/*.yaml"
+  - "tasks/edge-cases/*.yaml"

evals/openrouter-images/tasks/discovery/list-models.yaml

@@ -0,0 +1,24 @@
+id: discovery-001
+name: List available image models
+description: >
+  User asks which image models are available. The agent should run discover.ts
+  (no arguments) and summarize model IDs and capabilities rather than guessing.
+
+tags:
+  - discovery
+  - basic
+  - happy-path
+
+inputs:
+  prompt: "What image generation models can I use through OpenRouter, and which of them support image editing?"
+
+expected:
+  outcomes:
+    - type: task_completed
+  behavior:
+    required_tools:
+      - bash
+  output_contains_any:
+    - "discover"
+    - "input_modalities"
+    - "image models"

evals/openrouter-images/tasks/discovery/model-params.yaml

@@ -0,0 +1,24 @@
+id: discovery-002
+name: Check a specific model's parameters
+description: >
+  User asks whether a specific model supports a capability (4K resolution).
+  The agent should run discover.ts with the model ID and answer from the
+  per-endpoint supported_parameters, not from memory.
+
+tags:
+  - discovery
+  - happy-path
+
+inputs:
+  prompt: "Does bytedance-seed/seedream-4.5 support 4K output? What aspect ratios does it accept?"
+
+expected:
+  outcomes:
+    - type: task_completed
+  behavior:
+    required_tools:
+      - bash
+  output_contains_any:
+    - "4K"
+    - "supported_parameters"
+    - "aspect"

evals/openrouter-images/tasks/discovery/provider-passthrough.yaml

@@ -0,0 +1,25 @@
+id: discovery-003
+name: Discover provider passthrough options
+description: >
+  User asks how to control provider-specific knobs (steps/guidance) for a FLUX
+  model. The agent should inspect the model's endpoints with discover.ts and
+  explain allowed_passthrough_parameters and the --provider-options flag keyed
+  by provider_slug.
+
+tags:
+  - discovery
+  - advanced
+
+inputs:
+  prompt: "I want to tune the number of diffusion steps and guidance for black-forest-labs/flux.2-pro image generation. How do I do that and what values are allowed?"
+
+expected:
+  outcomes:
+    - type: task_completed
+  behavior:
+    required_tools:
+      - bash
+  output_contains_any:
+    - "provider-options"
+    - "provider_options"
+    - "passthrough"

evals/openrouter-images/tasks/edge-cases/cost-reporting.yaml

@@ -0,0 +1,25 @@
+id: edge-005
+name: Cost awareness and reporting
+description: >
+  User asks about cost before generating. The agent should look up per-image
+  pricing via discover.ts endpoint data and report the cost printed to stderr
+  after generation, per the skill's presentation guidance.
+
+tags:
+  - edge-case
+  - presentation
+  - pricing
+
+inputs:
+  prompt: "How much will it cost to generate one image with the default model? Check the pricing, then generate a picture of a hot air balloon and tell me what it actually cost."
+
+expected:
+  outcomes:
+    - type: task_completed
+  behavior:
+    required_tools:
+      - bash
+  output_contains_any:
+    - "$"
+    - "cost"
+    - "pricing"

evals/openrouter-images/tasks/edge-cases/invalid-model.yaml

@@ -0,0 +1,23 @@
+id: edge-004
+name: Nonexistent model requested
+description: >
+  User names a model that does not exist. The agent should check available
+  image models with discover.ts and suggest a close valid alternative instead
+  of passing the bogus ID through and failing.
+
+tags:
+  - edge-case
+  - negative
+  - model-selection
+
+inputs:
+  prompt: "Generate a photo of a beach at sunset using the dall-e-9-ultra model."
+
+expected:
+  outcomes:
+    - type: task_completed
+  behavior:
+    required_tools:
+      - bash
+  output_not_contains:
+    - "dall-e-9-ultra generated"

evals/openrouter-images/tasks/edge-cases/missing-source-image.yaml

@@ -0,0 +1,24 @@
+id: edge-002
+name: Missing source image
+description: >
+  User asks to edit a file that doesn't exist in the workspace. The agent
+  should notice the file is missing and ask for it or report the problem
+  clearly, not fabricate a result or generate a new image instead.
+
+tags:
+  - edge-case
+  - negative
+
+inputs:
+  prompt: "Remove the background from holiday-photo.png."
+
+expected:
+  outcomes:
+    - type: task_completed
+  output_contains_any:
+    - "not found"
+    - "doesn't exist"
+    - "does not exist"
+    - "couldn't find"
+    - "could not find"
+    - "no file"

evals/openrouter-images/tasks/edge-cases/unsupported-format.yaml

@@ -0,0 +1,24 @@
+id: edge-003
+name: Unsupported input format
+description: >
+  User asks to edit a file type edit.ts does not accept (.tiff — supported
+  formats are png/jpg/jpeg/webp/gif). The agent should surface the format
+  limitation, ideally offering to convert first, rather than failing opaquely.
+
+tags:
+  - edge-case
+  - negative
+
+inputs:
+  prompt: "Take scan.tiff and make it look like an oil painting."
+  files:
+    - path: scan.tiff
+      content: "placeholder tiff bytes"
+
+expected:
+  outcomes:
+    - type: task_completed
+  output_contains_any:
+    - "tiff"
+    - "format"
+    - "convert"

evals/openrouter-images/tasks/edge-cases/unsupported-parameter.yaml

@@ -0,0 +1,24 @@
+id: edge-001
+name: Unsupported parameter handling
+description: >
+  User requests an option the default model likely doesn't support. Instead of
+  blindly sending the flag and hitting a 400, the agent should discover the
+  model's supported_parameters and either pick a capable model or explain the
+  limitation.
+
+tags:
+  - edge-case
+  - capability-check
+
+inputs:
+  prompt: "Generate an 8K SVG vector illustration of a mountain range using the default model."
+
+expected:
+  outcomes:
+    - type: task_completed
+  behavior:
+    required_tools:
+      - bash
+  output_not_contains:
+    - "traceback"
+    - "unhandled"

evals/openrouter-images/tasks/editing/basic-edit.yaml

@@ -0,0 +1,29 @@
+id: editing-001
+name: Basic image edit
+description: >
+  User asks to modify an existing image. The agent should call edit.ts with
+  the source path and prompt, and must NOT use generate.ts (which would
+  discard the source image).
+
+tags:
+  - editing
+  - basic
+  - happy-path
+  - golden
+
+golden: true
+
+inputs:
+  prompt: "Make the sky purple in photo.png."
+  files:
+    - path: photo.png
+
+expected:
+  outcomes:
+    - type: task_completed
+  behavior:
+    required_tools:
+      - bash
+  output_contains_any:
+    - "edit"
+    - "photo.png"

evals/openrouter-images/tasks/editing/edit-model-capability.yaml

@@ -0,0 +1,27 @@
+id: editing-003
+name: Edit requires an image-input model
+description: >
+  User asks to edit with a specific model. The agent should verify the model's
+  input_modalities include image (via discover.ts) before editing, and switch
+  to or recommend a capable model if it does not.
+
+tags:
+  - editing
+  - capability-check
+  - advanced
+
+inputs:
+  prompt: "Convert sprite.webp to a watercolor style. I'd like to use the cheapest model that can actually do image-to-image editing — figure out which models support that first."
+  files:
+    - path: sprite.webp
+
+expected:
+  outcomes:
+    - type: task_completed
+  behavior:
+    required_tools:
+      - bash
+  output_contains_any:
+    - "input_modalities"
+    - "image-to-image"
+    - "image input"