Skip to content

test(openrouter-images): add waza eval suite for the images skill#87

Open
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1784094764-openrouter-images-eval
Open

test(openrouter-images): add waza eval suite for the images skill#87
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1784094764-openrouter-images-eval

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Summary

Adds a waza evaluation suite for the openrouter-images skill under evals/openrouter-images/.

Two specs:

  • eval.yaml — 19-task capability suite (skill body injected) covering:
    • discovery/ — list models, per-model supported_parameters, allowed_passthrough_parameters / --provider-options
    • generation/ — basic generate (golden), aspect ratio, model override, --output, --n variations, transparent background, --seed reproducibility
    • editing/ — basic edit (golden), edit with new output path, image-input capability check, multi-turn generate→edit via follow_up_prompts
    • edge-cases/ — unsupported params/formats, missing source file, nonexistent model, cost/pricing reporting
    • Plus an eval-level prompt (LLM-judge) grader correct_script_usage checking the agent picks the right script (discover vs generate vs edit), verifies flag support before sending, and doesn't hand-roll HTTP calls.
  • eval-trigger.yaml — trigger-precision suite with inject_skill_body: false: 3 positive tasks (direct generate, edit phrasing, indirect "logo" phrasing) and 3 negative tasks (vision analysis, video request, text-only).

Tasks use per-task expected checks (output_contains[_any], outcomes, behavior.required_tools/forbidden_tools), tags for filtering, and small image fixtures (photo.png, avatar.jpg, sprite.webp) copied into each task workspace. Golden tasks are enforceable via waza gate. Run instructions in the suite README (requires OPENROUTER_API_KEY for generate/edit tasks; discovery is keyless).

Link to Devin session: https://openrouter.devinenterprise.com/sessions/a653a8c7bdee4affa0cd2b7fd29dbe93

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author
Original prompt from kenny.rogers

i am creating a new skill designed to help people use our image functionality, create a comprehensive set of test tasks we should use to evaluate the effectiveness of the skill, we are using waza to eval the skills so use this guide https://microsoft.github.io/waza/guides/eval-yaml/

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant