Skip to content

EvalBuilder can retain stale attachments across prompt/case reconfiguration #3

Description

@MaestroError

Summary

EvalBuilder can retain previously configured attachments even after a later reconfiguration appears to replace them.

Why this matters

This can make evals behave unpredictably when a builder instance is reused through helpers or shared setup. A later prompt or case may unintentionally send stale files from an earlier configuration.

Evidence

  • src/EvalBuilder.php:96-98 only replaces attachments inside prompt() when the incoming attachments array is non-empty.
  • src/EvalBuilder.php:111-113 only copies attachments from withCase() when the case has attachments.
  • src/EvalBuilder.php:133-136 sets attachments directly, so once attachments are set there is currently no obvious clearing path through prompt(..., attachments: []) or withCase($caseWithoutAttachments).

Repro shape

A reused builder like this can keep old files attached:

evaluate(MyAgent::class)
    ->attachments([$oldFile])
    ->prompt('first prompt')
    ->run();

// Later reuse / reconfiguration
$builder
    ->prompt('second prompt', attachments: [])
    ->run();

The second run can still include $oldFile.

Suggested fix

This is only a suggestion; the actual fix should be planned carefully.

Possible directions:

  • Make prompt(..., attachments: []) explicitly replace attachments with an empty array.
  • Make withCase() fully replace builder attachments, including replacing them with [] when the case has none.
  • Add focused tests covering attachment clearing and builder reuse semantics.

Notes

This is primarily a developer-experience and correctness issue because the current behavior is surprising and hard to notice until eval inputs drift.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions