Skip to content

feat: run the same prompt multiple times into separate numbered output directories#65

Open
floo-dck wants to merge 1 commit into
ISG-Siegen:developfrom
leonlenz:feat/loop-program
Open

feat: run the same prompt multiple times into separate numbered output directories#65
floo-dck wants to merge 1 commit into
ISG-Siegen:developfrom
leonlenz:feat/loop-program

Conversation

@floo-dck

Copy link
Copy Markdown

Summary

Adds a new --runs CLI flag that executes AutoRecLab multiple times with the
same prompt in a single command. Each run is written to its own numbered
subfolder inside the configured out_dir, so multiple experiment runs no longer
overwrite each other.

Motivation: To analyze the stability/variance of results, we need to run the
exact same prompt several times (e.g. 10x) and compare the outputs.

Changes

  • --runs N argument (default 1) in main.py. The prompt is read once and
    reused for every run.
  • Numbered output folders: with multiple runs, output goes to
    out/run_01, out/run_02, … The numbering continues from the highest
    existing run_* folder, so previous results are never overwritten. The
    zero-padding adapts to the total run count (e.g. run_01run_10).
  • Refactor: extracted the single-run logic into a run_once() helper and the
    prompt input into get_user_request() for readability.
  • Isolated trackers per run: added reset() methods to CostsTracker and
    StatisticsTracker so costs and statistics start clean for every run.
  • Per-run logging: attach_file_handler() now removes previously attached
    file handlers, so each run logs to its own debug.log in its own folder.
  • README: documented the new --runs option.

Backwards compatibility

With --runs 1 (the default) the behavior is unchanged — output is written
directly to out/ as before.

How to test

uv run main.py --prompt "Build a recommender..." --runs 10

@eisenbahnhero eisenbahnhero left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a great idea, and I believe it aligns very well with how AutoRecLab will operate in the future.

However, in my opinion, there are still some key aspects missing from the implementation:

  • I wouldn’t see AutoRecLab handle multiple attempts by itself; instead, I’d would build a much simpler wrapper, for example, as a shell script or PS1 script, that simply calls AutoRecLab as it currently stands. This has the advantage of keeping ARL lightweight, while still ensuring that the functionality works fully even with the many changes in some key components.
  • It’s well known that AutoRecLab has very long runtime when processing large datasets. These runtimes are now multiplied by n. There should be an option to start the runs in parallel. I think this would be much simpler to implement via a wrapper.
  • What happens if one of the individual runs encounters an error? Or if n-x runs have already completed and n-x+1 then encounters an error? There should be error handling in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants