-
Notifications
You must be signed in to change notification settings - Fork 0
Issues
is:issue state:open
is:issue state:open
Issue creation is restricted in this repository
Search results
[Bug]: compare.sh bundle mispairs trajectories/logs with wrong run's results when a mid-sequence run is interrupted
bugSomething isn't workingSomething isn't workingcomplexity: mediumModerate scope — multiple files or non-trivial logicModerate scope — multiple files or non-trivial logicevaluation: vakraVakra (m3) evaluationVakra (m3) evaluationneeds-triageNewly filed, not yet reviewedNewly filed, not yet reviewedpriority: mediumNormal priorityNormal priorityStatus: Open.#103 In cuga-project/cuga-eval;[Feature]: curated --eval-key subsets for reliably-passing M3 capability 2/3 tasks
complexity: lowSmall, focused change — easy to reviewSmall, focused change — easy to reviewenhancementNew feature or requestNew feature or requestevaluation: vakraVakra (m3) evaluationVakra (m3) evaluationpriority: lowNice to have, no urgencyNice to have, no urgencyStatus: Open.#102 In cuga-project/cuga-eval;[Bug]: --model-profile is silently ignored when running a benchmark's eval.sh directly
bugSomething isn't workingSomething isn't workingneeds-triageNewly filed, not yet reviewedNewly filed, not yet reviewedpriority: highImportant, address soonImportant, address soonStatus: Open.#101 In cuga-project/cuga-eval;[Feature]: Consistent LLM token tracking via callbacks (cache-aware) with tests
enhancementNew feature or requestNew feature or requestevaluation: appworldAppworld EvalutionAppworld Evalutionpriority: mediumNormal priorityNormal priorityStatus: Open.#95 In cuga-project/cuga-eval;[Bug]: Agent crashes treating an already-parsed dict tool response as a JSON string
bugSomething isn't workingSomething isn't workingcomplexity: mediumModerate scope — multiple files or non-trivial logicModerate scope — multiple files or non-trivial logicevaluation: vakraVakra (m3) evaluationVakra (m3) evaluationpriority: mediumNormal priorityNormal priorityStatus: Open.#91 In cuga-project/cuga-eval;[Bug]: Wrong evaluation bundle and report names in --dotenv regime
bugSomething isn't workingSomething isn't workingpriority: highImportant, address soonImportant, address soonStatus: Open.#89 In cuga-project/cuga-eval;[Evaluation experiment]: Appworld baseline on NVIDIA-Nemotron-3-Super-120B-A12B-FP8
priority: highImportant, address soonImportant, address soonStatus: Open.#88 In cuga-project/cuga-eval;[Evaluation experiment]: Appworld Baselines on open-source models on RITS
evaluation: appworldAppworld EvalutionAppworld Evalutionpriority: highImportant, address soonImportant, address soonStatus: Open.#85 In cuga-project/cuga-eval;[Benchmark]: Include Tau-2 Bench into evaluation repository
priority: mediumNormal priorityNormal priorityStatus: Open.#84 In cuga-project/cuga-eval;[Feature]: Enable running of different CUGA configurations
enhancementNew feature or requestNew feature or requestevaluation: appworldAppworld EvalutionAppworld Evalutionpriority: mediumNormal priorityNormal priorityStatus: Open.#83 In cuga-project/cuga-eval;[Feature]: M3 bundle - Langfuse score polling, react/cuga parity beyond report.md, end-to-end bundle-content tests (follow-up to #56)
enhancementNew feature or requestNew feature or requestpriority: highImportant, address soonImportant, address soonStatus: Open.#71 In cuga-project/cuga-eval;[Feature]: Group compare reports into per-agent/model/policy sections (report.md)
enhancementNew feature or requestNew feature or requestpriority: highImportant, address soonImportant, address soonStatus: Open.#68 In cuga-project/cuga-eval;