Automate federated external OpenCode eval launch#46
Merged
penfever merged 1 commit intoJul 23, 2026
Conversation
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
penfever
force-pushed
the
penfever/federated-external-eval-launcher
branch
from
July 23, 2026 08:26
7addef4 to
49ca9b3
Compare
penfever
added a commit
that referenced
this pull request
Jul 23, 2026
* fix(iris): override stale external eval credentials * perf(iris): append-delta sync Ray logs * fix(iris): stream Ray delta plans over stdin * fix(iris): parent-mint federated external eval ingress * docs(iris): document federated external eval wrapper * [iris] Skip vanished trace objects during watcher sync * fix(iris): scale external opencode eval defaults * fix(iris): persist external OpenCode eval artifacts to S3 (#45) * fix(iris): persist external eval Harbor artifacts to S3 * style: format durable external eval launcher test --------- Co-authored-by: Benjamin Feuer <[email protected]> * feat(iris): automate federated external eval launch (#46) Co-authored-by: Benjamin Feuer <[email protected]> * feat(iris): track evalchemy artifact harvester * fix(iris): use locked S3 dependencies for GPU evals --------- Co-authored-by: Benjamin Feuer <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The external OpenCode eval launcher now submits the parent-delegated CoreWeave serve, waits for the mirrored ready endpoint, mints the parent-scoped capability URL without logging it, and submits the durable S3 Harbor eval. The Grug profile supplies the established serve, resource, concurrency, storage, dataset, and HF trace defaults while retaining explicit overrides and an existing-endpoint escape hatch.\n\nTests: ============================= test session starts ==============================
platform darwin -- Python 3.13.5, pytest-8.4.2, pluggy-1.5.0
rootdir: /Users/benjaminfeuer/Documents/.worktrees/otagent-federated-external-eval-launcher
configfile: pyproject.toml
plugins: jaxtyping-0.3.9, anyio-4.10.0, asyncio-1.3.0, cov-7.0.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 12 items / 1 error
==================================== ERRORS ====================================
____________ ERROR collecting tests/hpc/test_iris_launcher_args.py _____________
ImportError while importing test module '/Users/benjaminfeuer/Documents/.worktrees/otagent-federated-external-eval-launcher/tests/hpc/test_iris_launcher_args.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../miniconda3/lib/python3.13/importlib/init.py:88: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/hpc/test_iris_launcher_args.py:6: in
from eval.cloud.launch_eval_iris import EvalIrisLauncher
eval/cloud/launch_eval_iris.py:42: in
from hpc.datagen_config_utils import parse_datagen_config
hpc/datagen_config_utils.py:19: in
from data.generation.utils import (
data/generation/init.py:23: in
from .base import BaseDataGenerator
data/generation/base.py:33: in
from scripts.harbor.job_config_utils import (
scripts/harbor/job_config_utils.py:8: in
from harbor.models.job.config import JobConfig
../../harbor/src/harbor/models/job/config.py:14: in
from upath import UPath
E ModuleNotFoundError: No module named 'upath'
=========================== short test summary info ============================
ERROR tests/hpc/test_iris_launcher_args.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.35s =============================== (20 passed; 12 third-party deprecation warnings).