Setting UV_CACHE_DIR to agent.venvs_dir generate experiments conflicts in docker with multiple agents.

[ffvogmv]

By default, UV_CACHE_DIR is set to agent.venvs, which is also the directory where the experiment task repository is cloned and executed. In Docker environments, if we want this folder to be shared (e.g., mounted as a volume), we typically do something like:

-v host_venvs_build:/root/.clearml/venvs-builds

However, this setup causes conflicts when multiple experiments run concurrently on the same host. This is because the repository is cloned into:

/root/.clearml/venvs-builds/3.11/task_repository/repository

If this folder is shared across experiments, files may be modified simultaneously by different tasks using the same repository, leading to race conditions and unexpected behavior.

To resolve this, I used the uv_sync_extra_args configuration option:

uv_sync_extra_args: ["--cache-dir", "/root/.clearml/venvs-cache"]

This forces uv to use /root/.clearml/venvs-cache as the cache directory, which I believe is the intended default used by the framework.

Suggested Fix:

In clearml_agent/helper/package/uv_api.py, I suggest changing:

cache_dir = self.session.config.get("agent.venvs_dir", None)
if cache_dir is not None:
    kwargs["env"]["UV_CACHE_DIR"] = cache_dir

to:

cache_dir = self.session.config.get("agent.venvs_cache.path", None)
if cache_dir is not None:
    kwargs["env"]["UV_CACHE_DIR"] = cache_dir

Is my understanding correct, or is there something I'm missing about how UV_CACHE_DIR is intended to be used in ClearML?

[ainoam]

Thanks for reaching out @ffvogmv.
Your proposal makes perfect sense - Care to issue a PR?

[ainoam]

Upon further reflection, @ffvogmv, shouldn't the agent be mounting to a different directory for each container?
Can you provide an example of two containers mounted to the same place?

[ffvogmv]

At least in my case, which I believe follows the default ClearML configuration, the task repository is cloned inside the container at:

/root/.clearml/venvs-builds/3.11/task_repository/repository

Currently the framework maps as UV_CACHE_DIR the agent.venvs_dir, which as defined by the default config is /root/.clearml/venvs-builds/.

As I mentioned earlier, if I want to persist the cache generated during agent execution, I need to map this directory to the host machine. To do that, I use the following volume mount defined in extra_docker_arguments in clearml.conf:

-v host_venvs_build:/root/.clearml/venvs-builds

This setup worked fine until I started running two agents concurrently on the same host, both using the same repository. Since the agents clone the repository into the same path (/venvs-builds/3.11/task_repository/repo_name), the second agent's run overwrites files from the first one. Additionally, files generated by one agent become visible to the other, which leads to errors and inconsistent experiment behavior.

I'm not sure if I'm misconfiguring something, perhaps the agents are supposed to clone the repository into a different location than venvs-builds. But based on my understanding, that path is the expected default in ClearML for cloning and executing tasks. And for me makes more sense to change the location of UV_CACHE_DIR to a folder that is mounted automatically by clearml-agents like venvs_cache.

[daniel-bogdoll]

@ainoam Maybe related: When I use

package_manager: {
        type: uv,
        ...
        }

I get errors like

error: The project directory `.` is inside the cache directory `/root/.clearml/venvs-builds`
INFO: failed installing using lock file, trying without
error: The project directory `.` is inside the cache directory `/root/.clearml/venvs-builds`
Failed installing uv requirements from lock file: Command '['/root/.clearml/venvs-builds/3.13_uv/bin/uv', 'sync']' returned non-zero exit status 2.
clearml_agent: ERROR: Command '['/root/.clearml/venvs-builds/3.13_uv/bin/uv', 'sync']' returned non-zero exit status 2.

I can fix this by setting

extra_docker_arguments: ["-e", "UV_CACHE_DIR=/tmp/uv-cache"]

but it feels odd that I need to change anything else for uv to work. Am I missing something?