Bug: File renaming in a child dataset causes ClearML to expect the renamed file in the parent dataset for some files

[nvbln]

Hi,

We are using ClearML server version 2.2.0 in a Docker container hosted on a Windows machine, with the fileserver on a Synology NAS (mounted with NFS in the Docker container). All ClearML servers are on the same 2.2.0 version. Redis is on 7.4.1, Elasticsearch on 8.17.0, and Mongo on 6.0.19.

We encountered the following error (which gets repeated for all missing polygons files) after creating a new child dataset:

[WinError 2] The system cannot find the file specified
Failed copying file C:/Users/XXXX/.clearml/cache/storage_manager/datasets/ds_6fab9dee5fb14e7c907bb9d315cb07a0/Train/1450-polygons.txt to C:/Users/XXXX/.clearml/cache/storage_manager/datasets/ds_3b0e5094ad1040eaaacbc0fd1fbd6d43/Train/1450-polygons.txt

Where:

• Parent dataset: ds_6fab9dee5fb14e7c907bb9d315cb07a0
• Child dataset: ds_3b0e5094ad1040eaaacbc0fd1fbd6d43

To create the child dataset I did the following:

1. Get a parent dataset with samples (XXX-Sample.bmp) and ground truth files (XXX-masks.txt) (where XXX is the sample ID).
2. Update the metadata file (same filename, different data inside).
3. Loop through the files, renaming XXX-Sample.bmp to XXX-sample.bmp, and XXX-masks.txt to XXX-polygons.txt.

I have tried pushing these changes to ClearML in two ways, both cases ended up with the same error shown above:

1. Remove the old files (child.remove_files(dataset_path=folder.name + "/*-Sample.bmp") and child.remove_files(dataset_path=folder.name + "/*-masks.txt"), followed by added the new files from the directory that they were stored in (child.add_files(path=output_path, dataset_path=folder.name) and finally child.upload()).
2. Recreate the whole dataset with the renamed files and call child.sync_folder().

We looked into the fileserver to see if we could find the missing polygon files.
This is a small excerpt from the state.json (dataset_file_entries) (this is from case 1, in case 2 all polygons files refer to the parent dataset):

        {
            "artifact_name": "data_009",
            "hash": "1b69c9d4a2eb0936d06c79560b5b594c65484818658211826745f8aa3ff3b47b",
            "parent_dataset_id": "3b0e5094ad1040eaaacbc0fd1fbd6d43",
            "relative_path": "Test/840-polygons.txt",
            "size": 91
        },
        {
            "artifact_name": "data_001",
            "hash": "8424d2fe935b12337a029e6ccf85a2631bae4e79179075a1a1358aac3c58a9c3",
            "parent_dataset_id": "6fab9dee5fb14e7c907bb9d315cb07a0",
            "relative_path": "Test/709-polygons.txt",
            "size": 91
        },
        {
            "artifact_name": "data_001",
            "hash": "8fc06f9a3f94891f9dd3add3a7251186ada3aa9a482efa4897138c2a362498e2",
            "parent_dataset_id": "3b0e5094ad1040eaaacbc0fd1fbd6d43",
            "relative_path": "Test/60-contrast.bmp",
            "size": 3932214
        },
        {
            "artifact_name": "data_001",
            "hash": "1b3e421bf3a4a4e8b730d866d2fd6d77d65f93d6e614ce0aa30deed748abc8b4",
            "parent_dataset_id": "6fab9dee5fb14e7c907bb9d315cb07a0",
            "relative_path": "Test/629-polygons.txt",
            "size": 91
        },

It can be seen that specifically for 'data_001', it refers to the parent dataset for the polygons. For 'data_009' it does not, and looking at the data inside 'data_001' and 'data_009' we do indeed see that 'data_009' does contain the polygons files as expected, while 'data_001' does not.

Although we could not find any errors in the logs, we added them here:
webserver_docker_log.txt
fileserver_docker_log.txt

Thank you for your time!