Inconsistent `acccuracy_1` results with `av>=9.1.1`

[v-iashin]

With higher versions of av (>=9.1.1), one could get significantly lower performance:

test (67) metrics: {'test/accuracy_1': **0.3622**, 'test/accuracy_5': 0.8747, 'test/accuracy_1_tol1': 0.7768, 'test/accuracy_5_tol1': 0.9408, 'test/mAP': 0.3798, 'test/mROCAUC': 0.912, 'test/dprime': 1.9133, 'test/loss': 2.0175};

instead of

test (67) metrics: {‘test/accuracy_1': 0.5581, 'test/accuracy_5': 0.8907, 'test/accuracy_1_tol1': 0.7745, 'test/accuracy_5_tol1': 0.9522, 'test/mAP': 0.6309, 'test/mROCAUC': 0.9485, 'test/dprime': 2.3058, 'test/loss': 1.4253};

Interestingly, the *tol_1 numbers looked on-par.

A quicker way of testing that something is off, is to run the single video example and get

Prediction Results:
p=0.9482 (12.1250), "1.60" (18)   # av>=9.1.1
...
# instead of 
p=0.8076 (11.5469), "1.60" (18)   # av==[8.1.0,9.0.0]

---

To get to the bottom of it, one needs to figure out what changes in >8.1.0 are causing the inconsistency, and if these can be reverted somehow, perhaps, in .py scripts that rely on av.

Basically, I want to know:

1. What changes in later versions of av made the results inconsistent with the ones with av<=9.0.0? Like what happened between 8.1.0 and 9.1.1.
2. How can we reproduce the results of av<=9.0.0 with later versions of av, i.e. what code change we need to make
3. Does it hang during training with DDP.
4. Do you get consistent results for example.py on a CPU and GPU?

[v-iashin]

conda activate synchformer
pip install av==9.0.0

fixes the issue, but this may cause other issues related to ddp hangs.

A bit of context, i found myself downgrading av for all of my ongoing video projects, because with higher versions of av my DDP training hangs after one epoch. Thus, I tend to keep av==8.1.0.

Also, as it was discovered in #6, the results of this repo is not reproducible with later versions of av (>=9.1.1), with 9.0.0 it is reproducing the numbers.

It seems that these days it's no longer possible to install 8.1.0 version anymore due to the error above.

[v-iashin]

• Google colab is also impacted

I couldn't install an av < 10.0.0 to colab, hence colab will output the numbers that are slightly off.

However, it still works ok otherwise, and good for playing around. So I will keep it unchanged (with av==10.0.0).

[Ashayan97]

I tried to replicate the paper's results using the code checkpoints; however, my results are typically about 2-4% different from the reported results. I would like to ask whether the checkpoint results can be non-deterministic across systems, or if there may be a problem?

I am testing the checkpoints on H100 GPUs with 21 trial runs.