Alignment from meeteval.wer.wer.time_constrained.align

[m-wiesner]

I was playing around with ways of computing WER metrics on specified subsets of words. Perhaps there is already a way to do this, but I noticed some strange behavior and I wanted to confirm it was expected. I am just using the first few lines of some STM files I have.

import metal
meeteval.wer.wer.time_constrained.align(
    [
        {"words": "hi", "start_time": 0.93, "end_time": 2.03},
        {"words": "good how are you", "start_time": 3.15, "end_time": 5.36},
        {"words": "i'm leigh", "start_time": 7.24, "end_time": 8.36},
        {"words": "pretty good", "start_time": 9.44, "end_time": 12.27},
        {"words": "yeah", "start_time": 15.49, "end_time": 16.95},
    ],
    [
        {"words": "hi", "start_time": 0.93, "end_time": 2.03},
        {"words": "how are you", "start_time": 3.15, "end_time": 5.36},
        {"words": "i'm lee", "start_time": 7.24, "end_time": 8.36},
        {"words": "thank you", "start_time": 9.44, "end_time": 12.27},
        {"words": "yep", "start_time": 15.49, "end_time": 16.95},
    ]
)

Which produces:

[('hi', 'hi'),
('good', 'how'),
('how', 'are'),
('are', '*'),
('you', 'you'),
("i'm", "i'm"),
('leigh', 'lee'),
('pretty', 'thank'),
('good', 'you'),
('yeah', 'yep')]

There errors are:

1. good --> how
2. how --> are
3. are --> *
4. leigh --> lee
5. pretty --> thank
6. good --> you
7. yeah --> yep

I would have expected:

1. good --> *
2. leigh --> lee
3. pretty --> thank
4. good --> you
5. yeah --> yep

Which has 2 fewer errors.

Here are some more examples I constructed that I considered strange. Any explanation for why this occurs and confirmation that this is expected would be appreciated. I suspect there might be some weird pruning going on or something with time constraints that I am not fully understanding.

meeteval.wer.wer.time_constrained.align(
    [
        {"words": "hi", "start_time": 0.93, "end_time": 2.03},
        {"words": "good how are you", "start_time": 3.15, "end_time": 5.36},
        {"words": "i'm leigh adams", "start_time": 7.24, "end_time": 8.36},
        {"words": "pretty good now", "start_time": 9.44, "end_time": 12.27},
        {"words": "yeah", "start_time": 15.49, "end_time": 16.95},
    ],
    [
        {"words": "hi", "start_time": 0.93, "end_time": 2.03},
        {"words": "are you", "start_time": 3.15, "end_time": 5.36},
        {"words": "leigh adams", "start_time": 7.24, "end_time": 8.36},
        {"words": "good now", "start_time": 9.44, "end_time": 12.27},
        {"words": "yep", "start_time": 15.49, "end_time": 16.95},
    ]
)

[('hi', 'hi'),
('good', 'are'),
('how', ''),
('are', 'you'),
('you', ''),
("i'm", ''),
('leigh', 'leigh'),
('adams', 'adams'),
('pretty', 'good'),
('good', ''),
('now', 'now'),
('yeah', 'yep')]

2 errors:
pretty good now
good * now

when I could have done
1 error:
pretty good now
* good now

import meeteval
meeteval.wer.wer.time_constrained.align(
    [
        {"words": "hi", "start_time": 0.93, "end_time": 2.03},
        {"words": "good how are you", "start_time": 3.15, "end_time": 5.36},
        {"words": "i'm leigh adams", "start_time": 7.24, "end_time": 8.36},
        {"words": "pretty good now and you", "start_time": 9.44, "end_time": 12.27},
        {"words": "yeah", "start_time": 15.49, "end_time": 16.95},
    ],
    [
        {"words": "hi", "start_time": 0.93, "end_time": 2.03},
        {"words": "are you", "start_time": 3.15, "end_time": 5.36},
        {"words": "leigh adams", "start_time": 7.24, "end_time": 8.36},
        {"words": "good now and", "start_time": 9.44, "end_time": 12.27},
        {"words": "yep", "start_time": 15.49, "end_time": 16.95},
    ]
)

[('hi', 'hi'),
('good', 'are'),
('how', ''),
('are', 'you'),
('you', ''),
("i'm", ''),
('leigh', 'leigh'),
('adams', 'adams'),
('pretty', 'good'),
('good', ''),
('now', 'now'),
('and', '*'),
('you', 'and'),
('yeah', 'yep')]

[boeddeker]

Short: When you use meeteval.wer.wer.time_constrained.align(..., collar=5) it does what you expect.

Long:

I suspect there might be some weird pruning going on or something with time constraints that I am not fully understanding.

Yes and yes.

You call the align function with its default arguments. The default value for collar is 0 (for the command line interface, we recently removed the default value to minimize the confusion that you hit, when you forget that. (For the visualization code, it is 5 seconds.)

The constraint works as follows:
First, each sentence is divided into words. There we use the heuristic, that the number of characters correlates with the duration (controlled with the *_pseudo_word_level_timing arguments).
For the reference, the start and end timings are used, but for the hyp words, only the center is used to prevent a fooling of the metric.
Next, the hyp start end value are extended by the collar value (in your case zero)

The constraint is then: If a hyp word lies in a reference word, a substitution and correct alignment is allowed, otherwise only deletion and insertion.

Now, looking at your first example (left is ref, right is hyp):

We see, that the center of the word are on the right side (hyp) doesn't lie inside the timings of the word are on the left side (ref).

To get reasonable results, you should set the collar value to 5 seconds. We found that 5 seconds is a solid value to avoid such confusions (see Fig 3 of https://arxiv.org/pdf/2307.11394).

Sorry, that we selected a bad default.
@thequilo: Should we remove collar=0 from the complete codebase or, at least, add a warning, when someone uses collar=0?

[m-wiesner]

I see now. Thank you for the very quick reply. I think this is quite clear and I suspected something like that was happening.

[thequilo]

I'm fine with removing the default. It may break some code, but for everyone who uses the recommend es collar of 5s, nothing changes