`string_match_all` only evaluates recall but not precision

[seyuboglu]

I may be missing something, but it appears that string_match_all is only evaluating recall, but not precision.
So in the multi-key setting, a strategy for achieving a perfect score is to just output all values all the time.

For example, the following returns a score of 100%.

def string_match_all(preds, refs):
    score = sum([sum([1.0 if r.lower() in pred.lower() else 0.0 for r in ref]) / len(ref) for pred, ref in zip(preds, refs)]) / len(preds) * 100
    return round(score, 2)

preds =[
    "a b c d e f g h i j k l m n o p q r s t u v w x y z",
    "a b c d e f g h i j k l m n o p q r s t u v w x y z",
    "a b c d e f g h i j k l m n o p q r s t u v w x y z",
]
refs = [
    ["a", "b", "c"],
    ["x", "y", "z"],
    ["m", "n", "o"],
]

string_match_all(preds, refs)

This metric should balance recall and precision, no? Seems like reporting either F1-score or exact match would be more appropriate. But perhaps I'm missing something.

Thanks!

[hsiehjackson]

Yes, we intend to only evaluate recall. It is hard to evaluate precision since models will output tokens more than answers and we need to extract the correct and wrong answers from these tokens. I think maybe we can change prompt to enforce model giving answers with some special tokens to wrap them (\boxed in math). In this way, we can calculate precision and F1-score.

[seyuboglu]

thanks for the response!

Yes, that's what I've done in my code (just use \boxed or xml tags or something to denote final answer). I have found that in the multi-key, multi-value setting, the precision is sometimes quite low (i.e. the models get confused between the keys)

Of course, this is less important in the single-key, multi-value setting (which is, I believe, what you have reported in the paper).