Skip to content

update default for the filter_both_ac parameter in IAA score calculation#18

Open
ypislon wants to merge 4 commits into
mainfrom
fix-filter-both-ac-parameter-defaults
Open

update default for the filter_both_ac parameter in IAA score calculation#18
ypislon wants to merge 4 commits into
mainfrom
fix-filter-both-ac-parameter-defaults

Conversation

@ypislon

@ypislon ypislon commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

This updates the default value for the parameter filter_both_ac in the calculation of IAA scores (Scotts Pi and Cohens Kappa).

Now, the functions get_iaa,calculate_scotts_pi and calculate_cohens_kappa would filter both annotation collections by the given tag. This can still be overridden by setting filter_both_ac manually.

This behavior reflects the defaults in the underlying tag filter function (filter_ac_by_tag).

@ypislon

ypislon commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator Author

While working through the demo notebook again, I understood a use case for the filter_both_ac being False by default in all IAA score calculations. When filtering by a certain tag, you want to check how the respective tag might be labeled with another tag in the second collections. The calculation of an IAA score has the goal to give you feedback about how the two compared annotation collections are different. When we filter both collections by a certain, single tag by default, we end up with two collections where the same tags are at the same text span or where the tags are not set in one of the two collections.

tl;dr: we cannot see mismatches in two collections when we filter both collections by the same tag.

imo the default being False would make sense when you filter only one tag in two annotation collections, but if you would filter more than two tags, filtering and comparing those respective tags in both annotation collections would be the sensible default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant